Skip to content

Troubleshooting

Generally to be done with the assistance of Altair Support.

Checking status of Altair SLC Hub services

Note

In a Windows environment, the hubctl command must be run from a command prompt that has been started 'as Administrator'.

To check that the Altair SLC Hub services are running, use the hubctl command:

hubctl service status

This should print a summary table of all of the Altair SLC Hub services and whether they are active (running).

If any are marked as inactive try restarting them with

hubctl service start <name>

If they are still inactive, or marked as failed, then view the logs of the service.

Viewing service logs

The logs from the services are captured by systemd/journald and, they can most easily be accessed using the hubctl log command. For more details see Logging.

Note

In most scenarios the logs can be retrieved by the hubctl log command. However, if a service fails unexpectedly, systemd can fail to associate the final log messages with the relevant service, in this case it is necessary to use journalctl to view the systemd output.

Log viewing tips on Linux

By default, hubctl log pipes the log entries through a pager. Piping the output to a file, and then editing the file using an editor such as vi can be a useful alternative way of viewing the log files.

Rather than limiting the display to a fixed number of entries, the output can be limited based on the timestamp of the log record. To return the log entries for all Altair SLC Hub services that have happened in the last 5 minutes, use the command:

hubctl log --since -5m

The logs from the services are located in [var directory]\log. Your user must be in the 'Administrators' group on the server. Then, give yourself access to this folder by navigating to it in Windows Explorer.

Investigating Jobs and deployments with Cluster Workloads

The Cluster Workloads page in Altair SLC Hub can be used to investigate jobs and deployments when status or error information is not easily available elsewhere in the UI. Because execution details and logs are organized across several levels, you might need to click through different tabs to find the relevant information.

Use Cluster Workloads to investigate problems such as:

  • jobs that remain in Pending state
  • jobs that fail before starting
  • tasks that terminate with errors
  • jobs that appear active elsewhere in Altair SLC Hub but are not running correctly

Open the job

  1. In Altair SLC Hub, go to Hub Administration > Hub Management > Cluster Workloads.
  2. Locate the relevant job by name, type, or status.
  3. Click the job name to open its details.
  4. Review the Job Information and Job Executions tabs.

Inspect the execution and tasks

  1. In the Job Executions tab, click the execution ID to open the execution details.
  2. Review the available execution tabs:
    • Job Execution Information for status and execution details
    • Resource Utilization for resource usage while the execution is running
    • Tasks for the tasks that belong to the execution
    • Files for execution files and task-specific output, such as procmon.out, which can sometimes contain useful diagnostic information.
  3. In the Tasks tab, click a task name to open task details.
  4. Review the available task tabs:
    • Task Information for task status and details
    • Task Events for startup and failure events
    • Log Files to review or download logs such as stdout (standard output) and stderr (standard error).

A workload can have more than one execution. For example, a service workload can have multiple executions if it has been restarted.

Useful tabs for troubleshooting

The following tabs are often useful when troubleshooting:

  • Job Information shows the current state and basic details of the workload.
  • Job Executions shows individual executions of the workload.
  • Job Execution Information shows details for a specific execution.
  • Resource Utilization can help when the workload started and you need to review its resource usage.
  • Tasks shows the tasks that belong to the execution. Most workloads have a main task. Some batch workloads can contain additional tasks.
  • Task Events is often the best place to check first when a workload did not start or failed early.
  • Log Files is usually the next place to check after Task Events. This tab can contain stdout, stderr, and additional log files depending on the workload type.
  • Files can contain execution files (such as procmon.out) and task-specific output. This tab is less commonly needed, but can be useful if support asks you to inspect or share additional files.

Things to keep in mind

  • If a job is still Pending, it might not yet have a job execution.
  • If a process started but failed, the most useful details are often in Task Events or Log Files.
  • Depending on the job type, some tabs or log files can differ.
  • Some failures are only visible in Cluster Workloads, even if another page in Altair SLC Hub shows the workload as active.

Missing Nomad logs on worker nodes

Nomad has a garbage collector which by default deletes nomad log files when disk space usage exceeds 80%.

This can lead to nomad deleting log files as soon as a task completes, making it extremely difficult to diagnose the reason for a task failure.

This is unlikely to occur in a production environment.

If it does occur and there is an urgent need to diagnose a task failure, as a short term measure add a file named 90-gc-config.hcl to the [etc directory]/nomad.d directory of the Altair SLC Hub installation with this content: 

client {
  gc_disk_usage_threshold = 99
}
Then restart nomad before continuing: 
hubctl service restart nomad

A proper remedy is to increase the disk space available, for example on Linux putting the [var directory]/nomad directory of the Altair SLC Hub installation on its own volume.

Jobs remain pending with "Dimension 'disk' exhausted"

This error indicates that Nomad cannot schedule the job because it believes there is insufficient disk capacity available.

Error message: Dimension 'disk' exhausted on X nodes

Jobs remain in a Pending state and hubcli job status reports a message similar to the following:

State    Pending
Reason   Resources exhausted on 2 nodes
         Dimension 'disk' exhausted on 2 nodes

This issue can occur if Nomad was started while the disk was already full.

Nomad determines the available disk capacity when the Nomad agent starts and stores this value. It does not monitor disk capacity changes while running. As a result, if disk space is later freed (for example by cleaning up files) or the disk is resized, Nomad does not automatically detect the change.

The updated disk state is only recognized after restarting Nomad.

To solve it, restart Nomad so that it re-evaluates the available disk capacity:

hubctl service restart nomad

hubctl bootstrap fails with "Child process 'cmd (xxxx)' finished with code -1073741515" on Windows

This error indicates that a required DLL was not found (STATUS_DLL_NOT_FOUND). This usually happens when Microsoft Visual C++ Redistributable is not installed, or when OpenSSL 3 is not available on the system.

See Initial Windows Installation.

Nomad fails to start with "Failed to resolve Serf advertise address"

Nomad fails to start because it cannot determine which IP address to use for cluster communication.

Error message: Failed to resolve Serf advertise address ":4648"

This can occur on Windows systems, including disaster recovery (DR) environments, where Nomad cannot automatically detect the correct IP address to use because it can’t find the default private network.

To solve it, configure Nomad to use the machine's fixed IP address explicitly. Add a file to the etc/nomad.d directory of the Altair SLC Hub installation with the following content:

addresses {
  http = "<ip>"
  rpc  = "<ip>"
  serf = "<ip>"
}

Replace <ip> with the fixed IP address of the machine. Then restart Nomad:

hubctl service restart nomad

Note

This workaround requires the machine to have a fixed IP address. If the machine's IP address changes, update the configuration accordingly.

For more information on Nomad address configuration, see the Nomad configuration reference.

hubctl bootstrap fails with "invalid_grant: Invalid user credentials" during Keycloak initialization

This can happen when Altair SLC Hub is reinstalled while reusing an external database from a previous installation.

Error message during hubctl bootstrap:

[10/22] Initialise Keycloak admin user
    Admin user already exists
[11/22] Initialise Keycloak realm
    Error refreshing client auth token error="400 Bad Request: invalid_grant: Invalid user credentials"

Recovery procedure

Keycloak does not allow resetting the password of an existing admin account directly. Instead, a temporary admin account is created, used to reset the password of HubServiceAdmin, and then removed.

The kc.sh and kcadm.sh scripts used below are located in <installation root>/libexec/keycloak/bin (<installation root>\libexec\keycloak\bin on Windows, using kc.bat and kcadm.bat).

Follow Keycloak: Bootstrapping and recovering an admin account to:

  1. Stop the Keycloak service: 
    hubctl service stop keycloak
  2. Create a temporary admin user with the bootstrap-admin command, using the database connection details from [var directory]/lib/keycloak/config/keycloak.conf. Use a username other than HubServiceAdmin, since bootstrap-admin fails if the username already exists.
  3. Start the Keycloak service again: 
    hubctl service start keycloak
  4. Log in with kcadm.sh (or kcadm.bat on Windows) as the temporary admin, following the same Keycloak documentation.

Once logged in as the temporary admin, look up the HubServiceAdmin user ID and set a new password for it:

kcadm.sh get users -r master -q username=HubServiceAdmin
kcadm.sh set-password -r master --userid <id> --new-password <new-password>

Update the local file so Altair SLC Hub uses the same password:

sudo bash -c "printf '%s' '<new-password>' > [var directory]/data/keycloak/admin.txt"
sudo chown slchub-auth:slchub [var directory]/data/keycloak/admin.txt
sudo chmod 440 [var directory]/data/keycloak/admin.txt

Open [var directory]\data\keycloak\admin.txt in a text editor and replace its contents with <new-password>. Save the file without adding a trailing newline.

Warning

On Linux, admin.txt must be owned by slchub-auth:slchub with permissions 440.

Remove the temporary admin account, as described in the Keycloak documentation linked above.

Run bootstrap again:

hubctl bootstrap

Verification

hubctl bootstrap completes all steps successfully, including Keycloak admin and realm initialization.

hubctl bootstrap --nohttps fails with "HTTPS required" during Keycloak initialization on Windows

This can happen when the machine hostname resolves to a non-loopback address during bootstrap.

Error message during hubctl bootstrap --nohttps:

[14/25] Initialise Keycloak realm
        ... Failed to refresh HTTP JWK Set ... url=https://<hostname>:9091/keycloak/realms/slchub/protocol/openid-connect/certs
        Error refreshing client auth token error="403 Forbidden: invalid_request: HTTPS required"

Why this happens

During Keycloak initialization, Altair SLC Hub uses the internal ingress URL (https://<hostname>:9091/...) for service discovery. On some Windows environments, <hostname> resolves to the machine network interface address (often IPv6 first) rather than loopback. In this case, Keycloak can reject the request with HTTPS required. For background on Keycloak HTTPS requirements, see Keycloak Server Administration Guide: SSL mode.

Checks

Verify hostname resolution:

Resolve-DnsName <hostname>
ping <hostname>

Workaround for local --nohttps bootstrap

To solve it, set internalnetwork.hostname to a value that is not the Windows computer name.

You can use one of the following options in [etc directory]\config.d\internalnetwork_custom.yaml:

Option 1: Use loopback directly.

internalnetwork:
    hostname: localhost

Option 2: Use a custom local alias.

internalnetwork:
    hostname: hub-local

If you use a custom alias, add entries for the alias in C:\Windows\System32\drivers\etc\hosts (run editor as Administrator):

127.0.0.1 hub-local
::1 hub-local

Run bootstrap again:

hubctl bootstrap --nohttps

Verification

hubctl bootstrap --nohttps completes all steps successfully, including Keycloak realm initialization.

Workloads fail with "Logon failure: user has not been granted the requested logon type"

This occurs when a workload is started with an OS password, and the Windows worker node logs the user on with the supplied password using a batch logon.

Error message:

Logon failure: user has not been granted the requested logon type

To solve it, grant the "Log on as a batch job" right to the affected user or group on the Windows worker node(s) via secpol.msc (Local Security Policy > Security Settings > Local Policies > User Rights Assignment). See OS Passwords: Prerequisites.

This occurs when OS passwords are required (ospasswords.required), but the user has not set their OS password.

To solve it, set an OS password in the Altair SLC Hub Portal under Account Menu > Preferences > OS User Credentials, then retry starting the link session from the Excel Plugin.