Skip to content

Failed and Orphaned Workloads

When a workload becomes unreachable — due to repeated connection failures, a worker node going offline, or an interrupted Link Session — Altair SLC Hub will attempt to re-establish communication. If the workload remains unreachable for an extended period (a timeout), Altair SLC Hub marks it as Failed or Dead. Failed workloads are not restarted automatically and must be resubmitted by the user.

Note

Altair SLC Hub uses HTTP-based communication and does not maintain persistent connections between components. A temporary loss of communication does not immediately cause a workload to fail — Altair SLC Hub will continue retrying and will recover automatically if communication is restored in time.

Two common scenarios where a workload may eventually be marked as failed:

  • If a Altair SLC Hub is unable to reach a running workload after an extended period (timeout), the job is marked as failed and must be started again by the user.
  • If an interrupted Link Session cannot be re-established within the allowed time (timeout), any job running at the time will be marked as failed and must be restarted once a Link Session can be established again.

Automatic Restart Exceptions

Two workload types will attempt to restart automatically if they fail:

  • On-Demand runtime servers
  • Published library servers

Garbage Collection

Garbage collection (GC) is the process by which Altair SLC Hub identifies and cleans up workloads that are in a terminal state, such as failed, dead, completed, or orphaned workloads that are no longer active.

All workloads in a terminal state are subject to GC. Failed workloads are marked for GC automatically and will not be retriggered — a user must explicitly restart the job.

Garbage collection can be triggered manually at Hub Administration > Hub Management > Cluster Workloads.

Configuring Garbage Collection

The timing and behavior of garbage collection can be managed using the following Altair SLC Hub configuration values:

Parameter Description
jobs.gcInterval How frequently garbage collection runs. During problem diagnosis, it can be useful to increase this interval.
jobs.gcThreshold How long a job is retained before being garbage collected. During problem diagnosis, it can be useful to increase this duration. If jobs are submitted at high frequency, consider decreasing it — though it should not be set below approximately 30 seconds.