Skip to content

Job Orchestration

Altair SLC Hub uses HashiCorp Nomad to orchestrate workloads across worker nodes.

For background on Nomad and its scheduling model, see:

Overview

A workload can be submitted to Altair SLC Hub from different entry points, including:

  • the Altair SLC Hub portal
  • Altair Analytics Workbench
  • pipelines
  • hubcli
  • third-party schedulers that invoke hubcli.

Once the workload reaches Altair SLC Hub, the submission route does not change how it is scheduled.

Note

In Altair SLC Hub, the main control over workload placement is the Execution Profile used for the workload. An Execution Profile is required to run a workload on a worker node in the cluster.

For more information, see:

How orchestration works

At a high level, orchestration works as follows:

  1. A workload is submitted to Altair SLC Hub.
  2. Altair SLC Hub determines the execution settings that apply to the workload.
  3. The selected execution profile supplies scheduling-related settings such as constraints and resource requirements.
  4. Nomad evaluates the available worker nodes.
  5. Nomad places the workload on a suitable worker node, or the workload remains queued until placement becomes possible.
  6. The workload runs on the selected worker node.
  7. When the workload completes, the scheduled capacity becomes available for future workloads.

A workload is placed only when an eligible worker node is available and has sufficient capacity according to the scheduler's view. Otherwise, the workload remains queued until placement becomes possible.

Role of Execution Profiles

An Execution Profile is required to run any workload on a worker node in the cluster.

Execution profiles are the main mechanism in Altair SLC Hub for influencing workload placement and are a Altair SLC Hub concept. Altair SLC Hub uses the selected execution profile to prepare the workload that is submitted to Nomad.

An execution profile can define:

  • constraints that restrict which worker nodes are eligible
  • resource requirements, such as expected CPU and memory usage

Constraints limit where workloads can run

Constraints are used to control where a workload can run.

For example, constraints can be used to:

  • target workloads to worker nodes with specific labels
  • restrict workloads to a required operating system

This means a workload may remain queued if its constraints limit it to worker nodes that are not currently able to run it.

Resource Requirements guide scheduling

Execution Profiles can define Resource-Requirements such as CPU and memory. These values define the expected CPU and memory usage of a workload.

Resource Requirements are used to guide scheduling. They do not define hard upper resource limits for a workload. This means a workload can use more or less CPU or memory than the values defined in its Resource Requirements.

If no eligible worker node currently has sufficient available capacity for the workload, the workload remains queued until capacity becomes available.

Why a workload might stay queued or fail

To understand why a workload did not start or failed after it started, it helps to look at:

  • which worker nodes are eligible to run the workload
  • what CPU and memory the workload is expected to use
  • whether the worker where the workload runs is also under pressure from other processes

What happens if there is not enough capacity?

A workload is not always started immediately after submission.

If no eligible worker node currently has sufficient capacity for the workload, the workload can remain queued until placement becomes possible.

When that workload later runs and completes, the capacity used for it becomes available for future workloads.

What happens when other processes on a worker compete with Altair SLC Hub for resources?

Workloads started through Altair SLC Hub share CPU and memory with other processes running on the same worker node.

This includes processes that were started directly on the worker outside Altair SLC Hub orchestration.

Because Resource Requirements are used to guide scheduling rather than enforce hard runtime limits, these processes can compete for the same machine resources.

For example, on a worker with 8 GB of RAM:

  • three jobs started through Altair SLC Hub each eventually use 2 GB of RAM
  • at the same time, a Python process started directly on the same worker eventually uses 4 GB of RAM

In this situation, the Linux kernel can terminate any one of these four processes because of out-of-memory conditions.