Data Storage Architecture

Altair SLC Hub uses three storage components:

• PostgreSQL for application data, information about jobs, pipelines, users, and system state
• Object storage for job results, logs, and uploaded artefacts
• Shared filesystem for files that must be available across the Altair SLC Hub server and workers during runtime

For example, a pipeline definition and run status are stored in PostgreSQL, while the pipeline results and logs are stored in object storage.

Storage roles

Storage type	Purpose	Examples
PostgreSQL	Stores application data and current system state	users, pipeline definitions, job metadata, job status, access control logs, file references
Object storage	Stores large files created or uploaded in Altair SLC Hub	job results, job logs, uploaded artefacts
Shared filesystem	Makes files available across the Altair SLC Hub server and workers during runtime	trigger files, shared resources, files shared between pipeline steps

PostgreSQL

PostgreSQL stores the structured data that Altair SLC Hub needs to operate.

This includes:

• users
• pipeline definitions and configuration
• job records and status
• application state
• access control logs
• references to files stored in object storage

Note

PostgreSQL stores metadata and state, not the file contents produced by jobs.

Object storage

Object storage stores file-based data generated or used by Altair SLC Hub.

This includes:

• job results
• job log files
• uploaded artefacts and packages

Note

PostgreSQL stores metadata and state, while object storage stores the actual files.

Altair SLC Hub can use either:

• an external S3-compatible object storage backend, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage
• the internal object store provided with Altair SLC Hub

Shared filesystem

A shared filesystem is used when files must be available by path across the Altair SLC Hub server and worker nodes during execution.

Typical uses include:

• pipeline trigger files
• Altair Analytics Workbench file access
• files shared between pipeline steps
• shared resources required by jobs

Job execution flow

During a typical job run:

1. The job request and job state are recorded in PostgreSQL.
2. If the job has inputs, they are uploaded to object storage.
3. The controller reads and updates job status in PostgreSQL.
4. The workload is submitted to a worker for execution.
5. If the job requires shared runtime files, they are accessed through the shared filesystem.
6. When the job completes, its output files and captured logs are written to object storage.
7. References to those stored outputs remain in PostgreSQL.

What data goes where

Data type	Stored in
users, jobs, pipelines, system state	PostgreSQL
access control logs	PostgreSQL
job results	Object storage
job logs captured as part of a job	Object storage
uploaded artefacts	Object storage
runtime shared files	Shared filesystem