Pipelines¶

Pipelines provide the facility to execute a batch of programs.

As well as basic sequential chaining of Altair SLC programs, pipelines support nodes implemented by operating system scripts, parallel execution of nodes (with synchronization upon completion) and conditional execution of nodes.

Pipelines interface¶

The Pipelines page allows you to create, view, and run pipelines defined in Hub.

The pipelines displayed may be filtered by type (that is, pipeline or folder) and ordered using the created and modified dates.

You can also organize the pipelines in folders by clicking the Folder button and import a pipeline by clicking the Import Pipeline button.

To create a new pipeline, click the Pipeline button. This presents a canvas populated with the mandatory Start node. This canvas is also called the Pipeline Editor.

The Pipeline Editor and all the other tabs in the Pipelines page are described below.

Pipeline Editor¶

The Pipeline Editor tab (Editor tab hereafter) serves as your main work area for building your pipeline.

It is composed of the canvas, Node Palette, and Node Properties panel.

On the canvas:

Drag and drop nodes from the Node Palette on the right hand side to start creating your pipeline.
These nodes can represent programs, shell scripts, or defined completion status to enable testing parts of the pipeline.
Nodes can be linked together by drawing arrows between them.
The pipeline begins from the Start node. Links can be made between nodes on the canvas, but the links between nodes must form a directed acyclic graph.

You can also undo or redo your changes, zoom in and out of the canvas, view the full canvas, and lock or unlock the canvas.

An overview of the entire canvas is displayed on the lower right hand side of the canvas.

Node Palette¶

On the right hand side of the Editor tab is the Node Palette, which contains the following nodes:

SLC: Use this node to run an SLC program by writing code or selecting from a list of supported sources.
Script: Use this node to run a script by selecting from the following options: Bash, CMD, or PowerShell.
Control: Use this node to control the flow of your pipeline and handle conditional outputs.
Raw Exec: Use this node to execute a raw command on the target system.

You can drag and drop these nodes to the canvas to build your pipeline. You can also add a sticky note to your canvas to add a note or comment to your pipeline.

Clicking on a node on the canvas activates the Node Properties panel from where you can modify each node's properties.

Node Properties¶

The Node Properties panel contains all properties of a node that you can modify as you build your pipeline.

You can also use it to add inputs to the nodes (except Control Nodes) and results from the node. Results are files generated by the node that will be stored after it has run. Results can only come from the working directory of the node, and the path to the result is a relative path within the working directory.

You can modify the names of all nodes that you will use. All other properties are described further here.

SLC Node

Type: The SLC Node may be a program or package.

When you select the package type, you need to specify the source type (i.e., Hub artefact, git repository, or file path) and provide the name of the program that the node will execute.

When you choose the program type, you can further select the Allow warning return code checkbox so that the node is not marked as failed if the SLC program returns the return code that is normally generated a warning in the log.

You can also choose to specify whether the node is conditional or non-conditional using the Is Condition checkbox. This checkbox allows you to specify whether the node is conditional or non-conditional.

When not selected, the node shows the following conditions:
- Success. The destination node of the link is only executed if the completion status of the source node is "Success".
- Error. The destination node of the link is only executed if the completion status of the source node is "Error".
- Any. The destination node of the link is always executed regardless of the completion status of the source node.
When selected, the node shows the following conditions:
- True. The destination node of the link is only executed if the completion status of the source node is "True".
- False. The destination node of the link is only executed if the completion status of the source node is "False".
- Error. The destination node of the link is only executed if the completion status of the source node is "Error".
- Any. The destination node of the link is always executed regardless of the completion status of the source node.
By default, the final status of a node will be either "Success" or "Failure." If the conditional option is selected, the final node status becomes "True" or "False" instead.
Source Type: The source of the SLC Node may be a text editor, file path, Hub artefact, or git repository (see On Demand for how to upload an artefact to the Altair SLC Hub).
The properties you can further modify depend on the source type you select. For example, when you select text editor as your source type, you can enter your code directly into the field provided.

You can also specify environment variable values and system options for the SLC Node.

Script Node

Type: The Script Node may be a shell script, Windows batch script, or a PowerShell script.

Note

The default execution policy on Windows restricts the execution of PowerShell scripts. If a Windows PowerShell Script node does not execute as expected, consult the Windows Execution Policy documentation to ensure that the operating system is configured to allow execution of the node.
Source Type: Similar to the SLC Node, the Script Node may have a text editor, file path, Hub artefact, or git repository as its source.
Is Condition: Use this checkbox to specify whether the node is conditional or non-conditional.
You may refer to the conditions described under SLC Node.

When using the Script Node, you may also add environment variables and arguments.

Control Node

Control nodes provide a mechanism for synchronising the completion of nodes that execute in parallel: the nodes running in parallel all have a link to the control node.

In the Node Properties panel, you can select the type of Control Node to use. The available types are as follows:

Wait: The node waits until all the source nodes have completed execution.
If all the source nodes complete successfully, the completion status of the node is "Success."
If any of the source nodes complete unsuccessfully, the completion status of the node is "Failure."
All: This type is similar to Wait except that the node is a conditional one.
If all the source nodes complete successfully, the completion status of the node is "True."
If any of the source nodes complete unsuccessfully, the completion status of the node is "False."
Any: The node is set as a conditional node.
If any of the source nodes complete successfully, the completion status of the node is "True."
If none of the source nodes complete successfully, the completion status of the node is "False."

Note

The source node for the Any and All types must be conditional in order for the node to complete with a "True" or "False" result.

Raw Exec Node

When using this node, you need to specify the operating system command to be executed.

You can also specify whether the node is conditional or non-conditional by using the Is Condition checkbox.

Similarly, you may add environment variable values and arguments.

Properties tab¶

In the Pipelines page, you can switch to the Properties tab to modify the properties of the pipeline.

These properties are the name of the pipeline and the Continue on unhandled error checkbox, which allows you to specify whether to have all nodes continue when there are unhandled errors, except for the descendants of nodes with such errors, which are consequently skipped.

Triggers tab¶

In the Triggers tab, you can define trigger conditions or schedules to automatically run the pipeline.

This tab also displays existing triggers, which you can edit, delete, or disable by clicking more (...) to the right of each entry.

To create a trigger:

Click the New button.
In the dialog box that displays, choose between a time-based trigger type or file event trigger type.
Click Next.

Modify the properties that display, which may vary with the selected trigger type:
For time-based triggers, set the schedule as follows and then specify the time zone:

Trigger	Effect
Recurring	Select the desired recurrence type from the following options: Minutes, Hourly, Daily, Weekly, Monthly, and Yearly. Modify further the properties that become available depending on your selected recurrence type.
Once	Specify the time and date when the trigger will run once.
Advanced	Provide the cron expression to use for the schedule.

For file event triggers, modify the following:

Trigger	Effect
Filename	Enter the path to the files on which the trigger event is based. The path may contain "?" or * wildcard characters.
Trigger pipeline when file arrives/exists	Choose to trigger the pipeline when the specified file arrives or when the file exists.

Click Next.
Select the Execution Profile to use from the dropdown list and then click Next.
Review the settings you made and then click Finish.
If you need to make changes, you may return to the previous options by clicking Previous.

More information is available here.

Pipeline Runs¶

In the Pipeline Runs tab, you can view the status of the pipeline run, including the status of each node.

Click the Download button on the canvas to select the pipeline run you wish to view from the selected Execution Profile.

The right hand side displays the selected pipeline run's execution information, execution timeline, and node results.

Click on any node to view its execution information, results, and properties.

You can also download the results or view it with your desired text encoding.

More information is available here.

Executing the pipeline¶

Select the Execution Profile from the dropdown list and then click the Save and run button to save the pipeline and begin executing it. The Pipeline Runs page is subsequently displayed.

You may choose to save the pipeline without running it by clicking Save.