Pipelines
Overview¶
Pipelines provide the facility to execute a batch of programs.
As well as basic sequential chaining of Altair SLC programs, pipelines support nodes implemented by operating system scripts, parallel execution of nodes (with synchronization upon completion) and conditional execution of nodes.
Portal interface¶
Pipelines¶
On the Pipelines page, click the Pipeline button to create a pipeline. This presents a canvas populated with the mandatory Start node.
Items can be dragged and dropped from the palette on the left hand side to created nodes on the canvas. Nodes can be linked together by drawing arrows between them (the links must form a directed acyclic graph).
Program Nodes¶
Double click a Pipeline or Script node to open an editor for the node. The implementation may be provided in the Text Editor, or as a File Path, Hub Artefact or Git Repository (see On Demand for how to upload an artefact to the Altair SLC Hub).
The editor also supports entry of environment variable values, the definition of Inputs to the node, and Results from the node.
Results are files generated by the node that will be stored after it has run. Results can only come from the working directory of the node, and the path to the result is a relative path within the working directory. The Results from one node may become Inputs to a subsequent node.
There is also the option to select whether the node is conditional. By default, the final status of a node will be either "Success" or "Failure". If the conditional option is selected, the final node status becomes "True" or "False" instead.
Note
The default execution policy on Windows restricts the execution of Powershell scripts. If a Windows Powershell Script node does not execute as expected, consult the Windows Execution Policy documentation to ensure that the operating system is configured to allow execution of the node.
Links¶
A node may have multiple links to other nodes, and the type of each link may be edited.
For links from non-conditional nodes, the type may be "Success" (the destination node of the link is only executed if the completion status of the source node is "Success"), "Failure" (the destination node of the link is only executed if the completion status of the source node is "Failure") or "Any" (the destination node of the link is always executed no matter what the completion status of the source node).
For links from conditional nodes, the type may be "True" (the destination node of the link is only executed if the completion status of the source node is "True"), "False" (the destination node of the link is only executed if the completion status of the source node is "False") or "Any" (the destination node of the link is always executed no matter what the completion status of the source node).
Similarly multiple nodes may link to a single node.
Control Nodes¶
Control nodes provide a mechanism for synchronising the completion of nodes that execute in parallel: the nodes running in parallel all have a link to the control node.
The Wait node waits until all the source nodes have completed execution. If all the source nodes complete successfully the completion status of the Wait node is "Success". If any of the source nodes completes unsuccessfully the completion status of the Wait node is "Failure".
The All node is like the Wait node except it is a conditional node. If all the source nodes complete successfully the completion status of the All node is "True". If any of the source nodes completes unsuccessfully the completion status of the All node is "False".
The Any node is also a conditional node. If any of the source nodes completes successfully the completion status of the Any node is "True". If none of the source nodes completes successfully the completion status of the Any node is "False".
Basic Nodes¶
The completion status of the Start node is always "Any".
The completion status of the Success node is always "Success".
The completion status of the Failure node is always "Failure".
The completion status of the True node is always "True".
The completion status of the False node is always "False".
Executing the pipeline¶
Click the run button to save the pipeline, begin executing it and switch to the Pipeline Runs page.
Pipeline Runs¶
When a pipeline run is started from a pipeline the Pipeline Runs page displays a graphical representation of the executing pipeline. The current status of each node in the pipeline is represented by a coloured bar on the left hand side of the node. The results from the node can be viewed on completion of the node.
There is a Tabular tab; when this is selected the same information is presented in a textual format.
The Pipeline Runs page maintains a list of all pipeline executions.
Pipeline Triggers¶
The Pipelines page maintains a list of all pipelines.
The ... (more information button) at the right of each pipeline can be used to View More information about a pipeline.
The New button in the Triggers section creates a new trigger for the pipeline.
There are two types of trigger. Time based triggers and file based triggers.
Time based triggers can be Recurring or single use (Once) triggers.
There is also an Advanced option to specify a trigger in cron format for more complex time based triggers.
File based triggers can be configured to trigger either on the arrival or existence of a file.
File triggers are checked for on the Altair SLC Hub server machine, so trigger files must be on the server, or on a shared drive mounted to the server.
The file trigger path can make use of wildcard characters: ? matches any non-separator character; * matches a sequence of any non-separator characters.