Pipelines

Pipelines are a core concept in Phaedra, as they are the “glue” that bind data processing steps together in an automated fashion. While they are entirely optional, and you can perform each step manually, you will save a lot of time and effort by using pipelines, especially in large-volume projects or assays.

But before a pipeline can be used, it must be set up correctly, a task which requires some knowledge about the Phaedra system and some careful configuration. This section describes the more technical details of pipeline setup.

The Pipeline Model

Pipelines have some evident properties such as a name, description, version number, etc. But at the heart of the pipeline model lies the definition, which is expressed as a JSON document.

For more information about the JSON syntax, see:

• https://www.json.org/json-en.html
• https://www.w3schools.com/js/js_json_intro.asp

The pipeline definition describes a series of steps, where each step consists of a trigger and an action.

• The trigger describes the event that should initiate the execution of the current step.
• The action describes what should be done when the step gets executed.

However, you’ll find that many steps actually omit the trigger part. This is because most actions have an implicit trigger that fires automatically when the action is complete. The notable exception is the first step, which must always have an explicit trigger.

Consider the definition below:

{
    "steps": [
        {
            "trigger": {
                "type": "MeasurementCaptured",
                "config": {
                    "pattern": ".*Barcode1234.*"
                }
            },
            "action": {
                "type": "LinkPlateMeasurement",
                "config": {
                    "projectId": 500
                }
            }
        },
        {
            "action": {
                "type": "LinkPlateDefinition",
                "config": {
                    "name": "My 8 x 12 Layout"
                }
            }
        },
        {
            "action": {
                "type": "CalculateProtocol",
                "config": {
                    "id": 200
                }
            }
        }
    ]
}

The first step contains both a trigger (MeasurementCaptured) and an action (LinkPlateMeasurement). This is pretty straightforward: whenever a new measurement is captured into Phaedra whose name matches the given pattern, a new pipeline execution will begin for this pipeline. The first thing that will happen in this execution, is that the new measurement will be linked to a plate in the project with ID 500. More details about the LinkPlateMeasurement action can be found further down in this document.

The second step does not have an explicit trigger. This means that the step will be invoked automatically when the action of the previous step (LinkPlateMeasurement) is completed. At that moment, action LinkPlateDefinition is invoked, taking the name of a layout definition (My 8 x 12 Layout) as a parameter.

The third and last step also uses an implicit trigger, and invokes the CalculateProtocol action with protocol ID 200 as soon as the previous step’s LinkPlateDefinition action is completed.

Pipeline Execution Variables

During pipeline execution, variables are obtained and stored, to be passed on and used between the different steps of the pipeline.

For example, the MeasurementCaptured trigger will automatically capture two variables: barcode and measurementId, both obtained from the new measurement that was captured. These variables are available to all subsequent triggers and actions of the pipeline, so that they can know which measurement is being dealt with.

Similarly, the LinkPlateMeasurement action will capture an additional variable plateId, referencing the plate that the measurement was linked to.

Available Trigger Types

When designing a new pipeline, you can choose between several trigger types to start off pipeline execution.

MeasurementCaptured

This is a trigger that fires whenever a measurement gets captured by the Phaedra’s DataCapture system.

Event

This is a generic trigger that can fire whenever a Kafka event is received matching certain conditions. Please note that this is a more advanced trigger type.

Available Action Types

Below is a list of available action types you can choose from when building a pipeline. The same action type can be used repeatedly in a pipeline.

Each action type will list, in addition to its available configuration settings, a list of required variables. These variables should be made available by preceding steps. For example, using the MeasurementCaptured trigger ensures the required variables for the LinkPlateMeasurementAction action will be available. If any required variables are not available at the time of execution, the execution will fail with an error message.

LinkPlateMeasurementAction

This action will link a measurement to a plate whose barcode matches the measurement’s.

LinkPlateDefinitionAction

This action will link a plate with a plate definition.

CalculateProtocolAction

This action will execute a protocol on a plate.