How-to Guide: Creating Nested Workflows in CWL¶

This guide explains how to build and use nested workflows in CWL by leveraging the SubworkflowFeatureRequirement.

The focus is on the workflow composition and the integration of subworkflows to create reusable components.

Objective¶

Main Workflow: Accepts inputs and calls a subworkflow (rgb-composite) as a single step.
Subworkflow (rgb-composite): Performs a series of steps to process data and produce the desired output.

Key Blocks¶

SubworkflowFeatureRequirement

The SubworkflowFeatureRequirement allows workflows to include other workflows as steps.

In [ ]:

Copied!

cwl_file="../cwl-workflows/nested-workflow.cwl"

cat $cwl_file | yq e .'$graph[0]' | yq '.. | select(has("SubworkflowFeatureRequirement")) | {"requirements": {"SubworkflowFeatureRequirement": .SubworkflowFeatureRequirement}}' 
cwl_file="../cwl-workflows/nested-workflow.cwl"

cat $cwl_file | yq e .'$graph[0]' | yq '.. | select(has("SubworkflowFeatureRequirement")) | {"requirements": {"SubworkflowFeatureRequirement": .SubworkflowFeatureRequirement}}'

Subworkflow Step Definition

The main workflow calls the subworkflow using:

In [ ]:

Copied!

cat $cwl_file | yq e .'$graph[0].steps' 
cat $cwl_file | yq e .'$graph[0].steps'

run: "#rgb-composite": Links to the rgb-composite subworkflow.
Inputs and Outputs: The subworkflow accepts inputs (stac-item, bands) and produces an output (rgb-tif).

Steps¶

Define the Subworkflow

The rgb-composite subworkflow performs the following:

Fetch band-specific asset URLs using the stac tool.
Stack the asset TIFFs into a single file using the rio_stack tool.
Apply color correction to generate the RGB composite using the rio_color tool.

Subworkflow Definition (rgb-composite)

In [ ]:

Copied!

cat $cwl_file | yq e .'$graph[1]'  -
cat $cwl_file | yq e .'$graph[1]'  -

In [ ]:

Copied!

cwltool --print-dot $cwl_file#rgb-composite 2> /dev/null | dot -Tpng | display
cwltool --print-dot $cwl_file#rgb-composite 2> /dev/null | dot -Tpng | display

Define the Main Workflow

The main workflow invokes the rgb-composite subworkflow:

In [ ]:

Copied!

cat $cwl_file | yq e .'$graph[0]'  -
cat $cwl_file | yq e .'$graph[0]'  -

Requirements:
- SubworkflowFeatureRequirement: Enables the use of nested workflows.
- ScatterFeatureRequirement: Allows processing multiple bands simultaneously.
Inputs:
- stac-item: URL to a STAC item.
- bands: Array of band names (default: ["red", "green", "blue"]).
Outputs:
- rgb-tif: The RGB composite file produced by the subworkflow.

In [ ]:

Copied!

cwltool --print-dot $cwl_file 2> /dev/null | dot -Tpng | display
cwltool --print-dot $cwl_file 2> /dev/null | dot -Tpng | display

Run the Workflow

To execute the main workflow, use the following command:

cwltool nested-workflow.cwl \
  --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A

In [ ]:

Copied!





cat > input.json <<'JSON'
{
  "stac-item": {
    "value": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A"
  }
}
JSON
cwltool --podman "$cwl_file" input.json > output.json
cat > input.json <<'JSON'
{
  "stac-item": {
    "value": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A"
  }
}
JSON
cwltool --podman "$cwl_file" input.json > output.json

Expected Output

Intermediate Outputs:

URLs of band-specific TIFFs (hrefs).
Stacked TIFF file (stacked.tif).

Final Output:

RGB composite TIFF file (rgb-tif).

In [ ]:

Copied!

cat output.json | yq e . -o=json -
cat output.json | yq e . -o=json -

Key Takeaways¶

Modularity with Subworkflows:

Use SubworkflowFeatureRequirement to encapsulate reusable workflows.
Subworkflows simplify complex workflows by isolating specific logic.

Integration of Subworkflows:

Define subworkflow steps in the main workflow.
Use run to link the subworkflow.

Reusability:

Subworkflows can be reused in multiple workflows, promoting modularity and efficiency.

This approach makes it easy to manage and scale CWL workflows by leveraging nested subworkflows.

In [ ]:

Copied!

rm -f output.json
rm -f rgb.tif
rm -f output.json
rm -f rgb.tif