How-to Guide: Using Conditional Workflows¶

This guide explains how to create workflows with conditional execution using when and pickValue.

These features allow tasks to run conditionally based on inputs and dynamically select outputs.

Objective¶

Process raster files using rio_stack if the EPSG code is "native".
Use rio_warp_stack if the EPSG code is not "native".
Dynamically choose the appropriate output using pickValue: the_only_non_null.

Key Features¶

Conditional Execution with when

The when field controls whether a step is executed based on an input condition.

when: $( inputs.epsg_code == "native")

when: $( inputs.epsg_code != "native")

Executes the step only if the condition evaluates to true.

Dynamic Output Selection with pickValue

The pickValue field determines how to handle multiple output sources.

pickValue: the_only_non_null

pickValue: the_only_non_null

Selects the only non-null value from the list of outputs.
Ensures only one output is passed downstream.

Steps¶

Define the Workflow

Workflow Inputs and Outputs

The workflow accepts:

stac-item: A STAC item URL.
epsg_code: EPSG code for projection (default: "native").
bands: An array of band names (e.g., ["red", "green", "blue"]).

Workflow Definition

cwlVersion: v1.2

$graph:

  - class: Workflow

    id: main

    requirements:

      InlineJavascriptRequirement: {}

      NetworkAccess:

        networkAccess: true

      ScatterFeatureRequirement: {}

      MultipleInputFeatureRequirement: {}

    inputs:

      stac-item:

        type: string

      epsg_code:

        type: string

        default: "native"

      bands:

        type: string[]

    outputs:

      rgb-tif:

        outputSource: step_color/rgb

        type: File

      stack:

        outputSource:

          - step_stack/stacked

          - step_warp_stack/stacked

        pickValue: the_only_non_null

        type: File

    steps:

      step_curl:

in:

          stac_item: stac-item

          common_band_name: bands

        out:

          - hrefs

        run: "#stac"

        scatter: common_band_name

        scatterMethod: dotproduct

      step_stack:

in:

          tiffs:

            source: step_curl/hrefs

          epsg_code: epsg_code

        out:

          - stacked

        run: "#rio_stack"

        when: $( inputs.epsg_code == "native")

      step_warp_stack:

in:

          tiffs:

            source: step_curl/hrefs

          epsg_code: epsg_code

        out:

          - stacked

        run: "#rio_warp_stack"

        when: $( inputs.epsg_code != "native")

      step_color:

in:

          stacked:

            source:

              - step_stack/stacked

              - step_warp_stack/stacked

            pickValue: the_only_non_null

        out:

          - rgb

        run: "#rio_color"

  - class: CommandLineTool

    id: stac

    requirements:

      DockerRequirement:

        dockerPull: docker.io/curlimages/curl:latest

    baseCommand: curl

    stdout: message

    arguments:

      - $( inputs.stac_item )

    inputs:

      stac_item:

        type: string

      common_band_name:

        type: string

    outputs:

      hrefs:

        type: string

        outputBinding:

          glob: message

          loadContents: true

          outputEval: |

${

              const assets = JSON.parse(self[0].contents).assets;

              const bandKey = Object.keys(assets).find(key =>

                assets[key]['eo:bands'] &&

                assets[key]['eo:bands'].length === 1 &&

                assets[key]['eo:bands'].some(band => band.common_name === inputs.common_band_name)

);

              if (!bandKey) {

                throw new Error(`No valid asset found for band: ${inputs.common_band_name}`);

              return assets[bandKey].href;

  - class: CommandLineTool

    id: rio_stack

    requirements:

      DockerRequirement:

        dockerPull: ghcr.io/eoap/how-to/rio:1.0.0

      EnvVarRequirement:

        envDef:

          GDAL_TIFF_INTERNAL_MASK: YES

          GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: YES

          CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".tif"

      InitialWorkDirRequirement:

        listing:

          - entryname: run.sh

            entry: |-

              #!/bin/bash

              rio stack $@

    baseCommand: ["/bin/bash", "run.sh"]

    arguments:

      - valueFrom: "${  \n  var arr = [];\n  for(var i=0; i<inputs.tiffs.length; i++) {\n      arr.push(inputs.tiffs[i]); \n  }\n  return arr; \n  }\n"

      - stacked.tif

    inputs:

      tiffs:

        type: string[]

    outputs:

      stacked:

        type: File

        outputBinding:

          glob: stacked.tif

  - class: CommandLineTool

    id: rio_warp_stack

    requirements:

      DockerRequirement:

        dockerPull: ghcr.io/eoap/how-to/rio:1.0.0

      EnvVarRequirement:

        envDef:

          GDAL_TIFF_INTERNAL_MASK: YES

          GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: YES

          CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".tif"

      InitialWorkDirRequirement:

        listing:

          - entryname: run.sh

            entry: |-

              #!/bin/bash

              rio stack $@

              rio warp --dst-crs $(inputs.epsg_code) stacked.tif warped.tif

    baseCommand: ["/bin/bash", "run.sh"]

    arguments:

      - valueFrom: "${  \n  var arr = [];\n  for(var i=0; i<inputs.tiffs.length; i++) {\n      arr.push(inputs.tiffs[i]); \n  }\n  return arr; \n  }\n"

      - stacked.tif

    inputs:

      tiffs:

        type: string[]

      epsg_code:

        type: string

    outputs:

      stacked:

        type: File

        outputBinding:

          glob: warped.tif

  - class: CommandLineTool

    id: rio_color

    requirements:

      DockerRequirement:

        dockerPull: ghcr.io/eoap/how-to/rio:1.0.0

      InitialWorkDirRequirement:

        listing:

          - entryname: run.sh

            entry: |-

              #!/bin/bash

              rio color -j -1 --out-dtype uint8 $1 rgb.tif "gamma 3 0.95, sigmoidal rgb 35 0.13"

    baseCommand: ["/bin/bash", "run.sh"]

    arguments:

      - $( inputs.stacked.path )

    inputs:

      stacked:

        type: File

    outputs:

      rgb:

        type: File

        outputBinding:

          glob: rgb.tif

Its graphical representation:

No description has been provided for this image

Conditional Steps

Step 1: Use rio_stack for Native EPSG

step_stack:

in:

    tiffs:

      source: step_curl/hrefs

    epsg_code: epsg_code

  out:

    - stacked

  run: "#rio_stack"

  when: $( inputs.epsg_code == "native")

Executes only if epsg_code == "native".

Step 2: Use rio_warp_stack for Reprojection

step_warp_stack:

in:

    tiffs:

      source: step_curl/hrefs

    epsg_code: epsg_code

  out:

    - stacked

  run: "#rio_warp_stack"

  when: $( inputs.epsg_code != "native")

Executes only if epsg_code != "native".

Dynamic Output Selection

Stack Selection

stack:

  outputSource:

    - step_stack/stacked

    - step_warp_stack/stacked

  pickValue: the_only_non_null

  type: File

Dynamically selects the output from step_stack or step_warp_stack.
Ensures the downstream step (step_color) receives the correct output.

Run the Workflow

Run the workflow with the following commands based on the desired EPSG code:

Native EPSG Code:

cwltool conditional-workflows.cwl \
    --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A  \
    --bands red \
    --bands green \
    --bands blue \
    --epsg_code native

INFO /opt/hostedtoolcache/Python/3.12.8/x64/bin/cwltool 3.1.20241217163858

INFO Resolved '../cwl-workflows/conditional-workflows.cwl' to 'file:///home/runner/work/how-to/how-to/cwl-workflows/conditional-workflows.cwl'

INFO [workflow ] start

INFO [workflow ] starting step step_curl

INFO [step step_curl] start

INFO [job step_curl] /tmp/29mu0ppj$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/29mu0ppj,target=/tcfuqA \

    --mount=type=bind,source=/tmp/2g7nevqo,target=/tmp \

    --workdir=/tcfuqA \

    --read-only=true \

    --log-driver=none \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/9o4cd6xe/20250102120109-404543.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/tcfuqA \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/29mu0ppj/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  46214      0 --:--:-- --:--:-- --:--:-- 46163

100 10156  100 10156    0     0  46190      0 --:--:-- --:--:-- --:--:-- 46163

INFO [job step_curl] completed success

INFO [step step_curl] start

INFO [job step_curl_2] /tmp/ex2ewj9n$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/ex2ewj9n,target=/tcfuqA \

    --mount=type=bind,source=/tmp/0eeyxu8a,target=/tmp \

    --workdir=/tcfuqA \

    --read-only=true \

    --log-driver=none \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/dy496gsc/20250102120110-416396.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/tcfuqA \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/ex2ewj9n/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

 77 10156   77  7781    0     0  39915      0 --:--:-- --:--:-- --:--:-- 39902

100 10156  100 10156    0     0  52020      0 --:--:-- --:--:-- --:--:-- 51816

INFO [job step_curl_2] completed success

INFO [step step_curl] start

INFO [job step_curl_3] /tmp/8oukn0gr$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/8oukn0gr,target=/tcfuqA \

    --mount=type=bind,source=/tmp/rer54wgm,target=/tmp \

    --workdir=/tcfuqA \

    --read-only=true \

    --log-driver=none \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/qedzc_xe/20250102120111-427659.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/tcfuqA \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/8oukn0gr/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  84267      0 --:--:-- --:--:-- --:--:-- 84633

INFO [job step_curl_3] completed success

INFO [step step_curl] completed success

INFO [workflow ] starting step step_warp_stack

INFO [step step_warp_stack] will be skipped

INFO [step step_warp_stack] completed skipped

INFO [workflow ] starting step step_stack

INFO [step step_stack] start

INFO [job step_stack] /tmp/qar4m1_5$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/qar4m1_5,target=/tcfuqA \

    --mount=type=bind,source=/tmp/kcwfko56,target=/tmp \

    --workdir=/tcfuqA \

    --read-only=true \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/tg0aqdsb/20250102120112-456299.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/tcfuqA \

    --env=CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif \

    --env=GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES \

    --env=GDAL_TIFF_INTERNAL_MASK=YES \

    ghcr.io/eoap/how-to/rio:1.0.0 \

    /bin/bash \

    run.sh \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B04.tif \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B03.tif \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B02.tif \

    stacked.tif

INFO [job step_stack] Max memory used: 1257MiB

INFO [job step_stack] completed success

INFO [step step_stack] completed success

INFO [workflow ] starting step step_color

INFO [step step_color] start

INFO [job step_color] /tmp/btdwe0ll$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/btdwe0ll,target=/tcfuqA \

    --mount=type=bind,source=/tmp/24ariqp6,target=/tmp \

    --mount=type=bind,source=/tmp/qar4m1_5/stacked.tif,target=/var/lib/cwl/stg30f0136f-2fb6-4bc2-9c55-71f332c1b77b/stacked.tif,readonly \

    --workdir=/tcfuqA \

    --read-only=true \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/vj8cprdr/20250102120149-512398.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/tcfuqA \

    ghcr.io/eoap/how-to/rio:1.0.0 \

    /bin/bash \

    run.sh \

    /var/lib/cwl/stg30f0136f-2fb6-4bc2-9c55-71f332c1b77b/stacked.tif

INFO [job step_color] Max memory used: 714MiB

INFO [job step_color] completed success

INFO [step step_color] completed success

INFO [workflow ] completed success

INFO Final process status is success

Reprojected EPSG Code

cwltool conditional-workflows.cwl \
    --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A  \
    --bands red \
    --bands green \
    --bands blue \
    --epsg_code "EPSG:4326"

Expected Output

Intermediate Outputs:

stacked file from either step_stack or step_warp_stack.

Final Output:

rgb-tif: RGB composite file produced by step_color.

  "rgb-tif": {

    "location": "file:///home/runner/work/how-to/how-to/docs/rgb.tif",

    "basename": "rgb.tif",

    "class": "File",

    "checksum": "sha1$538196e1ff7fe82b762e62bcada6222169a53437",

    "size": 361747464,

    "path": "/home/runner/work/how-to/how-to/docs/rgb.tif"

},

  "stack": {

    "location": "file:///home/runner/work/how-to/how-to/docs/stacked.tif",

    "basename": "stacked.tif",

    "class": "File",

    "checksum": "sha1$a9fcfcac28c669f3714eaa4cc59858a28f1e715b",

    "size": 723450636,

    "path": "/home/runner/work/how-to/how-to/docs/stacked.tif"

Key Takeaways¶

Conditional Steps:

Use when to execute steps based on input conditions.
Allows workflows to adapt to varying requirements dynamically.

Dynamic Output Handling:

Use pickValue to select the appropriate output among multiple sources.

Flexible Workflows:

This approach creates workflows that handle both native and reprojected data seamlessly.

This guide demonstrates how to build adaptable workflows in CWL, leveraging conditional execution and dynamic output selection.