How-to Guide: Creating Files at Runtime¶

This guide explains how to create files dynamically at runtime in a CWL workflow using the InitialWorkDirRequirement.

The focus will be on defining a shell script (run.sh) as part of the workflow to handle runtime commands.

Objective¶

Dynamically create a run.sh script at runtime that executes specific commands for each tool in the workflow.

Relevant Block:

InitialWorkDirRequirement:

  listing:

    - entryname: run.sh

      entry: |-

        #!/bin/bash

        rio stack $@

InitialWorkDirRequirement: Specifies files or directories to be created dynamically in the working directory.
entryname: The name of the file to be created (run.sh).
entry: The contents of the file, which in this case is a shell script.

Steps¶

Understand the Workflow Structure

The workflow (runtime-file.cwl) processes a Sentinel-2 STAC item to:

Fetch band URLs (stac tool).
Stack TIFFs (rio_stack tool) using a runtime script.
Apply color correction (rio_color tool) using another runtime script.

Define the InitialWorkDirRequirement

For rio_stack Tool

The run.sh script stacks the provided TIFFs into a single file (stacked.tif):

InitialWorkDirRequirement:

  listing:

    - entryname: run.sh

      entry: |-

        #!/bin/bash

        rio stack $@

Contents:
- The script (run.sh) executes rio stack with all TIFF file paths passed as arguments ($@).

For rio_color Tool

The run.sh script applies color correction to the stacked TIFF file and produces an RGB TIFF (rgb.tif):

InitialWorkDirRequirement:

  listing:

    - entryname: run.sh

      entry: |-

        #!/bin/bash

        rio color -j -1 --out-dtype uint8 $1 rgb.tif "gamma 3 0.95, sigmoidal rgb 35 0.13"

Contents:
- The script (run.sh) applies color correction to the file passed as the first argument ($1).

Specify the baseCommand

Each tool uses /bin/bash to execute the dynamically created run.sh script:

baseCommand: ["/bin/bash", "run.sh"]

This ensures the created script run.sh is executed in a shell environment.

Complete Workflow Example

Here’s the full CWL example (runtime-file.cwl):

cwlVersion: v1.2

$graph:

  - class: Workflow

    id: main

    requirements:

      InlineJavascriptRequirement: {}

      NetworkAccess:

        networkAccess: true

      ScatterFeatureRequirement: {}

    inputs:

      stac-item:

        type: string

      bands:

        type: string[]

    outputs:

      rgb-tif:

        outputSource: step_color/rgb

        type: File

    steps:

      step_curl:

in:

          stac_item: stac-item

          common_band_name: bands

        out:

          - hrefs

        run: "#stac"

        scatter: common_band_name

        scatterMethod: dotproduct

      step_stack:

in:

          tiffs:

            source: step_curl/hrefs

        out:

          - stacked

        run: "#rio_stack"

      step_color:

in:

          stacked:

            source: step_stack/stacked

        out:

          - rgb

        run: "#rio_color"

  - class: CommandLineTool

    id: stac

    requirements:

      DockerRequirement:

        dockerPull: docker.io/curlimages/curl:latest

    baseCommand: curl

    stdout: message

    arguments:

      - $( inputs.stac_item )

    inputs:

      stac_item:

        type: string

      common_band_name:

        type: string

    outputs:

      hrefs:

        type: string

        outputBinding:

          glob: message

          loadContents: true

          outputEval: |

${

              const assets = JSON.parse(self[0].contents).assets;

              const bandKey = Object.keys(assets).find(key =>

                assets[key]['eo:bands'] &&

                assets[key]['eo:bands'].length === 1 &&

                assets[key]['eo:bands'].some(band => band.common_name === inputs.common_band_name)

);

              if (!bandKey) {

                throw new Error(`No valid asset found for band: ${inputs.common_band_name}`);

              return assets[bandKey].href;

  - class: CommandLineTool

    id: rio_stack

    requirements:

      DockerRequirement:

        dockerPull: ghcr.io/eoap/how-to/rio:1.0.0

      EnvVarRequirement:

        envDef:

          GDAL_TIFF_INTERNAL_MASK: YES

          GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: YES

          CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".tif"

      InitialWorkDirRequirement:

        listing:

          - entryname: run.sh

            entry: |-

              #!/bin/bash

              rio stack $@

    baseCommand: ["/bin/bash", "run.sh"]

    arguments:

      - valueFrom: "${  \n  var arr = [];\n  for(var i=0; i<inputs.tiffs.length; i++) {\n      arr.push(inputs.tiffs[i]); \n  }\n  return arr; \n  }\n"

      - stacked.tif

    inputs:

      tiffs:

        type: string[]

    outputs:

      stacked:

        type: File

        outputBinding:

          glob: stacked.tif

  - class: CommandLineTool

    id: rio_color

    requirements:

      DockerRequirement:

        dockerPull: ghcr.io/eoap/how-to/rio:1.0.0

      InitialWorkDirRequirement:

        listing:

          - entryname: run.sh

            entry: |-

              #!/bin/bash

              rio color -j -1 --out-dtype uint8 $1 rgb.tif "gamma 3 0.95, sigmoidal rgb 35 0.13"

    baseCommand: ["/bin/bash", "run.sh"]

    arguments:

      - $( inputs.stacked.path )

    inputs:

      stacked:

        type: File

    outputs:

      rgb:

        type: File

        outputBinding:

          glob: rgb.tif

Graphical representation:

No description has been provided for this image

Run the Workflow

Run the workflow with the following command:

cwltool runtime-file.cwl \
  --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
  --bands red \
  --bands green \
  --bands blue

INFO /opt/hostedtoolcache/Python/3.12.8/x64/bin/cwltool 3.1.20241217163858

INFO Resolved '../cwl-workflows/runtime-file.cwl' to 'file:///home/runner/work/how-to/how-to/cwl-workflows/runtime-file.cwl'

INFO [workflow ] start

INFO [workflow ] starting step step_curl

INFO [step step_curl] start

INFO [job step_curl] /tmp/oif1vrvi$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/oif1vrvi,target=/wrAEHP \

    --mount=type=bind,source=/tmp/ru549mv9,target=/tmp \

    --workdir=/wrAEHP \

    --read-only=true \

    --log-driver=none \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/0ke8fkut/20250102120501-862192.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/wrAEHP \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/oif1vrvi/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  21395      0 --:--:-- --:--:-- --:--:-- 21426

INFO [job step_curl] completed success

INFO [step step_curl] start

INFO [job step_curl_2] /tmp/if19pzv0$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/if19pzv0,target=/wrAEHP \

    --mount=type=bind,source=/tmp/k0cn3q8i,target=/tmp \

    --workdir=/wrAEHP \

    --read-only=true \

    --log-driver=none \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/jiyxy26r/20250102120502-874101.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/wrAEHP \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/if19pzv0/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  37080      0 --:--:-- --:--:-- --:--:-- 37201

INFO [job step_curl_2] completed success

INFO [step step_curl] start

INFO [job step_curl_3] /tmp/pltd57uu$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/pltd57uu,target=/wrAEHP \

    --mount=type=bind,source=/tmp/8629qatr,target=/tmp \

    --workdir=/wrAEHP \

    --read-only=true \

    --log-driver=none \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/oovpxbwp/20250102120503-885513.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/wrAEHP \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/pltd57uu/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  30161      0 --:--:-- --:--:-- --:--:-- 30226

INFO [job step_curl_3] completed success

INFO [step step_curl] completed success

INFO [workflow ] starting step step_stack

INFO [step step_stack] start

INFO [job step_stack] /tmp/4146qj1m$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/4146qj1m,target=/wrAEHP \

    --mount=type=bind,source=/tmp/g06y2p7e,target=/tmp \

    --workdir=/wrAEHP \

    --read-only=true \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/rrbkhq6z/20250102120504-911646.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/wrAEHP \

    --env=CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif \

    --env=GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES \

    --env=GDAL_TIFF_INTERNAL_MASK=YES \

    ghcr.io/eoap/how-to/rio:1.0.0 \

    /bin/bash \

    run.sh \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B04.tif \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B03.tif \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B02.tif \

    stacked.tif

INFO [job step_stack] Max memory used: 1273MiB

INFO [job step_stack] completed success

INFO [step step_stack] completed success

INFO [workflow ] starting step step_color

INFO [step step_color] start

INFO [job step_color] /tmp/gdswivye$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/gdswivye,target=/wrAEHP \

    --mount=type=bind,source=/tmp/4rn8d137,target=/tmp \

    --mount=type=bind,source=/tmp/4146qj1m/stacked.tif,target=/var/lib/cwl/stg47374e55-23ed-4316-8f74-f2104dd2a5c6/stacked.tif,readonly \

    --workdir=/wrAEHP \

    --read-only=true \

    --user=1001:128 \

    --rm \

    --cidfile=/tmp/502uyoum/20250102120535-584930.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/wrAEHP \

    ghcr.io/eoap/how-to/rio:1.0.0 \

    /bin/bash \

    run.sh \

    /var/lib/cwl/stg47374e55-23ed-4316-8f74-f2104dd2a5c6/stacked.tif

INFO [job step_color] Max memory used: 725MiB

INFO [job step_color] completed success

INFO [step step_color] completed success

INFO [workflow ] completed success

INFO Final process status is success

Expected Output

Final Output (rgb.tif): An RGB TIFF file generated by the rio_color tool.

  "rgb-tif": {

    "location": "file:///home/runner/work/how-to/how-to/docs/rgb.tif",

    "basename": "rgb.tif",

    "class": "File",

    "checksum": "sha1$7ec1fa2b4daa464eda9f57ad308009aff0b32a30",

    "size": 361747464,

    "path": "/home/runner/work/how-to/how-to/docs/rgb.tif"

Key Takeaways¶

InitialWorkDirRequirement:

Dynamically creates files like scripts or configuration files.
Use entryname to define the file name and entry for the content.

Flexibility:

Simplifies command-line definitions by embedding logic in runtime scripts.

This approach allows you to customize workflows with runtime behavior using dynamically created files.