How-to Guide: Creating Files at Runtime¶

This guide explains how to create files dynamically at runtime in a CWL workflow using the InitialWorkDirRequirement.

The focus will be on defining a shell script (run.sh) as part of the workflow to handle runtime commands.

Objective¶

Dynamically create a run.sh script at runtime that executes specific commands for each tool in the workflow.

Relevant Block:

In [ ]:

Copied!

cwl_file="../cwl-workflows/runtime-file.cwl"

cat  $cwl_file | yq e .'$graph[2]' - | yq '.. | select(has("InitialWorkDirRequirement")) | {"InitialWorkDirRequirement": .InitialWorkDirRequirement}'
cwl_file="../cwl-workflows/runtime-file.cwl"

cat  $cwl_file | yq e .'$graph[2]' - | yq '.. | select(has("InitialWorkDirRequirement")) | {"InitialWorkDirRequirement": .InitialWorkDirRequirement}'

InitialWorkDirRequirement: Specifies files or directories to be created dynamically in the working directory.
entryname: The name of the file to be created (run.sh).
entry: The contents of the file, which in this case is a shell script.

Steps¶

Understand the Workflow Structure

The workflow (runtime-file.cwl) processes a Sentinel-2 STAC item to:

Fetch band URLs (stac tool).
Stack TIFFs (rio_stack tool) using a runtime script.
Apply color correction (rio_color tool) using another runtime script.

Define the InitialWorkDirRequirement

For rio_stack Tool

The run.sh script stacks the provided TIFFs into a single file (stacked.tif):

In [ ]:

Copied!

cat  $cwl_file | yq e .'$graph[2]' - | yq '.. | select(has("InitialWorkDirRequirement")) | {"InitialWorkDirRequirement": .InitialWorkDirRequirement}'
cat  $cwl_file | yq e .'$graph[2]' - | yq '.. | select(has("InitialWorkDirRequirement")) | {"InitialWorkDirRequirement": .InitialWorkDirRequirement}'

Contents:
- The script (run.sh) executes rio stack with all TIFF file paths passed as arguments ($@).

For rio_color Tool

The run.sh script applies color correction to the stacked TIFF file and produces an RGB TIFF (rgb.tif):

In [ ]:

Copied!

cat  $cwl_file | yq e .'$graph[3]' - | yq '.. | select(has("InitialWorkDirRequirement")) | {"InitialWorkDirRequirement": .InitialWorkDirRequirement}'
cat  $cwl_file | yq e .'$graph[3]' - | yq '.. | select(has("InitialWorkDirRequirement")) | {"InitialWorkDirRequirement": .InitialWorkDirRequirement}'

Contents:
- The script (run.sh) applies color correction to the file passed as the first argument ($1).

Specify the baseCommand

Each tool uses /bin/bash to execute the dynamically created run.sh script:

In [ ]:

Copied!

cat  $cwl_file | yq e .'$graph[3]' - | yq '.. | select(has("baseCommand")) | {"baseCommand": .baseCommand}'
cat  $cwl_file | yq e .'$graph[3]' - | yq '.. | select(has("baseCommand")) | {"baseCommand": .baseCommand}'

This ensures the created script run.sh is executed in a shell environment.

Complete Workflow Example

Here’s the full CWL example (runtime-file.cwl):

In [ ]:

Copied!

cat  $cwl_file | yq e . - 
cat  $cwl_file | yq e . -

Graphical representation:

In [ ]:

Copied!

cwltool --print-dot $cwl_file 2> /dev/null | dot -Tpng | display
cwltool --print-dot $cwl_file 2> /dev/null | dot -Tpng | display

Run the Workflow

Run the workflow with the following command:

cwltool runtime-file.cwl \
  --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
  --bands red \
  --bands green \
  --bands blue

In [ ]:

Copied!





cat > input.json <<'JSON'
{
  "stac-item": {
    "value": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A"
  },
  "bands": [
    "red",
    "green",
    "blue"
  ]
}
JSON
cwltool --podman "$cwl_file" input.json > output.json
cat > input.json <<'JSON'
{
  "stac-item": {
    "value": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A"
  },
  "bands": [
    "red",
    "green",
    "blue"
  ]
}
JSON
cwltool --podman "$cwl_file" input.json > output.json

Expected Output

Final Output (rgb.tif): An RGB TIFF file generated by the rio_color tool.

In [ ]:

Copied!

cat output.json | yq e . -o=json -
cat output.json | yq e . -o=json -

Key Takeaways¶

InitialWorkDirRequirement:

Dynamically creates files like scripts or configuration files.
Use entryname to define the file name and entry for the content.

Flexibility:

Simplifies command-line definitions by embedding logic in runtime scripts.

This approach allows you to customize workflows with runtime behavior using dynamically created files.