How-to Guide: Setting Environment Variables¶

This guide explains how to use EnvVarRequirement to set environment variables in a CWL workflow.

We will focus on the EnvVarRequirement block and its usage in the example provided.

Objective¶

Set specific environment variables for a command-line tool in CWL to configure its behavior dynamically.

Relevant Block:

In [ ]:

Copied!

cwl_file="../cwl-workflows/environment-variable.cwl"

cat  $cwl_file | yq e .'$graph[2]' - | yq '.. | select(has("EnvVarRequirement")) | {"EnvVarRequirement": .EnvVarRequirement}'
cwl_file="../cwl-workflows/environment-variable.cwl"

cat  $cwl_file | yq e .'$graph[2]' - | yq '.. | select(has("EnvVarRequirement")) | {"EnvVarRequirement": .EnvVarRequirement}'

This configuration:

GDAL_TIFF_INTERNAL_MASK: Enables internal masking for TIFF files.
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: Optimizes HTTP range requests.
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Restricts allowed file extensions to .tif.

Steps¶

Understand the CWL Workflow Structure

The workflow (environment-variable.cwl) processes a Sentinel-2 STAC item to:

Fetch band URLs: The stac tool retrieves asset URLs for specified bands.
Stack TIFFs: The rio_stack tool stacks these bands into a single file.
Apply color correction: The rio_color tool processes the stacked file to create an RGB TIFF.

Define the Environment Variables

The EnvVarRequirement is applied in the rio_stack tool:

Relevant Block

In [ ]:

Copied!

cat  $cwl_file | yq e .'$graph[2]' -
cat  $cwl_file | yq e .'$graph[2]' -

Key Elements

EnvVarRequirement: Specifies environment variables.

envDef: Key-value pairs of environment variables.

Variables Defined:

GDAL_TIFF_INTERNAL_MASK: Enables internal TIFF masks.
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: Ensures efficient HTTP range fetching.
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Restricts file access to .tif.

Workflow Overview

The workflow orchestrates the process with three steps:

Step 1: Fetch Band URLs

The stac tool fetches asset URLs for the specified bands.

Step 2: Stack TIFF Files

The rio_stack tool stacks the band TIFFs into a single file (stacked.tif) using the environment variables defined in EnvVarRequirement.

Step 3: Apply Color Correction

The rio_color tool applies color correction and generates an RGB TIFF (rgb.tif).

In [ ]:

Copied!

cwltool --print-dot $cwl_file 2> /dev/null | dot -Tpng | display
cwltool --print-dot $cwl_file 2> /dev/null | dot -Tpng | display

Run the Workflow

Use the following command to execute the workflow:

cwltool environment-variable.cwl \
    --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
    --bands red \
    --bands green \
    --bands blue

In [ ]:

Copied!





cat > input.json <<'JSON'
{
  "stac-item": {
    "value": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A"
  },
  "bands": [
    "red",
    "green",
    "blue"
  ]
}
JSON
cwltool --podman "$cwl_file" input.json > output.json
cat > input.json <<'JSON'
{
  "stac-item": {
    "value": "https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A"
  },
  "bands": [
    "red",
    "green",
    "blue"
  ]
}
JSON
cwltool --podman "$cwl_file" input.json > output.json

Expected Output

The workflow processes the STAC item and generates:

Final Output (rgb.tif): The RGB TIFF file with color correction applied.

In [ ]:

Copied!

cat output.json | yq -e . -o=json -
cat output.json | yq -e . -o=json -

Key Takeaways¶

Setting Environment Variables:

Use EnvVarRequirement to configure the runtime environment for a tool.
Define key-value pairs in the envDef field.

Optimizing with Environment Variables:

GDAL_TIFF_INTERNAL_MASK: Improves TIFF file processing.
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: Reduces HTTP overhead.
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Ensures only .tif files are accessed.

By focusing on the EnvVarRequirement block, you can customize tool behavior to meet specific requirements in your CWL workflows.

In [ ]:

Copied!

rm -f output.json
rm -f rgb.tif
rm -f output.json
rm -f rgb.tif