How-to Guide: Setting Environment Variables¶

This guide explains how to use EnvVarRequirement to set environment variables in a CWL workflow.

We will focus on the EnvVarRequirement block and its usage in the example provided.

Objective¶

Set specific environment variables for a command-line tool in CWL to configure its behavior dynamically.

Relevant Block:

EnvVarRequirement:

  envDef:

    GDAL_TIFF_INTERNAL_MASK: YES

    GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: YES

    CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".tif"

This configuration:

GDAL_TIFF_INTERNAL_MASK: Enables internal masking for TIFF files.
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: Optimizes HTTP range requests.
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Restricts allowed file extensions to .tif.

Steps¶

Understand the CWL Workflow Structure

The workflow (environment-variable.cwl) processes a Sentinel-2 STAC item to:

Fetch band URLs: The stac tool retrieves asset URLs for specified bands.
Stack TIFFs: The rio_stack tool stacks these bands into a single file.
Apply color correction: The rio_color tool processes the stacked file to create an RGB TIFF.

Define the Environment Variables

The EnvVarRequirement is applied in the rio_stack tool:

Relevant Block

class: CommandLineTool

id: rio_stack

requirements:

  DockerRequirement:

    dockerPull: ghcr.io/eoap/how-to/rio:1.0.0

  EnvVarRequirement:

    envDef:

      GDAL_TIFF_INTERNAL_MASK: YES

      GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: YES

      CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".tif"

baseCommand: rio

arguments:

  - stack

  - valueFrom: "${  \n  var arr = [];\n  for(var i=0; i<inputs.tiffs.length; i++) {\n      arr.push(inputs.tiffs[i]); \n  }\n  return arr; \n  }\n"

  - stacked.tif

inputs:

  tiffs:

    type: string[]

outputs:

  stacked:

    type: File

    outputBinding:

      glob: stacked.tif

Key Elements

EnvVarRequirement: Specifies environment variables.

envDef: Key-value pairs of environment variables.

Variables Defined:

GDAL_TIFF_INTERNAL_MASK: Enables internal TIFF masks.
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: Ensures efficient HTTP range fetching.
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Restricts file access to .tif.

Workflow Overview

The workflow orchestrates the process with three steps:

Step 1: Fetch Band URLs

The stac tool fetches asset URLs for the specified bands.

Step 2: Stack TIFF Files

The rio_stack tool stacks the band TIFFs into a single file (stacked.tif) using the environment variables defined in EnvVarRequirement.

Step 3: Apply Color Correction

The rio_color tool applies color correction and generates an RGB TIFF (rgb.tif).

No description has been provided for this image

Run the Workflow

Use the following command to execute the workflow:

cwltool environment-variable.cwl \
    --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
    --bands red \
    --bands green \
    --bands blue

INFO /opt/hostedtoolcache/Python/3.13.2/x64/bin/cwltool 3.1.20250110105449

INFO Resolved '../cwl-workflows/environment-variable.cwl' to 'file:///home/runner/work/how-to/how-to/cwl-workflows/environment-variable.cwl'

INFO [workflow ] start

INFO [workflow ] starting step step_curl

INFO [step step_curl] start

INFO [job step_curl] /tmp/xsf7vggl$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/xsf7vggl,target=/jujDzZ \

    --mount=type=bind,source=/tmp/7vj2xpfu,target=/tmp \

    --workdir=/jujDzZ \

    --read-only=true \

    --log-driver=none \

    --user=1001:118 \

    --rm \

    --cidfile=/tmp/bdq5ylz4/20250303112138-379413.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/jujDzZ \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/xsf7vggl/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  44848      0 --:--:-- --:--:-- --:--:-- 44938

INFO [job step_curl] completed success

INFO [step step_curl] start

INFO [job step_curl_2] /tmp/2gm9ghef$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/2gm9ghef,target=/jujDzZ \

    --mount=type=bind,source=/tmp/tcffuor9,target=/tmp \

    --workdir=/jujDzZ \

    --read-only=true \

    --log-driver=none \

    --user=1001:118 \

    --rm \

    --cidfile=/tmp/zicrct_a/20250303112139-387070.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/jujDzZ \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/2gm9ghef/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  39124      0 --:--:-- --:--:-- --:--:-- 39212

INFO [job step_curl_2] completed success

INFO [step step_curl] start

INFO [job step_curl_3] /tmp/kq4kewzz$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/kq4kewzz,target=/jujDzZ \

    --mount=type=bind,source=/tmp/q_rcb_jf,target=/tmp \

    --workdir=/jujDzZ \

    --read-only=true \

    --log-driver=none \

    --user=1001:118 \

    --rm \

    --cidfile=/tmp/kdcsvbxp/20250303112140-394850.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/jujDzZ \

    docker.io/curlimages/curl:latest \

    curl \

    https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/kq4kewzz/message

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 10156  100 10156    0     0  39318      0 --:--:-- --:--:-- --:--:-- 39364

INFO [job step_curl_3] completed success

INFO [step step_curl] completed success

INFO [workflow ] starting step step_stack

INFO [step step_stack] start

INFO [job step_stack] /tmp/1y_uuo9z$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/1y_uuo9z,target=/jujDzZ \

    --mount=type=bind,source=/tmp/faj82wxx,target=/tmp \

    --workdir=/jujDzZ \

    --read-only=true \

    --user=1001:118 \

    --rm \

    --cidfile=/tmp/z0uu133w/20250303112141-416250.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/jujDzZ \

    --env=CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif \

    --env=GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES \

    --env=GDAL_TIFF_INTERNAL_MASK=YES \

    ghcr.io/eoap/how-to/rio:1.0.0 \

    rio \

    stack \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B04.tif \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B03.tif \

    https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B02.tif \

    stacked.tif

INFO [job step_stack] Max memory used: 1210MiB

INFO [job step_stack] completed success

INFO [step step_stack] completed success

INFO [workflow ] starting step step_color

INFO [step step_color] start

INFO [job step_color] /tmp/o_tyf_9f$ docker \

    run \

    -i \

    --mount=type=bind,source=/tmp/o_tyf_9f,target=/jujDzZ \

    --mount=type=bind,source=/tmp/bmoj4543,target=/tmp \

    --mount=type=bind,source=/tmp/1y_uuo9z/stacked.tif,target=/var/lib/cwl/stg4a4af87d-518b-406a-8c84-64fc4b911bff/stacked.tif,readonly \

    --workdir=/jujDzZ \

    --read-only=true \

    --user=1001:118 \

    --rm \

    --cidfile=/tmp/7xp7wefz/20250303112206-088136.cid \

    --env=TMPDIR=/tmp \

    --env=HOME=/jujDzZ \

    ghcr.io/eoap/how-to/rio:1.0.0 \

    rio \

    color \

    -j \

    -1 \

    --out-dtype \

    uint8 \

    /var/lib/cwl/stg4a4af87d-518b-406a-8c84-64fc4b911bff/stacked.tif \

    rgb.tif \

    'gamma 3 0.95, sigmoidal rgb 35 0.13'

INFO [job step_color] Max memory used: 708MiB

INFO [job step_color] completed success

INFO [step step_color] completed success

INFO [workflow ] completed success

INFO Final process status is success

Expected Output

The workflow processes the STAC item and generates:

Final Output (rgb.tif): The RGB TIFF file with color correction applied.

  "rgb-tif": {

    "location": "file:///home/runner/work/how-to/how-to/docs/rgb.tif",

    "basename": "rgb.tif",

    "class": "File",

    "checksum": "sha1$8c9710324da63e8483db9afb74ac1b26092aa49f",

    "size": 361747464,

    "path": "/home/runner/work/how-to/how-to/docs/rgb.tif"

Key Takeaways¶

Setting Environment Variables:

Use EnvVarRequirement to configure the runtime environment for a tool.
Define key-value pairs in the envDef field.

Optimizing with Environment Variables:

GDAL_TIFF_INTERNAL_MASK: Improves TIFF file processing.
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: Reduces HTTP overhead.
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Ensures only .tif files are accessed.

By focusing on the EnvVarRequirement block, you can customize tool behavior to meet specific requirements in your CWL workflows.