How-to Guide: Capturing JSON from stdout and Reusing It in Another CWL Tool¶
This guide demonstrates how to capture a JSON response from the stdout of one CWL tool and reuse it as input for another tool.
We'll focus on two key sections:
- Capturing JSON output:
stdout: message
outputs:
asset:
type: Any
outputBinding:
glob: message
loadContents: true
outputEval: ${ return JSON.parse(self[0].contents).assets; }
- Reusing captured JSON in another tool:
arguments:
- valueFrom: "${\n let redKey = Object.keys(inputs.asset).find(key => \n inputs.asset[key]['eo:bands'] && \n inputs.asset[key]['eo:bands'].length === 1 &&\n inputs.asset[key]['eo:bands'].some(band => band.common_name === inputs.common_band_name)\n );\n return inputs.asset[redKey].href;\n}\n"
- valueFrom: |
${
return inputs.common_band_name + ".png";
}
Objective¶
- Tool 1 (
stac): Fetch a STAC item in JSON format and extract itsassets. - Tool 2 (
rio): Process the extracted asset to generate a PNG image based on the specified band.
stdout: message
outputs:
asset:
type: Any
outputBinding:
glob: message
loadContents: true
outputEval: ${ return JSON.parse(self[0].contents).assets; }
stdout: message: Redirects the tool'sstdoutto a file namedmessage.glob: message: Locates themessagefile.loadContents: true: Reads the content of the file into memory.outputEval: Parses the JSON content to extract theassetsfield using JavaScript (JSON.parse).
How It Works
- The
stactool fetches a STAC item as JSON usingcurl. - The output (
stdout) is saved tomessage. - The
outputsblock processes this JSON and makes theassetsfield available for downstream steps.
- Reusing Captured JSON in Another Tool
In the rio tool:
Relevant Block
arguments:
- valueFrom: "${\n let redKey = Object.keys(inputs.asset).find(key => \n inputs.asset[key]['eo:bands'] && \n inputs.asset[key]['eo:bands'].length === 1 &&\n inputs.asset[key]['eo:bands'].some(band => band.common_name === inputs.common_band_name)\n );\n return inputs.asset[redKey].href;\n}\n"
- valueFrom: |
${
return inputs.common_band_name + ".png";
}
- First
valueFrom: Selects the asset whoseeo:bandsarray contains a band with acommon_namematchinginputs.common_band_nameand returns itshref. - Second
valueFrom: Generates the output filename for the PNG (red.pngfor theredband).
How It Works
- The
riotool processes the asset URL (href) extracted in the first step.` - It generates a PNG file using the
common-band-nameand saves it as `.png.
- Complete CWL Example
Here’s the full capture-json-stdout.cwl file:
cwlVersion: v1.2
$graph:
- class: Workflow
id: main
requirements:
InlineJavascriptRequirement: {}
NetworkAccess:
networkAccess: true
inputs:
stac-item:
type: string
common-band-name:
type: string
outputs:
preview:
outputSource: step_translate/preview
type: File
json_output:
outputSource: step_stac/asset
type: Any
steps:
step_stac:
in:
stac_item: stac-item
out:
- asset
run: "#stac"
step_translate:
in:
asset:
source: step_stac/asset
common_band_name: common-band-name
out:
- preview
run: "#rio"
- class: CommandLineTool
id: stac
requirements:
DockerRequirement:
dockerPull: docker.io/curlimages/curl:latest
baseCommand: curl
arguments:
- -s
- $( inputs.stac_item )
inputs:
stac_item:
type: string
stdout: message
outputs:
asset:
type: Any
outputBinding:
glob: message
loadContents: true
outputEval: ${ return JSON.parse(self[0].contents).assets; }
- class: CommandLineTool
id: rio
requirements:
DockerRequirement:
dockerPull: ghcr.io/eoap/how-to/rio:1.0.0
baseCommand: rio
arguments:
- convert
- --driver
- PNG
- --dtype
- uint8
- valueFrom: "${\n let redKey = Object.keys(inputs.asset).find(key => \n inputs.asset[key]['eo:bands'] && \n inputs.asset[key]['eo:bands'].length === 1 &&\n inputs.asset[key]['eo:bands'].some(band => band.common_name === inputs.common_band_name)\n );\n return inputs.asset[redKey].href;\n}\n"
- valueFrom: |
${
return inputs.common_band_name + ".png";
}
inputs:
asset:
type: Any
common_band_name:
type: string
outputs:
preview:
type: File
outputBinding:
glob: "*.png"
It's graphical representation:
- Run the Example
Command to execute the workflow:
cwltool capture-json-stdout.cwl \
--stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
--common-band-name red
- Execute the CWL Tool
Run the command in your terminal:
cwltool capture-json-stdout.cwl \
--stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
--common-band-name red
INFO /opt/hostedtoolcache/Python/3.12.8/x64/bin/cwltool 3.1.20241217163858
INFO Resolved '../cwl-workflows/capture-json-stdout.cwl' to 'file:///home/runner/work/how-to/how-to/cwl-workflows/capture-json-stdout.cwl'
INFO [workflow ] start
INFO [workflow ] starting step step_stac
INFO [step step_stac] start
Error: No such object: docker.io/curlimages/curl:latest
INFO ['docker', 'pull', 'docker.io/curlimages/curl:latest']
latest: Pulling from curlimages/curl
38a8310d387e: Already exists
fa22e15491be: Pulling fs layer
4ca545ee6d5d: Pulling fs layer
fa22e15491be: Downloading 63.13kB/5.913MB
4ca545ee6d5d: Downloading 42B/42B
4ca545ee6d5d: Verifying Checksum
4ca545ee6d5d: Download complete
fa22e15491be: Download complete
fa22e15491be: Extracting 65.54kB/5.913MB
fa22e15491be: Extracting 5.913MB/5.913MB
fa22e15491be: Pull complete
4ca545ee6d5d: Extracting 42B/42B
4ca545ee6d5d: Extracting 42B/42B
4ca545ee6d5d: Pull complete
Digest: sha256:c1fe1679c34d9784c1b0d1e5f62ac0a79fca01fb6377cdd33e90473c6f9f9a69
Status: Downloaded newer image for curlimages/curl:latest
docker.io/curlimages/curl:latest
INFO [job step_stac] /tmp/z9bb0rhe$ docker \
run \
-i \
--mount=type=bind,source=/tmp/z9bb0rhe,target=/XHOJtZ \
--mount=type=bind,source=/tmp/elyi4pxl,target=/tmp \
--workdir=/XHOJtZ \
--read-only=true \
--log-driver=none \
--user=1001:128 \
--rm \
--cidfile=/tmp/658gpk88/20250102120019-315869.cid \
--env=TMPDIR=/tmp \
--env=HOME=/XHOJtZ \
docker.io/curlimages/curl:latest \
curl \
-s \
https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A > /tmp/z9bb0rhe/message
INFO [job step_stac] completed success
INFO [step step_stac] completed success
INFO [workflow ] starting step step_translate
INFO [step step_translate] start
INFO [job step_translate] /tmp/nblgy8pq$ docker \
run \
-i \
--mount=type=bind,source=/tmp/nblgy8pq,target=/XHOJtZ \
--mount=type=bind,source=/tmp/yjf0m2ci,target=/tmp \
--workdir=/XHOJtZ \
--read-only=true \
--user=1001:128 \
--rm \
--cidfile=/tmp/1m93evuz/20250102120020-352527.cid \
--env=TMPDIR=/tmp \
--env=HOME=/XHOJtZ \
ghcr.io/eoap/how-to/rio:1.0.0 \
rio \
convert \
--driver \
PNG \
--dtype \
uint8 \
https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B04.tif \
red.png
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option BLOCKXSIZE
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option BLOCKYSIZE
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option TILED
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option COMPRESS
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option INTERLEAVE
INFO [job step_translate] Max memory used: 781MiB
INFO [job step_translate] completed success
INFO [step step_translate] completed success
INFO [workflow ] completed success
INFO Final process status is success
Expected Output¶
- Intermediate Output (
json_output): Extracted assets from the STAC item JSON. - Final Output (
preview): PNG file for the specified band (e.g.,red.png).
{
"thumbnail": {
"title": "Thumbnail",
"type": "image/png",
"roles": [
"thumbnail"
],
"href": "https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/53/H/PA/2021/7/23/0/preview.jpg"
},
"overview": {
"title": "True color image",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"overview"
],
"gsd": 10,
"eo:bands": [
{
"name": "B04",
"common_name": "red",
"center_wavelength": 0.6645,
"full_width_half_max": 0.038
},
{
"name": "B03",
"common_name": "green",
"center_wavelength": 0.56,
"full_width_half_max": 0.045
},
{
"name": "B02",
"common_name": "blue",
"center_wavelength": 0.4966,
"full_width_half_max": 0.098
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/L2A_PVI.tif",
"proj:shape": [
343,
343
],
"proj:transform": [
320,
0,
600000,
0,
-320,
6100000,
0,
0,
1
]
},
"info": {
"title": "Original JSON metadata",
"type": "application/json",
"roles": [
"metadata"
],
"href": "https://roda.sentinel-hub.com/sentinel-s2-l2a/tiles/53/H/PA/2021/7/23/0/tileInfo.json"
},
"metadata": {
"title": "Original XML metadata",
"type": "application/xml",
"roles": [
"metadata"
],
"href": "https://roda.sentinel-hub.com/sentinel-s2-l2a/tiles/53/H/PA/2021/7/23/0/metadata.xml"
},
"visual": {
"title": "True color image",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"overview"
],
"gsd": 10,
"eo:bands": [
{
"name": "B04",
"common_name": "red",
"center_wavelength": 0.6645,
"full_width_half_max": 0.038
},
{
"name": "B03",
"common_name": "green",
"center_wavelength": 0.56,
"full_width_half_max": 0.045
},
{
"name": "B02",
"common_name": "blue",
"center_wavelength": 0.4966,
"full_width_half_max": 0.098
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/TCI.tif",
"proj:shape": [
10980,
10980
],
"proj:transform": [
10,
0,
600000,
0,
-10,
6100000,
0,
0,
1
]
},
"B01": {
"title": "Band 1 (coastal)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 60,
"eo:bands": [
{
"name": "B01",
"common_name": "coastal",
"center_wavelength": 0.4439,
"full_width_half_max": 0.027
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B01.tif",
"proj:shape": [
1830,
1830
],
"proj:transform": [
60,
0,
600000,
0,
-60,
6100000,
0,
0,
1
]
},
"B02": {
"title": "Band 2 (blue)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 10,
"eo:bands": [
{
"name": "B02",
"common_name": "blue",
"center_wavelength": 0.4966,
"full_width_half_max": 0.098
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B02.tif",
"proj:shape": [
10980,
10980
],
"proj:transform": [
10,
0,
600000,
0,
-10,
6100000,
0,
0,
1
]
},
"B03": {
"title": "Band 3 (green)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 10,
"eo:bands": [
{
"name": "B03",
"common_name": "green",
"center_wavelength": 0.56,
"full_width_half_max": 0.045
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B03.tif",
"proj:shape": [
10980,
10980
],
"proj:transform": [
10,
0,
600000,
0,
-10,
6100000,
0,
0,
1
]
},
"B04": {
"title": "Band 4 (red)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 10,
"eo:bands": [
{
"name": "B04",
"common_name": "red",
"center_wavelength": 0.6645,
"full_width_half_max": 0.038
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B04.tif",
"proj:shape": [
10980,
10980
],
"proj:transform": [
10,
0,
600000,
0,
-10,
6100000,
0,
0,
1
]
},
"B05": {
"title": "Band 5",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 20,
"eo:bands": [
{
"name": "B05",
"center_wavelength": 0.7039,
"full_width_half_max": 0.019
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B05.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
},
"B06": {
"title": "Band 6",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 20,
"eo:bands": [
{
"name": "B06",
"center_wavelength": 0.7402,
"full_width_half_max": 0.018
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B06.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
},
"B07": {
"title": "Band 7",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 20,
"eo:bands": [
{
"name": "B07",
"center_wavelength": 0.7825,
"full_width_half_max": 0.028
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B07.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
},
"B08": {
"title": "Band 8 (nir)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 10,
"eo:bands": [
{
"name": "B08",
"common_name": "nir",
"center_wavelength": 0.8351,
"full_width_half_max": 0.145
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B08.tif",
"proj:shape": [
10980,
10980
],
"proj:transform": [
10,
0,
600000,
0,
-10,
6100000,
0,
0,
1
]
},
"B8A": {
"title": "Band 8A",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 20,
"eo:bands": [
{
"name": "B8A",
"center_wavelength": 0.8648,
"full_width_half_max": 0.033
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B8A.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
},
"B09": {
"title": "Band 9",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 60,
"eo:bands": [
{
"name": "B09",
"center_wavelength": 0.945,
"full_width_half_max": 0.026
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B09.tif",
"proj:shape": [
1830,
1830
],
"proj:transform": [
60,
0,
600000,
0,
-60,
6100000,
0,
0,
1
]
},
"B11": {
"title": "Band 11 (swir16)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 20,
"eo:bands": [
{
"name": "B11",
"common_name": "swir16",
"center_wavelength": 1.6137,
"full_width_half_max": 0.143
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B11.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
},
"B12": {
"title": "Band 12 (swir22)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"gsd": 20,
"eo:bands": [
{
"name": "B12",
"common_name": "swir22",
"center_wavelength": 2.22024,
"full_width_half_max": 0.242
}
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B12.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
},
"AOT": {
"title": "Aerosol Optical Thickness (AOT)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/AOT.tif",
"proj:shape": [
1830,
1830
],
"proj:transform": [
60,
0,
600000,
0,
-60,
6100000,
0,
0,
1
]
},
"WVP": {
"title": "Water Vapour (WVP)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/WVP.tif",
"proj:shape": [
10980,
10980
],
"proj:transform": [
10,
0,
600000,
0,
-10,
6100000,
0,
0,
1
]
},
"SCL": {
"title": "Scene Classification Map (SCL)",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": [
"data"
],
"href": "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/SCL.tif",
"proj:shape": [
5490,
5490
],
"proj:transform": [
20,
0,
600000,
0,
-20,
6100000,
0,
0,
1
]
}
}
{
"location": "file:///home/runner/work/how-to/how-to/docs/red.png",
"basename": "red.png",
"class": "File",
"checksum": "sha1$3b0ab5ca89bc640e276ac48c6d37a900a4493708",
"size": 103995863,
"path": "/home/runner/work/how-to/how-to/docs/red.png"
}
Key Takeaways¶
Capturing JSON:
- Use
stdoutto redirect the JSON output to a file. - Parse and extract specific fields using
outputEval.
- Use
Reusing JSON:
- Use
valueFromin arguments to select fields dynamically from the JSON input. - Process the selected field (e.g., generate a file based on its href).
- Use
By focusing on stdout and outputBinding, this guide highlights how CWL facilitates JSON data flow between tools.