Skip to content

Landsat-9

Goal

Run the app-water-body.1.0.0.cwl released application package using cwltool on a staged Landsat-9 acquisition.

Relationship to the current lab

This page demonstrates a historical released package using the download and execution scripts below. Its input format and container versions belong to that release.

The current notebook, practice-labs/4-CWL-Workflows/2.2-staged.ipynb, runs the checked-out typed CWL and locally built images instead. Follow the current workflow guide for its job format and commands.

Step 1 - Configure the workspace

The results produced will be available in the local folder /workspace/mastering-app-package/runs

terminal
1
2
3
4
export WORKSPACE=/workspace/mastering-app-package
export RUNTIME=${WORKSPACE}/runs
mkdir -p ${RUNTIME}
cd ${RUNTIME}

source /workspace/mastering-app-package/scripts/setup.sh

Step 2 - Download the released Application package

scripts/download-app-water-body.sh
1
2
3
4
5
version="1.0.0"

wget \
    -O ${WORKSPACE}/runs/app-water-body.${version}.cwl \
    https://github.com/eoap/mastering-app-package/releases/download/${version}/app-water-body.${version}.cwl

sh ${WORKSPACE}/scripts/download-app-water-body.sh

Step 3 - Stage-in the Landsat-9 acquisition

The Landsat-9 acquistion "https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2/items/LC09_L2SP_042033_20231015_02_T1" is staged with: 

1
2
3
4
5
6
7
8
ls9_ref="https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-c2-l2/items/LC09_L2SP_042033_20231015_02_T1"

cwltool \
    --podman \
    --outdir ${WORKSPACE}/runs \
    ${WORKSPACE}/cwl-cli/stage-in-released.cwl \
    --reference \
    ${ls9_ref} > staged.json

sh ${WORKSPACE}/scripts/cwl-cli-stage-in-released.sh

The expected content after this step is:

(base) jovyan@coder-fbrito:~/runs$ tree
.
├── app-water-body.1.0.0.cwl
├── lwft9j0o
│   ├── LC09_L2SP_042033_20231015_02_T1
│   │   ├── LC09_L2SP_042033_20231015_02_T1.json
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ANG.txt
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_MTL.json
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_MTL.txt
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_MTL.xml
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_QA_PIXEL.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_QA_RADSAT.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B1.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B2.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B3.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B4.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B5.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B6.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B7.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_QA_AEROSOL.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_ATRAN.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_B10.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_CDIST.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_DRAD.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_EMIS.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_EMSD.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_QA.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_TRAD.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_URAD.TIF
│   │   ├── preview.png
│   │   └── tilejson.json
│   ├── catalog.json
│   └── stage.py
└── staged.json

2 directories, 30 files

The result is redirected to a file named staged.json as we use jq to get the path of the staged product:

terminal
cat staged.json | jq -r .staged.path

This returns a path like /workspace/mastering-app-package/runs/921x91vw

Step 4 - Execute the Application Package

Use cwltool to run the application package using the staged Landsat-9 acquisition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
version="1.0.0"

cwltool \
    --podman \
    --outdir ${WORKSPACE}/runs \
    ${WORKSPACE}/runs/app-water-body.${version}.cwl \
    --item $( cat staged.json | jq -r .staged.path ) \
    --aoi="-118.985,38.432,-118.183,38.938" \
    --epsg "EPSG:4326" \
    --bands green \
    --bands nir08 > results.json

sh ${WORKSPACE}/scripts/exec-app-water-body.sh

Step 5 - Stage-out to S3

This step uploads the STAC Catalog to S3.

terminal
cwltool \
    --podman \
    ${WORKSPACE}/cwl-cli/stage-out-released.cwl \
    --aws_access_key_id $( cat ~/.aws/credentials | grep aws_access_key_id | cut -d "=" -f 2 ) \
    --aws_secret_access_key $( cat ~/.aws/credentials | grep aws_secret_access_key | cut -d "=" -f 2 ) \
    --endpoint_url $( cat ~/.aws/config | grep endpoint_url | head -n 1 | cut -d "=" -f 2 ) \
    --region_name $( cat ~/.aws/config | grep region | cut -d "=" -f 2 ) \
    --s3_bucket processing-results \
    --sub_path processing-results/$( cat /proc/sys/kernel/random/uuid ) \
    --stac_catalog /workspace/mastering-app-package/runs

sh ${WORKSPACE}/scripts/cwl-cli-stage-out-released.sh

Check the staged results on S3:

terminal
aws s3 ls $( cat staged-s3.json | jq -r .s3_catalog_output | xargs dirname )/

Expected outcome

The folder /workspace/mastering-app-package/runs contains: 

(base) jovyan@coder-mrossi:~/runs$ tree .
.
├── app-water-body.1.0.0.cwl
├── cg8n2dnu
│   ├── LC09_L2SP_042033_20231015_02_T1
│   │   ├── LC09_L2SP_042033_20231015_02_T1.json
│   │   └── otsu.tif
│   └── catalog.json
├── lwft9j0o
│   ├── LC09_L2SP_042033_20231015_02_T1
│   │   ├── LC09_L2SP_042033_20231015_02_T1.json
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ANG.txt
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_MTL.json
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_MTL.txt
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_MTL.xml
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_QA_PIXEL.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_QA_RADSAT.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B1.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B2.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B3.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B4.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B5.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B6.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_B7.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_SR_QA_AEROSOL.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_ATRAN.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_B10.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_CDIST.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_DRAD.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_EMIS.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_EMSD.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_QA.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_TRAD.TIF
│   │   ├── LC09_L2SP_042033_20231015_20231016_02_T1_ST_URAD.TIF
│   │   ├── preview.png
│   │   └── tilejson.json
│   ├── catalog.json
│   └── stage.py
└── staged.json

4 directories, 33 files

and 

(base) jovyan@coder-mrossi:~/runs$ aws s3 ls $( cat staged-s3.json | jq -r .s3_catalog_output | xargs dirname )/
                           PRE LC09_L2SP_042033_20231015_02_T1/
2023-11-02 08:46:04        549 catalog.json

These commands target downloaded release packages and retain those releases' input format. For local typed CWL files, use the YAML job examples in the workflow authoring labs.