Stage and cog

Goal

Run the stage-and-cog.cwl CWL Workflow that that stages a Sentinel-2 Level-1C from the CDSE and converts/resamples its assets to COG.

Lab

This step has a dedicated lab available at /workspace/inference-eoap/practice-labs/Application steps/01 - Stage-and-cog.ipynb

Dependencies

This steps requires generating a CDSE access token with the Python script and your own CDSE credentials:

#!/usr/bin/env python3

import click
import requests

def get_access_token(username: str, password: str) -> str:
    data = {
        "client_id": "cdse-public",
        "username": username,
        "password": password,
        "grant_type": "password",
    }
    try:
        r = requests.post(
            "https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token",
            data=data,
        )
        r.raise_for_status()
    except Exception as e:
        raise Exception(
            f"Access token creation failed. Response from the server was: {r.json()}"
        )
    return r.json()["access_token"]

@click.command(
    short_help="Get bearer token",
    help="Get a bearer token for the Copernicus Data Space Ecosystem ",
)
@click.option(
    "--username",
    help="Username for the Copernicus Data Space Ecosystem",
    required=True,
)
@click.option(
    "--password",
    help="Password for the Copernicus Data Space Ecosystem",
    required=True,
)
def get_bearer_token(username: str, password: str) -> str:

    return print(get_access_token(username, password))

if __name__ == "__main__":
    get_bearer_token()

Lab

This step has a dedicated lab available at /workspace/inference-eoap/practice-labs/Application steps/01 - Stage-and-cog.ipynb

Step 1 - Run the stage-and-cog.cwl_ CWL Workflow

Use cwltool to run the stage-and-cog.cwl CWL Workflow with a Sentinel-2 Level-1C acquisition reference

The CWL Workflow is shown below and the lines highlighted chain the stage-in and conversion to COG steps:

cwlVersion: v1.0
$namespaces:
  s: https://schema.org/
s:softwareVersion: 1.0.0
schemas:
  - http://schema.org/version/9.0/schemaorg-current-http.rdf
$graph:
  - class: Workflow
    id: main
    label: Stage-in and cog-ify a Sentinel-2 L1C acquisition
    doc: Stage-in and cog-ify a Sentinel-2 L1C acquisition
    requirements: {}
    inputs:
      item:
        doc: Reference to a STAC item
        label: STAC item reference
        type: string
      access_token:
        doc: CDSE bearer token
        label: CDSE bearer token
        type: string
    outputs:
      - id: stac_catalog
        outputSource:
          - node_sen2cog/stac_catalog
        type: Directory
    steps:
      node_stage_in:
        run: "#stage-in"
        in:
          item: item
          access_token: access_token
        out:
          - staged
      node_sen2cog:
        run: "#sen2cog"
        in:
          staged:
            source: node_stage_in/staged
        out:
          - stac_catalog

  - class: CommandLineTool
    id: stage-in
    requirements:
      InlineJavascriptRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          PYTHONPATH: /app
          CDSE_ACCES_TOKEN: $(inputs.access_token)
      ResourceRequirement:
        coresMax: 1
        ramMax: 1024
    hints:
      DockerRequirement:
        dockerPull: localhost/stage-in:latest
    baseCommand: ["python", "-m", "app"]
    arguments: []
    inputs:
      item:
        type: string
        inputBinding:
          prefix: --input-item
      access_token:
        type: string
    outputs:
      staged:
        outputBinding:
          glob: .
        type: Directory
  - class: CommandLineTool
    id: sen2cog
    requirements:
      InlineJavascriptRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          PYTHONPATH: /app
      ResourceRequirement:
        coresMax: 2
        ramMax: 4096
    hints:
      DockerRequirement:
        dockerPull: docker.io/library/sen2cog:latest
    baseCommand: ["python", "-m", "app"]
    arguments: []
    inputs:
      staged:
        type: Directory
        inputBinding:
          prefix: --input-catalog
    outputs:
      stac_catalog:
        outputBinding:
          glob: .
        type: Directory

To run this CWL document run the script below to:

• provide your CDSE username and password to generate and access token
• Use cwltool to run stage-and-cog.cwl CWL description

terminal

version="1.0.0"

export WORKSPACE=/workspace/inference-eoap

export PATH=${WORKSPACE}/bin:$PATH

read -p "Enter the CDSE username: " username
read -sp "Enter the CDSE password: " password

BEARER_TOKEN=$( get-bearer-token ${username} ${password} )

cwltool \
    --podman \
    --outdir ${WORKSPACE}/runs \
    https://github.com/eoap/inference-eoap/releases/download/1.0.0/stage-and-cog.1.0.0.cwl \
    --access_token ${BEARER_TOKEN} \
    --item "https://catalogue.dataspace.copernicus.eu/stac/collections/SENTINEL-2/items/S2A_MSIL1C_20240125T100311_N0510_R122_T33TVM_20240125T104959.SAFE" > ${WORKSPACE}/runs/results.json

Expected outcome

The folder /workspace/inference-eoap/runs contains:

(base) jovyan@jupyter-fbrito--training:~/inference-eoap$ tree runs
runs
├── results.json
└── ydmns4od
    ├── S2A_T33TVM_20240125T100359_L1C
    │   ├── S2A_T33TVM_20240125T100359_L1C.json
    │   ├── blue.tif
    │   ├── cirrus.tif
    │   ├── coastal.tif
    │   ├── green.tif
    │   ├── nir.tif
    │   ├── nir08.tif
    │   ├── nir09.tif
    │   ├── red.tif
    │   ├── rededge1.tif
    │   ├── rededge2.tif
    │   ├── rededge3.tif
    │   ├── swir16.tif
    │   └── swir22.tif
    └── catalog.json

2 directories, 16 files