Processing a staged Landsat-9 acquisition

The non-Cloud native Workflow chains the crop, norm_diff, otsu and stac steps taking a staged EO acquisition as a STAC Catalog in a directory as input parameters:

a local STAC Catalog in a directory
a GeoJSON Polygon area of interest (AOI), with coordinates and bbox
the EPSG enum value "4326"
a list of common band names (["green", "nir08"])

graph TB S[(local storage)] S -- STAC Catalog --> B(("crop(green)")); S -- STAC Catalog --> C(("crop(nir)")); S -- STAC Catalog --> F P[bands] Q[EPSG code] R[AOI] subgraph scatter on bands P --> B(("crop(green)")) P --> C(("crop(nir08)")) Q --> B(("crop(green)")) Q --> C(("crop(nir08)")) R --> B(("crop(green)")) R --> C(("crop(nir08)")) end B(("crop(green)")) --> D C(("crop(nir08)")) --> D D(("`Normalized difference`")); D --> E(("`Otsu threshold`")) E --> F(("`Create STAC Catalog`"))

The CWL Workflow is shown below and the lines highlighted show the changes to manage a staged STAC Catalog instead of an URL to a STAC Item:

app-water-body.cwl
cwlVersion: v1.2
$namespaces:
  s: https://schema.org/
s:softwareVersion: 2.0.2
# Training-example authors and creation date mirror codemeta.json.
s:name: Water body detection based on NDWI and the otsu threshold
s:description: Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 or Landsat-9 staged acquisitions
s:dateCreated: '2022-09-01'
s:license:
  s:name: Creative Commons Attribution-ShareAlike 4.0 International
  s:url: https://creativecommons.org/licenses/by-sa/4.0/
  s:identifier: CC-BY-SA-4.0
s:softwareHelp:
  s:name: Mastering Earth Observation Application Packaging with CWL
  s:url: https://eoap.github.io/mastering-app-package
s:publisher:
  s:name: EO Application Packaging
  s:identifier: https://github.com/eoap
s:author:
  - s:givenName: Jane
    s:familyName: Doe
    s:email: jane.doe@acme.earth
    s:affiliation:
      s:name: ACME
  - s:givenName: John
    s:familyName: Doe
    s:email: john.doe@acme.earth
    s:affiliation:
      s:name: ACME
s:softwareRequirements:
  - container runtime
  - cwl runner
schemas:
  - http://schema.org/version/9.0/schemaorg-current-http.rdf
$graph:
  - class: Workflow
    id: water-bodies
    label: Water body detection based on NDWI and the otsu threshold
    doc: Water bodies detection based on NDWI and otsu threshold applied to Sentinel-2 or Landsat-9 staged acquisitions
    requirements:
      - class: ScatterFeatureRequirement
      - class: SchemaDefRequirement
        types:
          - $import: https://raw.githubusercontent.com/eoap/schemas/main/geojson.yaml
          - name: '#EPSGCode'
            type: enum
            symbols: ["4326"]
          - name: '#SpectralBand'
            type: enum
            symbols: [green, nir, nir08]
    inputs:
      aoi:
        label: area of interest
        doc: GeoJSON Polygon defining the area to crop; raster pixels outside the polygon are masked.
        type: https://raw.githubusercontent.com/eoap/schemas/main/geojson.yaml#Polygon
      epsg:
        label: EPSG code
        doc: EPSG code
        type: 'EPSGCode'
        default: "4326"
      bands:
        label: bands used for the NDWI
        doc: bands used for the NDWI
        type:
          type: array
          items: 'SpectralBand'
        default: ["green", "nir"]
      item:
        doc: Reference to a STAC item
        label: STAC item reference
        type: Directory
    outputs:
      - id: stac_catalog
        outputSource:
          - node_stac/stac_catalog
        type: Directory
        label: Results STAC catalog
        doc: Directory containing the results STAC catalog and its referenced raster assets.
    steps:
      node_crop:
        run: "#crop"
        in:
          item: item
          aoi: aoi
          epsg: epsg
          band: bands
        out:
          - cropped
        scatter: band
        scatterMethod: dotproduct
        label: Crop bands
        doc: Crop the selected bands to the area of interest.
      node_normalized_difference:
        run: "#norm_diff"
        in:
          rasters:
            source: node_crop/cropped
        out:
          - ndwi
        label: Compute NDWI
        doc: Compute the normalized difference water index from the cropped bands.
      node_otsu:
        run: "#otsu"
        in:
          raster:
            source: node_normalized_difference/ndwi
        out:
          - binary_mask_item
        label: Detect water bodies
        doc: Apply the Otsu threshold to produce a binary water body mask.
      node_stac:
        run: "#stac"
        in:
          item: item
          rasters:
            source: node_otsu/binary_mask_item
        out:
          - stac_catalog
        label: Catalog results
        doc: Create a STAC catalog describing the detected water bodies.
  - class: CommandLineTool
    id: crop
    requirements:
      InlineJavascriptRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: /app/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ResourceRequirement:
        coresMin: 1
        coresMax: 1
        ramMin: 512
        ramMax: 512
      NetworkAccess:
        networkAccess: true
      DockerRequirement:
        dockerPull: localhost/crop:latest
      SchemaDefRequirement:
        types:
          - $import: https://raw.githubusercontent.com/eoap/schemas/main/geojson.yaml
          - name: '#EPSGCode'
            type: enum
            symbols: ["4326"]
          - name: '#SpectralBand'
            type: enum
            symbols: [green, nir, nir08]
    baseCommand: crop
    arguments: []
    inputs:
      item:
        type: Directory
        inputBinding:
          prefix: --input-item
        label: STAC item
        doc: Directory containing the staged acquisition STAC catalog and assets.
      aoi:
        type: https://raw.githubusercontent.com/eoap/schemas/main/geojson.yaml#Polygon
        inputBinding:
          prefix: --aoi
          valueFrom: $(JSON.stringify(self))
        label: Area of interest
        doc: GeoJSON Polygon defining the area to crop; raster pixels outside the polygon are masked.
      epsg:
        type: 'EPSGCode'
        inputBinding:
          prefix: --epsg
          valueFrom: $(self.split(":").pop())
        label: Coordinate reference system
        doc: EPSG code of the coordinate reference system used for the area of interest.
      band:
        type: 'SpectralBand'
        inputBinding:
          prefix: --band
        label: Raster band
        doc: Name of the STAC asset band to crop.
    outputs:
      cropped:
        outputBinding:
          glob: '*.tif'
        type: File
        label: Cropped raster
        doc: Raster band cropped to the area of interest.
  - class: CommandLineTool
    id: norm_diff
    requirements:
      InlineJavascriptRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: /app/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ResourceRequirement:
        coresMin: 1
        coresMax: 1
        ramMin: 2048
        ramMax: 2048
      NetworkAccess:
        networkAccess: false
      DockerRequirement:
        dockerPull: localhost/norm-diff:latest
    baseCommand: norm_diff
    arguments: []
    inputs:
      rasters:
        type:
          type: array
          items: File
          inputBinding:
            prefix: --rasters
        inputBinding:
          position: 1
        label: Input rasters
        doc: Raster files used for normalized difference computation or catalog generation.
    outputs:
      ndwi:
        outputBinding:
          glob: '*.tif'
        type: File
        label: Normalized difference water index
        doc: Water index raster computed from the input bands.
  - class: CommandLineTool
    id: otsu
    requirements:
      InlineJavascriptRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: /app/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ResourceRequirement:
        coresMin: 1
        coresMax: 1
        ramMin: 1024
        ramMax: 1024
      NetworkAccess:
        networkAccess: false
      DockerRequirement:
        dockerPull: localhost/otsu:latest
    baseCommand: otsu
    arguments: []
    inputs:
      raster:
        type: File
        inputBinding:
          position: 1
          prefix: --raster
        label: Water index raster
        doc: Normalized difference water index raster to threshold.
    outputs:
      binary_mask_item:
        outputBinding:
          glob: '*.tif'
        type: File
        label: Water body mask
        doc: Binary raster mask identifying detected water bodies.
  - class: CommandLineTool
    id: stac
    requirements:
      InlineJavascriptRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: /app/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      ResourceRequirement:
        coresMin: 1
        coresMax: 1
        ramMin: 512
        ramMax: 512
      NetworkAccess:
        networkAccess: true
      DockerRequirement:
        dockerPull: localhost/stac:latest
    baseCommand: stac
    arguments: []
    inputs:
      item:
        type: Directory
        inputBinding:
          prefix: --item
        label: STAC item
        doc: Directory containing the staged acquisition STAC catalog and assets.
      rasters:
        type: File
        inputBinding:
          prefix: --rasters
        label: Input rasters
        doc: Raster files used for normalized difference computation or catalog generation.
    outputs:
      stac_catalog:
        outputBinding:
          glob: .
        type: Directory
        label: Results STAC catalog
        doc: Directory containing the results STAC catalog and its referenced raster assets.

Typed inputs¶

Use a YAML job document for the local CWL definitions. AOIs require a Polygon record with type, coordinates, and bbox; the crop command uses the supplied bounding box. STAC references use URI records with a value field. Workflow EPSG values are "4326", and supported bands are green, nir, and nir08. Staged input acquisitions retain their Directory type.

The staged processing lab (practice-labs/4-CWL-Workflows/2.2-staged.ipynb) creates a job using the directory returned in staged.json.

cat <<EOF > staged-params.yaml
item:
  class: Directory
  path: "$(jq -r .staged.path staged.json)"
aoi:
  type: Polygon
  coordinates:
    -   -   - -118.985
            - 38.432
        -   - -118.183
            - 38.432
        -   - -118.183
            - 38.938
        -   - -118.985
            - 38.938
        -   - -118.985
            - 38.432
  bbox:
    - -118.985
    - 38.432
    - -118.183
    - 38.938
epsg: '4326'
bands:
  - green
  - nir08
EOF

cwltool \
    --podman \
    --outdir "${WORKSPACE}/runs" \
    "${WORKSPACE}/cwl-workflow/app-water-body.cwl#water-bodies" \
    staged-params.yaml