TL;DR¶

Zarr Cloud native format in Earth Observation Application Packages.

Application Package Overview¶

See https://github.com/eoap/zarr-cloud-native-format/releases for the latest Application Packages

Producer Application Package¶

The workflow is based on one of the workflows of the https://github.com/eoap#mastering-earth-observation-application-packaging-with-cwl module extended to provide the temporal element.

The steps are:

STAC API Discovery: defines a STAC API search request and queries a STAC API endpoint returning a FeatureCollection
SearchResults: extracts the discovered STAC Items self href.
Water bodies detection: a sub-workflow that runs:

Cropping: Crops Sentinel-2 imagery to the Area of Interest (AOI).
Normalized Difference Water Index (NDWI): Computes NDWI to identify water bodies.
Otsu Thresholding: Applies Otsu's thresholding method to binarize NDWI values.

Zarr dataset creation and STAC Metadata: Converts the results into a Zarr dataset and generates the STAC Collection including STAC Datacube Extension.

In [1]:

Copied!





import os
import sys

module_path = os.path.abspath(os.path.join("."))
sys.path.insert(0, module_path)

from helpers import WorkflowViewer
import cwl_loader
cwl_loader.logger.remove()
cwl_loader.logger.add(sys.stderr, level="INFO")
import os
import sys

module_path = os.path.abspath(os.path.join("."))
sys.path.insert(0, module_path)

from helpers import WorkflowViewer
import cwl_loader
cwl_loader.logger.remove()
cwl_loader.logger.add(sys.stderr, level="INFO")

Out[1]:

In [2]:

Copied!

version = "0.3.0"
wf = WorkflowViewer.from_file(f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies.{version}.cwl", "water-bodies")
version = "0.3.0"
wf = WorkflowViewer.from_file(f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies.{version}.cwl", "water-bodies")

Inputs¶

In [3]:

Copied!

wf.display_inputs()
wf.display_inputs()

Id	Type	Label	Doc
`stac_api_endpoint`	`https://raw.githubusercontent.com/eoap/schemas/main/experimental/api-endpoint.yaml#APIEndpoint`	STAC API endpoint	STAC API endpoint
`search_request`	`https://raw.githubusercontent.com/eoap/schemas/main/experimental/discovery.yaml#STACSearchSettings`	STAC search request	STAC search request
`bands`	`string[]`	bands used for the NDWI	bands used for the NDWI

Steps¶

In [4]:

Copied!

wf.display_steps()
wf.display_steps()

Id	Runs	Label	Doc
`discovery`	`#stac-client`	STAC API discovery	Discover STAC items from a STAC API endpoint based on a search request
`convert_search`	`#convert-search`	Convert Search	Convert Search results to get the item self hrefs and the area of interest
`water_bodies`	`#detect_water_body`	Water bodies detection	Water bodies detection based on NDWI and otsu threshold applied to each STAC item (sub-workflow)
`stac`	`#stac`	Create a STAC catalog with COG outputs	Create a STAC catalog with the detected water bodies COG outputs
`stac_zarr`	`#stac-zarr`	Create a STAC Catalog for the Zarr store	Create a STAC Catalog for the Zarr store from the STAC catalog with COG outputs

Outputs¶

In [5]:

Copied!

wf.display_outputs()
wf.display_outputs()

Id	Type	Label	Doc
`zarr_stac_catalog`	`Directory`	None	None
`stac_catalog`	`Directory`	None	None

Component diagram¶

In [6]:

Copied!

wf.display_components_diagram(entrypoint="water-bodies")
wf.display_components_diagram(entrypoint="water-bodies")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[6], line 1
----> 1 wf.display_components_diagram(entrypoint="water-bodies")

File ~/work/zarr-cloud-native-format/zarr-cloud-native-format/docs/helpers.py:92, in WorkflowViewer.display_components_diagram(self, entrypoint)
     89     wf = self.workflow
     90     entrypoint = self.entrypoint
---> 92 self._display_puml(DiagramType.COMPONENT, wf=wf, workflow_id=entrypoint)

File ~/work/zarr-cloud-native-format/zarr-cloud-native-format/docs/helpers.py:68, in WorkflowViewer._display_puml(self, diagram_type, wf, workflow_id)
     65 def _display_puml(self, diagram_type: DiagramType, wf, workflow_id=None):
     67     out = StringIO()
---> 68     to_puml(
     69         cwl_document=wf,
     70         diagram_type=diagram_type,
     71         output_stream=out,
     72         workflow_id=workflow_id,
     73     )
     75     clear_output = out.getvalue()
     76     encoded = deflate_and_encode(clear_output)

File /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/cwl2puml/__init__.py:224, in to_puml(cwl_document, diagram_type, output_stream, workflow_id)
    211 """
    212 Converts a CWL, given its document model, to a PlantUML diagram.
    213 
   (...)    220     `None`: none
    221 """
    222 assert_process_contained(process=cwl_document, process_id=workflow_id)
--> 224 assert_connected_graph(cwl_document)
    226 index = (
    227     to_index(cwl_document)
    228     if isinstance(cwl_document, list)
    229     else {workflow_id: cwl_document}
    230 )
    232 template = _jinja_environment.get_template(f"{diagram_type.name.lower()}.puml")

File /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/cwl_loader/utils.py:120, in assert_connected_graph(process)
    118 if issues:
    119     nl = "\n"
--> 120     raise ValueError(
    121         f"Detected unresolved links in the input $graph:\n{nl.join(issues)}"
    122     )

ValueError: Detected unresolved links in the input $graph:
- water-bodies.steps.discovery = #stac-client
- water-bodies.steps.convert_search = #convert-search
- water-bodies.steps.water_bodies = #detect_water_body
- water-bodies.steps.stac = #stac
- water-bodies.steps.stac_zarr = #stac-zarr

Class diagram¶

In [7]:

Copied!

wf.display_class_diagram()
wf.display_class_diagram()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], line 1
----> 1 wf.display_class_diagram()

File ~/work/zarr-cloud-native-format/zarr-cloud-native-format/docs/helpers.py:99, in WorkflowViewer.display_class_diagram(self, entrypoint)
     96     entrypoint = self.entrypoint
     98 wf = search_process(process_id=entrypoint, process=self.workflow)
---> 99 self._display_puml(DiagramType.CLASS, wf=wf, workflow_id=entrypoint)

File ~/work/zarr-cloud-native-format/zarr-cloud-native-format/docs/helpers.py:68, in WorkflowViewer._display_puml(self, diagram_type, wf, workflow_id)
     65 def _display_puml(self, diagram_type: DiagramType, wf, workflow_id=None):
     67     out = StringIO()
---> 68     to_puml(
     69         cwl_document=wf,
     70         diagram_type=diagram_type,
     71         output_stream=out,
     72         workflow_id=workflow_id,
     73     )
     75     clear_output = out.getvalue()
     76     encoded = deflate_and_encode(clear_output)

File /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/cwl2puml/__init__.py:224, in to_puml(cwl_document, diagram_type, output_stream, workflow_id)
    211 """
    212 Converts a CWL, given its document model, to a PlantUML diagram.
    213 
   (...)    220     `None`: none
    221 """
    222 assert_process_contained(process=cwl_document, process_id=workflow_id)
--> 224 assert_connected_graph(cwl_document)
    226 index = (
    227     to_index(cwl_document)
    228     if isinstance(cwl_document, list)
    229     else {workflow_id: cwl_document}
    230 )
    232 template = _jinja_environment.get_template(f"{diagram_type.name.lower()}.puml")

File /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/cwl_loader/utils.py:120, in assert_connected_graph(process)
    118 if issues:
    119     nl = "\n"
--> 120     raise ValueError(
    121         f"Detected unresolved links in the input $graph:\n{nl.join(issues)}"
    122     )

ValueError: Detected unresolved links in the input $graph:
- water-bodies.steps.discovery = #stac-client
- water-bodies.steps.convert_search = #convert-search
- water-bodies.steps.water_bodies = #detect_water_body
- water-bodies.steps.stac = #stac
- water-bodies.steps.stac_zarr = #stac-zarr

Consumer Application Package¶

The workflow reads the produced STAC Catalog describing the detected water bodies Zarr store and produces the mean over the time dimension.

There's a single step that:

Read STAC Catalog: read the STAC Catalog and inspect the STAC Collection
Read the Zarr store: read the Zarr store STAC Asset
mean calculation: use xarray to calculate the mean over the time dimension
export to GeoTIFF: user rioxarray to write a GeoTIFF
Generate STAC Catalog: create a STAC Catalog describing the result

In [8]:

Copied!

wf = WorkflowViewer.from_file(f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies-occurrence.{version}.cwl", "water-bodies-occurrence")
wf = WorkflowViewer.from_file(f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies-occurrence.{version}.cwl", "water-bodies-occurrence")

Inputs¶

In [9]:

Copied!

wf.display_inputs()
wf.display_inputs()

Id	Type	Label	Doc
`zarr-stac-catalog`	`Directory`	Zarr store STAC Catalog	Input STAC catalog with datacube

Steps¶

In [10]:

Copied!

wf.display_steps()
wf.display_steps()

Id	Runs	Label	Doc
`step_occurrence`	`#occurrence`	Water bodies occurrence	Water bodies occurrence based on NDWI and otsu threshold

Outputs¶

In [11]:

Copied!

wf.display_outputs()
wf.display_outputs()

Id	Type	Label	Doc
`stac-catalog`	`Directory`	STAC catalog	Output STAC catalog with water bodies occurrence

Component diagram¶

In [12]:

Copied!

wf.display_components_diagram()
wf.display_components_diagram()

No description has been provided for this image

Class diagram¶

In [13]:

Copied!

wf.display_class_diagram()
wf.display_class_diagram()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 1
----> 1 wf.display_class_diagram()

File ~/work/zarr-cloud-native-format/zarr-cloud-native-format/docs/helpers.py:99, in WorkflowViewer.display_class_diagram(self, entrypoint)
     96     entrypoint = self.entrypoint
     98 wf = search_process(process_id=entrypoint, process=self.workflow)
---> 99 self._display_puml(DiagramType.CLASS, wf=wf, workflow_id=entrypoint)

File ~/work/zarr-cloud-native-format/zarr-cloud-native-format/docs/helpers.py:68, in WorkflowViewer._display_puml(self, diagram_type, wf, workflow_id)
     65 def _display_puml(self, diagram_type: DiagramType, wf, workflow_id=None):
     67     out = StringIO()
---> 68     to_puml(
     69         cwl_document=wf,
     70         diagram_type=diagram_type,
     71         output_stream=out,
     72         workflow_id=workflow_id,
     73     )
     75     clear_output = out.getvalue()
     76     encoded = deflate_and_encode(clear_output)

File /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/cwl2puml/__init__.py:224, in to_puml(cwl_document, diagram_type, output_stream, workflow_id)
    211 """
    212 Converts a CWL, given its document model, to a PlantUML diagram.
    213 
   (...)    220     `None`: none
    221 """
    222 assert_process_contained(process=cwl_document, process_id=workflow_id)
--> 224 assert_connected_graph(cwl_document)
    226 index = (
    227     to_index(cwl_document)
    228     if isinstance(cwl_document, list)
    229     else {workflow_id: cwl_document}
    230 )
    232 template = _jinja_environment.get_template(f"{diagram_type.name.lower()}.puml")

File /opt/hostedtoolcache/Python/3.12.13/x64/lib/python3.12/site-packages/cwl_loader/utils.py:120, in assert_connected_graph(process)
    118 if issues:
    119     nl = "\n"
--> 120     raise ValueError(
    121         f"Detected unresolved links in the input $graph:\n{nl.join(issues)}"
    122     )

ValueError: Detected unresolved links in the input $graph:
- water-bodies-occurrence.steps.step_occurrence = #occurrence

Pre-requisites¶

To run this notebook, the pre-requisites are:

A container runtime like docker or podman
The cwltool CWL runner
The Python libraries shutil, pystac, xarray, rasterio and matplotlib

Part 1 - Zarr store producer: Application Package execution¶

Use cwltool to run the producer Application Package.

Modules used¶

In [14]:

Copied!





import yaml
from shutil import which
import nest_asyncio
from cwltool.main import main
from io import StringIO
import argparse
import json
nest_asyncio.apply()
import yaml
from shutil import which
import nest_asyncio
from cwltool.main import main
from io import StringIO
import argparse
import json
nest_asyncio.apply()

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[14], line 3
      1 import yaml
      2 from shutil import which
----> 3 import nest_asyncio
      4 from cwltool.main import main
      5 from io import StringIO
      6 import argparse

ModuleNotFoundError: No module named 'nest_asyncio'

Container runtime¶

Check if podman or docker are available:

In [15]:

Copied!





if which("podman"):
    podman = True
    print("Using podman")
elif which("docker"):
    podman = False
    print("Using docker")
else:
    raise ValueError("No container engine")

podman = False
if which("podman"):
    podman = True
    print("Using podman")
elif which("docker"):
    podman = False
    print("Using docker")
else:
    raise ValueError("No container engine")

podman = False

Using podman

Prepare the producer Application Package execution¶

Use cwltool and Python to execute the Application Package.

First create a YAML file with the parameters:

In [16]:

Copied!





params = {}

params["bands"] = ["green", "nir"]

params["stac_api_endpoint"] = {
    "url": {"value": "https://earth-search.aws.element84.com/v1/"},
    "headers": [],
}

params["search_request"] = {
    "limit": 20,
    "collections": ["sentinel-2-l2a"],
    "datetime_interval": {
        "start": {"value": "2021-06-01T00:00:00"},
        "end": {"value": "2021-08-01T23:59:59"},
    },
    "bbox": [-121.399, 39.834, -120.74, 40.472],
    "max-items": 10
}

with open("producer-params.yaml", "w") as file:
    print(yaml.dump(params), file=file)
params = {}

params["bands"] = ["green", "nir"]

params["stac_api_endpoint"] = {
    "url": {"value": "https://earth-search.aws.element84.com/v1/"},
    "headers": [],
}

params["search_request"] = {
    "limit": 20,
    "collections": ["sentinel-2-l2a"],
    "datetime_interval": {
        "start": {"value": "2021-06-01T00:00:00"},
        "end": {"value": "2021-08-01T23:59:59"},
    },
    "bbox": [-121.399, 39.834, -120.74, 40.472],
    "max-items": 10
}

with open("producer-params.yaml", "w") as file:
    print(yaml.dump(params), file=file)

In [17]:

Copied!

print(yaml.dump(params))
print(yaml.dump(params))

bands:
- green
- nir
search_request:
  bbox:
  - -121.399
  - 39.834
  - -120.74
  - 40.472
  collections:
  - sentinel-2-l2a
  datetime_interval:
    end:
      value: '2021-08-01T23:59:59'
    start:
      value: '2021-06-01T00:00:00'
  limit: 20
  max-items: 10
stac_api_endpoint:
  headers: []
  url:
    value: https://earth-search.aws.element84.com/v1/

Producer Application Package execution¶

We use the released application package available here https://github.com/eoap/zarr-cloud-native-format/releases and cwltool APIs to run the Application Package.

Any error messages such as:

Error: No such object: ghcr.io/eoap/zarr-cloud-native-format/stac-zarr@sha256:72fdc2757ace393b3d94995bd3e300e0f32c6a3906fe5e659f8a9efb95de76ea

is a not a real error, it's just that the container image isn't available locally and must be pulled.

In [18]:

Copied!





parsed_args = argparse.Namespace(
    podman=podman,
    parallel=False,
    validate=False,
    debug=False,
    outdir="./runs",
    quiet=True,
    workflow=f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies.{version}.cwl#water-bodies",
    job_order=["producer-params.yaml"],
)

stream_out = StringIO()
stream_err = StringIO()

res = main(
    args=parsed_args,
    stdout=stream_out,
    stderr=stream_err,
)

assert res == 0
parsed_args = argparse.Namespace(
    podman=podman,
    parallel=False,
    validate=False,
    debug=False,
    outdir="./runs",
    quiet=True,
    workflow=f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies.{version}.cwl#water-bodies",
    job_order=["producer-params.yaml"],
)

stream_out = StringIO()
stream_err = StringIO()

res = main(
    args=parsed_args,
    stdout=stream_out,
    stderr=stream_err,
)

assert res == 0

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[18], line 1
----> 1 parsed_args = argparse.Namespace(
      2     podman=podman,
      3     parallel=False,
      4     validate=False,

NameError: name 'argparse' is not defined

The stream_out object contains the cwltool stdout.

We can list the results keys with:

In [19]:

Copied!

producer_results = json.loads(stream_out.getvalue())

list(producer_results.keys())
producer_results = json.loads(stream_out.getvalue())

list(producer_results.keys())

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[19], line 1
----> 1 producer_results = json.loads(stream_out.getvalue())
      2 
      3 list(producer_results.keys())

NameError: name 'json' is not defined

And find the path of the directory containing the STAC Catalog with the Zarr store:

In [20]:

Copied!

producer_results["zarr_stac_catalog"]["path"]
producer_results["zarr_stac_catalog"]["path"]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[20], line 1
----> 1 producer_results["zarr_stac_catalog"]["path"]

NameError: name 'producer_results' is not defined

Part 2 - Inspect Application Package results¶

Read the STAC Catalog containing the Zarr store

Modules¶

In [21]:

Copied!

import pystac
import os
from pystac.extensions.datacube import DatacubeExtension
import pystac
import os
from pystac.extensions.datacube import DatacubeExtension

Inspect the STAC Collection metadata¶

Open the STAC Catalog generated as an output of the producer Application Package execution:

In [22]:

Copied!





cat = pystac.Catalog.from_file(
    os.path.join(
        producer_results["zarr_stac_catalog"]["path"], "catalog.json"
    )
)
cat = pystac.Catalog.from_file(
    os.path.join(
        producer_results["zarr_stac_catalog"]["path"], "catalog.json"
    )
)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[22], line 3
      1 cat = pystac.Catalog.from_file(
      2     os.path.join(
----> 3         producer_results["zarr_stac_catalog"]["path"], "catalog.json"
      4     )
      5 )

NameError: name 'producer_results' is not defined

In [23]:

Copied!

cat.describe()
cat.describe()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[23], line 1
----> 1 cat.describe()

NameError: name 'cat' is not defined

Now open the STAC Collection named water-bodies:

In [24]:

Copied!

collection = cat.get_child("water-bodies")

collection.describe()
collection = cat.get_child("water-bodies")

collection.describe()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[24], line 1
----> 1 collection = cat.get_child("water-bodies")
      2 
      3 collection.describe()

NameError: name 'cat' is not defined

In [25]:

Copied!

collection
collection

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[25], line 1
----> 1 collection

NameError: name 'collection' is not defined

The STAC Collection contains an asset:

In [26]:

Copied!

zarr_asset = collection.get_assets()["data"]

print(f"Zarr asset media type: {zarr_asset.media_type}")

zarr_asset
zarr_asset = collection.get_assets()["data"]

print(f"Zarr asset media type: {zarr_asset.media_type}")

zarr_asset

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[26], line 1
----> 1 zarr_asset = collection.get_assets()["data"]
      2 
      3 print(f"Zarr asset media type: {zarr_asset.media_type}")
      4 

NameError: name 'collection' is not defined

Inspect the datacube extension metadata included in the STAC Collection:

In [27]:

Copied!

dc_collection = DatacubeExtension.ext(collection)
dc_collection = DatacubeExtension.ext(collection)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[27], line 1
----> 1 dc_collection = DatacubeExtension.ext(collection)

NameError: name 'collection' is not defined

List and describe the dimensions

In [28]:

Copied!

for key, value in dc_collection.dimensions.items():

    print(key, dc_collection.dimensions[key].to_dict())
for key, value in dc_collection.dimensions.items():

    print(key, dc_collection.dimensions[key].to_dict())

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[28], line 1
----> 1 for key, value in dc_collection.dimensions.items():
      2 
      3     print(key, dc_collection.dimensions[key].to_dict())

NameError: name 'dc_collection' is not defined

List and describe the variables

In [29]:

Copied!

for key, value in dc_collection.variables.items():
    print(key, dc_collection.variables[key].to_dict())
for key, value in dc_collection.variables.items():
    print(key, dc_collection.variables[key].to_dict())

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[29], line 1
----> 1 for key, value in dc_collection.variables.items():
      2     print(key, dc_collection.variables[key].to_dict())

NameError: name 'dc_collection' is not defined

In [30]:

Copied!

dc_collection.variables["data"].to_dict()
dc_collection.variables["data"].to_dict()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[30], line 1
----> 1 dc_collection.variables["data"].to_dict()

NameError: name 'dc_collection' is not defined

Part 3 - Inspect the zarr store¶

Open the Zarr store with xarray to access its metadata and data.

Modules¶

In [31]:

Copied!

import xarray as xr
import xarray as xr

In [32]:

Copied!

zarr_asset.extra_fields["xarray:open_kwargs"]
zarr_asset.extra_fields["xarray:open_kwargs"]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[32], line 1
----> 1 zarr_asset.extra_fields["xarray:open_kwargs"]

NameError: name 'zarr_asset' is not defined

Open the Zarr result with xarray¶

In [33]:

Copied!

water_bodies = xr.open_zarr(zarr_asset.get_absolute_href(), **zarr_asset.extra_fields["xarray:open_kwargs"])
water_bodies = xr.open_zarr(zarr_asset.get_absolute_href(), **zarr_asset.extra_fields["xarray:open_kwargs"])

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[33], line 1
----> 1 water_bodies = xr.open_zarr(zarr_asset.get_absolute_href(), **zarr_asset.extra_fields["xarray:open_kwargs"])

NameError: name 'zarr_asset' is not defined

In [34]:

Copied!

xr.set_options(display_style="text")

water_bodies
xr.set_options(display_style="text")

water_bodies

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[34], line 3
      1 xr.set_options(display_style="text")
      2 
----> 3 water_bodies

NameError: name 'water_bodies' is not defined

Inspect the data variable:

In [35]:

Copied!

water_bodies.data
water_bodies.data

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[35], line 1
----> 1 water_bodies.data

NameError: name 'water_bodies' is not defined

Check the geospatial metadata

In [36]:

Copied!

water_bodies.data_vars["spatial_ref"]
water_bodies.data_vars["spatial_ref"]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[36], line 1
----> 1 water_bodies.data_vars["spatial_ref"]

NameError: name 'water_bodies' is not defined

In [37]:

Copied!

str(water_bodies.data_vars["spatial_ref"].values)
str(water_bodies.data_vars["spatial_ref"].values)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[37], line 1
----> 1 str(water_bodies.data_vars["spatial_ref"].values)

NameError: name 'water_bodies' is not defined

Inspect the Zarr coordinates

In [38]:

Copied!

water_bodies.coords
water_bodies.coords

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[38], line 1
----> 1 water_bodies.coords

NameError: name 'water_bodies' is not defined

List the values of the time coordinate:

In [39]:

Copied!

water_bodies.coords["time"].values
water_bodies.coords["time"].values

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[39], line 1
----> 1 water_bodies.coords["time"].values

NameError: name 'water_bodies' is not defined

Plot the water bodies detected on '2021-07-13T19:03:24.000000000'

In [40]:

Copied!





_ = (
    water_bodies.isel(time=1)
    .to_array("data")
    .plot.imshow(
        col="data",
        size=4,
        vmin=0,
        vmax=1,
    )
)
_ = (
    water_bodies.isel(time=1)
    .to_array("data")
    .plot.imshow(
        col="data",
        size=4,
        vmin=0,
        vmax=1,
    )
)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[40], line 2
      1 _ = (
----> 2     water_bodies.isel(time=1)
      3     .to_array("data")
      4     .plot.imshow(
      5         col="data",

NameError: name 'water_bodies' is not defined

Part 4 - Consume the Zarr store using the consumer Application Package¶

Goal: Invoke an Application Package that consumes the detected water bodies Zarr store.

Run the consumer Application Package¶

First, create a YAML file with the parameters: the path to the STAC Catalog describing the detected water bodies Zarr store.

In [41]:

Copied!





consumer_params = {}

consumer_params["zarr-stac-catalog"] = {
  "class": "Directory",
  "path": producer_results["zarr_stac_catalog"]["path"]
}

with open("consumer-params.yaml", "w") as file:
    print(yaml.dump(consumer_params), file=file)

print(yaml.dump(consumer_params))
consumer_params = {}

consumer_params["zarr-stac-catalog"] = {
  "class": "Directory",
  "path": producer_results["zarr_stac_catalog"]["path"]
}

with open("consumer-params.yaml", "w") as file:
    print(yaml.dump(consumer_params), file=file)

print(yaml.dump(consumer_params))

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[41], line 5
      1 consumer_params = {}
      2 
      3 consumer_params["zarr-stac-catalog"] = {
      4   "class": "Directory",
----> 5   "path": producer_results["zarr_stac_catalog"]["path"]
      6 }
      7 
      8 with open("consumer-params.yaml", "w") as file:

NameError: name 'producer_results' is not defined

Now, use cwltool APIs to run the consumer Application Package.

Again, any error messages such as:

Error: No such object: ghcr.io/eoap/zarr-cloud-native-format/occurrence@sha256:256f8672ff74f41b97ab307162b09e07412eddd1d9658a07106379597a7c9fad

is a not a real error, it's just that the container image isn't available locally and must be pulled.

In [42]:

Copied!





version = "0.3.0"

parsed_args = argparse.Namespace(
    podman=podman,
    parallel=False,
    validate=False,
    debug=False,
    outdir="./runs",
    quiet=True,
    workflow=f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies-occurrence.{version}.cwl#water-bodies-occurrence",
    job_order=["consumer-params.yaml"],
)

stream_out = StringIO()
stream_err = StringIO()

res = main(
    args=parsed_args,
    stdout=stream_out,
    stderr=stream_err,
)

assert res == 0
version = "0.3.0"

parsed_args = argparse.Namespace(
    podman=podman,
    parallel=False,
    validate=False,
    debug=False,
    outdir="./runs",
    quiet=True,
    workflow=f"https://github.com/eoap/zarr-cloud-native-format/releases/download/{version}/app-water-bodies-occurrence.{version}.cwl#water-bodies-occurrence",
    job_order=["consumer-params.yaml"],
)

stream_out = StringIO()
stream_err = StringIO()

res = main(
    args=parsed_args,
    stdout=stream_out,
    stderr=stream_err,
)

assert res == 0

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[42], line 3
      1 version = "0.3.0"
      2 
----> 3 parsed_args = argparse.Namespace(
      4     podman=podman,
      5     parallel=False,
      6     validate=False,

NameError: name 'argparse' is not defined

In [43]:

Copied!

print(stream_err.getvalue())
print(stream_err.getvalue())

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[43], line 1
----> 1 print(stream_err.getvalue())

NameError: name 'stream_err' is not defined

Inspect the generate STAC Catalog¶

The execution produces a STAC Catalog:

In [44]:

Copied!

consumer_results = json.loads(stream_out.getvalue())

consumer_results["stac-catalog"]["path"]
consumer_results = json.loads(stream_out.getvalue())

consumer_results["stac-catalog"]["path"]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[44], line 1
----> 1 consumer_results = json.loads(stream_out.getvalue())
      2 
      3 consumer_results["stac-catalog"]["path"]

NameError: name 'json' is not defined

In [45]:

Copied!





occurence_cat = pystac.Catalog.from_file(
    os.path.join(
        consumer_results["stac-catalog"]["path"], "catalog.json"
    )
)
occurence_cat = pystac.Catalog.from_file(
    os.path.join(
        consumer_results["stac-catalog"]["path"], "catalog.json"
    )
)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[45], line 3
      1 occurence_cat = pystac.Catalog.from_file(
      2     os.path.join(
----> 3         consumer_results["stac-catalog"]["path"], "catalog.json"
      4     )
      5 )

NameError: name 'consumer_results' is not defined

The STAC Catalog contains a STAC Item:

In [46]:

Copied!

occurence_cat.describe()
occurence_cat.describe()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[46], line 1
----> 1 occurence_cat.describe()

NameError: name 'occurence_cat' is not defined

In [47]:

Copied!

occurence_item = next(occurence_cat.get_all_items())

occurence_item
occurence_item = next(occurence_cat.get_all_items())

occurence_item

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[47], line 1
----> 1 occurence_item = next(occurence_cat.get_all_items())
      2 
      3 occurence_item

NameError: name 'occurence_cat' is not defined

Plot the mean GeoTiff¶

Open the data STAC Asset and plot the GeoTiff:

In [48]:

Copied!

occurence_item.get_assets()
occurence_item.get_assets()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[48], line 1
----> 1 occurence_item.get_assets()

NameError: name 'occurence_item' is not defined

In [49]:

Copied!





import rasterio
import matplotlib.pyplot as plt

with rasterio.open(occurence_item.get_assets()["data"].get_absolute_href()) as src:
    img = src.read(1)

plt.imshow(img, cmap="gray")
plt.colorbar()
plt.title("Water bodies occurrence")
plt.show()
import rasterio
import matplotlib.pyplot as plt

with rasterio.open(occurence_item.get_assets()["data"].get_absolute_href()) as src:
    img = src.read(1)

plt.imshow(img, cmap="gray")
plt.colorbar()
plt.title("Water bodies occurrence")
plt.show()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[49], line 4
      1 import rasterio
      2 import matplotlib.pyplot as plt
      3 
----> 4 with rasterio.open(occurence_item.get_assets()["data"].get_absolute_href()) as src:
      5     img = src.read(1)
      6 
      7 plt.imshow(img, cmap="gray")

NameError: name 'occurence_item' is not defined

In [ ]: