How-to Guide: Scattering Workflows¶

This guide demonstrates how to use scattering in CWL workflows, allowing parallel execution of tasks on multiple inputs.

The focus is on the ScatterFeatureRequirement and scatterMethod.

Objective¶

• Execute a subworkflow (rgb-composite) in parallel for multiple stac-items.
• Scatter the input stac-items to create multiple RGB composite outputs.

Key Concepts¶

1. ScatterFeatureRequirement

The ScatterFeatureRequirement enables scattering, allowing a step to process multiple inputs in parallel.

This makes it possible to process an array of inputs independently, where each item is executed as a separate job.

2. Scatter and ScatterMethod

• scatter: Specifies the input field(s) to be scattered.
• scatterMethod: Defines how the inputs are combined when multiple fields are scattered:
  • dotproduct: Matches corresponding elements of input arrays (e.g., the first element of each array is processed together).
  • nested_crossproduct: Creates a Cartesian product of inputs, producing combinations of all elements.
  • flat_crossproduct: Similar to nested_crossproduct but flattens the structure of results.

• scatter: stac-item: The stac-item array is scattered, creating one subworkflow execution per item.
• scatterMethod: dotproduct: Matches each stac-item with the same bands input for each subworkflow execution.

Steps¶

1. Define the Workflow

The main workflow (scatter-workflows.cwl) processes multiple stac-items in parallel:

Workflow Definition:

Its graphical representation:

Inputs:

• stac-items: An array of URLs to STAC items.
• bands: Array of band names (default: ["red", "green", "blue"]).

Output:

• rgb-tif: An array of RGB TIFF files (one per stac-item).

Steps:

• step_rgb_composite: Calls the rgb-composite subworkflow with scattered stac-items.

2. Define the Subworkflow

The subworkflow (rgb-composite) processes a single stac-item to create an RGB composite:

Subworkflow Definition

Its graphical representation:

Scatter in Subworkflow:

• step_curl scatters bands (e.g., red, green, blue).
• Each band is processed independently to retrieve its corresponding asset.

3. Run the Workflow

Execute the main workflow with multiple stac-items:

cwltool scatter-workflows.cwl \
    --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_53HPA_20210723_0_L2A \
    --stac-item https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2A_53HPA_20210728_0_L2A

4. Expected Output

The workflow creates an array of RGB composite TIFF files, one for each stac-item:

Output Files:

• rgb-tif[0]: RGB composite for the first stac-item.
• rgb-tif[1]: RGB composite for the second stac-item.

Key Takeaways¶

1. ScatterFeatureRequirement:

• Enables parallel processing of array inputs.

2. Scatter and ScatterMethod:

• scatter: Specifies the input to scatter.
• scatterMethod:
  • dotproduct: Matches corresponding elements in input arrays.
  • nested_crossproduct: Generates all combinations of input elements.

3 .Scalable Workflows:

• Scattering simplifies large-scale processing by distributing tasks over multiple inputs, improving performance and modularity.

This guide demonstrates how scattering in CWL can streamline workflows by enabling parallel execution of steps on array inputs.