Continuous integration

Application Package Software Configuration Management¶

The SCM has the task of tracking and controlling changes in the software as a part of the larger cross-disciplinary field of configuration management.

SCM practices include revision control and the establishment of baselines.

The Application Package code is hosted on a repository publicly accessible (Github, Bitbucket, a GitLab instance, an institutional software forge, etc.) using one of the version control systems supported by (Subversion, Mercurial and Git)

The Application Package code include, at the top level of the source code tree, the following files:

README containing a description of the software (name, purpose, pointers to website, documentation, development platform, contact, and support information, …)
AUTHORS, a list of all the persons to be credited for the software.
LICENSE, the project license terms. For Open Source Licenses, the standard SPDX license names are used. For large software projects and developers, the REUSE (https://reuse.software/) process and tools can be an option to look at.
codemeta.json, a linked data metadata file that helps index the source code in the Software Heritage archive and provides an easy way to link to other related research outputs.

The codemeta.json includes metadata information to support the Continuous Integration phase and it is shown below:

codemeta.json
{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "@type": "SoftwareSourceCode",
    "license": "https://spdx.org/licenses/CC-BY-SA-4.0",
    "codeRepository": "https://github.com/eoap/mastering-app-package.git",
    "dateCreated": "2022-09-01",
    "datePublished": "2022-09-25",
    "dateModified": "2026-10-12",
    "name": "Water Bodies Detection",
    "version": "2.0.2",
    "description": "The Water Bodies Detection is an Application that uses the NDWI index and the Otsu threshold to detect water bodies using Sentinel-2 or Landsat-9 data",
    "developmentStatus": "active",
    "downloadUrl": "https://github.com/eoap/mastering-app-package/releases/tag/2.0.2",
    "relatedLink": [
        "https://eoap.github.io/mastering-app-package"
    ],
    "funder": {
        "@type": "Organization",
        "name": "Terradue"
    },
    "keywords": [
        "NDWI",
        "Landsat-9",
        "Sentinel-2",
        "Water Bodies"
    ],
    "programmingLanguage": [
        "Python",
        "CWL"
    ],
    "softwareRequirements": [
        "container runtime",
        "cwl runner"
    ],
    "author": [
        {
            "@type": "Person",
            "givenName": "Jane",
            "familyName": "Doe",
            "email": "jane.doe@acme.earth",
            "affiliation": {
                "@type": "Organization",
                "name": "ACME"
            }
        },
        {
            "@type": "Person",
            "givenName": "John",
            "familyName": "Doe",
            "email": "john.doe@acme.earth",
            "affiliation": {
                "@type": "Organization",
                "name": "ACME"
            }
        }
    ]
}

Application Package Continuous Integration¶

A typical Continuous Integration scenario for an Application Package includes the release of the CWL document(s) and publishing the container images to a container registry.

This is depicted below:

graph TB SCM[(software repository)] --> T(Test packages and validate CWL) T --> V(Check new release version) V --> B(Build images) B --> S(Scan images) S --> CR[(Push images to GHCR)] CR --> P(Update metadata and image digests) P --> C(Pack and validate CWL) C --> A(Generate annotations with cwl2oci) A --> O[(Publish CWL with ORAS)] O --> R(Pull by digest and verify) R --> G(Publish GitHub release and assets)

Below an example of a GitHub CI configuration implementing the scenario:

.github/workflows/build.yaml
name: Release application packages
on:
  push:
    branches: [master, main]
    paths:
      - 'water-bodies/**'
      - '.github/**'
      - 'cwl-cli/**'
      - 'cwl-workflow/**'
      - 'scripts/**'
      - 'tests/**'
      - 'codemeta.json'
      - 'CHANGELOG.md'
  workflow_dispatch:
    inputs:
      channel:
        description: Publish a candidate for Kubernetes lab testing or a stable release
        type: choice
        options: [candidate, stable]
        default: candidate
permissions:
  contents: read
concurrency:
  group: application-release
  cancel-in-progress: false
jobs:
  tests:
    uses: ./.github/workflows/processing-tests.yaml

  version:
    needs: tests
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.metadata.outputs.version }}
      registry: ${{ steps.metadata.outputs.registry }}
      base-version: ${{ steps.metadata.outputs.base-version }}
      candidate: ${{ steps.metadata.outputs.candidate }}
    steps:
      - uses: actions/checkout@v4
      - id: metadata
        name: Check version and publication targets
        env:
          GH_TOKEN: ${{ github.token }}
          CHANNEL: ${{ inputs.channel }}
        shell: bash
        run: |
          version=$(jq -er '.version' codemeta.json)
          [[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?$ ]]
          grep -Fq "## [$version]" CHANGELOG.md || {
            echo "::error::CHANGELOG.md must describe release $version."
            exit 1
          }
          echo "base-version=$version" >> "$GITHUB_OUTPUT"
          candidate=false
          if [[ "$GITHUB_EVENT_NAME" == workflow_dispatch && "$CHANNEL" == candidate ]]; then
            version="$version-rc.$GITHUB_RUN_NUMBER.$GITHUB_RUN_ATTEMPT"
            candidate=true
          elif [[ "$GITHUB_REF" != refs/heads/main && "$GITHUB_REF" != refs/heads/master ]]; then
            echo "::error::Stable releases must run from main or master. Use candidate on develop."
            exit 1
          fi
          echo "candidate=$candidate" >> "$GITHUB_OUTPUT"
          registry="ghcr.io/${GITHUB_REPOSITORY,,}"
          echo "version=$version" >> "$GITHUB_OUTPUT"
          echo "registry=$registry" >> "$GITHUB_OUTPUT"
          gh api "repos/$GITHUB_REPOSITORY/releases" --paginate --jq '.[].tag_name' > /tmp/release-tags
          gh api "repos/$GITHUB_REPOSITORY/git/matching-refs/tags/$version" --jq '.[].ref' > /tmp/tag-refs
          if grep -Fxq "$version" /tmp/release-tags || grep -Fxq "refs/tags/$version" /tmp/tag-refs; then
            echo "::error::Version $version already exists. Bump codemeta.json before releasing."
            exit 1
          fi

  container-build:
    needs: version
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    strategy:
      fail-fast: false
      matrix:
        step: [crop, norm_diff, otsu, stac, stage-in, stage-out]
    env:
      IMAGE: ${{ needs.version.outputs.registry }}/${{ matrix.step }}
      VERSION: ${{ needs.version.outputs.version }}
    steps:
      - uses: actions/checkout@v4
      - uses: oras-project/setup-oras@v1
        with:
          version: '1.3.0'
      - name: Authenticate
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          echo "$GH_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
          echo "$GH_TOKEN" | oras login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
      - name: Refuse to replace an existing image tag
        run: |
          oras repo tags "$IMAGE" > /tmp/image-tags 2>/tmp/registry-error || {
            # A repository that has never been published has no tags.
            grep -Eq 'NAME_UNKNOWN|name unknown|404' /tmp/registry-error || { cat /tmp/registry-error; exit 1; }
          }
          if grep -Fxq "$VERSION" /tmp/image-tags; then
            echo "::error::Image $IMAGE:$VERSION already exists. Use a new release version."
            exit 1
          fi
      - name: Build image
        run: docker build -t "$IMAGE:$VERSION" "water-bodies/command-line-tools/${{ matrix.step }}"
      - name: Scan image (block fixable HIGH and CRITICAL vulnerabilities)
        uses: aquasecurity/trivy-action@0.35.0
        with:
          version: 'v0.69.3'
          image-ref: ${{ env.IMAGE }}:${{ env.VERSION }}
          scanners: vuln
          format: json
          output: scan-${{ matrix.step }}.json
          severity: HIGH,CRITICAL
          ignore-unfixed: true
          exit-code: '1'
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: scan-${{ matrix.step }}
          path: scan-${{ matrix.step }}.json
          if-no-files-found: ignore
      - name: Push and record immutable image reference
        run: |
          docker push "$IMAGE:$VERSION"
          mkdir -p digests
          digest=$(oras resolve "$IMAGE:$VERSION")
          [[ "$digest" =~ ^sha256:[a-f0-9]{64}$ ]]
          printf '%s@%s\n' "$IMAGE" "$digest" > "digests/${{ matrix.step }}.txt"
      - uses: actions/upload-artifact@v4
        with:
          name: digest-${{ matrix.step }}
          path: digests/*.txt
          if-no-files-found: error

  publish-artifacts:
    needs: [version, container-build]
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    env:
      VERSION: ${{ needs.version.outputs.version }}
      REGISTRY: ${{ needs.version.outputs.registry }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: python -m pip install -r scripts/requirements-cwl-release.txt
      - uses: oras-project/setup-oras@v1
        with:
          version: '1.3.0'
      - uses: actions/download-artifact@v4
        with:
          pattern: digest-*
          merge-multiple: true
          path: digests
      - name: Prepare and validate standalone release CWL
        run: python scripts/prepare-cwl-release.py --digests digests --output downloads --version "$VERSION"
      - name: Authenticate to GHCR
        env:
          GH_TOKEN: ${{ github.token }}
        run: echo "$GH_TOKEN" | oras login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
      - name: Annotate, publish, and verify CWL artifacts
        shell: bash
        run: |
          : > downloads/oci-artifacts.txt
          for name in app-water-bodies-cloud-native app-water-body-cloud-native app-water-body stage-in stage-out; do
            file="$name.$VERSION.cwl"
            annotations="$name.$VERSION.annotations.json"
            transpiler-mate cwl2oci \
              --image-source "https://github.com/$GITHUB_REPOSITORY" \
              --image-revision "$GITHUB_SHA" \
              --output "downloads/$annotations" "downloads/$file#main"
            cwl_type=Workflow
            if [[ "$name" == stage-in || "$name" == stage-out ]]; then cwl_type=CommandLineTool; fi
            python scripts/check-oci-annotations.py "downloads/$annotations" "$file" "$VERSION" "$cwl_type"
            target="$REGISTRY/cwl/$name:$VERSION"
            oras repo tags "$REGISTRY/cwl/$name" > /tmp/cwl-tags 2>/tmp/registry-error || {
              grep -Eq 'NAME_UNKNOWN|name unknown|404' /tmp/registry-error || { cat /tmp/registry-error; exit 1; }
            }
            if grep -Fxq "$VERSION" /tmp/cwl-tags; then
              echo "::error::Artifact $target already exists. Use a new release version."
              exit 1
            fi
            (cd downloads && oras push --artifact-type application/cwl \
              --annotation-file "$annotations" "$target" "$file:application/cwl")
            digest=$(oras resolve "$target")
            [[ "$digest" =~ ^sha256:[a-f0-9]{64}$ ]]
            reference="$REGISTRY/cwl/$name@$digest"
            oras manifest fetch "$reference" > "downloads/$name.$VERSION.manifest.json"
            jq -e --slurpfile expected "downloads/$annotations" \
              '.annotations as $actual | $expected[0]."$manifest" | to_entries | all(.[]; $actual[.key] == .value)' \
              "downloads/$name.$VERSION.manifest.json"
            mkdir -p "verified/$name"
            oras pull "$reference" -o "verified/$name"
            cmp "downloads/$file" "verified/$name/$file"
            cwltool --validate "verified/$name/$file#main"
            printf '%s\n' "$reference" >> downloads/oci-artifacts.txt
          done
      - uses: actions/upload-artifact@v4
        with:
          name: application-packages
          path: downloads/*
          if-no-files-found: error

  create-release:
    needs: [version, publish-artifacts]
    runs-on: ubuntu-latest
    permissions:
      contents: write
    env:
      VERSION: ${{ needs.version.outputs.version }}
      GH_TOKEN: ${{ github.token }}
      BASE_VERSION: ${{ needs.version.outputs.base-version }}
      CANDIDATE: ${{ needs.version.outputs.candidate }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: application-packages
          path: downloads
      - name: Create draft, attach verified assets, and publish
        run: |
          awk -v version="$BASE_VERSION" '
            index($0, "## [" version "]") == 1 { found=1; active=1 }
            active && /^## / && index($0, "## [" version "]") != 1 { active=0 }
            active { print }
            END { if (!found) exit 1 }
          ' CHANGELOG.md > /tmp/release-notes.md
          printf '\n### OCI artifacts\n\n' >> /tmp/release-notes.md
          cat downloads/oci-artifacts.txt >> /tmp/release-notes.md
          flags=()
          if [[ "$CANDIDATE" == true ]]; then
            flags+=(--prerelease --latest=false)
          fi
          gh release create "$VERSION" --target "$GITHUB_SHA" --title "$VERSION" \
            --draft "${flags[@]}" --notes-file /tmp/release-notes.md
          gh release upload "$VERSION" downloads/*
          gh release edit "$VERSION" --draft=false

The processing test workflow installs and tests the six Python packages, runs a synthetic processing chain, validates the CWL, and builds wheels and images. Release image digests are written to mandatory requirements.DockerRequirement entries. scripts/pack-cwl-release.py selects the workflow explicitly, embeds the schema imports, preserves package metadata, and validates the downloaded artifact as a standalone CWL file.

The release workflow calls the processing test workflow before publishing anything. Set a new version in codemeta.json; an existing GitHub release or tag is rejected. The workflow runs on changes to release inputs on main or master, or manually with workflow_dispatch. Concurrent releases are serialized.

Each image is built and scanned with Trivy before it is pushed. Fixable HIGH and CRITICAL vulnerabilities block publication; the JSON report is retained as a run artifact, including on scan failure. Unfixed vulnerabilities do not block this gate. The image jobs record immutable GHCR references for the packaging job, which updates mandatory Docker requirements and release metadata in isolated copies of the workflows. Repository source files are not rewritten.

After packing and validation, pinned transpiler-mate-runtime and cwl2oci generate the OCI annotations from each final CWL document's #main entrypoint. The generated $manifest structure is passed directly to ORAS. The entrypoint annotation is normalized from the runtime's process identifier to the artifact filename plus #main; source and revision identify the repository and release commit. These annotations describe the CWL artifacts; they are not Docker image labels.

ORAS publishes the three workflows at ghcr.io/<owner>/<repository>/cwl/<workflow>:<version>, using the project's application/cwl artifact and layer media type. Existing image and CWL tags are rejected rather than replaced. Each artifact is pulled by digest, compared with the packed file, checked against the generated annotations, and validated again. oci-artifacts.txt records immutable references. CWL files, annotation JSON files, OCI manifests, and that inventory are attached to a GitHub draft release, which is published only after every verification and asset upload succeeds. A failed upload leaves a draft for inspection.

The workflow uses GITHUB_TOKEN with packages: write only for registry jobs and contents: write only for release creation. GHCR must allow the repository's token to publish these packages. A failure after image or OCI publication can leave registry artifacts without a GitHub release; inspect the failed run and use a new version for the next release. Registry publication is not transactional.

For the prepared 2.0.2 release, the changelog documents the typed-input and console-command migration. The release job requires a matching version section in CHANGELOG.md and uses it as the GitHub release notes, followed by the verified OCI artifact references. Package versions and current CWL metadata are aligned to 2.0.2; historical released CWL examples retain their old metadata.

Validate chapter 5 before a stable release¶

Publish a candidate from develop with the manual release workflow:

gh workflow run build.yaml --ref develop -f channel=candidate

This command publishes registry artifacts and a GitHub prerelease after the usual tests and image scans. It does not publish the stable version or mark the candidate as the latest release. Candidates use a unique version such as 2.0.0-rc.<run-number>.<run-attempt>. Find the actual tag in the completed run and GitHub Releases, then set APPLICATION_PACKAGE_VERSION in chapter 5. The release includes typed-scatter-inputs.yaml so inputs match the selected CWL.

Test the candidate's direct Calrissian execution, Kubernetes Job, and both benchmark runs on the target cluster using registry-pulled images. Check package visibility or image-pull credentials; publication alone does not grant a cluster access to a private GHCR package. Record the candidate tag, source commit, cluster configuration, digests, outputs and reports before promoting changes.

Only after these tests pass should the tested changes be merged into main or master for stable publication. Manual channel=stable also requires one of those branches. Stable CI rebuilds images and repeats its gates, so verify the final stable artifacts with a cluster smoke test as well. Candidate testing is an explicit release procedure; this workflow does not automatically execute the Kubernetes notebooks or enforce a cluster approval gate.