Skip to content

Continuous Integration

Application Package Software Configuration Management

The SCM has the task of tracking and controlling changes in the software as a part of the larger cross-disciplinary field of configuration management.

SCM practices include revision control and the establishment of baselines.

The Application Package code is hosted on a repository publicly accessible (Github, Bitbucket, a GitLab instance, an institutional software forge, etc.) using one of the version control systems supported by (Subversion, Mercurial and Git)

The Application Package code include, at the top level of the source code tree, the following files:

  • README containing a description of the software (name, purpose, pointers to website, documentation, development platform, contact, and support information, …)
  • AUTHORS, a list of all the persons to be credited for the software.
  • LICENSE, the project license terms. For Open Source Licenses, the standard SPDX license names are used. For large software projects and developers, the REUSE (https://reuse.software/) process and tools can be an option to look at.
  • codemeta.json, a linked data metadata file that helps index the source code in the Software Heritage archive and provides an easy way to link to other related research outputs.

The codemeta.json includes metadata information to support the Continuous Integration phase and it is shown below:

codemeta.json
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "@type": "SoftwareSourceCode",
    "license": "https://spdx.org/licenses/CC-BY-NC-SA-4.0",
    "codeRepository": "https://github.com/eoap/quickwin.git",
    "dateCreated": "2022-09-01",
    "datePublished": "2022-09-25",
    "dateModified": "2022-09-25",
    "name": "Water Bodies Detection",
    "version": "1.0.0",
    "description": "The Water Bodies Detection is an Application that uses the NDWI index and the Otsu threshold to detect water bodies using Sentinel-2 or Landsat-9 data",
    "developmentStatus": "active",
    "downloadUrl": "https://github.com/eoap/quickwin/releases/tag/1.0.0",
    "relatedLink": [
        "https://eoap.github.io/quickwin"
    ],
    "funder": {
        "@type": "Organization",
        "name": "Terradue"
    },
    "keywords": [
        "NDWI", "Landsat-9", "Sentinel-2", "Water Bodies"
    ],
    "programmingLanguage": [
        "Python", "CWL"
    ],
    "softwareRequirements": [
        "container runtime",
        "cwl runner"
    ],
    "author": [
        {
            "@type": "Person",
            "givenName": "Jane",
            "familyName": "Doe",
            "email": "jane.doe@acme.earth",
            "affiliation": {
                "@type": "Organization",
                "name": "ACME"
            }
        },
        {
            "@type": "Person",
            "givenName": "John",
            "familyName": "Doe",
            "email": "john.doe@acme.earth",
            "affiliation": {
                "@type": "Organization",
                "name": "ACME"
            }
        }
    ]
}

Application Package Continuous Integration

A typical Continuous Integration scenario for an Application Package includes the release of the CWL document(s) and publishing the container images to a container registry.

This is depicted below:

graph TB SCM[(software repository)] SCM -- CWL Workflow --> A SCM -- codemeta.json --> B A(validate CWL Workflow) --> B(extract version) B --> C subgraph Build containers SCM -- Dockerfiles --> C C(build container) --> D(push container) end D -- push --> CR[(Container Registry)] D -- container sha256 --> F("update Dockerpull/metadata in CWL Workflows") F -- push --> AR[(Artifact Registry)] SCM -- codemeta.json --> F

Below an example of a GitHub CI configuration implementing the scenario:

.github/workflows/build.yaml
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
name: build
on:
  push:
    branches:
    - master
    - main
    - bids23

    paths:
    # Only rebuild website when apps have changed
    - 'water-bodies/**'
    - .github/**
    - docs/**
    - cwl-workflow/*.cwl
    - codemeta.json
    - practice-labs/**

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: 3.x
    - run: pip install cwltool
    - run: cwltool --validate cwl-workflow/app-water-body-cloud-native.cwl

  version:
    needs: validate

    runs-on: ubuntu-latest

    outputs:

      app-version: ${{ steps.set-version.outputs.version }}

    steps:
    - uses: actions/checkout@v2
    - run: echo "APP_VERSION=$(cat codemeta.json | jq -r .version )" >> $GITHUB_ENV
    - run: echo app version is $APP_VERSION
    - id: set-version
      run: echo "::set-output name=version::$APP_VERSION"

  container-build:

    needs: version

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2   
    - run: echo version ${{needs.version.outputs.app-version}}
    - run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
    - name: build & push image
      run: |
        IMAGE_ID=ghcr.io/eoap/quickwin/detect-water-body
        docker build water-bodies/command-line-tools/detect-water-body --file water-bodies/command-line-tools/detect-water-body/Dockerfile --tag detect-water-body
        docker tag detect-water-body $IMAGE_ID:${{needs.version.outputs.app-version}}
        docker push $IMAGE_ID:${{needs.version.outputs.app-version}}

  create-release:

    needs:
    - container-build
    - version

    runs-on: ubuntu-latest

    outputs:

      upload_url: ${{ steps.set-upload-url.outputs.upload_url }}

    steps:
      - name: release
        uses: actions/create-release@v1
        id: create_release
        with:
          draft: false
          prerelease: false
          release_name: ${{needs.version.outputs.app-version}}
          tag_name: ${{needs.version.outputs.app-version}}
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - id: set-upload-url
        run: echo "::set-output name=upload_url::${{ steps.create_release.outputs.upload_url }}"

  publish-artifacts:

    needs:
    - create-release
    - version

    runs-on: ubuntu-latest

    steps:   
    - uses: actions/checkout@v2 
    - run: |

        tag="ghcr.io/eoap/quickwin/detect-water-body:${{needs.version.outputs.app-version}}"
        docker pull ${tag}
        shatag=$( docker inspect ${tag} | yq -r '.[0]["RepoDigests"][0]' )
        s="detect-water-body" t="${shatag}" yq -i eval '(.$graph[] | select (.id == env(s)) ).hints.DockerRequirement.dockerPull = env(t)' cwl-workflow/app-water-body-cloud-native.cwl

    - run: |
        r=$( cat codemeta.json | jq -r ".codeRepository" ) yq -i eval '."s:codeRepository" = {"URL" : env(r)}' cwl-workflow/app-water-body-cloud-native.cwl
        v="${{needs.version.outputs.app-version}}" yq -i eval '."s:softwareVersion" = env(v)' cwl-workflow/app-water-body-cloud-native.cwl
        n=$(cat codemeta.json | jq -r '(.author[0].givenName + " " + .author[0].familyName)') \
        e=$(cat codemeta.json | jq -r '.author[0].email') \
        a=$(cat codemeta.json | jq -r '.author[0].affiliation["name"]') \
        yq eval -i  '."s:author" += [{"class": "s:Person", "s.name": env(n), "s.email": env(e), "s.affiliation": env(a)}]' cwl-workflow/app-water-body-cloud-native.cwl
    - run: |
        mkdir downloads
        cp cwl-workflow/app-water-body-cloud-native.cwl downloads/app-water-body-cloud-native.${{needs.version.outputs.app-version}}.cwl
    - uses: actions/upload-artifact@v4
      with:
        name: application-package
        path: downloads
    - name: upload linux artifact 1
      uses: actions/upload-release-asset@v1
      env:
        GITHUB_TOKEN: ${{ github.token }}
      with:
        upload_url: ${{needs.create-release.outputs.upload_url}}
        asset_path: downloads/app-water-body-cloud-native.${{needs.version.outputs.app-version}}.cwl
        asset_name: app-water-body-cloud-native.${{needs.version.outputs.app-version}}.cwl
        asset_content_type: text/yaml

Artefacts

The released application package is published here: https://github.com/eoap/quickwin/releases/tag/1.0.0

The container is available at: https://github.com/eoap/quickwin/pkgs/container/quickwin%2Fdetect-water-body