Configuration

The Hopic build configuration is stored in a file in your repository. The default location where it will look for this is ${repo}/hopic-ci-config.yaml.

Pre-defined Hopic variables

WORKSPACE

Contains the absolute path of the git repository in which Hopic was called.

CFGDIR

Contains the absolute path of the directory in which the hopic-ci-config.yaml file is located (this can differ from WORKSPACE if Hopic was called with the --config option).

VERSION

Contains the current version of the workspace. See version for configuration and bump settings. The format is based on git describe and is formatted as:

TAGGED_VERSION[-COMMIT_DISTANCE][.dirty.TIMESTAMP]+gCOMMIT_ID

For example:

Version Meaning
1.8.1+g3269de8 tag 1.8.1, clean, unmodified, HEAD is at commit 3269de8
1.8.2-0.dirty.20210607131656+g3269de8 tag 1.8.1, with uncommited changes
1.8.2-1+gdaffe9c tag 1.8.1, with one commit on top of it (-1)
1.8.2-1.dirty.20210607131703+g4dbb6b4 tag 1.8.1, with one commit on top of it, as well as uncommitted changes

Note

When the current HEAD is dirty or does not have a version tag, the latest tag does not uniquely describe HEAD.

This is technically a “post-release” of the last tag. To implement this, Hopic will define the version as a pre-release for the next patch-version, using the commit distance to the latest tag as the pre-release identifier. This is the closest approximation of a “post-release” while still remaining SemVer-compliant in format as well as version precedence.

Since the ‘pre-release-behavior’ can be experienced as non-intuitive, an extra example is provided below.

Consider the following repository history:

ea47ce2 (HEAD -> master) test: add extra unit test for feature X
9f8f270 ci: add ci configuration
fe6b6fa (tag: 0.1.0) feat: add feature X
cd0aec6 (tag: 0.0.1) chore: initial commit

The latest tag for this repository is 0.1.0. However, since two non-bumping commits were done on top of that tag, the tag 0.1.0 no longer uniquely describes the code in the repository. The version for HEAD will therefore be a pre-release for the next patch commit, the pre-release field being defined as the distance to that last tag: 0.1.1-2.

When a fix-commit is submitted thereafter, the version will be bumped to 0.1.1. This version, not being a pre-release (that is, not containing a hyphen: -), has a higher precedence than any previously published pre-release version (0.1.1-N).

PURE_VERSION

Contains the VERSION (see above), but without build metadata (everything after the +).

For example:

Version Meaning
1.8.1 tag 1.8.1, clean, unmodified
1.8.2-0.dirty.20200609131656 tag 1.8.1, with uncommited changes
1.8.2-1 tag 1.8.1, with one commit on top of it
1.8.2-1.dirty.20200609152429 tag 1.8.1, with one commit on top of it, as well as uncommitted changes
DEBVERSION

Contains a Debian version-compliant representation of the VERSION as outlined above. Its specification can be found at: https://www.debian.org/doc/debian-policy/ch-controlfields.html#version

For example:

Version Meaning
1.8.1+g3269de8 tag 1.8.1, clean, unmodified, HEAD is at commit 3269de8
1.8.2~0+dirty20200609131656+g3269de8 tag 1.8.1, with uncommited changes
1.8.2~1+gdaffe9c tag 1.8.1, with one commit on top of it (~1)
1.8.2~1+dirty20200609152429+g4dbb6b4 tag 1.8.1, with one commit on top of it, as well as uncommitted changes
PUBLISH_VERSION

Based on the VERSION (see above) by default, with the following differences:

  • Where COMMIT_ID is part part of the prerelease (-COMMIT_ID) instead of build metadata (+gCommit_ID)
  • Where build is present in the Versioning configuration this is added, e.g. +build (only to PUBLISH_VERSION)

During a build that is publishing, this is based on PURE_VERSION. Currently the version build property only affects PUBLISH_VERSION. The build property value is appended to the build metadata.

Some package managers (e.g Maven) do not correctly implement SemVer 2.0.0. Those package managers fail to perform range checks with a ‘build metadata’ component, because they treat the build metadata as an alphanumeric part of the version instead of ignoring it.

GIT_COMMIT

GIT_COMMIT contains the full commit hash of the current commit, also known symbolically as HEAD.

GIT_COMMIT_TIME

GIT_COMMIT_TIME contains the committer time of GIT_COMMIT formatted according to RFC 3339.

GIT_BRANCH

Iff the current repository was checked out from a branch with checkout-source-tree then GIT_BRANCH will contain that branch’s name.

BUILD_NAME

BUILD_NAME will contain the same value as the same named environment variable, if it exists. If that environment variable doesn’t exist it will default to "unknown". On Jenkins that environment variable will contain the “job name”.

BUILD_NUMBER

BUILD_NUMBER will contain the same value as the same named environment variable, if it exists. If that environment variable doesn’t exist it will default to "NaN". On Jenkins that environment variable will contain the “build number”. For pull-requests that build number will be prefixed with “PR-<n>”, e.g. “PR-123 42”.

BUILD_URL

BUILD_URL will contain the same value as the same named environment variable, if it exists. If that environment variable doesn’t exist it will default to to the empty string. On Jenkins it will contain the URL for the build’s overview page.

During a build the following configuration variables are available.

BUILD_DURATION

BUILD_DURATION will contain the amount of seconds passed since GIT_COMMIT_TIME. Under the assumption that GIT_COMMIT_TIME is (close enough) to the start of the build this represents the time the build has taken so far.

HOPIC_PHASE

HOPIC_PHASE will contain the name of the current executing phase. This name is only available during phases specified within phases!

HOPIC_VARIANT

HOPIC_VARIANT will contain the name of the current executing variant. This name is only available during variants specified within phases!

Pre-defined environment variables

The VERSION, PURE_VERSION, DEBVERSION and PUBLISH_VERSION variables (documented above) are also available in the environment of any process spawned by Hopic.

Build Phases

phases

Hopic’s build flow is divided in phases, during which a set of commands can be executed for different variants. The phases option is a dictionary of dictionaries. It’s top-level key specifies the name of each phase. The keys within each phase specify the names of variants to be executed within that phase. Note that the variant name post-submit is reserved for internal use and is not permitted to be specified by users.

Phases are executed in the order in which they appear in the configuration. Within a phase each variant may be executed in parallel, possibly on different executors. Every next phase only starts executing when each variant within the previous phase finished successfully. I.e. the execution flow “forks” to each variant at the start of a phase and “joins” at the end.

A variant, identified by its name, may appear in multiple phases. Variants appearing in multiple phases are guaranteed to run on the same executor within each phase. This provides a stable environment (workspace) to work in and allows incremental steps, such as building in phase A and running built tests in phase B.

For example, the configuration example listed results in an execution flow as shown after that.

phases:
  style:
    commit-message-checker:
      # Require each commit message to adhere to our requirements
      - foreach: AUTOSQUASHED_COMMIT
        sh: .ci/verify-commit-message.py ${AUTOSQUASHED_COMMIT}
    clang-format:
      # Show all changes necessary to have each commit formatted properly
      - foreach: AUTOSQUASHED_COMMIT
        sh: GIT_PAGER="tee ${AUTOSQUASHED_COMMIT}.format.diff" git clang-format --style=file --diff ${AUTOSQUASHED_COMMIT}~1 ${AUTOSQUASHED_COMMIT}
      # Assert that no changes were necessary
      - foreach: AUTOSQUASHED_COMMIT
        sh: 'test ! -s ${AUTOSQUASHED_COMMIT}.format.diff'

  build:
    x64-release:
      - cmake -B build-x64-release -S . -DCMAKE_BUILD_TYPE=RelWithDebInfo
      - cmake --build build-x64-release
    x64-debug:
      - cmake -B build-x64-debug -S . -DCMAKE_BUILD_TYPE=Debug
      - cmake --build build-x64-debug
    arm64-release:
      - cmake -B build-arm64-release -S . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE:PATH=${CMAKE_AGL_AARCH64_TOOLCHAIN_FILE}
      - cmake --build build-arm64-release
    arm64-debug:
      - cmake -B build-arm64-debug -S . -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE:PATH=${CMAKE_AGL_AARCH64_TOOLCHAIN_FILE}
      - cmake --build build-arm64-debug

  test:
    x64-debug:
      - junit: build-x64-debug/test-unit.xml
      - cmake --build build-x64-debug --target test

  package:
    x64-release:
      - fingerprint:
          artifacts:
            - build-x64-release/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
      - cmake --build build-x64-release --target package
    x64-debug:
      - fingerprint:
          artifacts:
            - build-x64-debug/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
      - cmake --build build-x64-debug --target package
    arm64-release:
      - fingerprint:
          artifacts:
            - build-arm64-release/TomTom-Stacktrace-${VERSION}-Linux-aarch64.tar.gz
      - cmake --build build-arm64-release --target package
    arm64-debug:
      - fingerprint:
          artifacts:
            - build-arm64-debug/TomTom-Stacktrace-${VERSION}-Linux-aarch64.tar.gz
      - cmake --build build-arm64-debug --target package

  upload:
    x64-release:
      - run-on-change: only
      - build-x64-release/do-upload.sh
    x64-debug:
      - run-on-change: only
      - build-x64-debug/do-upload.sh
    arm64-release:
      - run-on-change: only
      - build-arm64-release/do-upload.sh
    arm64-debug:
      - run-on-change: only
      - build-arm64-debug/do-upload.sh
_images/parallel-phases.svg

Example execution flow of a Hopic build configuration.

Post Submission Phases

post-submit

Optionally, as part of the submission, Hopic provides the opportunity to execute another list of phases. These will be executed during the submit subcommand, but just after the actual submission to the revision control system has been performed. The post-submit option is a dictionary of phases. The keys specify the name of each phase.

The content of each phase has the same syntax, and is interpreted the same, as the content of a variant for phases. Only the following subset of options however, is permitted to be used within these phases:

The node-label option has an additional restriction. If specified multiple times, it is only allowed to contain the same value for each location it is specified. This is necessary to keep the complexity, and thus chance of failure, of the submit command low.

version:
  tag:    'v{version.major}.{version.minor}.{version.patch}'
  format: semver
  bump:
    policy: conventional-commits
    strict: true
    on-every-change: false

project-name: PIPE

phases:
  style:
    flake8:
      - tox -e flake8

  test:
    python3.8:
      - tox -r -e py38

post-submit:
  publish:
    - run-on-change: new-version-only
      with-credentials:
        id: pypi
        type: username-password
      sh: tox -e publish

Environment

environment

Sometimes it’s necessary to execute a command with altered environment variables. For that purpose it’s possible to use leading name=value words in the sh command. This requires shell escaping and because of that may not always be convenient enough to use. To address this command entries may instead use the environment option. Note that using the environment option disables processing of leading name=value words.

This option specifies a mapping of environment variables and the values to override them with for the current command only. The null value can be used to require removal of the specified environment variable. This is different from specifying the empty string which will still allow the environment variable to exist (but be empty).

phases:
  a:
    x:
      # leading name=value environment variable overrides
      - ENV_1=one ENV_2="two and three" ./command.sh --option=four
      # explicit environment mapping override
      - environment:
          ENV_1: one
          ENV_2: two and three
          SOURCE_DATE_EPOCH: null
        sh: ./command.sh --option=four

CI-locks

ci-locks

When within a Hopic file multiple git repo’s are changed and submitted it might be required to have a lock on every git repo. The current repo where Hopic is used is automatically protected with a lock. The :option: ci-locks option can be used to specify additional repositories whose locks should be acquired during the merge submit flow. This assumes that the other git repo’s are protected with a named lock while merging a change.

ci-locks requires a list of of branch and repo-name:

branch
Target git branch name that needs to be protected
repo-name
Target git repo-name that needs to be protected
lock-on-change (optional)
Specifies when additional lock is acquired. Can only be one of the following values:
  • always (default)
  • never
  • new-version-only (only acquire lock when version is bumped)
from-phase-onward (optional)
Acquire the lock only at the start of the specified phase By default it would be acquired at the start of the build instead.

The named lock that Hopic will acquire is formatted as repo-name/branch If all merge checks pass and Hopic is going to submit the merge, all specified addition locks will be acquired at the beginning of a build, alongside the repository’s own lock.

ci-locks:
  - branch: "master"
    repo-name: "PIPE/ci-lock"

phases:
  phase-1:
    variant-1:
      - sh -c "mkdir -p /tmp/ci-lock-dir && cd /tmp/ci-lock-dir && git init"
      - touch new_file.txt
      - sh -c "git add new_file.txt && git commit -m 'add new_file'"
      - rm -r /tmp/ci-lock-dir

Clean

clean

Hopic provides built-in clean check-out functionality by executing git clean -fxd on the ${WORKSPACE}. It is possible to add a list of commands that is executed before the git clean -fxd is executed. Within the list of commands $HOME and ~ will be expanded to the home directory of the current user.

clean:
  - rm -rf ~/.cache
  - rm -rf ${HOME}/.other_cache

Config

config

Some features in Hopic are using YAML’s explicit tags functionality to support custom defined YAML parsing behavior. Features that use this functionality can be recognized by <dict_key>: !<custom_function>. Since explicit tags need to be specified as a value of a dictonary, the config is introduced to specify explicit tags on the top level. config transparently adds a top level dictionary which allows making a global explicit tag.

pip:
  - with-extra-index:
      - https://test.pypi.org/simple
    packages:
      - hopic-templates-test[full-pipeline-template]>=0.2.3,<1
        
config: !template "full-pipeline-template"

Credentials

with-credentials

Sometimes it’s necessary to execute commands with privileged access. For that purpose the with-credentials configuration option can be used for a variant within a phase. You need to specify an identifier (id), used for looking up the credential and its type (type). In addition to that you can specify the name of the config and environment variable that should be set to contain them. with-credentials value can be specified as a list to have multiple credentials for a variant within a phase. Username/password credentials can be optionally encoded for use in a url by configuring encoding to (url).

The supported types of credential are:

Username/password credential

  • type: username-password
  • username-variable default: USERNAME
  • password-variable default: PASSWORD
  • encoding: default: plain

File credential

  • type: file
  • filename-variable default: SECRET_FILE

String credential

  • type: string
  • string-variable default: SECRET

SSH key credential

  • type: ssh-key
  • ssh-command-variable default: SSH

This provides, in the variable specified by ssh-command-variable, an executable that will behave like ssh. It will however be pre-authenticated with the credential’s SSH key and, if known, associated user.

phases:
  download:
    toolchain:
      - with-credentials:
          id: access_key
          type: string
      - download --api-key=${SECRET}

  build:
    toolchain:
      - with-credentials:
          id: license_file
          type: file
      - build --license-key=${SECRET_FILE}

  test:
    toolchain:
      - with-credentials:
          - id: squish_cred
            type: username-password
            username-variable: SQUISH_USERNAME
            password-variable: SQUISH_PASSWORD
          - id: artifactory_creds
            type: username-password
            username-variable: ARTIFACTORY_USERNAME
            password-variable: ARTIFACTORY_PASSWORD
      - download test-certificate --username=${ARTIFACTORY_USERNAME} --password=${ARTIFACTORY_PASSWORD}
      - test --username=${SQUISH_USERNAME} --password=${SQUISH_PASSWORD}

  deploy:
    toolchain:
      - run-on-change: only
        with-credentials:
          id: artifactory_creds
          type: username-password
      - publish --user=${USERNAME} --password=${PASSWORD}

Keyring Support

For local, interactive, use Hopic also supports credentials stored in your keyring using the keyring library. Note that keyring is only an optional dependency of Hopic and will only get installed when the interactive extra feature is selected.

If you didn’t install Hopic with the interactive feature you can either reinstall it or install the keyring library yourself. Note on macOS you will also need to install the netstruct library.

pip3 install keyring

This has been tested and confirmed to work with at least these keyring implementations:

  • gnome-keyring-daemon: GNOME’s builtin keyring
  • KeepassXC 2.6
project-name

If properly configured, Hopic will attempt to obtain all username-password credentials used during execution from your keyring. In order to do that, Hopic needs to know a project scope within which to look for your credentials. This scope can be configured with the project-name option. We suggest using your projects Jira keyword for this purpose as it’s the most likely to be unique enough.

project-name: JIRA_KEY

phases:
  deploy:
    toolchain:
      - run-on-change: only
        with-credentials:
          id: artifactory-creds
          type: username-password
      - publish --user=${USERNAME} --password=${PASSWORD}

This example will cause Hopic to look for a credential with a value for the service field of JIRA_KEY-artifactory-creds. When found, it will use its username and password fields for the publish command.

When Hopic doesn’t find the credential in the keyring, its behavior depends on whether it’s running in an interactive terminal or not.

When Hopic runs in an interactive terminal, it will prompt the user for the username and password. It will store these in your keyring and continue execution with those values.

When Hopic doesn’t run in an interactive terminal, like on your CI system, it will attempt to obtain the credentials from your CI system’s credential store. The project-name will not be taken into account when looking in the CI system’s credential store. That option applies to keyring lookups only. If it cannot find them there, it will fail with an error message indicating it couldn’t find the specific credential.

Container Image

image

In order to execute commands within a Docker container Hopic needs to be told what image to use for creating a container. This option can either contain a string in which case every variant will execute in a container constructed from that image. Alternatively it can contain a mapping where the keys and values are the names of the variant and the image to execute those in respectively. If using the mapping form, the default key will be used for variants that don’t have an image specified explicitly. The image can also be specified within a phase for a variant, this will override the global specified image for the specific variant for only the phase where it is specified.

An example of the mapping style where two different variants are executed in containers based on different images:

image:
  default: buildpack-deps:testing
  whoami:  buildpack-deps:testing-curl

phases:
  prepare:
    hello:
      - cc -o hello hello.c
    whoami:
      - curl --output ip-address.txt https://api.ipify.org

  run:
    hello:
      - image: busybox:latest
      - ./hello

For the purpose of using an image (name and version) specified in an Ivy dependency manifest file the !image-from-ivy-manifest type constructor exists. When used its contents are a mapping with these keys:

manifest

Path to the Ivy manifest file. Relative paths are interpreted relative to the first of the ${CFGDIR} or ${WORKSPACE} directories that exists. This defaults to the first of these to exist:

  • ${WORKSPACE}/dependency_manifest.xml
  • ${CFGDIR}/dependency_manifest.xml
repository
Docker repository to fetch the image from.
path
Directory within the repository to fetch from.
name
Name of the image to fetch. This defaults to the content of the name attribute in the Ivy manifest.
rev
Version of the image to fetch. This defaults to the content of the rev attribute in the ivy manifest.

When used this will get treated as if the expansion of {repository}/{path}/{name}:{rev} was specified as a string value of this field. This allows using Ivy as a mechanism for automatically keeping the Docker image up to date.

For example, when using this dependency manifest in ${WORKSPACE}/dependency_manifest.xml:

<ivy-module version="2.0">
  <info module="p1cms" organisation="com.tomtom" revision="dont-care" />
  <dependencies>
    <dependency name="python" org="com.tomtom.toolchains" rev="3.6.5" revConstraint="[3.5,4.0[">
      <!-- identify this as the dependency specifying the Docker image for Hopic -->
      <conf mapped="toolchain" name="default" />
    </dependency>
  </dependencies>
</ivy-module>

And this Hopic config file:

image:
  default: !image-from-ivy-manifest
    repository: hub.docker.com
    path: tomtom
  Linux_AGL-aarch64: !image-from-ivy-manifest
    repository: hub.docker.com
    path: tomtom
    name: agl-build-aarch64-toolchain

The result will be to use the hub.docker.com/tomtom/python:3.6.5 image by default. The PyPy build will instead use the hub.docker.com/tomtom/pypy:3.6.5 image. I.e. for that build the image name is overridden from that used in the Ivy manifest, while still using the version from it.

Extra Docker arguments

extra-docker-args

There is a limited subset of docker run arguments that can be specified for a variant:

add-host
This options accepts one value or a list of values. Translates into docker run argument --add-host=<value>, allowing the user to add a custom DNS entry to the container. If a list is provided, the option shall be expanded to multiple --add-host arguments, e.g. --add-host=<list-item-1> --add-host=<list-item-2>.
device
This options accepts one value or a list of values. Translates into docker run argument --device=<value>, allowing the user to forward a device from the host machine to the container. If a list is provided, the option shall be expanded to multiple --device arguments.
dns
Translates into docker run argument --dns=<value>, allowing the user to specify custom DNS servers to be used in the container.
entrypoint
Translates into docker run argument --entrypoint=<value>, allowing the user to override the entrypoint as defined in the Docker image.
hostname
Translates into docker run argument --hostname=<value>, allowing the user to specify the hostname that the container shall use.
init
This option only accepts boolean value True. When provided, this translates into docker run argument --init, which adds an init daemon to the container.
image:
  default: buildpack-deps:testing

phases:
  docker-phase:
    variant-with-extra-docker-args:
      - extra-docker-args:
          hostname: buildhost
          init: yes
          device:
            - /dev/ttyS0
            - /dev/kvm
          add-host: my-test-host:10.1.2.3
          dns: 9.9.9.9
      - sh -c 'test "$$HOSTNAME" = "buildhost"'

Docker in Docker

docker-in-docker

Sometimes it may be necessary to perform operations on the Docker daemon from within a Docker container. E.g. trying to use some tool that resides in a different Docker container from within a build running in a container. If that happens this option can be used to make the required Docker socket available within the first container.

image:
  default: buildpack-deps:testing

phases:
  prepare:
    whoami:
      - image: buildpack-deps:testing-curl
        docker-in-docker: true
        volumes:
          - /usr/bin/docker:/usr/bin/docker:ro
      - docker pull buildpack-deps:testing-curl

Volumes

volumes

In order to execute commands within Docker it is often required to mount directories or a file to the docker container. This can be done by specifying volumes. volumes doesn’t have any effect when there is no image specified.

There are two formats how a volume can be specified:

Format 1

The volume can be specified as host-src[:container-dest][:<options>]. The options are [rw|ro] The host-src is an absolute path or a name value.

Format 2

The volume can be specified using a dictionary with the following keys:
  • source
  • [target]
  • [read-only]

Where source is equal to host-src, target is equal to container-dest and read-only reflects the possible options with a boolean value.

By default the host-src is mounted rw.

When the given host-src doesn’t exist Hopic will create the path as a directory to prevent dockerd from creating the directory as root user. If container-dest is not specified, it will take the same value as host-src. For the host-src path, $HOME or ~ will be expanded to the home directory of the current user. While for the container-dest, $HOME or ~ will be expanded to /home/sandbox.

The following directories and files are mounted by default:

host-src container-dest <options>
WORKSPACE [*] /code read-write
[*]WORKSPACE/code for repositories referring to other repositories for their code.

These defaults can be disabled by specifying a null source for them:

volumes:
  - source: null
    target: /code

volumes can be declared in every scope and will be used during the specified scopes e.g. volumes specified in global scope are used with every command. In case an inherited bind mount needs to be overridden, that can be accomplished by adding a volume with the same target_location. Consider the following example where /tmp/downloads is overridden:

example:

image: buildpack-deps:testing-curl

volumes:
  - source: ~/
    target: /tools/devenv
    read-only: true
  - /tmp
  - /tmp/Music:/home/sandbox/temp

phases:
  list:
    jira:
      - with-credentials:
          id: netrc
          type: file
        volumes:
          - ${SECRET_FILE}:~/.netrc:ro
      - curl -v --netrc --silent "https://jira.atlassian.com/rest/api/2/search?maxResults=100&fields=key,summary,assignee&jql=resolution+%3D+unresolved+AND+%28project+%3D+BSERV+OR+assignee+%3D+currentUser()%29+ORDER+BY+priority+DESC%2C+created+ASC"
    print:
      - echo "only global volumes will be mounted"
  mounts:
    print:
      - sh: echo "/tmp/Music will be overridden with /tmp/Downloads"
        volumes:
          - /tmp/Downloads:/home/sandbox/temp
      - volumes: 
        - /tmp/Pictures:/home/sandbox/temp
        - /tmp/Desktop
      - echo "override /tmp/Downloads with /tmp/Pictures and /tmp/Desktop is added"

Environment Variables

pass-through-environment-vars

This option allows passing environment variables of the host environment through into containers. This is a list of strings. Each string is the name of an environment variable. If the named environment variable exists in the host environment, it will be set to the same value inside the container.

image: build-image:4.2.1

pass-through-environment-vars:
  - JENKINS_TEST_INSTANCE

phases:
  build:
    x86:
      - ./build.sh

  upload:
    x86:
      - ./upload-from-non-test-only.sh

Mounting Volumes From Other Containers

volumes-from

The option volumes-from allows you to mount volumes that are defined in an external Docker image. The behavior translates directly to a --volumes-from Docker-run option; the volumes are mapped to the path as originally specified in the external image.

Note that this option does nothing if you haven’t specified a Docker image (see the image option).

The option requires two keys to be specified:

image-name
The full name of the Docker image.
image-version
The targeted version of the Docker image.

The combination of <image-name>:<image-version> should result in a correct, downloadable Docker image.

example:

phases:
  coverity:
    x64:
      - description: Coverity run
      - volumes-from:
        - image-name: hub.docker.com/tomtom/coverity
          image-version: 2.4.7
      - sh -c 'cd build/x64-rel;
          /opt/coverity/bin/cov-configure --template --compiler c++ --comptype gcc --version 7.3 --config my_config.conf;
          /opt/coverity/bin/cov-build --config my_config.conf --dir cov-int ninja;
          echo etc.'

Publish From Branch

publish-from-branch

The publish-from-branch option, when provided, specifies a regular expression matching the names of branches from which to allow publication. Publication includes version bumping (see version) and the execution of any steps marked with run-on-change as only.

If this option is omitted, Hopic allows publication from any branch.

The example below configures Hopic to only publish from the master branch or any branch starting with release/ or rel-.

example:

publish-from-branch: '^master$|^release/.*|^rel-.*'

Versioning

version

Hopic provides some support for determining and bumping of the currently checked out version.

It currently supports the syntax, sorting and bumping strategies of these versioning policies. The policy to use can be specified in the format option of the version option section.

semver

Semantic Versioning. This is the default when no policy is explicitly specified.

The default tag format is {version.major}.{version.minor}.{version.patch}.

The default component to bump is the pre-release label.

carver

Caruso variation on Semantic Version for branching

The default tag format is {version.major}{version.minor}{version.patch}+PI{version.increment}.{version.fix}.

The default component to bump is the pre-release label.

The version can be read from and stored in two locations:

version.file
When this option is specified Hopic will always use this as the primary source for reading and storing the version. The first line to contain only a syntactically valid version, optionally prefixed with version=, is assumed to be the version. When reading the version it’ll use this verbatim. When storing a (likely bumped) version it’ll only modify the version portion of that file.
version.tag

When this option is set to true or a non-empty string Hopic will, when storing, create a tag every time it creates a new version. When this option is set to a string it will be interpreted according to Python Format Specification with the named variable version containing the version. When this option is set and file is not set it will use git describe to read the current version from tags. When used for reading, it will mark commits that don’t have a tag a virtual prerelease of the predicted next version.

Setting this option to a string can, for example, be used to add a prefix like v to tags, e.g. by using v{version}. Having it set to true instead uses the version policy’s default formatting.

version.build
When using semver it is possible to define custom build metadata (see https://semver.org/#spec-item-10), by setting the build property. When this option is set, the build value will be appended to the tag according to the semver build metadata spec (+<version.build>). Currently, this setting will only affect the PUBLISH_VERSION as described in the Pre-defined Hopic variables chapter.

When and what to bump can be controlled by the version.bump option. When set to false it disables automated bumping completely. Otherwise it describes a version bumping policy in its version.bump.policy member.

There are currently two version bumping policies available:

constant
Its version.bump.field property specifies what field to bump. It will always bump that field and no other. When not specified it defaults to bumping the default-to-bump part of the used version policy.
version:
  format: semver
  tag: 'v.{version.major}.{version.minor}.{version.patch}'
  bump:
    policy: constant
    field: patch
conventional-commits

When used this policy determines the version field to bump based on commit messages formatted according to Conventional Commits. It searches all to-merge commits for breaking changes, new features and fixes. If any of those are present it will bump, in order of precedence, the major, minor or patch number.

The version.bump.strict option of this policy controls whether each commit message is required to parse as a valid Conventional Commit. If set to false invalidly formatted messages are just ignored and not taken into account to determine what to bump. Otherwise the merge will fail when encountering invalidly formatted messages. Additionally, if version.bump.strict is set to true, the merge-change-request title version bump is validated against the corresponding commit messages version bump and the merge will fail if the version bump mismatches.

The version.bump.reject-breaking-changes-on and version.bump.reject-new-features-on options specify regular expressions matching branch names. Merges into these branches will be rejected if, respectively, they contain breaking changes or new features. The purpose of this is to prevent accidental inclusion of these kinds of changes into release branches.

version:
  format: semver
  tag: 'v.{version.major}.{version.minor}.{version.patch}'
  bump:
    policy: conventional-commits
    strict: false # default
    reject-breaking-changes-on: 'release/.*|rel-.*' # default
    reject-new-features-on: 'release/\d+\..*|rel-\d+\..*' # default

In order to configure a version bumping policy without automatically bumping for every change the version.bump.on-every-change option can be set to false (defaults to true).

When bumping is enabled, Hopic bumps each time that it applies a change. Usually this means when it’s merging a pull request. Another option is when it’s performing a modality change (currently only UPDATE_DEPENDENCY_MANIFEST).

hotfix-branch

Hotfix versioning is supported and performed on hotfix branches only. If and only if a branch’s name matches the regular expression configured in this option will hotfix versioning be performed. The regular expression in this option MUST either contain an id or ID named capture group or have exactly one capture group. This capture group will be used as the hotfix ID.

This defaults to matching branch names like hotfix/x.y.z-{hotfix-id}.

hotfix-allowed-start-tags
This contains a list of commit tags to allow after the version tag that the hotfix-branch has been split off from. This will only get used when using the Conventional Commits bumping policy. This defaults to an empty set.

Todo

Describe after-submit. Maybe?

Modality Changes

modality-source-preparation

The modality-source-preparation option allows for influencing the build according to the MODALITY parameter. If Hopic is called with a MODALITY that is present in the configuration file, then the commands as specified in that section are executed before the other phases.

See the description of the apply-modality-change parameter on the Usage page for the calling syntax.

Note that this is, above all, a remnant of the previous generation pipeline; it is currently only used to perform UPDATE_DEPENDENCY_MANIFEST builds.

Note

Defining new functionality using this option is discouraged.

description
An optional description for the command, which will be printed in the logs.
sh
The actual command to be run. Variables will be expanded, similar to commands defined in the phases.
changed-files

Specifies the files that are changed by the command, which are to be added to the commit.

If omitted, Hopic forces a clean repository before running the command specified by sh. Upon completion of the command, all files that are changed, removed and/or previously untracked are added to the commit.

commit-message

The message that will be used to commit the changes when this modality is run.

This option is mutually exclusive with commit-message-cmd.

If omitted, the value of the MODALITY parameter is used as the commit message.

commit-message-cmd

A command, or a mapping with an sh option, of which stdout is used as the commit message of this modality.

This option is mutually exclusive with commit-message.

If omitted, the value of the MODALITY parameter is used as the commit message.

example:

modality-source-preparation:
  UPDATE_DEPENDENCY_MANIFEST:
    - sh: update_dependency_manifest.py ${CFGDIR}/dependency_manifest.xml ${CFGDIR}/ivysettings.xml
      changed-files:
        - ${CFGDIR}/dependency_manifest.xml
      commit-message: Update of dependency manifest

Extension Installation

pip

Hopic can be extended with extra Python packages. For this purpose it provides the ability to install packages, with pip, before building.

The pip option contains a list of requirement strings or package installation specifications. Each of those specifications may contain these options:

packages

This option is mandatory.

It contains a list of pip requirement strings. E.g. click>=7,<8 or python-keyring.

from-index

This must be a single URL string.

When specified this causes pip to install from this package index instead of the default one.

with-extra-index

This must be a single URL string or a list of URL strings.

When specified this causes pip to look in this index when the primary one doesn’t contain the specified packages.

pip:
  - with-extra-index:
      - https://test.pypi.org/simple/
    packages:
      - commisery>=0.2,<1

phases:
  style:
    commit-messages: !template "commisery"

Restricting Variants to Specific Build Nodes

node-label

The option node-label executes the specified steps on an agent available in the Jenkins environment with the provided label(s).

example:

phases:
  build:
    Linux-x86_64-Release:
      - node-label: Linux && Docker && Release
      - Build/build.py -f Build/Linux-x86_64.yaml -m Release

Restricting Steps to Changes or Not

run-on-change
class hopic.config_reader.RunOnChange

The run-on-change option allows you to specify when a step needs to be executed. The value of this option can be one of:

always = 'always'

The steps will always be performed. (Default if not specified).

never = 'never'

The steps will never be performed for a change.

new_version_only = 'new-version-only'

The steps will only be performed when the change is on a new version and is to be submitted in the current execution.

only = 'only'

The steps will only be performed when the change is to be submitted in the current execution.

example:

phases:
  build:
    x64-release:
      - cmake --build build-x64-release
    x64-debug:
      - cmake --build build-x64-debug

  upload:
    x64-release:
      - run-on-change: only
      - build-x64-release/do-upload.sh
    x64-debug:
      - run-on-change: only
      - build-x64-debug/do-upload.sh

Limiting Execution Time for Steps

timeout

The option timeout allows specifying an upper limit to how long a step is allowed to execute. When this time gets exceeded the process will get aborted and the build will stop with an error.

When this option is specified combined with a command the timeout applies to that command’s execution only. Otherwise this option has to be specified separately and before any command, in which case it applies to every command in that variant.

The current implementation only accepts timeouts expressed as positive real numbers interpreted as seconds. In the future other units may be added as a suffix. Without an explicit unit present or some other notation the interpretation will always be as seconds.

example timeout combined with command:

phases:
  build:
    x64-release:
      - timeout: 180
        sh: cmake --build build-x64-release
    x64-debug:
      - timeout: 300
        sh: cmake --build build-x64-debug

  upload:
    x64-release:
      - run-on-change: only
        timeout: 30.5
        sh: build-x64-release/do-upload.sh
    x64-debug:
      - run-on-change: only
        timeout: 45.75
        sh: build-x64-debug/do-upload.sh

example timeout per variant:

phases:
  build:
    x64-release:
      - timeout: 180
      - timeout: 15
        sh: ./configure
      - make
      - make test

post-submit:
  upload:
    - timeout: 30
    - make upload

Sharing Output Data Between Variants

stash

The option stash will save files to be used in another Hopic phase. The stashed files are available for every executor/node in every phase and workspace after the current one.

Note

This option is not allowed to be used within a post-submit phase.

Use the option includes to identify the files to be stashed. Use Wildcards like module/dist/**/*.zip. A * expands only to a single directory entry, where ** expands to multiple directory levels deep.

Use the option dir to change the working directory that is being used while stashing.

example:

phases:
  builld:
    stash:
      - stash:
          includes: stash/stash.txt
      - mkdir -p stash
      - sh -c 'echo stashed_file > stash/stash.txt'
      - ls -l
      - ls -l stash

    stash-dir:
    - stash:
        includes: stashed_dir.txt
        dir: stash/stashed_dir
    - mkdir -p stash/stashed_dir
    - sh -c 'echo stashed_dir > stash/stashed_dir/stashed_dir.txt' 

  upload:
    stash:
      - ls -l $WORKSPACE/stash
      - ls -l $WORKSPACE/stashed_dir

Customizing Step Description

description

The option description adds a description to the step which will be printed in the logs.

example:

phases:
  quality:
    Coverity:
      - description: Coverity run
      - volumes-from:
        - image-name: hub.docker.com/tomtom/coverity
          image-version: 2.4.6
      - echo -e "Run Coverity here, eg.:\n  cov-configure --template --compiler gcc --comptype gcc\n  cov-configure -co /tools/devenv/Linux/Linux/gcc-4.8.2/bin/c++ -- -msse -mfpmath=sse\netc.."

Branches in Subdirectory Worktrees

worktrees

Note

This option is not allowed to be used within a post-submit phase.

Todo

Document worktrees option.

Repeating Steps for Commits

foreach

Todo

Document foreach option.

  • SOURCE_COMMIT
  • SOURCE_COMMITS
  • AUTOSQUASHED_COMMIT
  • AUTOSQUASHED_COMMITS

Sub SCM

scm

Todo

Document scm option.

JUnit Test Results

junit

The option junit triggers Jenkins Unit Testing. For more information about this feature see: https://www.tutorialspoint.com/jenkins/jenkins_unit_testing.htm

example:

phases:
  build:
    x64-release:
      - cmake --build build-x64-release

  test:
    x64-debug:
      - junit: build-x64-debug/test-unit.xml
      - cmake --build build-x64-debug --target test

JUnit allow-missing

allow-missing

Normally, a build fails if none of the specified JUnit XML files are found. Setting this option’s value to true allows the build to continue even if no JUnit XML files were found. When this option is specified, JUnit configuration must contain also a test-results option which specifies which XML files to be uploaded.

example:

phases:
  build:
    example:
      - junit:
          test-results: 
            doesnotexistjunitresult.xml
          allow-missing: true

JUnit allow-failures

allow-failures

Normally, a build fails if junit XML report file contains failed tests. Setting this option’s value to true allows the build to continue even if junit XML report file contains failed tests. The variant will still be marked as “failed”, but the pipeline will continue.

Note: This option cannot suppress failures in test execution. If an executable that produces the junit report fails, the pipeline will still fail. This option is useful when junit represents static analysis report: a tool succeeds, but produces a report which can be uploaded.

example:

phases:
  build:
    example:
      - junit:
          test-results: 
            doesnotexistjunitresult.xml
          allow-failures: true

Artifact Archiving

archive

The option archive allows you to archive build artifacts. The artifacts can be stored on Jenkins and/or archived to Artifactory.

Note

This option is not allowed to be used within a post-submit phase.

The base directory is the workspace. Artifacts specified are discovered relative to the workspace.

Use Wildcards like module/dist/**/*.zip. A * expands only to a single directory entry, where ** expands to multiple directory levels deep.

Use the pattern option to identify and upload a specific artifact. The specific artifact can then be uploaded to artifactory with the option target.

example:

phases:
  build:
    x64-release:
      - cmake -B build-x64-release -S . -DCMAKE_BUILD_TYPE=RelWithDebInfo

  upload:
    x64-release:
      - cmake --build build-x64-release --target package
      - archive:
          artifacts:
            - Build/Output/x64/release/**
            - pattern: Build/Output/x64/release/TomTom-Package-${VERSION}-x64.tar.gz
              target: cs-snapshot/com.tomtom.package/Package/x64/release/${VERSION}/TomTom-Package-x64-release-${VERSION}.tar.gz
        run-on-change: only
upload-artifactory

The option upload-artifactory allows you to archive build artifacts to Artifactory. The option id is the named identifier referring to a preconfigured Artifactory server from Jenkins’ global configuration. The artifacts to be uploaded are specified with the artifacts option.

example:

phases:
  upload:
    Linux-x86_64:
      - archive:
          artifacts:
            - pattern: build-x86/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
              target: cs-psa-p1cms-snapshot/com.tomtom.stacktrace/Stacktrace/linux/x86_64/release/${VERSION}/Stacktrace-linux-x86_64-release-custom-${VERSION}.tar.gz
            - pattern: build-x86/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
              target: cs-psa-p1cms-snapshot/com.tomtom.stacktrace/Stacktrace/linux/x86_64/debug/${VERSION}/Stacktrace-linux-x86_64-release-custom-${VERSION}.tar.gz
          upload-artifactory:
            id: artifactory-navkit
        run-on-change: only
artifactory

The option artifactory lets the user create promotion definitions for specific Artifactory repositories. The option promotion is used to promote build artifacts. After this option the Artifactory server ID is specified. The option target-repo specifies the Artifactory target repository.

Only build artifacts that were uploaded with the option upload-artifactory will be promoted.

example:

artifactory:
  promotion:
    artifactory-navkit:
      target-repo: cs-psa-p1cms-release

phases:
  build:
    Linux-x86_64:
      - cmake --build build-x86
    Linux_AGL-aarch64:
      - cmake --build build-agl-aarch64

  package:
    Linux-x86_64:
      - fingerprint:
          artifacts:
            - build-x86/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
      - cmake --build build-x86 --target package

  upload:
    Linux-x86_64:
      - archive:
          artifacts:
            - pattern: build-x86/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
              target: cs-psa-p1cms-snapshot/com.tomtom.stacktrace/Stacktrace/linux/x86_64/release/${VERSION}/Stacktrace-linux-x86_64-release-custom-${VERSION}.tar.gz
          upload-artifactory:
            id: artifactory-navkit
        run-on-change: only

Artifact Fingerprint

fingerprint

The option fingerprint triggers fingerprinting of build artifacts on Jenkins. Use the option artifacts to identify the artifacts to be fingerprinted.

Note

This option is not allowed to be used within a post-submit phase.

With the fingerprint option, the Jenkins “fingerprint” feature is invoked. For more information about this feature see: https://jenkins.io/doc/pipeline/steps/core/#fingerprint-record-fingerprints-of-files-to-track-usage

example:

phases:
  build:
    x64-release:
      - cmake --build build-x64-release

  package:
    x64-release:
      - fingerprint:
          artifacts:
            - build-x64-release/TomTom-Stacktrace-${VERSION}-Linux-x86_64.tar.gz
      - cmake --build build-x64-release --target package

Artifact upload-on-fail

upload-on-failure

The option upload-on-failure allows artifacts to be uploaded regardless of the build status. By default option is disabled (set to false), which means that artifacts are not published when build is not marked as success. Set this value to true to enable this option.

example:

version:
  bump: false

phases:
  build:
    example:
      - archive:
          artifacts:
            produce_output.txt
          upload-on-failure: true
      - touch produce_output.txt
      - "false"

Artifact allow-empty-archive

Note

allow-empty-archive is a deprecated name for the allow-missing option.

Avoid its use as it will be removed in the next release. Enabling both allow-empty-archive and allow-missing will result in a configuration error.

Artifact allow-missing

allow-missing

Normally, a build fails if archiving returns zero artifacts. This option allows the archiving process to return nothing without failing the build. Instead, the archive step will simply throw a warning. Set this value to true to enable this option.

example:

phases:
  build:
    example:
      - archive:
          artifacts:
            doesnotexist.txt
          allow-missing: true

Embed scripts

!embed

The option !embed allows Hopic to embed a part of the configuration file from an external source. This can be via a command line script (sed, cat, etc), or by evaluating a script. The !embed option requires to have a cmd argument which is the command to execute. The command should output the part of the configuration file to standard output.

The location of a script should be specified either from the workspace directory or the configuration file directory.

example:

phases:
  build: 
    x86_64:
      - echo "building..."

  generate: !embed
    cmd: yaml-generation.py variant

With yaml-generation.py:

#!/usr/bin/env python3

import sys
from textwrap import dedent


def main(argv):
    print(dedent(f"""\
    test-{argv[0]}:
      - echo generated test variant
    """), end='')


if __name__ == "__main__":
    main(sys.argv[1:])

Finally

finally

The option finally can be used within a variant to specify actions that should be executed regardless of an earlier failure. Intended actions for finally could be metrics reporting or resource cleanup actions.

Finally is syntactically the same as a variant, however the following keywords are forbidden:

When there is no sh block next to the finally block, the finally block is considered global for the variant. It is not allowed to specify a sh command after the global finally is defined. The global finally is always executed last within the variant.

If the finally is specified on the same level as an sh command, the finally block is only executed when the corresponding sh block was executed, but not if an early command failed.

timeout is supported, but only when attached to a command.

phases:
  a:
    x:
      - sh: echo "hello world"
        finally: 
          - echo "hello world finally"
      - sh: invalid_cmd
        finally: 
          - echo "invalid cmd finally"
      - sh: echo "never reach this point"
        finally:
          - echo "never execute finally"
      - finally:
          - echo "last finally"