Building a CI Pipeline with CTest and CDash

We can use CMake, CTest and CDash to build a basic Continuous Integration (CI) pipeline. The pipeline builds the applications, runs the tests, collects coverage information and uploads all the results to a dashboard on a web server. All this is controlled by one CMake script executed by CTest.

This pipeline could be triggered by a CI system like Github Actions or Jenkins, when developers try to integrate their code changes into main (e.g., by push or by merge request). Although the CTest pipeline could update the git repository (see the CTest Update Step), we let the CI system do this. If all tests pass, the CI system automatically integrates the changes into the main branch.

The CI system could run more checks before integration. For example, it could check that all code changes are covered by tests or that no cyclic dependencies between libraries exist. Integrating the CTest pipeline into a CI system will be the topic of a later post.

Installing the CDash Server

CDash is web server. When we run the pipeline, CTest uploads the build, test and coverage results to CDash. We can access the dashboard of the results from our browser.

We can install CDash on our own server by spinning up a Docker container.

$ git clone https://github.com/Kitware/CDash
$ cd CDash

We clone the CDash repository, change to the CDash directory and edit the environment section of the file docker-compose.yml. Of course, we adapt the secrets and email addresses to our needs.

    environment:
      CDASH_ROOT_ADMIN_PASS: secret

      CDASH_STATIC_USERS: |
        USER jdoe@acme.com jdoe_secret

        ADMIN admin@example.org admin_secret

We start the Docker container with the CDash server in the background. The first run takes a couple of minutes, because Docker builds the CDash server. Subsequent runs start the CDash server right away without building.

$ docker-compose up -d

After starting CDash for the first time, we configure it with the values from docker-compose.yml.

$ docker-compose run --rm cdash install configure

We don’t need this command for subsequent starts of CDash.

We stop the CDash server with the command

$ docker-compose down

Creating a Project on the CDash Server

We enter localhost:8080 in the address field of our browser to access the CDash dashboard. We click on Login in the upper left corner of the page. We enter the email address and password of the USER from docker-compose.yml above to log in.

We click on My CDash in the upper left corner of the page to reach our dashboard. The dashboard shows sections My Projects, Authentication Tokens and Administration. We choose the action Administration > Create new project. As a minimum we fill out the project Name and the Home URL. For example:

Name:      TerminalApps
Home URL:  http://localhost:8080

We click on the blue right arrow multiple times to reach the Miscellaneous page. There we click the Create Project button. The CDash client, CTest, needs the name and the home URL to upload the results of a pipeline run to CDash.

Running a CI Pipeline with CTest

We use a CTest script, TerminalAppsPipeline.cmake, to describe a CI pipeline. I use the example from Chapter 25.9.3 CTest Configuration of Craig Scott’s book Professional CMake: A Practical Guide (11th edition) as the starting point for my script. The script is located in the top-level directory of the terminal-apps repository.

/path/to/               # working directory
    terminal-apps/      # source directory
        TerminalAppsPipeline.cmake
        CTestConfig.cmake
    build-test/         # binary directory

In the working directory, we run the pipeline with the command

$ ctest -S terminal-apps/TerminalAppsPipeline.cmake

The command reads the source files from directory terminal-apps and writes all the build artifacts to build-test. It deletes the contents of build-test from the last run, configures the build with CMake, builds the applications and the unit tests with Ninja, runs the unit tests, generates the coverage information and uploads the results to CDash. It runs the complete pipeline as specified in TerminalAppsPipeline.cmake.

# Test configuration
include(${CTEST_SCRIPT_DIRECTORY}/CTestConfig.cmake)

# General settings
site_name(CTEST_SITE)
set(CTEST_BUILD_NAME "linux-x86_64-gcc")
set(CTEST_SOURCE_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}")
set(CTEST_BINARY_DIRECTORY "${CTEST_SCRIPT_DIRECTORY}/../build-test")

# Build settings
set(CTEST_CMAKE_GENERATOR Ninja)
set(CTEST_CONFIGURATION_TYPE RelWithDebInfo)
set(configureOpts
    "-DCMAKE_CXX_FLAGS_INIT=--coverage"
    "-DCMAKE_PREFIX_PATH=/public/Qt/host-qt6.2.4"
    )

# Coverage settings
set(CTEST_COVERAGE_COMMAND gcov)
set(CTEST_COVERAGE_EXTRA_FLAGS "--demangled-names")
set(CTEST_CUSTOM_COVERAGE_EXCLUDE "_autogen;machine/tests")

# Step 1 - Clean previous pipeline run
ctest_empty_binary_directory(${CTEST_BINARY_DIRECTORY})

# Step 2 - Configure
ctest_start(Experimental)
ctest_configure(OPTIONS "${configureOpts}")
ctest_submit(PARTS Start Configure)

# Step 3 - Build
ctest_build(PARALLEL_LEVEL 16)
ctest_submit(PARTS Build)

# Step 4 - Run unit tests
ctest_test(PARALLEL_LEVEL 16)
ctest_submit(PARTS Test)

# Step 5 - Collect coverage information
ctest_coverage()
ctest_submit(PARTS Coverage Done)

CTEST_SCRIPT_DIRECTORY points to the working directory, which contains the pipeline script. The Test Configuration tells the pipeline script (the CDash client) how to communicate with the CDash server. The file CTestConfig.cmake in the working directory contains the test configuration.

set(CTEST_PROJECT_NAME "TerminalApps")
set(CTEST_SUBMIT_URL 
    "http://localhost:8080/submit.php?project=${CTEST_PROJECT_NAME}")
set(CTEST_USE_LAUNCHERS YES)

We must set CTEST_PROJECT_NAME to the same name that we used when we created the project on the CDash server. It’s TerminalApps in our case. CTEST_SUBMIT_URL is the URL to which CTest uploads the results of the pipeline run. The flag CTEST_USE_LAUNCHERS writes the command for each pipeline step into the CDash logs, which is useful for debugging.

Let us go back to the pipeline script. The General Settings describe the computer, on which the pipeline runs, by the site name (the host name by default) and the build name linux-x86_64-gcc. It also sets CTEST_SOURCE_DIRECTORY and CTEST_BINARY_DIRECTORY to /path/to/terminal-apps and /path/to/build-test, as described at the beginning of this section. The source and binary directories are used by the pipeline steps.

The Build Settings determine Ninja as the CMake generator. Setting CTEST_CONFIGURATION_TYPE to RelWithDebInfo and CMAKE_CXX_FLAGS_INIT to --coverage ensures that the compiler instruments the binaries for coverage collection in Step 5. We tell the Configure step where to find Qt by setting CMAKE_PREFIX_PATH to the Qt installation directory (e.g., /public/Qt/host-qt6.2.4).

In the Coverage Settings, we define that gcov is our tool for collecting coverage information and that we want human-readable symbol names (option --demangle-names). The option CTEST_CUSTOM_COVERAGE_EXCLUDE takes a semicolon-separated list of regular expressions excluding file and directory paths from the coverage result. The regular expression _autogen excludes files generated by moc and rcc. The build places these files in directories containing the substring _autogen. The regular expression machine/tests excludes the directories containing the test cases. This leaves us with the coverage information for the source files in machine/src.

The remaining steps specify the steps of the CI pipeline.

Step 1 deletes all the files and directories in the binary directory from the previous pipeline run.
Step 2 tells the CDash server that a new Experimental pipeline run starts (ctest_start), configures the build by running CMake (ctest_configure), and sends the results of the Start and Configure steps to CDash (ctest_submit). As we define a custom pipeline, we pass Experimental as the model to ctest_start (see Dashboard Client Modes for an explanation of the Experimental, Continuous and Nightly models). Experimental doesn’t update the source code.
Step 3 builds the CMake project including the applications, libraries and test executables (ctest_build). It performs the build on 16 cores in parallel. It submits the build results to CDash.
Step 4 runs all the tests created with add_test on 16 cores in parallel (ctest_test) and submits the results to CDash.
Step 5 collects the coverage information (ctest_coverage) and sends the results to CDash.

Now, we are ready to run the pipeline from the working directory.

$ ctest -S terminal-apps/TerminalAppsPipeline.cmake
   Each . represents 1024 bytes of output
    .. Size of output: 1K
   Each symbol represents 1024 bytes of output.
    ......... Size of output: 9K

By default, CTest only prints the above progress information. We get more telling information with the option -V and a lot more information with -VV. CTest writes the results for each step, the coverage info for each header and source file, and the tool logs into subdirectories below build-test/Testing.

/path/to/               # working directory
    build-test/Testing/
        20220501-1536/       # <date>-<time>
            <step>.xml       # Results of each step uploaded to CDash
        CoverageInfo/
            <src-file>.gcov  # Each *.cpp/*.h file annotated with coverage info
        Temporary/
            Last<step>_20220501-1536.log    # Log file for each step

Build, Test and Coverage Results

We go to My CDash in the browser and click on the project My Projects > TerminalApps to see the build summary (section Experimental) and the coverage summary (section Coverage).

CDash dashboard with build and coverage summaries

Nearly 92% line coverage looks very promising. However, it’s not surprising as I use TDD to write code. For more detailed coverage information, we click on the last coverage percentage and then on the directory (e.g., machine/src), in which we are interested.

Line coverage for the files in a selected directory

Acting on the Coverage Gaps

Code coverage tells us more about test gaps than about code quality. For every uncovered piece of code, we must decide whether and what do to about it. Here are three different examples.

A closer look at j1939_message.cpp reveals that I didn’t write a test for the function setPayload.

Identifying an untested and unused function

A quick search through the code reveals that setPayload is never used. We remove the function. Rerunning the pipeline shows no test failures and an increased coverage of nearly 93%.

The file quantity_object.cpp has only a coverage of 74%. What’s the problem?

Testing trivial functions like getters and setters is optional. I wouldn’t bother and focus on more important uncovered functions. The file can_bus_router.cpp provides such an example.

The second uncovered constructor requires tests

The first covered constructor is only used in test code. The second uncovered constructor is only used in production code. Yes, guilty as charged! I haven’t written tests for the second constructor yet. The uncovered code tells me that I must fix this omission, although the fix takes some effort.

References

Craig Scott, Professional CMake: A Practical Guide (11th edition). Chapter 25. Testing covers CTest and CDash in detail. The explanation of a CDash pipeline can be found in Section 25.9 CDash Integration.
CMake Documentation: ctest(1). Describes the functions for the pipeline steps and the CTEST_* variables.