Check Code Coverage in Pull Requests and Merge Requests¶

⚠ Attention: This document describes early access features. The code coverage gates are in Beta stage, and available for opt-in customers on GitHub Cloud. Contact your CodeScene Account Executive to get early access, or for more info on when code coverage gates are available for other platforms. (We’re working on it 👍).

Use CodeScene to enforce code coverage where it matters most: in PRs/MRs. CodeScene supports two complementary gates:

New and Changed Code: enforce coverage on the lines of code you just modified or added.
Overall Code Coverage: ensure the project’s total code coverage doesn’t slip below your configured threshold, e.g 95% or 80%.

CodeScene posts a status check on the PR/MR and you can block merges when gates fail.

An example of CodeScene's automated code coverage gates in pull and merge requests. — Fig. 65 An example of CodeScene’s automated code coverage gates in pull and merge requests.¶

This document covers two topics:

Detailed instructions on how to Enable the Code Coverage Gates.
A discussion and definition of the rules behind each gate in the section Code Coverage Gates: Rules and Metrics.

In addition, we include a FAQ on Code Coverage gates, as well as a guide for teams that aren’t using PRs. (See Not using PRs? Check Code Coverage in a Build Step).

Enable the Code Coverage Gates¶

The code coverage gates are checked as part of your CI flow, e.g. via GitHub actions. To enable them, you need to follow three steps:

Your build generates a coverage report (e.g., LCOV, JaCoCo, Cobertura, OpenCover – see Getting started with Code Coverage in CodeScene for supported coverage formats).
You have integrated CodeScene’s coverage CLI tool in a GitHub Action. (CodeScene provides a command-line tool running in your build pipeline + templates for the GitHub actions).
You have enabled CodeScene’s Code Coverage gates in your project’s configuration.

We’ll look at each of these steps in details, but let’s start with a conceptual overview of how code coverage gates are integrated:

Fig. 66 The code coverage gates are integrated into your build. CodeScene offers GitHub Action templates + the cs-coverage tool.¶

Step 1: Generate Code Coverage Reports in your CI¶

Most testing frameworks don’t generate coverage reports by default, so you need to enable that. This depends on your environment and programming language, so check the documentation for your test framework.

Note: Larger codebases can have multiple test frameworks and, consequently, you might have to generate multiple coverage reports.

After the tests are run, your GitHub action needs to upload the coverage reports as artefacts.

Example script: Here’s an example on a GitHub Action job which runs tests using Maven and uploads the coverage artefacts.

Step 2: Integrate CodeScene’s coverage CLI tool in a GitHub action¶

This step downloads the generated coverage reports, and runs a command-line tool – cs-coverage provided by CodeScene. The cs-coverage tool includes all the logic for checking the coverage gates, and handles all the communication with your CodeScene server.

Template script: We offer a script which automatically downloads the cs-coverage tool and knows how to execute it. Grab a copy of the script and place it under .github/actions/check-coverage/action.yml in your Git repository.

⑂ Alternative option: If your repo is built as multiple source code components, then check out Enabling Coverage Gates for Repositories with Multiple Source Components.

Using the template script above, you extend your workflow with one more job – cs-coverage-check. This job is responsible for downloading the coverage artefacts and invoking the cs-coverage tool. Here’s an example on how it could look:

cs-coverage-check:
runs-on: ubuntu-24.04
needs: [ execute-tests ]
if: always()
steps:
    - uses: actions/checkout@v4
    with:
        # fetch the repo history so that `cs-coverage check` can find merge-base.
        fetch-depth: 0
    # Download all artefacts/coverage files: https://github.com/actions/download-artifact#download-all-artifacts
    # this also affects the `coverage-files` property in the subsequent step:
    - name: download coverage
    uses: actions/download-artifact@v5
    with:
        path: coverage/
    # Here we reuse the action offered as a teamplate for your consumption, too.
    # This step will a) download the lastest cs-coverage too, b) run the coverage check, and
    # c) POST the results to the CodeScene server to complete the workflow.
    - uses: ./.github/actions/check-coverage
    if: always()
    with:
        access-token: ${{ secrets.PROD_CS_API_TOKEN }}
        # Specify the URL to your CodeScene server. We use this to figure out
        # which configuration to use:
        project-url: "https://api.codescene.io/v2/projects/<YOUR_PROJECT_ID>"
        # The cs-coverage tool uses a glob pattern to identify code
        # coverage reports. This mechanism simplifies the scenario where
        # there are many reports for different parts of your codebase:
        coverage-files: "**/coverage/jacoco.xml"

Example script: Here’s an example on a cs-coverage-check job .

👉 In the previous code example, you want to tweak coverage-files to match the format of your code coverage reports. You also want to change the project-url so that it points to _your_ CodeScene project. (This URL would looks something like https://api.codescene.io/v2/projects/71803).

🎗️ Get an access-token: You create an access-token as described in Public API (See the section on REST API Tokens).

Step 3: Enable CodeScene’s Code Coverage gates in your project’s configuration¶

CodeScene offers configuration at two levels:

A general configuration that applies to all CodeScene projects in your organization. That is, specify the desired code coverage gates once, and have all projects inherit those settings.
Override the general configuration inside a specific CodeScene project. Typically, you’d do this in order to raise the bar and raise the thresholds for a successful gate.

The configuration lets you specify the thresholds for the gates, as well as enabling/disabling individual gates.

NOTE The default setting enables the gate for New and Changed Code. We find that this is the most impactful and actionable gate; it checks the code you just worked on. The Overall Code Coverage gate is less actionable, and might be noisy in a PR context. (Unless your codebase has been strict on unit testing since day one).

Now, with code coverage reports being generated via your tests (Step 1), the cs-coverage tool integrated in your GitHub workflow (Step 2), and a CodeScene configuration for the coverage gates (Step 3), you’re ready to go. Open a PR, and you should get a comment on the coverage outcome:

Fig. 67 Example on a failed code coverage gate with detailed and actionable information included.¶

In the preceding example, you see that in case of a failed coverage gate, CodeScene includes info on files and lines that led to the failure. Covering those with tests makes the gate pass.

Code Coverage Gates: Rules and Metrics¶

This section defines the rules and semantics for the two code coverage gates.

New and Changed Code¶

This gate checks the added and changed lines of code in your PR. Added/changed lines of code are identified by diffing your PR against its merge base. (I.e. the point on the Git main branch from which you created the PR branch).

The gate uses the following rules:

The gate only considers executable lines of code. That is, modifying a comment of global declaration won’t impact the coverage check. (See the PR gate for New and Changed Code here for an example ).
If we identify added/changed code that isn’t executed by _any_ test, then CodeScene estimates the number of uncovered executable lines of code.
The success criteria is simple: covered changed/added executable lines of code / total changed/added executable lines of code has to be larger of equal to the threshold configured in the CodeScene UI.
The gate respects the exclusion filters set via the CodeScene UI configuration. Any code excluded from the coverage check – or analysis – is ignored in the gate.
Non-code content (e.g. JSON, infrastructure like Terraform, markdown documentation, etc.) is ignored in the gate.

⚠ Attention: If you use the coverage gates in a Build context (e.g. trunk-based development) – not a PR – then the added/changed lines of code are considered to be the content of the last commit.

Overall Code Coverage¶

This gate checks the total code coverage across your codebase.

The gate uses the following rules:

The gate only considers executable lines of code. We deduce this number from the code coverage reports.
If we identify any application code that lacks coverage, then we estimate the total number of executable lines of code in that uncovered file or function. Note that this might overestimate the actual number of executable lines of code. (A typical example is local type declarations).
The gate respects the exclusion filters set via the CodeScene UI configuration. Any code excluded from the coverage check – or analysis – is ignored in the gate.
Non-code content (e.g. JSON, infrastructure like Terraform, markdown documentation, etc.) is ignored in the gate.

Enabling Coverage Gates for Repositories with Multiple Source Components¶

The examples so far have been for a repository where all code is built in one step. Another version is to have a repository hosting multiple different source code components - each with a separate build.

In that case, you need to generate and upload the coverage reports for each component you build. You then use a single GitHub action job for invoking the cs-coverage tool and checking the coverage gates. Note that you only want to invoke the cs-coverage check once, and have it check all built source components.

Example script: We have three separate examples on how to enable code coverage gates in repositories with multiple source code components.

⚠ Attention: If you only build some of the source components – for example only components changed in a PR – then the scope of the Overall Code Coverage gate adapts; that gate will only check the coverage ratio of the tested components, not all code in the repository. The PR comment generated by CodeScene will show the scope, though.

Not using PRs? Check Code Coverage in a Build Step¶

You can also enable code coverage gates in a pure build step without any PRs.

The process is similar to how you enable the PR gates described above. The only difference is that you do _not_ enable the PR check in CodeScene’s UI configuration; just specify the gates you want to use together with their thresholds.

Example script: Example on integrating coverage gates in a build pipeline .

FAQ on Code Coverage gates¶

How do I debug my code coverage configuration?¶

The cs-coverage tool offers a debug option, the –verbose flag. Invoke the tool like this:

$(pwd)/cs-coverage check --verbose --coverage-files "${{ inputs.coverage-files }}"

The cs-coverage tool will now output detailed logs about the content it finds, as well as its progress.

Which code coverage gates should I use?¶

We recommend using the gate for New and Changed Code. We find that this is the most impactful and actionable gate; it checks the code you just worked on. So, no matter your current coverage level, this gate ensures that – starting now – each change to the code has proper coverage.

What thresholds should I use for the code coverage?¶

We recommend a high bar for New and Changed Code. Make it at least 90% so that you get a decent safety-net for future development.

What should I do when a code coverage gate fails?¶

Ideally, you’d expand your PR with new tests that cover your changeset. You find an example on a PR doing that here .

In some cases, you might have modified code that you don’t expect to be checked. For example, we usually don’t have tests for the tests themselves. Consequently, you’d expect zero coverage for your unit tests files. CodeScene offers a mechanism for excluding content from the coverage gate via the configuration UI. Ensure your test files are excluded.

Finally, we should acknowledge that a coverage number is just a number. Important as a guidance, but lacking context. So there might be situations where you are OK with accepting uncovered code. In those situations, make it visible and explicit rather than just ignoring the gate: add a comment on your PR where you motivate _why_ the coverage gate is ignored. That way, you get traceability and documentation for your decision. Your team’s going to be grateful.

What are the semantics for the New and Changed Code in the context of a build?¶

If you use the coverage gates in a Build context (e.g. trunk-based development) – not a PR – then the added/changed lines of code are considered to be the content of the last commit.

Can I exclude certain files or folders from the code coverage gates?¶

Yes, you can exclude any content via the CodeScene configuration UI. CodeScene aims to provide sensible default exclusions (e.g. identify and exclude unit test files from the gates), and you can expand the exclusions yourself:

Exclude content from the code coverage gates via CodeScene's project configuration. — Fig. 68 Exclude content from the code coverage gates via CodeScene’s project configuration.¶

Do I have to upload code coverage reports to the CodeScene server when using the coverage gates?¶

Short answer: no. The code coverage gates are independent of the analysis.

Longer answer: CodeScene has UI views for analyzing code coverage trends, as well as visualizations of what the coverage looks like in your codebase. To activate those features, you need to upload code coverage data to the CodeScene server from the builds.

However, the code coverage gates do not depend on coverage data being uploaded. So you can enable the code coverage gates without having the code coverage UI enabled.

Why is the coverage check failing, complaining that it cannot identify the merge base for the PR?¶

CodeScene identifies the branching point for your PR branch. We do that in order to identify the files that are changed on your branch (and your branch only). For this to work, the action that runs the cs-coverage tool must have access to the full Git repository history. You control that by setting fetch-depth: 0 in your GitHub action.

We’re replacing SonarQube with CodeScene, are the code coverage gates comparable?¶

Yes, CodeScene offers the same code coverage gates as SonarQube. There are minor differences due to the design decisions we have made:

New and Changed Code: CodeScene aims for explainability; then a gate fails, you probably want to know _why_ it failed and how to make it pass. For that reason, we use the Line Coverage metric only. This is different in SonarQube which uses a magical formula combining the ratios of line and branch coverage metrics. In practice, you won’t notice any larger discrepancies in the numbers.
Overall Code Coverage: In a PR, CodeScene’s gate behaves exactly as SonarQube’s. However, in a build context, Sonar offers additional options for defining the baseline (e.g. “how the coverage looked a month ago”, “compared to last release”). CodeScene instead compares against the latest commit.

Going forward, CodeScene aims to offer a new set of gates that offer more contextual feedback than these more traditional gates.

So what’s your roadmap for code coverage? Can I expect more context-aware gates in the future?¶

We’ll keep the existing gates, and augment them with a novel set of gates that are more context-aware.

Specifically, we want to offer a better alternative to the Overall Code Coverage gate. Our philosophy is that the level of coverage you need is context-dependent, taking the code’s rate of change (e.g. hotspots) and cost of failure into account:

Refactoring a complex hotspot? You want close to full coverage here.
Working within critical code with a high cost of failure? 80% is too low.
Making a small change to stable code? Well, perhaps it’s enough to only cover the change or – heresy – nothing at all.

In addition to the upcoming context-aware gates, we also plan to support a configuration-as-code approach for code coverage. That configuration allows more granular control, for example by putting different minimum thresholds on different parts of your codebase.