Check Code Coverage in Pull Requests and Merge Requests¶

Code Coverage PR gates are supported for GitHub and Azure DevOps. GitHub reports the outcome as a check run; Azure DevOps reports it as a PR status and a comment thread. For Azure Repos, configure a branch policy to require the status with genre CodeScene Code Coverage and name (-> <target branch>) before completing a PR.

Use CodeScene to enforce code coverage where it matters most: in PRs/MRs. CodeScene supports two complementary gates:

New and Changed Code: enforce coverage on the lines of code you just modified or added.
Overall Code Coverage: ensure the project’s total code coverage doesn’t slip below your configured threshold, e.g 95% or 80%.

CodeScene posts a status check on the PR/MR and you can block merges when gates fail.

An example of CodeScene's automated code coverage gates in pull and merge requests. — Fig. 81 An example of CodeScene’s automated code coverage gates in pull and merge requests.¶

This document covers two topics:

Detailed instructions on how to Enable the Code Coverage Gates.
A discussion and definition of the rules behind each gate in the section Code Coverage Gates: Rules and Metrics.

In addition, we include a FAQ on Code Coverage gates, as well as a guide for teams that aren’t using PRs. (See Not using PRs? Check Code Coverage in a Build Step).

Enable the Code Coverage Gates¶

The code coverage gates are checked as part of your CI flow, e.g. via GitHub Actions or Azure Pipelines. To enable them, you need to follow three steps:

Your build generates a coverage report (e.g., LCOV, JaCoCo, Cobertura, OpenCover – see Getting started with Code Coverage in CodeScene for supported coverage formats).
You have integrated CodeScene’s coverage CLI tool in your build pipeline. The CLI runs the gates and communicates the result to CodeScene.
You have enabled CodeScene’s Code Coverage gates in your project’s configuration.

We’ll look at each of these steps in details, but let’s start with a conceptual overview of how code coverage gates are integrated:

Fig. 82 The code coverage gates are integrated into your build. CodeScene offers GitHub Action templates + the cs-coverage tool.¶

Step 1: Generate Code Coverage Reports in your CI¶

Most testing frameworks don’t generate coverage reports by default, so you need to enable that. This depends on your environment and programming language, so check the documentation for your test framework.

Note: Larger codebases can have multiple test frameworks and, consequently, you might have to generate multiple coverage reports.

Run the coverage check after generating the reports. If generation and checking run in separate jobs, upload the reports as pipeline artifacts and download them in the checking job. When both steps run in the same job, as in the Azure Pipelines example below, use the generated files directly.

Example script: Here’s an example on a GitHub Action job which runs tests using Maven and uploads the coverage artefacts.

Step 2: Integrate CodeScene’s coverage CLI tool in your CI pipeline¶

This step downloads the generated coverage reports, and runs a command-line tool – cs-coverage provided by CodeScene. The cs-coverage tool includes all the logic for checking the coverage gates, and handles all the communication with your CodeScene server.

GitHub Actions¶

Template script: We offer a script which automatically downloads the cs-coverage tool and knows how to execute it. Grab a copy of the script and place it under .github/actions/check-coverage/action.yml in your Git repository.

⑂ Alternative option: If your repo is built as multiple source code components, then check out Enabling Coverage Gates for Repositories with Multiple Source Components.

Using the template script above, you extend your workflow with one more job – cs-coverage-check. This job is responsible for downloading the coverage artefacts and invoking the cs-coverage tool. Here’s an example on how it could look:

cs-coverage-check:
runs-on: ubuntu-24.04
needs: [ execute-tests ]
if: always()
steps:
    - uses: actions/checkout@v4
    with:
        # fetch the repo history so that `cs-coverage check` can find merge-base.
        fetch-depth: 0
    # Download all artefacts/coverage files: https://github.com/actions/download-artifact#download-all-artifacts
    # this also affects the `coverage-files` property in the subsequent step:
    - name: download coverage
    uses: actions/download-artifact@v5
    with:
        path: coverage/
    # Here we reuse the action offered as a teamplate for your consumption, too.
    # This step will a) download the lastest cs-coverage too, b) run the coverage check, and
    # c) POST the results to the CodeScene server to complete the workflow.
    - uses: ./.github/actions/check-coverage
    if: always()
    with:
        access-token: ${{ secrets.PROD_CS_API_TOKEN }}
        # Select the CodeScene project whose coverage gates should be checked:
        project-id: "<YOUR_PROJECT_ID>"
        # The cs-coverage tool uses a glob pattern to identify code
        # coverage reports. This mechanism simplifies the scenario where
        # there are many reports for different parts of your codebase:
        coverage-files: "**/coverage/jacoco.xml"

Example script: Here’s an example on a cs-coverage-check job.

Azure Pipelines¶

For Azure Repos, configure PR validation through a branch policy rather than a YAML pr trigger. Configure these two policies on the target branch:

Build validation: select your pipeline, choose Automatic, and make it Required. This runs the pipeline when the PR changes. Choose path filters that include the code you want to check, or leave the filter empty.
Status checks: require the status with genre CodeScene Code Coverage and name (-> main) (replace main with your target branch). This blocks completion when coverage gates fail. If the status is not available in the selector yet, run the integration once so CodeScene posts it.

Important

A successful PR build does not mean the coverage gates passed. In a PR build, cs-coverage check can print Code coverage gates: FAIL and still exit successfully: CodeScene reports the gate outcome through the separate PR coverage status. Configuration or result-submission errors still fail the command. Requiring only the build validation policy does not enforce the coverage outcome. Outside PR builds, a coverage gate failure causes the command to exit with a nonzero code.

The CLI needs the full Git history and a remote-tracking ref for the target branch to find the PR merge base. Use fetchDepth: 0 and fetch that ref explicitly. Keep checkout credentials available for this fetch, including when the repository is private.

The PR validation build checks coverage for the PR’s source commit. Azure checks out a synthetic merge commit for PR builds, so do not upload that merge commit’s report as branch coverage. Upload reports on branch builds and run cs-coverage check for both PR and branch builds:

trigger:
  branches:
    include:
    - main # Include every branch for which you want to upload coverage.
# For Azure Repos, configure PR validation separately with a branch policy.

pool:
  vmImage: ubuntu-latest

steps:
- checkout: self
  fetchDepth: 0
  persistCredentials: true

- script: |
    target="${SYSTEM_PULLREQUEST_TARGETBRANCH#refs/heads/}"
    git fetch origin "+refs/heads/$target:refs/remotes/origin/$target"
  displayName: Fetch PR target branch for coverage diff
  condition: eq(variables['Build.Reason'], 'PullRequest')
  workingDirectory: $(Build.SourcesDirectory)

# Configure the JaCoCo Maven plugin to generate target/site/jacoco/jacoco.xml in your pom.xml.
- task: Maven@4
  displayName: Run tests and generate JaCoCo coverage
  inputs:
    mavenPomFile: pom.xml
    goals: clean install
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    javaHomeOption: JDKVersion
    jdkVersionOption: '1.17'

- script: |
    curl -fsSL https://downloads.codescene.io/enterprise/cli/install-cs-coverage-tool.sh | sh -s -- -y
  displayName: Install cs-coverage

- script: |
    "$HOME/.local/bin/cs-coverage" upload --verbose \
      --format jacoco --metric line-coverage \
      "$(Build.SourcesDirectory)/target/site/jacoco/jacoco.xml"
  displayName: Upload branch coverage to CodeScene
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
  workingDirectory: $(Build.SourcesDirectory)
  env:
    CS_ACCESS_TOKEN: $(CS_ACCESS_TOKEN)

- script: |
    "$HOME/.local/bin/cs-coverage" check --project-id 71803 --verbose \
      --coverage-files "target/site/jacoco/jacoco.xml"
  displayName: Check coverage gates
  workingDirectory: $(Build.SourcesDirectory)
  env:
    CS_ACCESS_TOKEN: $(CS_ACCESS_TOKEN)

Define CS_ACCESS_TOKEN as a secret pipeline variable and pass the CodeScene project ID to cs-coverage check using --project-id. For On-prem, set CS_ONPREM_URL to your instance’s base URL for both the upload and check steps, alongside CS_ACCESS_TOKEN. For example, use --project-id 42 with CS_ONPREM_URL=https://codescene.example.com. Do not configure CS_PROJECT_URL for the coverage check.

Azure PR check results are posted to the PR by the CodeScene Azure integration; grant the configured Azure identity permission to post PR statuses and threads. To block completion, require the status context with genre CodeScene Code Coverage and name (-> main) (or your target branch name) in the target branch policy.

CodeScene must receive the repository’s PR webhooks as well as the CLI’s results. Keep the Azure repository authorization valid so CodeScene can create the coverage status and post the result comment. The build’s CodeScene API token and the integration’s Azure credential serve different purposes; successful CLI authentication alone does not confirm that CodeScene can post to Azure.

Note

Configure coverage exclusions before testing the integration. JaCoCo normally reports production code, not the test classes themselves. Changed test files missing from the report count as uncovered unless excluded. For the Maven layout above, add */src/test/** to your project’s coverage exclusions. The initial * selects every repository; src/test/** selects that directory at each repository root. See How coverage exclusion patterns are matched for other layouts and repository-specific patterns.

Verify the Azure integration¶

Enable the coverage gates in CodeScene as described in Step 3 below.
Open a PR containing a real source-code change and its tests against the protected target branch.
Confirm Azure queues a new validation build. Check the build ID and, in the verbose coverage step log, verify that Running in PR context names the current PR source commit and target branch.
Confirm the separate CodeScene Code Coverage status completes and the PR comment explains the measured coverage, thresholds, and any failures. A failing coverage gate should block completion through the status policy even if the build succeeds.

When testing another revision, make a file change and verify that a new build actually starts. If the PR still links to an older successful build, use Re-queue on the PR’s build validation check and inspect the newly created build’s commit context.

👉 In the previous code example, you want to tweak coverage-files to match the format of your code coverage reports. Set the project-id action input to your CodeScene project ID. For On-prem, also set the action’s onprem-url input to your instance’s base URL.

🎗️ Get an access-token: You create an access-token as described in Public API (See the section on REST API Tokens).

Step 3: Enable CodeScene’s Code Coverage gates in your project’s configuration¶

CodeScene offers configuration at two levels:

A general configuration that applies to all CodeScene projects in your organization. That is, specify the desired code coverage gates once, and have all projects inherit those settings.
Override the general configuration inside a specific CodeScene project. Typically, you’d do this in order to raise the bar and raise the thresholds for a successful gate.

The configuration lets you specify the thresholds for the gates, as well as enabling/disabling individual gates.

NOTE The default setting enables the gate for New and Changed Code. We find that this is the most impactful and actionable gate; it checks the code you just worked on. The Overall Code Coverage gate is less actionable, and might be noisy in a PR context. (Unless your codebase has been strict on unit testing since day one).

👉 The final step is to enable the pull request gates in your CodeScene projects:

For On-prem, configure the project’s Azure connection under PR Integration first. Then open Configuration → Coverage Gates, enable coverage gates in pull requests, and save the configuration. Coverage gates can run independently of Code Health gates, using the same Azure connection and PR webhooks.

Fig. 83 Enable the code coverage gates in your CodeScene projects.¶

💡 Optional: Control When the GitHub Coverage Check Appears: By default, the coverage check is Always shown. Because it depends on specific GitHub Actions, it will fail if those workflows aren’t yet in your repo. In a large organization with separate groups managing CodeScene vs creating GitHub actions, choose the second option. It hides the coverage gates until the required GitHub Actions are in place.

Now, with code coverage reports being generated via your tests (Step 1), the cs-coverage tool integrated in your GitHub workflow (Step 2), and a CodeScene configuration for the coverage gates (Step 3), you’re ready to go. Open a PR, and you should get a comment on the coverage outcome:

Fig. 84 Example on a failed code coverage gate with detailed and actionable information included.¶

In the preceding example, you see that in case of a failed coverage gate, CodeScene includes info on files and lines that led to the failure. Covering those with tests makes the gate pass.

Code Coverage Gates: Rules and Metrics¶

This section defines the rules and semantics for the two code coverage gates.

New and Changed Code¶

This gate checks the added and changed lines of code in your PR. Added/changed lines of code are identified by diffing your PR against its merge base. (I.e. the point on the Git main branch from which you created the PR branch).

The gate uses the following rules:

The gate only considers executable lines of code. That is, modifying a comment of global declaration won’t impact the coverage check. (See the PR gate for New and Changed Code here for an example).
If we identify added/changed code that isn’t executed by _any_ test, then CodeScene estimates the number of uncovered executable lines of code.
The success criteria is simple: covered changed/added executable lines of code / total changed/added executable lines of code has to be larger of equal to the threshold configured in the CodeScene UI.
The gate respects the exclusion filters set via the CodeScene UI configuration. Any code excluded from the coverage check – or analysis – is ignored in the gate.
Non-code content (e.g. JSON, infrastructure like Terraform, markdown documentation, etc.) is ignored in the gate.

⚠ Attention: If you use the coverage gates in a Build context (e.g. trunk-based development) – not a PR – then the added/changed lines of code are considered to be the content of the last commit.

Overall Code Coverage¶

This gate checks the total code coverage across your codebase.

The gate uses the following rules:

The gate only considers executable lines of code. We deduce this number from the code coverage reports.
If we identify any application code that lacks coverage, then we estimate the total number of executable lines of code in that uncovered file or function. Note that this might overestimate the actual number of executable lines of code. (A typical example is local type declarations).
The gate respects the exclusion filters set via the CodeScene UI configuration. Any code excluded from the coverage check – or analysis – is ignored in the gate.
Non-code content (e.g. JSON, infrastructure like Terraform, markdown documentation, etc.) is ignored in the gate.

Enabling Coverage Gates for Repositories with Multiple Source Components¶

The examples so far have been for a repository where all code is built in one step. Another version is to have a repository hosting multiple different source code components - each with a separate build.

In that case, you need to generate and upload the coverage reports for each component you build. You then use a single GitHub action job for invoking the cs-coverage tool and checking the coverage gates. Note that you only want to invoke the cs-coverage check once, and have it check all built source components.

Example script: We have three separate examples on how to enable code coverage gates in repositories with multiple source code components.

⚠ Attention: If you only build some of the source components – for example only components changed in a PR – then the scope of the Overall Code Coverage gate adapts; that gate will only check the coverage ratio of the tested components, not all code in the repository. The PR comment generated by CodeScene will show the scope, though.

Components with identical relative paths: Coverage tools typically report paths relative to the component, such as src/lib/thing.ts. If several components contain a file with the same relative path, cs-coverage uses the location of the coverage report to pick the right one: a report in comp-a/coverage/lcov.info is matched to comp-a/src/lib/thing.ts. Keep each coverage report inside the directory of the component it belongs to. If a path still matches several files, cs-coverage logs a warning and uses the first match. A path that matches no file in the repository is ignored, and cs-coverage upload warns about it.

Not using PRs? Check Code Coverage in a Build Step¶

You can also enable code coverage gates in a pure build step without any PRs.

The process is similar to how you enable the PR gates described above. The only difference is that you do _not_ enable the PR check in CodeScene’s UI configuration; just specify the gates you want to use together with their thresholds.

Example script: Example on integrating coverage gates in a build pipeline.

FAQ on Code Coverage gates¶

How do I debug my code coverage configuration?¶

The cs-coverage tool offers a debug option, the –verbose flag. Invoke the tool like this:

$(pwd)/cs-coverage check --project-id "${{ inputs.project-id }}" --verbose --coverage-files "${{ inputs.coverage-files }}"

The cs-coverage tool will now output detailed logs about the content it finds, as well as its progress.

Which code coverage gates should I use?¶

We recommend using the gate for New and Changed Code. We find that this is the most impactful and actionable gate; it checks the code you just worked on. So, no matter your current coverage level, this gate ensures that – starting now – each change to the code has proper coverage.

What thresholds should I use for the code coverage?¶

We recommend a high bar for New and Changed Code. Make it at least 90% so that you get a decent safety-net for future development.

What should I do when a code coverage gate fails?¶

Ideally, you’d expand your PR with new tests that cover your changeset. You find an example on a PR doing that here.

In some cases, you might have modified code that you don’t expect to be checked. For example, we usually don’t have tests for the tests themselves. Consequently, you’d expect zero coverage for your unit tests files. CodeScene offers a mechanism for excluding content from the coverage gate via the configuration UI. Ensure your test files are excluded.

Finally, we should acknowledge that a coverage number is just a number. Important as a guidance, but lacking context. So there might be situations where you are OK with accepting uncovered code. In those situations, make it visible and explicit rather than just ignoring the gate: add a comment on your PR where you motivate _why_ the coverage gate is ignored. That way, you get traceability and documentation for your decision. Your team’s going to be grateful.

What are the semantics for the New and Changed Code in the context of a build?¶

If you use the coverage gates in a Build context (e.g. trunk-based development) – not a PR – then the added/changed lines of code are considered to be the content of the last commit.

Can I exclude certain files or folders from the code coverage gates?¶

Yes, you can exclude any content via the CodeScene configuration UI. CodeScene aims to provide sensible default exclusions (e.g. identify and exclude unit test files from the gates), and you can expand the exclusions yourself:

Exclude content from the code coverage gates via CodeScene's project configuration. — Fig. 85 Exclude content from the code coverage gates via CodeScene’s project configuration.¶

These are the same coverage exclusions that the project’s coverage views use, so a file excluded there is also left out of the gates. See How coverage exclusion patterns are matched for how a pattern selects repositories and paths, for example **/*.test.ts for test files at any depth in every repository.

A changed file that is not in the coverage report counts as uncovered in the New and Changed Code gate, unless it is excluded. Run cs-coverage check with --verbose to list the files that had no coverage data.

Do I have to upload code coverage reports to the CodeScene server when using the coverage gates?¶

Short answer: no. The code coverage gates are independent of the analysis.

Longer answer: CodeScene has UI views for analyzing code coverage trends, as well as visualizations of what the coverage looks like in your codebase. To activate those features, you need to upload code coverage data to the CodeScene server from the builds.

However, the code coverage gates do not depend on coverage data being uploaded. So you can enable the code coverage gates without having the code coverage UI enabled.

Why is the coverage check failing, complaining that it cannot identify the merge base for the PR?¶

CodeScene identifies the branching point for your PR branch. We do that in order to identify the files that are changed on your branch (and your branch only). For this to work, the action that runs the cs-coverage tool must have access to the full Git repository history. You control that by setting fetch-depth: 0 in your GitHub action. For Azure Pipelines, use fetchDepth: 0 and fetch the target branch as shown in the Azure example above.

Why does an Azure PR coverage check stay pending or time out?¶

The PR coverage check needs two inputs: an Azure PR webhook registers the check, and cs-coverage check submits the gate results from the pipeline. They must identify the same CodeScene project, repository, PR source commit SHA, and target branch.

Inspect the build linked from the current PR’s build validation check. Confirm a new build ran for the current revision, rather than relying on a previous successful build. Use Re-queue if needed.
Inspect the coverage step, including whether it ran or was skipped. With --verbose, compare its PR head and target branch with the current PR. A rerun for an older commit cannot complete the check registered for a newer commit.
Verify the configured project URL and inspect CLI errors when fetching configuration or posting results.
If the CLI reports No check registered yet, CodeScene accepted and stored the result but has not matched it to a registered PR check. The result will be reported when registration succeeds. Check Azure webhook delivery, the project’s PR coverage configuration, and the integration’s Azure authorization and permissions to post statuses and threads.

A completed Code Health Delta Analysis is a separate check; it does not supply the pipeline’s coverage results. A coverage timeout means no coverage result was available for that check before its deadline, not that the measured coverage was below the threshold.

We’re replacing SonarQube with CodeScene, are the code coverage gates comparable?¶

Yes, CodeScene offers the same code coverage gates as SonarQube. There are minor differences due to the design decisions we have made:

New and Changed Code: CodeScene aims for explainability; then a gate fails, you probably want to know _why_ it failed and how to make it pass. For that reason, we use the Line Coverage metric only. This is different in SonarQube which uses a magical formula combining the ratios of line and branch coverage metrics. In practice, you won’t notice any larger discrepancies in the numbers.
Overall Code Coverage: In a PR, CodeScene’s gate behaves exactly as SonarQube’s. However, in a build context, Sonar offers additional options for defining the baseline (e.g. “how the coverage looked a month ago”, “compared to last release”). CodeScene instead compares against the latest commit.

Going forward, CodeScene aims to offer a new set of gates that offer more contextual feedback than these more traditional gates.

So what’s your roadmap for code coverage? Can I expect more context-aware gates in the future?¶

We’ll keep the existing gates, and augment them with a novel set of gates that are more context-aware.

Specifically, we want to offer a better alternative to the Overall Code Coverage gate. Our philosophy is that the level of coverage you need is context-dependent, taking the code’s rate of change (e.g. hotspots) and cost of failure into account:

Refactoring a complex hotspot? You want close to full coverage here.
Working within critical code with a high cost of failure? 80% is too low.
Making a small change to stable code? Well, perhaps it’s enough to only cover the change or – heresy – nothing at all.

In addition to the upcoming context-aware gates, we also plan to support a configuration-as-code approach for code coverage. That configuration allows more granular control, for example by putting different minimum thresholds on different parts of your codebase.