Making Continuous Integration Tools Work

Budget ten minutes for the commit-triggered build, and move every check that will not fit into a job that runs apart from it; a pipeline slower than that gets worked around by the developers it is meant to serve.

Extreme Programming set a ten-minute bound on the integration cycle, and the F1 Group's 2026 UK review reports it as the working limit, for the reason that review gives: feedback arriving after a developer has switched task has already been discounted. The same review names the three numbers to watch first, which are pipeline duration, build success rate and change failure rate. Those three are the acceptance test for any product you install.

What a continuous integration server actually does

A continuous integration server fetches the current version of the software, rebuilds it, runs the tests, and reports the outcome to the team. The Software Sustainability Institute's guide describes that mechanism and isolates the part that changes behaviour: the server monitors revision control, and a change to the code starts a new build-and-test job. A cron job does the same work on a timer. A commit-triggered job makes the commit the unit of integration, review and release, whereas a timer provides no such guarantee.

The University of Cambridge's DevOps division defines its CI/CD standard in operational terms. A job is an automated process taking a specific commit plus configuration variables as input, and a pipeline is jobs collected together with declared dependencies between them. The stated purpose is that automation makes processes predictable, reviewable and repeatable, with less opportunity for error from human intervention, and the document anchors the whole set of rules in ISO/IEC/IEEE 32675:2022. Turn that into a test you can fail. Two runs of the same commit that produce different artefacts mean the automation has not been achieved, whatever the dashboard shows.

Keep the pipeline definition inside the repository

The commit is the unit, so its definition travels with it. The same Cambridge CI/CD standard places job and pipeline configuration inside the project's git repository, normally in a file such as .gitlab-ci.yml, and defines three triggers: a push, a tag, and a schedule. Every commit then includes a version-controlled description of what was supposed to happen to it, and an incident review has something to examine.

A minimum viable shape, with the pins that matter:

build:
  image: python:3.12.4-slim   # pinned, never :latest
  cache:
    key: $CI_COMMIT_REF_SLUG
    paths: [.cache/pip]
  script:
    - pip install --require-hashes -r requirements.txt
    - python -m build
    - python -m pytest --junitxml=report.xml
  artifacts:
    when: always
    reports: {junit: report.xml}
    paths: [dist/]

Three properties do the work here. The base image is pinned to a full version or a digest, so the environment does not drift between Tuesday and Friday. Dependencies resolve from a lock file with hashes, so a yanked or compromised release cannot enter the build. The test report is emitted as an artefact even when the job fails, which is the only way a reviewer sees why a merge request is blocked without opening a second console.

Latency determines the architecture

Once the ten-minute budget is fixed, the checks sort themselves. Anything returning in seconds runs on the commit: compile, unit tests, lint, and a dependency scan against a warm cache. Anything taking minutes runs on a schedule: the full integration suite, container image scanning across the entire layer history, and the infrastructure plan for every environment. The split keeps the commit-triggered path inside the budget, and the F1 Group review warns that a build which drags on encourages batching, which pushes the team back toward the integration pain continuous integration was meant to remove.

Two mechanical fixes recover most of the lost time. Cache package restoration keyed on the lock file, and parallelise jobs that share no artefacts. The Cambridge templates demonstrate the second part: dependency scanning, container scanning, secret detection, static analysis and code quality run as separate jobs whose reports are aggregated back onto the merge request, so no job waits on the others.

Permissions set the blast radius

A runner stores credentials, and that is the real threat surface. Coherent Solutions, in its review of recurring continuous integration problems, reports that every pipeline job should execute with the fewest privileges it can complete with, because a job granted more permissions than necessary can break production or open a back door. The same review identifies four failure classes that recur across projects: poor performance, flawed tests, unreliable security, and version drift in tooling.

Continuous integration and deployment should improve the efficiency of your project, not slow it down. If something is completed faster manually, then it's an indicator that you need to dig deeper to find the cause.

That is Alexander Simonov, a DevOps engineer at Coherent Solutions, and it applies to permissions as much as to speed. A pipeline that needs a production write token to run a linter has already handed an attacker the same token. Scope the token to the job, keep secrets in the platform secret store and out of the repository, and require approval before any job that mutates a running environment. Microsoft's Power Platform guidance makes the same case from the other end: failed tests should temporarily block a deployment and send the team into a deeper analysis of what happened.

Which three metrics to track first

Track pipeline duration, build success rate and change failure rate, and add deployment frequency once builds actually reach an environment. The F1 Group review argues for starting with those three because they answer whether the pipeline is usable, whether it is trusted, and whether it helps or harms production. Set the success threshold at nine runs in ten. A rate below that points at flaky tests or unpinned dependencies. A duration rising week on week means the cache stopped matching or a job grew without anyone measuring it.

Determining whether the pipeline changed how the team works needs more than a green badge, and the tooling-assisted development evidence sets out what that measurement looks like when the tool operates between the developer and the commit.

Where four classes of continuous integration tool actually differ

Once the loop, the artefact, the latency budget and the token scope are settled, products differ on four attributes and no other factor that matters operationally.

Hosted runners on a vendor's cloud. Queue latency follows the vendor's load, billing is per minute and scales with test suite size, the vendor handles isolation, and the machine image stays fixed to the versions the vendor publishes. This fits a team of under five engineers with no build touching regulated data.
Self-hosted runners on your own infrastructure. Queue latency follows your own capacity, cost is a fixed monthly figure per machine, you configure isolation, and you control the base image and the network path. The Cambridge standard takes this route, requiring a runner to be deployed through its own Terraform configuration before a project may use the shared templates.
Pipeline as code inside the source control platform. The trigger and the review surface are one product, so comments, status and required checks attach directly to the merge request, and the configuration lives in the repository by default. The cost is that pipeline upgrades arrive with the platform's release schedule.
A stand-alone build server you install and host. Parallel job execution runs across agents, plugins supply each integration, the server publishes its own results dashboard, and a version-compatibility matrix is yours to maintain against every build tool it calls. The IONOS comparison of continuous integration tools places Jenkins, Bamboo, TeamCity and CruiseControl in this class. The Software Sustainability Institute notes that some servers, TeamCity among them, run tests before new code reaches the repository and let the commit through only when the tests pass.

The choice of class is a decision about the guarantee you need at the boundary of the build, and the four attributes above answer it before any vendor demo does. Hold the class you pick to the ten-minute budget, the three metrics and the token scope. A tool that fails any of those three tests is a tool your team will route around.