Why your CI/CD investment isn't showing up in release velocity

The tooling is rarely the bottleneck

If you’ve invested heavily in CI/CD pipelines and release tooling and releases still crawl, the constraint is probably not in the tooling. The pattern is familiar in large transformation programmes; the dashboards are green, and changes still queue for a weekly approval forum. A one-line configuration change passes through the same governance as a database migration. Validation waits for an environment to be free. Two teams that could release independently take turns. And every release, whatever is in it, pays for a full penetration test before it can go live.

Everyone in the room agrees the symptoms are real. What’s harder is agreeing where they come from. In our experience the constraints live in the branching model, in who owns what, in how environments are shared, and in governance that was designed for the riskiest change and then applied to all of them. None of that is a criticism of the teams who built the pipelines. Controls accumulate for good reasons, and nobody is ever asked to remove one.

These are engineering problems with engineering fixes, and many of them are small. Some are not. Continuous delivery is not the same thing as pipeline automation. The pipeline is the visible part; the practices that make frequent, safe releases possible sit underneath it, in how teams branch, test, couple their systems and govern change.

That is also why the real problems are hard to find. A long-lived branch or two services that can’t ship apart never appears in the pipeline as itself, only as a rerun, a wait or a failed deploy, and the pipeline gets the blame.

Finding the cause means reading the configuration and the approval records with the people who run them, not scoring the programme against a maturity model.

What the review produces. A checklist of fixes, not just a DORA score

A maturity assessment has its uses. It gives a programme a shared vocabulary and a target state. What it doesn’t give the team is a change ticket. We do measure, with the DORA (DevOps Research and Assessment) metrics – deployment frequency, lead time for changes, change failure rate and time to recover, baselined at the start so that every fix on the list is tied to the number it should move. That is a scoreboard the programme can watch rather than a capability rating, but a score on its own, of either kind, still doesn’t tell the people who need to ship next month which file to change first.

A hands-on review is shaped differently. We read the pipeline YAML, the environment definitions, the branch protection rules and the approval records, and we look at how the pipelines actually run, not just how they are defined. Run history shows where the time goes; approval timestamps show whether a gate is a decision or a diary slot. Your platform and release teams already know most of this. The review’s job is to put evidence and an order against what they know, so the case for change doesn’t rest on anyone’s opinion.

The output is a ranked list of specific changes, typically forty to sixty of them, each with the same fields:

• Title. One sentence naming the file, gate, policy or role to change. ‘Split the production deployment job into a build stage and a deploy stage so the artefact is immutable and promotion is reproducible’ is a line item. ‘Mature your release culture’ is not.
• Evidence. The metric, wait time or pipeline behaviour that motivates it, sourced and reproducible.
• Effort. Hours, days or ‘multi-team initiative’. Nothing on the list is a year-long programme.
• Owner. A named role – platform team, release engineering, security, a specific service team.
• Success signal. The movement that proves it landed – lead time down by a day, rerun rate halved, approval queue shorter.
• Tier. This quarter, this half, or parked until the easier wins are banked.

The list lands in your own Jira board rather than in a report folder, and the team can start on the first items the day it arrives. Where a bottleneck can be sized in money (engineers waiting on an environment, a release that is ready but waiting for the next approval meeting), the cost goes on the line item, because that is what gets sponsors to underwrite the fix.

Pipeline structure, and how the pipelines are actually used

The first thing we read is the pipeline configuration itself, file by file, alongside its run history. The reference point is a staged pipeline that fails fast and fails cheap, in the shape described by the Minimum CD pipeline reference architecture – the cheapest gates run first and catch the most common defects, so the expensive gates only run on changes that have already earned them.

On every commit, the baseline is lint and formatting, static type checks, secret scanning, static analysis for injection patterns, build, unit tests, a dependency vulnerability scan and schema migration validation. That stage should complete in under five minutes. Contract tests, integration checks and environment parity follow in under ten. Deeper verification (mutation testing, dynamic security tests, policy-as-code checks, software bill of materials (SBOM) generation, container scanning) in under fifteen. Acceptance and load tests in a production-like environment in under twenty. A stage that blows its budget is a finding in its own right.

The structural anti-patterns show up quickly once you are reading the files:

• More than one path to production. Hot-fix branches that skip the standard pipeline, deploys run from a workstation, ops-only release scripts. The pipeline has to be the definitive judge of releasability, which means every path goes through it.
• Mutable artefacts. Building once per environment instead of building once and promoting. Three builds are three different artefacts, and nobody can say which one is in production.
• Heavy gates running first. A fifteen-minute acceptance suite ahead of fifteen seconds of unit tests.
• Overridable verdicts. ‘Continue-on-error’, optional gates, skipped workflows, and approval steps that let a red run ship anyway. Once an override exists it gets normalised, and the gate becomes theatre.
• Serial jobs that could run in parallel, and jobs with no explicit timeout.

The behavioural anti-patterns are the ones a configuration read alone would miss, and they are usually what people mean when they say the pipelines are brittle. High automatic-rerun rates hiding flaky tests. More than three manual reruns on a single change. Tests quarantined for months. Scans switched off for being noisy. [skip ci] on production-affecting changes. Force-pushes to protected branches. Each of these is visible in run history, and each is a sign that the team has stopped trusting the pipeline’s answer and started working around it.

Every finding in this section maps to one of the DORA metrics, which is what ties a line item on the checklist to a number the programme can watch move. The specific performance bands change with each annual DORA report, so a review quotes the current one rather than a remembered threshold.

Environments. Fewer of them, and closer to production

Environments slow programmes more than tooling does, and the usual response, more environments with better booking rules, makes it worse. A connected end-to-end environment where every team’s work meets before production is the biggest synchronisation point in most estates. Two teams cannot validate in parallel because it is contended, the booking spreadsheet becomes the real release calendar, and because it only resembles production it produces false confidence anyway. Continuous delivery goes the other way. The aim is to need fewer lower environments, and ideally none that are shared, long-lived or connected end to end.

What replaces them. Confidence comes from the pipeline rather than from a staging estate, and that starts with design – systems are built so that end-to-end testing is not needed. The weight sits in contract tests at each integration boundary, which prove that two services still agree without standing both up together, with a deliberately small set of integration tests behind them and test doubles or recorded responses wherever a test needs a dependency.

Where an environment is needed at all, the pipeline creates it – containers started for that run, from the same definition as production, used by the tests and torn down afterwards. Nothing is booked and nothing is shared. If standing one up takes more than fifteen minutes, that is itself a finding. Tests that need production-shaped data get a sanitised seed set in the repository, not a weekend refresh window. Parity is defined in code and checked by a gate that fails on drift, which is the only form of production-like that holds.

What the review does with the existing estate. We walk it end to end with the team that runs it – how many environments exist, what each is for, which tests genuinely need it, and which teams it serialises. Most lower environments turn out to exist because some test has no other way to run, and the fix is to give that test another way to run, not to govern the environment better. Where environments stay for now, the access approvals and change control around them are a cost to size rather than a control to preserve – a short-lived environment defined in code needs no change ticket, because it is reviewed where it is defined. The acceptance checks that remain move as close to production as possible. Progressive rollout, canaries and flags do in production, with real traffic, what a connected staging environment was only pretending to do.

Production. With less resting on lower environments, more rests on production verification, which has to be real rather than logged. Health checks with automatic rollback wired in, not just logged. A canary or progressive rollout that actually progresses (a canary stuck at 100% is not a canary). Service-level checks that gate promotion. Concurrency control so two deploys to the same environment cannot overlap. Audit logs that survive a restart. Each of these turns a failed deployment from an incident into a non-event, and together they are what lets a programme deploy more often without increasing risk.

Release management and approval flow, end to end

In a regulated programme the approval flow is almost always the most expensive constraint, and the one least often measured, because it lives in calendars and tickets rather than in the pipeline log. We trace a change through it end to end – who is on the chain, what each person actually looks at, how long each step takes, what proportion of approvals are waved through, and whether a rejection feeds back into the pipeline as a new check or just as a re-queue.

The signature of a synchronous forum is visible in the data before anyone describes it – deployments clustered into a weekly slot, pull requests that age for days and then close with no review comments, a required-reviewer list with one named individual on it. That last one is a single point of failure dressed up as a control.

From forum to gate. A weekly change board concentrates every approval into one throughput-limited hour. An asynchronous, policy-based gate (codified rules the pipeline evaluates on every change, with an exception route for the cases that need a human) moves the same approvals to a continuous stream. The migration pattern is simple and measurable; codify the rules, run them alongside the forum, measure the exception rate, and retire the meeting for that class of change once exceptions fall below 10%. The board is not abolished. It shrinks to the judgement calls, which is what it was for.

Risk-tiered change. A configuration tweak and a schema migration should not pass through the same gate set. The ITIL language of standard, normal, major and emergency changes is familiar to a regulated reader; what is usually missing is a definition the pipeline can apply automatically. The workable approach classifies a change by what it touches:

• Standard: application code or configuration only, no schema, infrastructure, authentication or external-interface change, all baseline gates green. Flows through the automated gates with no human approval.
• Normal: touches schema, infrastructure definitions or a feature flag’s default. Automated gates plus one asynchronous approval from a qualified reviewer who is not the author.
• Major: touches authentication, a new external surface, a new data flow or the system’s structure (a new service or integration). Automated gates, asynchronous approval, and a risk-triggered security review (see below).
• Emergency: the standard route run faster, not a bypass; it goes through the same pipeline with the approval recorded afterwards. If more than 10% of changes are using it, the standard route is too slow.

The classification is derived from the diff, not declared by the author, so it cannot be gamed and it is recorded with the run.

Deployment is not release. Feature flags let code reach production on the trunk pipeline without being visible to customers. That separation is the heart of continuous delivery. A deployment is a code change reaching production, and in a working pipeline it is a non-event – it happens automatically, as often as changes merge, because the pipeline has already enforced every test and quality gate the change’s risk tier calls for, and the team trusts that verdict.

A release is a different thing.

A decision to make those changes available to customers. You cannot release what has not been deployed, so the order matters – deploy continuously, release deliberately. For most programmes that is the real shift, more than any pipeline change. The batch of work stops being ‘the release’ and becomes a stream of small deployments, with release a decision taken on top of them. A product owner takes it when the organisation is ready, for everyone or for a small cohort first, and can reverse it in seconds by switching the flag off. Most deployments are therefore dark, and a dark deployment needs no release approval because nothing has been released. That breaks the assumption behind most forums, which treat every deployment as a release and so put a meeting in front of code changes the pipeline has already verified, when the decision that actually needs a person is the release. Decoupling the two needs a flag library, a short flag policy and a clean-up SLA so the flag count does not grow without limit.

Parallel releases. Three things block teams releasing independently:

1. Shared environments
2. Shared release branches and
3. Shared approval forums.

The fixes are short-lived environments, trunk-based development with daily integration and no long-lived release or staging branches, and the asynchronous gates above.

The fourth blocker, which the first three do not solve, is dependency between systems, and part of the review’s job is to tell two kinds of coupling apart.

Coupling in the process (the three blockers above) can be broken within the engagement. Coupling in the architecture (a shared database, synchronous call chains, a core system that several teams are changing at once) cannot, and pretending otherwise is how programmes end up with independent pipelines whose releases still break each other. Two teams whose services call each other can only release independently once the interface between them is protected – contract tests at the boundary, run in the pipeline, and backwards-compatible interface and schema changes (expand, migrate, then contract) so that either side can ship first. Where that work is needed, the review maps the dependencies, identifies which seam to cut first and sizes it, because loosening the architecture is a separate piece of engineering from fixing the pipeline.

The controls belong in the pipeline and people belong on the judgement calls. That gives up less assurance than it sounds. An approval records that someone looked, not what they checked, and the research bears this out – reviewers catch defects best on changes under a few hundred lines (SmartBear’s Cisco study, summarised by Atlassian), and reviews at Microsoft found fewer defects than developers expected, delivering most of their value as knowledge sharing (Bacchelli and Bird, 2013). An automated gate applies the same checks to every change, records that it did, and isn’t under pressure late on a Friday.

Security controls proportionate to the change

Security in a regulated programme accretes. Every incident adds a control, no incident ever removes one, and eventually every change pays the cost of every control ever added. A full penetration test on every release is the clearest example. A high-cost, slow gate applied uniformly, when the risk it addresses varies enormously from one release to the next. Running it on a release that contains a typo fix and a logging change is the security equivalent of re-running the whole test suite on failure.

We ask three questions of each security control.

1. What class of change does it actually protect against?
2. Is it running in the pipeline, or as a separate review pass that doubles the approval queue?
3. Does it produce evidence the next change can reuse?

What triggers a penetration test. Architectural change. A new external surface. A new data flow, or a change to how existing data is classified or stored. Any change to authentication or authorisation. Those are the same signals the risk-tier classification already reads from the diff, so the trigger is automatic and recorded. A change that touches none of them does not need a test team; it needs the automated gates to have run and passed.

What replaces it for standard changes. The baseline gates already in the pipeline – secret scanning, static analysis, dependency vulnerability scanning and licence checks on every commit; dynamic security tests, container image scanning and SBOM generation in the deeper stage. These run in minutes, on every change, and they fail the build rather than warn. A container scan that warns but never fails, or an SBOM that nothing consumes, is compliance theatre and goes on the checklist.

Evidence that compounds. A penetration test that produces a PDF the next test cannot read is a control with a memory leak. Findings should be structured (SARIF for scan results, CycloneDX for SBOMs), tied to the pipeline run and commit that produced them, and queryable. The next test then starts from the last one’s residual findings rather than from a blank page, which is most of what makes a quarterly or risk-triggered cadence defensible.

The gates security should be adding, not removing. Supply-chain exposure is the one most programmes have without knowing it. CI workflows referencing third-party actions by mutable tag, base images by tag rather than digest, long-lived static cloud credentials in the pipeline. SHA-pinned actions from an allowlist, digest-pinned images and short-lived credentials via OIDC are cheap, automated, and close a real gap.

Security runs in parallel with the other gates, not after them. A security team that reviews changes in its own forum after the engineering review has doubled the queue depth; embedding its checks in the pipeline and reserving its people for escalated findings is what lets it keep up once deployments become frequent.

Why automated gates are stronger evidence in a regulated environment

The objection to all of the above is usually ‘the regulator expects a sign-off’. What regulators and auditors actually expect is evidence that a control operated, and an automated gate produces better evidence than a meeting does.

Regulated industries already rely on validated processes rather than inspecting every output. The MHRA inspects pharmaceutical manufacturers against good manufacturing practice, including Annex 15 on qualification and validation, which treats validation as documented evidence that a process will consistently produce a product meeting its specification. That is a manufacturing rule, not a software one, but the principle carries. Assurance comes from validating and monitoring the process, not from inspecting every unit it produces. ISO/IEC 27001 certification works the same way. UKAS-accredited bodies audit whether controls are in place and operating by examining evidence and sampling records, not by re-inspecting every change. SOC 2 Type 2 reports rest on the same principle.

Mapped to the pipeline, this is concrete. The classic change-management evidence is a ticket with a manual approval on it. A continuous-delivery pipeline produces something stronger and automatic – every commit to trunk passes a defined set of gates before it can reach production, and each gate is recorded against the commit and the deployment it fed. Where a step genuinely requires a human approval, it is codified in the pipeline and recorded there. Segregation of duties is evidenced by the approver identity in the deploy record being different from the commit author, enforced by an environment rule that blocks self-approval.

A useful test on any programme. Pick three production deployments at random from the last quarter and produce the approver, the gates that ran and their results, in under a minute. Teams most often fail this because approvals live in chat. A message in a channel is not an artefact; it is hard to find and easy to dispute. A hot-fix route with no approver record is a finding waiting to happen.

None of this produces compliance on its own. The organisation still decides which controls its obligations require. What it does is change the answer to ‘how is this control met’ from a search through tickets, wikis and email to a query across a chain – requirement, test, gate, commit, deployment, with who approved what at each step.

What the engagement looks like

It’s short, and it’s done by senior engineers who build and run pipelines themselves, working alongside your platform and release teams rather than around them.

It needs read access to the pipeline configuration and run history, the environment definitions and protection rules, branch protection settings and the approval records for the review window, plus time with the people who live in the flow: the platform team, release engineering, security, and one or two service teams whose releases are typical.

What comes out is the checklist described above, in the programme’s own Jira, with a short covering note for sponsors that sizes the bottlenecks and sets out the order. Where the constraint is in the architecture rather than the pipeline, the output also includes a dependency map and a sequenced plan for loosening it, so the programme can see what releasing independently would actually take. The first items are typically small enough to be done in the week after the review lands, and the success signals on each line are what the programme reports against afterwards.

Catapult CX works this way itself. Our own delivery runs through automated, gate-enforced pipelines with an evidence chain from requirement to release, and we are externally audited on it under ISO/IEC 27001, ISO 9001 and ISO/IEC 20000-1. Our clients include financial-services firms, government bodies and telecoms providers.

Talk to an engineer

If your pipelines are mature on paper and slow in practice, the constraint is probably not in the tooling. Continuous delivery is a set of engineering practices, and the pipeline is only the part you can see; the fixes that matter usually sit in how teams work and how systems are coupled. Some are small. The ones that aren’t are worth knowing about before the next release train leaves.

Our CI/CD pipeline optimisation and DevOps health check and uplift services are built around exactly this review.

Talk to one of our engineers about what a hands-on review would read in your estate, and what the first ten items on the list would be.