Your Platform Readiness Score

Insights

Why Software Releases Fail in Production and What It Costs Retail and Commerce Businesses

Every release carries a tension. Deploy fast enough to keep pace with the business but stay in control enough to avoid the next production incident. Most engineering leaders in retail and commerce aren’t losing that tension because of a lack of discipline. They’re losing it because their testing process was built for a slower delivery cadence than the one the business is now running.

Trading windows don’t move for anyone, and the calendar doesn’t care how confident engineering feels about the next release. When speed and control fall out of balance, releases fail. In commerce, that failure reaches the customer before it reaches an incident channel.

Here’s why that happens, what it actually costs, including the part of the cost that never shows up in a Jira ticket, and what closing that distance looks like for teams that have already done it.

Why do software releases fail in production?

Ask most engineering leaders why a release failed and you’ll get a specific answer: a bad migration, an untested edge case, a third-party API that changed behaviour without notice. We hear a version of this in a lot of incident reviews we’re brought into, and it’s rarely the full story. The named defect is a symptom. Ask enough teams, over enough incidents, and a pattern emerges underneath it.

Production releases fail for all kinds of reasons; bad deployment tooling, configuration drift between environments, capacity nobody planned for, plain human error. The pattern worth focusing on, because it’s the one testing can actually fix, is the gap between how fast delivery moves and how testing, from regression testing through performance testing, was built to work.

Strip away the specific defect in most retail and commerce incidents and it tends to come down to one of five things:

The test suite that’s always a sprint behind
Manual regression is bound by headcount and hours: how many people can click through how many test cases before the release window closes. Sprint cadence has no such ceiling, so the two drift apart. Every sprint, the distance between “code complete” and “verified” gets a little wider.
Teams respond to that distance in one of two ways, and both cost something. Cut scope to protect the date, and you deploy with reduced confidence, hoping the untested paths don’t matter this time. Slip the date to protect coverage, and the business absorbs a delay it didn’t plan for. Most teams end up doing both, on rotation, without ever closing the distance for good. We see the same adaptation in almost every commerce team where the release calendar has outpaced the test calendar: the two or three flows anyone remembers under pressure, usually checkout and payment, get protected every time, and everything else rides on hope.
The seam nobody owns
Integration testing is supposed to catch this. In practice, a retail stack is rarely one system. Ecommerce platform, POS estate, ERP, payment providers, fulfilment: each on its own release schedule, with its own third-party dependencies, each tested in isolation by teams who assume the other systems will behave as documented.
That assumption is where most of the expensive defects live. Picture a promotions engine that’s been tested against every discount rule in the spec, deployed alongside a payment provider update that changes how partial refunds are calculated. Both passed their own test suites. Neither team tested them together, because neither owned the seam between them. The defect only exists at that seam, and it only shows up once real customers start using both systems at once. Without integration testing at that seam, it’s a common root cause of exactly the incidents that are hardest to untangle after the fact: not a broken system, but a boundary nobody was responsible for.
In commerce, this is expensive to fix for a specific reason: nobody owns the seam, so nobody’s first on the call when it breaks, and diagnosis starts from zero every time. Most QA providers test a checkout flow in isolation against its own spec. They don’t test it against the exact payment provider and POS combination live in your stack, which is precisely where seam defects hide.
Tested everywhere except where it matters
Most testing effort goes into whether a release works, not whether it still works under real trading load. A checkout flow can pass every functional test and still buckle the first time a flash sale or peak trading event creates volume past what anyone tested for. Testing under real peak-load conditions, not just staging conditions, is the layer that catches this before customers do, and it’s routinely the first thing cut when a release date is tight, because a system that “works” in a quiet staging environment looks finished even when it’s never been proven under load.
Picture a checkout flow that’s passed every functional test against a staging database seeded with a few hundred test orders. The first time it meets a flash-sale spike of real concurrent traffic, contention nobody load-tested for turns a working checkout into a queue, or a timeout. The code didn’t change between the successful test and the failed release. The load did.
That’s an expensive place to cut corners. The releases most likely to face genuine volume, seasonal peaks, promotional spikes, are exactly the releases where an untested performance ceiling is most expensive to discover live.
The fix that’s always next sprint
Ad hoc testing is fine for a small platform that a small team can hold in their heads. However, it stops being fine the moment the platform, the integration count, and the release frequency all grow past what one team’s memory can track. Nothing steps in to absorb that growth.
Without a regression suite designed to scale with the codebase, every release adds to a backlog of untested paths rather than shrinking it. That backlog is invisible until a release exposes one of the paths inside it, which is usually the first anyone hears about it. By then, closing that backlog manually is no longer realistic. The same automation investment the team postponed becomes the only way out, on a worse timeline than if it had started sooner. Skipping test automation is rarely a deliberate decision. It happens by drift, where the plan is always to build it properly once the current release calms down, and the current release never calms down.
This shortfall is widening faster than most teams have registered. DORA’s 2024 State of DevOps report found that teams who increased their use of AI coding tools by 25% saw software delivery stability drop by about 7.2%, with throughput down roughly 1.5% too: more AI-generated code, without the speed gains it’s supposed to deliver, and less stability on top. The fix isn’t slowing AI adoption down. It’s regression coverage intelligent enough to scale with a faster-growing codebase: risk-driven, AI-supported prioritisation that lets automated coverage grow in step with the code being deployed, rather than falling further behind it every sprint.
The QA team sized for last year’s roadmap
A lot of internal QA functions are sized for the roadmap the business was running eighteen months ago, not the one it’s running now. They cover more releases than they can safely handle, often without deep platform expertise for the specific commerce stack in play. Adobe Commerce, Shopify, and a bespoke POS integration each carry failure modes a generalist tester won’t necessarily catch.
Hiring to fix this takes months you don’t have, and the shortfall between capacity and demand persists for as long as that hiring process runs, not because the team in place isn’t good, but because the roadmap simply outgrew the size of team built to check it. PMC and Retail Economics’ own research among 100 senior retail and DTC leaders found the same pattern showing up in the wider data: skills and recruitment challenges are cited as a barrier to digital transformation generally, a smaller factor than legacy systems or ROI pressure, but a real and persistent one.
None of the five problems above are inherent to retail or commerce as a sector. They show up anywhere a delivery cadence outpaces the testing process built to support it. Retail and commerce just make the consequences visible faster, because the release calendar is tied directly to a trading calendar. These are failures in practice, not an unavoidable cost of the industry a team happens to be in.

What do failed releases actually cost retail and commerce businesses?

Two numbers tell you whether a delivery pipeline is actually under control: how often a release causes a problem, and how long it takes to recover when one does. Neither number improves just because the team works harder or gets bigger. Both are a direct read on release readiness, and both move quietly in the wrong direction for exactly as long as a team keeps testing the way it did when release frequency was lower.

Velocity cost
When testing can’t keep pace with development, it becomes the ceiling on how fast the business can actually deploy. Time-to-market slips even in sprints where development hit every deadline, because the constraint has simply moved downstream to a stage nobody is measuring as closely as they measure code velocity. That’s a cost that leadership usually feels months before it shows up in a retrospective: commitments start slipping, one sprint at a time, long before anyone names testing as the reason.
Change failure rate: the number that says whether you’re actually in control
How often a release causes a problem once it’s live is the most direct evidence available of whether a delivery pipeline is under control or just moving fast. It’s the clearest single measure of release readiness, and it’s rarely tracked as closely as deployment speed, because speed is visible in every sprint review and failure rate usually only gets calculated after an incident forces the question. Every weakness covered above pushes that number in the wrong direction, until something breaks.
Time to restore: why untested integrations make incidents longer, not just more likely
The cost of a defect isn’t fixed at the moment it happens, it’s shaped by how long it takes to find. A defect inside a single, well-tested component is usually fast to diagnose: the fault is isolated, the logs point somewhere specific. A defect at an untested integration seam is a different problem entirely. The on-call engineer is debugging across two or three systems simultaneously, none of which were tested together, trying to work out which one is lying about its own behaviour. Standing still on integration testing doesn’t just raise the odds of an incident. It extends every incident that does happen, because the diagnosis itself takes longer when nothing was tested at the seam.
This is where ongoing support matters as much as the initial build: 24x7x365 incident response and continuous test suite maintenance are what keep time-to-restore short after go-live, not just at it.
Customer experience cost: what you won’t see in a ticket queue
This is the cost most engineering metrics don’t capture, and it’s often the largest one. In retail and commerce, a customer who hits a broken checkout doesn’t file a support ticket. They close the tab and buy from a competitor who’s already open in the next one. There’s no complaint logged, no ticket to count, no line in an incident postmortem. Just a transaction that should have happened and didn’t, and a customer who suddenly stopped being a customer.
That asymmetry matters more in commerce than almost anywhere else. A shopper doesn’t need to evaluate an RFP or migrate a codebase to switch away from you: the alternative is one tab over, and a checkout failure at the moment someone was ready to pay leaves a stronger impression than a dozen smooth transactions before it. It’s also not a coincidence that the releases most likely to fail, deployed under time pressure with reduced coverage, are structurally the releases most likely to touch checkout, promotions, or payment: the areas under the most delivery pressure are exactly where testing shortfalls hit hardest, and where customers feel it first.
Trading, revenue, and team cost
Retail failures don’t distribute evenly across the calendar. A defect that surfaces during a promotional peak or a seasonal spike costs more in revenue than the same defect on an ordinary Tuesday, because trading volume at that moment is the whole point of the release calendar. Meanwhile, every hour an engineer spends diagnosing a live incident, is an hour not spent on the next feature. The more incidents a team absorbs, the less roadmap capacity survives to prevent the next one. Competitors who’ve closed this distance aren’t just deploying more reliably; they’re doing it with engineering capacity your team is spending on recovery instead of delivery. This widens with every release cycle it’s left unaddressed.

Move faster. Stay in control.

Our view, formed across 20+ years of testing commerce platforms under real trading pressure, is that speed and control were never actually in tension. The tension only appears when testing is treated as a gate at the end of delivery instead of a capability built into it. Teams that make that shift deploy faster and fail less, not despite moving quickly, but because verification and velocity have become the same investment rather than competing ones.

The instinct when a release goes wrong points the other way: slow down, add another sign-off, protect the next date at all costs, treat caution as the price of stability. That instinct trades away velocity and, most of the time, doesn’t buy back as much stability as it costs. The underlying mismatch between delivery pace and testing capacity is still there, just given more time to be worked around manually instead of closed for good.

Closing it properly tends to look like a handful of concrete shifts, all of which sit within PMC’s Testing and Automation Services.

Shift testing left, inside the sprint
Test cases get written alongside the feature, not after it’s marked done. Acceptance criteria are verified as part of the sprint, not in a separate phase tacked onto the end of it, and user acceptance testing (UAT) happens early enough to catch a misunderstanding of the requirement while it’s still cheap to fix, not after the business has already signed off on the wrong thing. The defect that would have surfaced in production instead gets caught by the developer who still remembers exactly why they wrote that line of code. It costs minutes then, not the days it costs once the context is gone.
Automate regression with risk-based prioritisation
Not every test needs to run against every release. Automated regression testing suites that prioritise by risk keep pace with delivery frequency instead of falling further behind it each sprint, and in commerce that means consistent coverage exactly where a missed regression is most expensive: checkout, promotions, product data, and payment integrations. These are the paths that sit directly on the route to revenue.
Validate integration and deployment, every release
Integration testing at that layer, paired with deployment validation before a release goes live, catches the failure category responsible for most of the hardest, slowest, and most expensive incidents before it reaches a live trading environment, not after. Treated as a standing part of the pipeline rather than a one-off exercise, it also builds the evidence base that go-live readiness actually needs: proof that the release behaves correctly across the full stack, not just within any one system.
Treat quality assurance as a managed capability
Defined coverage priorities, clear quality gates, and reporting that’s actually visible to the business turn testing from something the team scrambles to catch up on before every go-live into a function that scales with the delivery programme by design. That’s the difference between quality as a fire drill and quality as infrastructure, and it’s what makes continuous improvement possible: without a stable baseline to measure from, there’s no way to tell whether coverage, failure rate, or recovery time is actually getting better release over release.
That capability doesn’t need to look the same for every team. Whether QA is delivered project-based, embedded inside the sprint team, or run as a fully managed function, the structure flexes to the delivery model already in place, rather than asking engineering to adapt around it.

What this looks like once it's working

Across PMC’s testing and automation engagements: release time cut by 75%+. Manual testing effort down by around 40%. Releases deploying 50% faster. Automated coverage scaled up to 3x what manual testing alone could sustain.

Pluxee needed to launch a government-related voucher scheme against a fixed, non-negotiable deadline, with every user journey and edge case covered. Expertise-led manual testing and scripting got them to go-live readiness on time, without trading coverage for speed.

FitFlop’s manual regression process had become the bottleneck inside every sprint. Replacing it with automation didn’t just remove the bottleneck. It expanded coverage across browsers and devices while shortening the sprint cycle itself, turning testing from the constraint on delivery into the thing that no longer touches it.

The Health Lottery had testing running as an ad hoc, project-by-project cost, with quality depending on how much attention any given release happened to get. A permanently embedded testing resource turned that into a consistent, on-time capability, and freed the internal team to stop re-litigating test coverage every cycle and focus on the roadmap instead.

Where to look first

Is your regression suite growing faster than your backlog of untested paths, or the other way round?
Could you name, right now, which two systems in your stack have never been tested together?
Has your change failure rate been measured this quarter, or only after the last incident forced the question?
If a release failed during your next peak trading period, would you find out from monitoring, or from customers?

The takeaway for engineering leaders

A failed release is never one cost. It’s the velocity lost recovering from it, the change failure rate creeping upward with every incident, the customers who left without ever filing a ticket, and the revenue hit if the timing lands during a trading peak. Most of that cost is easy to underestimate individually and easy to miss entirely, because it’s absorbed quietly across releases rather than arriving as a single number anyone must explain.

Standing still on testing is a choice, even on the days it doesn’t feel like one. The alternative isn’t more caution or another sign-off gate. It’s verification, from regression testing to performance testing to deployment validation, built into delivery from the start, scaling with the roadmap instead of trailing behind it.

Our position, after 20+ years doing this work across commerce platforms under trading pressure, is straightforward. Testing that scales with delivery isn’t a cost sitting between engineering and the roadmap. It’s the fastest route to more roadmap capacity, not less, because every hour not spent recovering from a preventable incident is an hour back on the thing the business actually asked for.

For a fuller breakdown of how we approach this, PMC’s Testing and Automation Service Summary is worth a look.