Cost of Late Fixes

The cost of a late fix is the extra effort needed to remove a defect found long after it was made, compared with removing it in the phase that made it; the classic claim is that it grows steeply with every phase the defect survives [1].

1. What the measurements show

The longer a defect survives, the more work it takes to remove: a wrong requirement found in operation means changing the requirement, the design, the code, the tests and the deployed system. Boehm’s 1981 data, as NASA’s authors reproduce them, put the cost of fixing a requirements error in operation at 40 to 1,000 times the cost of fixing it during requirements [1], [2]. Twenty years later Boehm and Basili restated the rule carefully: a problem found after delivery is often 100 times more expensive to fix than one found in requirements and design, but for small, non-critical systems the ratio is “more like 5:1 than 100:1” [3].

NASA measured the effect on its own projects. On a major aircraft programme, Haskins, Stecklein and colleagues reduced more than 3,300 change records to 231 true errors that affected the hardware, and costed the fix of a requirements error by the phase in which it was found [2]. The paper calls this its second method, the total cost breakdown: average the actual cost of the fixes in each phase and compare the averages:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "width": 460, "height": 260,
  "data": {"values": [
    {"phase": "Requirements", "i": 0, "cost": 22632},
    {"phase": "Design", "i": 1, "cost": 87832},
    {"phase": "Build", "i": 2, "cost": 354808},
    {"phase": "Test", "i": 3, "cost": 1370888},
    {"phase": "Operations", "i": 4, "cost": 3558215}
  ]},
  "encoding": {
    "x": {"field": "phase", "type": "ordinal", "sort": {"field": "i"}, "axis": {"title": "Phase in which the requirements error was found", "labelAngle": 0}},
    "y": {"field": "cost", "type": "quantitative", "scale": {"type": "log"}, "axis": {"title": "Average cost to fix (US$, log scale)", "format": "~s"}}
  },
  "layer": [
    {"mark": {"type": "line", "color": "#2D6E2A", "strokeWidth": 2}},
    {"mark": {"type": "point", "filled": true, "size": 80, "color": "#019546"}},
    {"mark": {"type": "text", "dy": -12, "fontSize": 12}, "encoding": {"text": {"field": "cost", "type": "quantitative", "format": "$,"}}}
  ],
  "config": {"font": "Tahoma, sans-serif", "axis": {"labelFontSize": 13, "titleFontSize": 13}, "view": {"stroke": null}}
}

Average cost to fix one requirements error on an aircraft programme, by the phase in which it was found; values as printed in the paper for its total cost breakdown method (method 2). On the log scale the points lie close to a straight line: the cost grows about fourfold per phase up to test, and by 2.6 times from test to operations. [2]

From requirements to operations the cost rises 3,558,215 / 22,632 ≈ 157 times. The same study found about 29 times on spacecraft, so the multiplier depends on the kind of system; the authors limit their results to hardware-software projects of this character [2].

2. Example: one requirement, three moments

A booking service must reject reservations for past dates. If a reviewer spots the missing rule while the requirement is written, the fix is one sentence. If testing finds it, the rule must be added to the specification, the validation code and the tests, and the build rerun. If customers find it, the team must also find and repair the bad bookings already stored, notify the affected users and ship a hotfix. Each step adds work that the earlier fix did not need. (Constructed example.)

When to use it, and when not

• Use the figures to argue for reviews, inspections and early tests, and to explain why a defect’s origin and the phase that found it are both worth recording (see Defect Injection and Removal).
• Do not quote “100 times” as a law or apply NASA’s multipliers to a small web service: the ratio depends on the system, and the studies say so themselves.

How solid is this?

• Where it comes from. Boehm’s 1981 table, Boehm and Basili’s 2001 summary and NASA’s costed error data from three methods on hardware-software programmes [1], [3], [2].
• What is contested. No study measures exactly “100 times”: Boehm’s range is 40 to 1,000, NASA found about 29 on spacecraft and 157 on the aircraft programme, and Boehm and Basili name 5:1 for small systems. Menzies et al. found no delayed-issue effect in 171 software projects [4]. Teach the direction, not a fixed multiplier.
• What we do not hold. Comparable costed data for modern web or mobile software delivered continuously.

---

References

1. B. W. Boehm,
   Software Engineering Economics. Englewood Cliffs, NJ: Prentice-Hall, 1981.
2. B. Haskins, J. Stecklein, B. Dick, G. Moroney, R. Lovell, and J. Dabney,
   “Error Cost Escalation Through the Project Life Cycle,” in INCOSE International Symposium, 2004, pp. 1723–1737. doi: 10.1002/j.2334-5837.2004.tb00608.x.
3. B. Boehm and V. R. Basili,
   “Software Defect Reduction Top 10 List,” Computer, vol. 34, no. 1, pp. 135–137, 2001, doi: 10.1109/2.962984.
4. T. Menzies, W. Nichols, F. Shull, and L. Layman,
   “Are Delayed Issues Harder to Resolve? Revisiting Cost-to-Fix of Defects Throughout the Lifecycle,” Empirical Software Engineering, vol. 22, no. 4, pp. 1903–1935, 2017, doi: 10.1007/s10664-016-9469-x.

---

Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.