Defect Seeding
Defect seeding (also fault injection or error seeding) estimates how many real defects a product contains by planting a known number of artificial defects, running the inspection or test as usual, and assuming the real defects were found at the same rate as the planted ones [1]. Harlan Mills proposed it at IBM in 1972.
1. The estimate
The method rests on one ratio: if inspectors found a fraction FP / P of the planted defects, they probably found the same fraction of the real ones. With
- P the number of defects planted,
- FP the number of planted defects found,
- FR the number of real (original) defects found, counting every logged real find, including those first raised in the meeting, because the ratio uses all finds (capture-recapture leaves them out),
the estimated number of real defects is
N = FR × P / FP.
2. Worked example
A moderator plants 12 major defects in a 40-page design document before an inspection. After minor issues are dropped, the team’s log holds 15 major defects; checking against the moderator’s list shows that 4 of them are planted and 11 are real.
| Quantity | Value |
|---|---|
| Defects planted, P | 12 |
| Planted defects found, FP | 4 |
| Real defects found, FR | 11 |
| Estimated real defects, N = 11 × 12 / 4 | 33 |
| Estimated real defects still in the document, 33 − 11 | 22 |
The inspection found a third of what was planted (4 of 12), so the estimate assumes it also found a third of the real defects. Two thirds are still in the document. On 40 pages, 22 / 40 = 0.55 major defects per page remain, more than twice an exit criterion of 0.25 per page [2]. The product should not pass this inspection.
Where the inputs come from. The planted defects are written by someone outside the inspection team, best modelled on the organisation’s own defect history so that they are as hard to find as real ones. The counts of planted and real defects found come from the inspection’s defect log, with each entry marked planted or real after the meeting.
When to use it — and when not
Use seeding to measure how effective a team or a reading technique is (in experiments, in training, or to calibrate a new review process), and when too few independent inspectors are available for capture-recapture. Do not use it when the planted defects would be noticeably different from real ones: the estimate is only as good as that similarity.
Limitations
- Planted defects must be as hard to find as real ones. Artificial defects tend to be simpler and more visible; then FP / P is too high and N too low.
- Planting and removing cost effort, and every seeded defect must be removed afterwards; one left behind is a real defect.
- Inspectors who know about seeding behave differently, for example hunting for planted-looking defects.
- Few planted finds make the estimate unstable. With 3 planted defects found instead of 4, the same data give N = 11 × 12 / 3 = 44.
How solid is this?
- Where it comes from. Mills’ IBM report introduced the idea for statistical validation of programs [1].
- What is contested. Whether seeded defects can ever be representative; for inspections the usual alternative, which needs no planting, is capture-recapture.
- What we do not hold. The area’s literature review has no empirical study of seeding accuracy in inspections.
References
- H. D. Mills,“On the Statistical Validation of Computer Programs,” IBM Federal Systems Division, Gaithersburg, MD, FSC-72-6015, 1972.
- N. Malotaux,“One-Page Inspection.” 2008.
Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.