Random versus Partition Testing

Partition testing divides the input domain into subdomains expected to behave alike and tests each one, as equivalence partitioning and boundary value analysis do (Input Domain Testing); random testing samples the whole domain without such a division. The question of which finds more faults per unit of effort has been studied since 1984, and the answer is “it depends, by less than you would think”.

1. The surprising first result

Duran and Ntafos simulated both strategies on programs whose subdomains had random failure rates [1]. In one simulation of 50 trials, partition testing was more likely to find a failure in 36 trials and random testing in 14. The surprise was how close random testing came: on average its probability of detection was 0.932 of partition testing’s, with no analysis at all. Because random tests are much cheaper to produce than tests chosen per subdomain, they concluded that random testing “may often be more cost effective”. Their experiments on real programs also reached high statement and branch coverage with random inputs alone.

2. Twenty percent more tests

Hamlet’s survey of the comparisons gives the rule of thumb: partition testing can have the edge for the same number of tests, but “by taking 20% more points in a random test, any advantage a partition test might have enjoyed is wiped out” [2]. The only way to give partition testing a clear win is a partition whose subdomains really are homogeneous: either every input in a subdomain fails or none does.

3. When partitioning wins

Two later results moved the balance back.

  • Cost of failure. If failures in some subdomains are far more expensive than in others, partition testing can put more tests where failures matter; random testing “has no way to tune itself to the cost factors” [3].
  • Proportional allocation. Allocating partition tests in proportion to subdomain size guarantees that partition testing is never worse than random [4], but Ntafos showed the guaranteed margin is tiny: starting from 500 tests, the relative advantage was below 0.05% and quickly became insignificant [5].

The measures behind these results, and the exact conditions, are worked out on E-measure and P-measure.

4. When only random will do

Hamlet later named two situations in which systematic partitioning is not a real option [6]:

  • Large, unstructured input domains, where no meaningful subdomains suggest themselves. Calling inputs “the same” is then subjective.
  • Programs with state. A valid test is a sequence of inputs starting from initialisation, because a state sampled directly may never occur in practice. Random sequences reach only reachable states.

Worked example

A tax function has three brackets. A simplified partition suite picks one income per bracket plus one value at each of the two boundaries: five tests. Random testing draws 100 incomes from last year’s distribution of real declarations. If the bug is a wrong rate in the top bracket, which holds 2% of declarations, the 100 random tests miss it with probability 0.98100 ≈ 13%, while the partition suite catches it every time. If the bug is a rounding error that appears for one income in a thousand, spread evenly over the incomes people actually declare and unrelated to the brackets, the five partition tests, each hitting it with probability 0.001 independently, catch it with probability about 0.5% and the 100 random tests with about 10% (1 − 0.999100); to be 95% sure, random testing needs about 3,000 tests (1 − 0.9993000 ≈ 95%). Which strategy wins depends on where the bug is, which nobody knows in advance.

Where the inputs come from. Partitions come from the specification (brackets, limits, input classes) and from the code’s branches. A random distribution worth sampling comes from real inputs, such as last year’s declarations in the example, or from the operational profile; without usage data, random testing falls back to a uniform distribution, which says nothing about reliability.

When to use each — and when not

  • Use partitions when the specification names the subdomains and the boundaries, and when some failures are much costlier than others.
  • Use random sampling for broad coverage, for stateful sequences, and when a reliability estimate is needed.
  • Do not treat them as rivals. The usual practice is both: systematic tests for the specified cases and boundaries, random tests for everything nobody thought of.

How solid is this?

  • Where it comes from. The comparison rests on simulations and analytical models [1], [2], [5], [3], with small experiments on real programs.
  • What is contested. Every model assumes known failure rates per subdomain, which a tester never has. The results say which strategy wins under those assumptions, not on a given project.

References

  1. J. W. Duran and S. C. Ntafos,
    “An Evaluation of Random Testing,” IEEE Transactions on Software Engineering, vol. SE-10, no. 4, pp. 438–444, 1984, doi: 10.1109/TSE.1984.5010257.
  2. R. Hamlet,
    “Random Testing,” in Encyclopedia of Software Engineering, Wiley, 1994.
  3. M. Z. Tsoukalas, J. W. Duran, and S. C. Ntafos,
    “On Some Reliability Estimation Problems in Random and Partition Testing,” IEEE Transactions on Software Engineering, vol. 19, no. 7, pp. 687–697, 1993, doi: 10.1109/32.238569.
  4. T. Y. Chen and Y. T. Yu,
    “On the Relationship Between Partition and Random Testing,” IEEE Transactions on Software Engineering, vol. 20, no. 12, pp. 977–980, 1994, doi: 10.1109/32.368132.
  5. S. C. Ntafos,
    “On Comparisons of Random, Partition, and Proportional Partition Testing,” IEEE Transactions on Software Engineering, vol. 27, no. 10, pp. 949–960, 2001, doi: 10.1109/32.962563.
  6. D. Hamlet,
    “When only random testing will do,” in Proceedings of the 1st international workshop on Random testing, 2006, pp. 1–9. doi: 10.1145/1145735.1145737.

Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.

This site uses Just the Docs, a documentation theme for Jekyll.