Performance Measurement

Performance measurement connects three fundamental quantities — throughput, response time, and concurrency — through operational laws derived from observable parameters. This page covers the mathematical foundations that underpin all performance analysis, from Little’s Law through queuing theory to scalability modeling.

Operational Analysis

Operational analysis evaluates system performance using measured parameters and established mathematical relationships, without requiring assumptions about probability distributions [1].

Observed vs Derived Parameters

Type Parameter Symbol Definition
Observed Arrivals A Total requests entering the system
Observed Completions C Total requests leaving the system
Observed Busy time B Time the resource is not idle
Observed Observation period T Duration of measurement
Derived Throughput X = C/T Completions per unit time
Derived Utilization U = B/T Fraction of time busy
Derived Service time S = B/C Average time per completion
Derived Arrival rate λ = A/T Arrivals per unit time

The key assumption is job flow balance: A ≈ C over the observation period [2].

The Six Operational Laws

Hillston formalizes six laws relating these parameters [2]:

Law Formula Meaning
Little’s Law L = X · W Avg items = throughput × avg time in system
Utilization Law Ui = X · Di Utilization = throughput × service demand
Service Demand Law Di = Si · Vi Demand = service time × visit count
Forced Flow Law Xi = X · Vi Resource throughput = system throughput × visits
Residence Time Law W = Σ Wi · Vi Total time = sum of weighted resource times
Interactive Response Time R = L/X − Z Response time = residence time minus think time

Little’s Law: L = λW

Little’s Law is the single most important equation in performance analysis [3] [4]:

“The results are remarkably free of specific assumptions about arrival and service distributions, independence of interarrival times, number of channels, queue discipline, etc.” — Little (1961) [3]

Proof Intuition (Sample Path)

The 2008 proof by Little and Graves uses a geometric argument [4]:

  1. Plot cumulative arrivals A(t) and departures D(t) over time
  2. The area between the two curves can be computed two ways:
    • Integrating the number of items over time: L × T
    • Summing individual wait times: N × W
  3. Since both describe the same area: L = (N/T) × W = λW

Practical Examples

System Known Values Derived
Wine cellar L=160 bottles, λ=96/yr W = 160/96 = 1.67 years
Semiconductor fab λ=1000 wafers/day, L=45000 WIP W = 45000/1000 = 45 days
Hospital ward λ=5 births/day, W=2.5 days L = 5×2.5 = 12.5 beds

Non-Normal Distributions: Why Percentiles Beat Averages

Performance data is fundamentally non-normal with long right tails [1] [5]:

“Operational analysis leverages statistical means, or averages. This provides both strength and weakness. The weakness comes in the form of loss of information and the fact that means are influenced by outliers.” — Wilson (2008) [1]

The Problem with Averages

A system with mean response time of 1 second could mean:

  • Good: All requests between 0.8s and 1.2s (low variance)
  • Terrible: 90% at 0.5s, 10% at 5.5s (high variance, same mean)

Jain warns: “An average response time of 1 second with a range of 0.5 to 1.5 is very different from an average of 1 second with a range of 0.1 to 10” [6].

Percentile Guidelines

Metric Use Case
p50 (median) Typical user experience
p90 “What most users actually perceive” [1]
p99 Tail latency — important for SLAs
p99.9 Critical for large-scale systems [7]

Wilson recommends the 90th percentile over the mean as the primary reporting metric, and identifies practical thresholds [1]:

Resource Bottleneck Threshold
CPU utilization >70% average or queue >2/processor
Disk utilization >20%
Performance change <10% may be noise, not real improvement

The Rule of 8

Human productivity drops dramatically when system response times exceed 8 seconds [1] [8]. Users lose their train of thought, context-switch to other tasks, or abandon the interaction entirely.

Queuing Theory: The Hockey Stick Curve

When utilization increases, response time follows a characteristic hockey stick shape — flat at low load, then climbing exponentially [9]:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "title": {"text": "Response Time vs Utilization (M/M/1)", "fontSize": 14},
  "width": 450,
  "height": 280,
  "data": {
    "values": [
      {"u": 0.05, "rt": 1.05}, {"u": 0.10, "rt": 1.11}, {"u": 0.15, "rt": 1.18},
      {"u": 0.20, "rt": 1.25}, {"u": 0.25, "rt": 1.33}, {"u": 0.30, "rt": 1.43},
      {"u": 0.35, "rt": 1.54}, {"u": 0.40, "rt": 1.67}, {"u": 0.45, "rt": 1.82},
      {"u": 0.50, "rt": 2.00}, {"u": 0.55, "rt": 2.22}, {"u": 0.60, "rt": 2.50},
      {"u": 0.65, "rt": 2.86}, {"u": 0.70, "rt": 3.33}, {"u": 0.75, "rt": 4.00},
      {"u": 0.80, "rt": 5.00}, {"u": 0.85, "rt": 6.67}, {"u": 0.90, "rt": 10.0},
      {"u": 0.95, "rt": 20.0}
    ]
  },
  "layer": [
    {
      "mark": {"type": "line", "strokeWidth": 3, "interpolate": "monotone"},
      "encoding": {
        "x": {"field": "u", "type": "quantitative", "title": "Utilization (%)", "scale": {"domain": [0, 1]}, "axis": {"format": ".0%"}},
        "y": {"field": "rt", "type": "quantitative", "title": "Response Time (× service time)", "scale": {"domain": [0, 22]}},
        "color": {"value": "#d32f2f"}
      }
    },
    {
      "data": {"values": [{"u": 0.80, "rt": 5.0}]},
      "mark": {"type": "point", "size": 120, "filled": true},
      "encoding": {
        "x": {"field": "u", "type": "quantitative"},
        "y": {"field": "rt", "type": "quantitative"},
        "color": {"value": "#FF9800"}
      }
    },
    {
      "data": {"values": [{"u": 0.82, "rt": 7.0, "label": "← 80% \"elbow\""}]},
      "mark": {"type": "text", "fontSize": 12, "align": "left", "fontWeight": "bold"},
      "encoding": {
        "x": {"field": "u", "type": "quantitative"},
        "y": {"field": "rt", "type": "quantitative"},
        "text": {"field": "label"},
        "color": {"value": "#FF9800"}
      }
    }
  ],
  "config": {"font": "Tahoma, sans-serif", "view": {"stroke": null}}
}

The formula for M/M/1 response time: R = S / (1 − U) where S is service time and U is utilization [9].

The Three-Zone Conceptual Model

Haines (2006) presents a complementary conceptual view showing three metrics simultaneously [10]:

Zone Utilization (U) Throughput (X) Response Time (R)
Light Load Rising linearly Rising linearly Flat, near-minimal
Heavy Load Saturated (~100%) Plateau (max throughput) Starting to climb
Buckle Zone Stays at 100% Falling (resource thrashing) Explodes exponentially

The boundaries shift right with faster/more CPUs. The mathematical M/M/1 chart above shows the R curve in isolation; the three-zone model shows how all three metrics interact under increasing load.

Key Queuing Insights

Concept Implication
The “elbow” ~80% utilization — response time degrades sharply [9]
1×M/M/4 vs 4×M/M/1 Single queue, multiple servers always outperforms multiple independent queues [9]
Faster CPUs Shift curve down and to the right
More CPUs Shift curve to the right only
More arrivals Slide along the existing curve (no shift)

Kendall’s Notation

Queuing systems are described as A/B/m where A = arrival distribution, B = service distribution, m = number of servers [9]:

Symbol Meaning
M Markovian (exponential, memoryless)
D Deterministic (constant)
G General (any distribution)

Example: M/M/4 = Poisson arrivals, exponential service, 4 servers.

Bottleneck Analysis

The bottleneck resource has the greatest service demand Dmax = max{Di} [2]. It limits the entire system because it reaches 100% utilization first.

Asymptotic Bounds

Bound Formula Interpretation
Throughput upper bound X ≤ min{1/Dmax, N/(D+Z)} Can’t exceed bottleneck capacity
Response time lower bound R ≥ max{D, N·Dmax − Z} Can’t be faster than total service demand

The balanced system principle: maximum scalable performance is achieved when all resources have equal utilization [11]. An unbalanced system wastes capacity on non-bottleneck resources.

Scalability Laws

Three laws form a progression of realism for modeling how systems scale:

Amdahl’s Law (1967)

S(N) = 1 / (f + (1−f)/N) [12]

Where f is the serial fraction of work. The ceiling is 1/f — with 40% serial overhead (α≈0.4), maximum speedup is only 2.5× regardless of how many processors you add [12].

Gustafson’s Law (1988)

S = N + (1−N) × s [13]

Gustafson’s key insight: real users scale problem size with processor count, not problem time. On 1024 processors, Gustafson measured 1016–1021× speedup — near-linear scaling because the parallel portion grew with capacity [13].

Universal Scalability Law (2002)

S(N) = N / [1 + α(N−1) + βN(N−1)] [14]

The USL extends Amdahl by adding coherence costs:

Parameter Physical Meaning Example
α (contention) Waiting for shared resources Database locks, bus contention
β (coherence) Maintaining global consistency Cache invalidation, distributed sync

When β=0, USL reduces to Amdahl’s Law. When β>0, the system degrades beyond a peak — retrograde throughput [14]. This is critical for distributed systems where coherence costs grow quadratically.

graph LR
    A["Amdahl (1967)<br>Serial ceiling: 1/f<br>Plateau only"]
    G["Gustafson (1988)<br>Scaled speedup<br>Near-linear possible"]
    U["USL (2002)<br>Contention + Coherence<br>Predicts degradation"]
    A -->|"Fixed problem"| G
    A -->|"Add coherence"| U
    style A fill:#4CAF50,color:#fff
    style G fill:#2196F3,color:#fff
    style U fill:#d32f2f,color:#fff

Practical USL Application

Gunther shows that as few as 4 load test data points suffice to determine α and β via regression and predict the full scalability curve [14]. The capacity doubling time is Tdouble = ln(2)/λ [15].

Model Validation

Jain provides acceptance criteria for analytical model accuracy [6]:

Prediction Target Acceptable Error
Resource utilization Within 10%
System throughput Within 10%
Response time Within 30%

The higher tolerance for response time reflects the compounding of errors from multiple resource predictions. Wilson adds: changes of less than 10% may represent intrinsic fluctuation rather than real improvement [1].

References

  1. T. Wilson, “An Operational Analysis Primer.” Tutorial, 2008.
  2. J. Hillston, “Operational Laws.” Performance Modelling Lecture Notes, University of Edinburgh, 2014.
  3. J. D. C. Little, “A Proof for the Queuing Formula: L = λW,” Operations Research, vol. 9, no. 3, pp. 383–387, 1961, doi: 10.1287/opre.9.3.383.
  4. J. D. C. Little and S. C. Graves, “Little’s Law,” in Building Intuition: Insights from Basic Operations Management Models and Principles, Springer, 2008, pp. 81–100. doi: 10.1007/978-0-387-73699-0_5.
  5. G. Dobson and R. Shumsky, “Web-based Simulations for Teaching Queueing, Little’s Law, and Inventory Management,” INFORMS Transactions on Education, 2012.
  6. R. Jain, The Art of Computer Systems Performance Analysis. Wiley, 1991.
  7. J. Dean and L. A. Barroso, “The Tail at Scale,” Communications of the ACM, vol. 56, no. 2, pp. 74–80, 2013, doi: 10.1145/2408776.2408794.
  8. G. D. Everett and R. McLeod, “Performance Testing,” in Software Testing: Testing Across the Entire Software Development Life Cycle, Wiley, 2007.
  9. C. Shallahamer, “Practical Queuing Theory,” in Forecasting Oracle Performance, Apress, 2007.
  10. S. Haines, Pro Java EE 5 Performance Management and Optimization. Apress, 2006.
  11. H. H. Liu, Software Performance and Scalability: A Quantitative Approach. Wiley, 2009. doi: 10.1002/9780470465394.
  12. G. M. Amdahl, “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities,” in Proceedings of the AFIPS Spring Joint Computer Conference, 1967, pp. 483–485. doi: 10.1145/1465482.1465560.
  13. J. L. Gustafson, “Reevaluating Amdahl’s Law,” Communications of the ACM, vol. 31, no. 5, pp. 532–533, 1988, doi: 10.1145/42411.42415.
  14. N. J. Gunther, “Hit-and-Run Tactics Enable Guerrilla Capacity Planning,” IT Professional, vol. 4, no. 4, pp. 44–47, 2002, doi: 10.1109/MITP.2002.1046643.
  15. N. J. Gunther, Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services. Springer, 2007.

Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.

Table of contents

This site uses Just the Docs, a documentation theme for Jekyll.