Worked Case: A Profile from an Access Log

This worked case builds an operational profile for a small web shop from one day of its web server’s access log, drives a test with it, estimates the failures users would see per hour and per day, and decides whether the release meets its reliability objective. The terms (run, operation, mode) and the six steps are on Developing an Operational Profile; the test-count formula is on Random Testing Fundamentals. (Constructed example: the log was generated by a script for this page, with realistic noise; the runs against the shop are real.)

The case

The shop sells shoes and bags online. A new release is ready, and the product owner has set a reliability objective in user terms: at most about three failed user requests a day. The team has no test plan that reflects use, only feature tests. What it does have is the web server’s access log. All scripts ran on Python 3.12.9; the load run used Locust 2.47.0.

Step 1. Get the raw input

On a real system the input is the access log of the web server or load balancer (nginx, Apache, an AWS ALB), the request log of an API gateway, or page events from product analytics. Take at least one full day, so the night is in it, and preferably a week, so weekends are. For example, the shop’s log for 14 October 2026 has 55,546 lines in the combined log format:

10.0.37.43 - - [14/Oct/2026:00:00:00 +0000] "GET /health HTTP/1.1" 200 2 "-" "kube-probe/1.30"
10.0.0.157 - - [14/Oct/2026:00:00:01 +0000] "GET /products?category=shoes HTTP/1.1" 200 2018 "-" "Mozilla/5.0 ..."
10.0.20.66 - - [14/Oct/2026:00:00:03 +0000] "GET /products/605 HTTP/1.1" 200 4798 "-" "Mozilla/5.0 ..."
10.0.12.100 - - [14/Oct/2026:00:00:05 +0000] "GET /static/logo.png HTTP/1.1" 200 4096 "-" "Mozilla/5.0 ..."

Step 2. Decide what counts as a run

A log line is a request, not necessarily a run. Decision point: which lines are runs of a user-initiated operation? The criterion is Musa’s: a run is one execution of a task started from outside, and the profile is made per initiator [1]. Here the initiators are of two kinds: people using the web front end (shoppers, and staff running the sales report), and machines (the load balancer, crawlers, scanners, cron). The people share one profile, because they use the same server and the same pages; the machines are excluded. That gives four exclusions:

  • Static files (/static/...) are pulled by the browser as part of a page, so they are part of another run, not runs of their own.
  • Health checks come from the load balancer, a machine initiator; they test nothing users do. The script counts them under not a user, with bots, scanners and cron.
  • Bots and scanners (Googlebot, curl probing for /.env) are real traffic but not users. If crawler load matters, give it its own profile.
  • The nightly export (POST /internal/export, started by cron) is a system-initiated operation in a separate mode; it is tested separately.

Failed requests stay in: a 500 or a rejected login is still a run, and dropping them would hide exactly what the profile is meant to measure.

Step 3. Map requests to operations and count

Each operation is a class of runs the design treats alike, so URLs with different IDs map to one operation. The mapping is a list of method and path patterns:

import re, sys
from collections import Counter

LINE = re.compile(r'\[(?P<day>[^:]+):(?P<hour>\d\d)[^\]]*\] "(?P<method>\S+) (?P<path>[^ ?]+)\S* '
                  r'[^"]*" (?P<status>\d{3}) \S+ "[^"]*" "(?P<ua>[^"]*)"')
OPERATIONS = [                         # (operation, method, path regex); first match wins
    ("Browse catalogue", "GET", r"/products$"),
    ("View product", "GET", r"/products/\d+$"),
    ("Search", "GET", r"/search$"),
    ("Add to cart", "POST", r"/cart/items$"),
    ("View cart", "GET", r"/cart$"),
    ("Log in", "POST", r"/login$"),
    ("Check out", "POST", r"/checkout$"),
    ("Track order", "GET", r"/orders/\d+$"),
    ("Admin sales report", "GET", r"/admin/reports/sales$"),
]
NOT_USER = re.compile(r"bot|crawler|spider|kube-probe|curl|cron", re.I)

ops, skipped, per_hour = Counter(), Counter(), Counter()
for raw in open(sys.argv[1]):
    m = LINE.search(raw)
    if not m:
        skipped["unparsed"] += 1
    elif NOT_USER.search(m["ua"]):
        skipped["not a user (bot, probe, scanner, cron)"] += 1
    elif m["path"].startswith("/static/"):
        skipped["static file"] += 1
    else:
        for name, method, pattern in OPERATIONS:
            if m["method"] == method and re.match(pattern, m["path"]):
                ops[name] += 1
                per_hour[m["hour"]] += 1
                break
        else:
            skipped["unknown path " + m["path"]] += 1

total = sum(ops.values())
for name, n in ops.most_common():
    print(f"{name:<20}{n:>7}{n / total:>13.4f}")
print(f"{'Total':<20}{total:>7}")
print("Skipped:", dict(skipped))
peak_hour, peak = per_hour.most_common(1)[0]
print(f"Peak hour {peak_hour}:00 with {peak} runs; daily mean {total / 24:.0f} runs/hour")

$ python3 profile.py access.log
Browse catalogue       9996       0.3764
View product           6574       0.2475
Search                 3944       0.1485
Add to cart            2228       0.0839
View cart              1618       0.0609
Log in                 1083       0.0408
Check out               670       0.0252
Track order             358       0.0135
Admin sales report       89       0.0034
Total                 26560
Skipped: {'not a user (bot, probe, scanner, cron)': 2545, 'static file': 26441}
Peak hour 19:00 with 1920 runs; daily mean 1107 runs/hour

More than half the lines, 28,986 of 55,546 (52%), were not runs. An “unknown path” bucket that grows is the signal that the mapping has missed an operation; here it stayed empty.

Decision point: one mode or several? The criterion: a separate mode when the mix of operations differs, not merely the volume. Counting night (22:00 to 07:00, 3,360 runs) and day (23,200 runs) separately gave the same mix: no operation’s share differed by more than 0.8 percentage points (Browse catalogue 38.3% at night, 37.5% by day). So day and night share one profile; the cron export is the only separate mode.

Step 4. Write the weighted test driver

The driver draws each test run’s operation with the profile’s probability, so the test mix matches the field mix. It takes the number of runs and a random seed, sends each request to the shop over one HTTP connection, and counts server errors per operation:

# driver.py
"""Draw operations from the profile, run each against the shop, count failures."""
import http.client, json, random, sys
from collections import Counter

PROFILE = {   # operation: (probability, method, path maker)
    "Browse catalogue":   (0.3764, "GET",  lambda: f"/products?category={random.choice(['shoes', 'bags'])}"),
    "View product":       (0.2475, "GET",  lambda: f"/products/{random.randint(1, 900)}"),
    "Search":             (0.1485, "GET",  lambda: f"/search?q={random.choice(['red', 'wool'])}"),
    "Add to cart":        (0.0839, "POST", lambda: "/cart/items"),
    "View cart":          (0.0609, "GET",  lambda: "/cart"),
    "Log in":             (0.0408, "POST", lambda: "/login"),
    "Check out":          (0.0252, "POST", lambda: "/checkout"),
    "Track order":        (0.0135, "GET",  lambda: f"/orders/{random.randint(10000, 99999)}"),
    "Admin sales report": (0.0034, "GET",  lambda: "/admin/reports/sales"),
}
runs, seed = int(sys.argv[1]), int(sys.argv[2])
random.seed(seed)
names = list(PROFILE)
plan = random.choices(names, weights=[PROFILE[n][0] for n in names], k=runs)
conn = http.client.HTTPConnection("127.0.0.1", 8099)
done, failed = Counter(), Counter()
for name in plan:
    _, method, path = PROFILE[name]
    conn.request(method, path(), body=json.dumps({}) if method == "POST" else None)
    resp = conn.getresponse(); resp.read()
    done[name] += 1
    if resp.status >= 500:                 # the oracle: a server error is a failure
        failed[name] += 1
for name in names:
    print(f"{name:<20}{done[name]:>7}{failed[name]:>6}")
print(f"{'Total':<20}{runs:>7}{sum(failed.values()):>6}")

For this page the shop was a 30-line stand-in server on port 8099 (Python’s http.server) with the fault planted in its product data; on a real system the driver points at a test environment loaded with a copy of production data. With the shop running:

$ python3 driver.py 5000 1         # runs, seed
Browse catalogue       1871     0
View product           1255    23
Search                  685     0
Add to cart             449     0
View cart               319     0
Log in                  194     0
Check out               136     0
Track order              77     0
Admin sales report       14     0
Total                  5000    23

The same profile drives a load tool when the run should also measure response times. In Locust the probabilities become integer task weights (per 10,000):

# locustfile.py
import random
from locust import HttpUser, task, constant

class WebUser(HttpUser):            # one simulated human user (shopper or staff)
    wait_time = constant(0)

    @task(3764)
    def browse(self): self.client.get("/products?category=shoes", name="Browse catalogue")
    @task(2475)
    def product(self): self.client.get(f"/products/{random.randint(1, 900)}", name="View product")
    @task(1485)
    def search(self): self.client.get("/search?q=wool", name="Search")
    @task(839)
    def add(self): self.client.post("/cart/items", json={}, name="Add to cart")
    @task(609)
    def cart(self): self.client.get("/cart", name="View cart")
    @task(408)
    def login(self): self.client.post("/login", json={}, name="Log in")
    @task(252)
    def checkout(self): self.client.post("/checkout", json={}, name="Check out")
    @task(135)
    def track(self): self.client.get(f"/orders/{random.randint(10000, 99999)}", name="Track order")
    @task(34)
    def report(self): self.client.get("/admin/reports/sales", name="Admin sales report")

A 15-second headless run against the same, still unfixed shop (locust -f locustfile.py --headless -u 5 -r 5 -t 15s -H http://127.0.0.1:8099; -u 5 means five simulated users at once, -r 5 starts all five in the first second) sent 77,739 requests. Browse catalogue was 29,235 of them (37.6%, against 37.64% in the profile) and View product failed on 430 of 19,289 (2.23%). See Performance Testing for the load side.

Step 5. Estimate the failure probability and intensity

The share of failing test runs estimates the share of failing field runs, because the mixes match [2]:

  • failure probability per run: 23 / 5,000 = 0.0046;
  • failure intensity at the peak hour: 0.0046 × 1,920 = 8.8 failures per hour;
  • per day: 0.0046 × 26,560 = 122 failures, against an objective of about 3.

Decision point: release? No. All 23 failures were View product, all for product IDs above 880; products 881 to 900 had no image record, so their page crashed. Product data, not code: the fix was a fallback image. Without the profile, a feature test of one product page would have passed.

Step 6. How many runs prove the objective?

The objective is per day, the formula needs a probability per run, so divide the daily objective by the daily runs: three failures a day out of 26,560 runs is a failure probability of 3 / 26,560 = 0.00011; the team rounded the objective to F = 0.0001 per run, 2.7 a day. To claim F < 0.0001 with 90% confidence, the number of independent, failure-free runs is

N = ln(1 − 0.90) / ln(1 − 0.0001) = 23,024.7, rounded up to 23,025

which is the value in the table on Random Testing Fundamentals [3]. On the fixed version:

$ python3 driver.py 23025 2
Browse catalogue       8692     0
View product           5687     0
Search                 3371     0
Add to cart            1952     0
View cart              1428     0
Log in                  944     0
Check out               563     0
Track order             300     0
Admin sales report       88     0
Total                 23025     0

Decision

Criterion: 23,025 consecutive failure-free runs drawn from the profile on the release candidate. Met, so with 90% confidence the failure rate is below 0.0001 per run: under 2.7 failed requests a day, and under 0.2 in the peak hour. Decision: release, with one condition. Check out received only 563 of the runs and Log in 944; a failure in either loses money or locks customers out, so both get their own targeted tests (error paths, payment timeouts, empty cart) outside the profile, as Musa advises for rare critical operations [1]. Those extra tests do not count towards the 23,025, because they do not follow the profile.

The finished artifact

The profile as the team keeps it next to the driver; refresh it from the log each release.

Operation Request pattern Runs (14 Oct) Probability Runs in a 23,025 test Critical?
Browse catalogue GET /products?category= 9,996 0.3764 8,666 no
View product GET /products/{id} 6,574 0.2475 5,699 no
Search GET /search?q= 3,944 0.1485 3,419 no
Add to cart POST /cart/items 2,228 0.0839 1,931 no
View cart GET /cart 1,618 0.0609 1,403 no
Log in POST /login 1,083 0.0408 939 yes: extra tests
Check out POST /checkout 670 0.0252 581 yes: extra tests
Track order GET /orders/{id} 358 0.0135 310 no
Admin sales report GET /admin/reports/sales 89 0.0034 77 no
Total   26,560 ≈ 1.000 (rounding) 23,025  

Excluded: static files, health checks, bots, scanners; the cron export is a separate mode. Peak 1,920 runs per hour (19:00). The test-runs column is the expected number of runs, runs (14 Oct) / 26,560 × 23,025, rounded; multiplying the rounded probabilities gives numbers a few runs off, and one random draw differs again, as the fixed run shows.

Common mistakes

  1. Counting every log line as a run. Spot it: static files or /health in the operations table; here they would have been half the profile.
  2. Uniform parameters inside an operation. The driver picks product IDs uniformly, because they were uniform in this log; real shops have best-sellers. Spot it: compare the driver’s ID distribution with the log’s (GET /products/(\d+) counts) before trusting it.
  3. Converting with the daily average. Using 1,107 runs per hour instead of the peak 1,920 understates peak-hour failures by 42%. Spot it: the objective names an hour, and the rate used is not that hour’s.
  4. Claiming confidence from a run that had failures. The formula assumes N failure-free runs on the final version. Spot it: the 23,025 runs straddle a fix; restart the count after every fix.
  5. Letting the profile decide what critical operations get. Spot it: a money or safety operation whose expected runs are in the hundreds; add targeted tests outside the profile.

How solid is this?

  • Where it comes from. The procedure is Musa’s [1], [2]; the test count is Hamlet’s formula [3]. All scripts and both runs were executed for this page on 29 October 2026; the log and the shop’s fault were constructed.
  • What is contested. The confidence figure assumes runs are independent and the profile is right; a profile built from one day can miss weekly or seasonal patterns [4].
  • What we do not hold. Field data showing how far a profile built from a single day differs from a year’s use.

References

  1. J. D. Musa,
    “Operational Profiles in Software-Reliability Engineering,” IEEE Software, vol. 10, no. 2, pp. 14–32, 1993, doi: 10.1109/52.199724.
  2. J. D. Musa,
    Software Reliability Engineering: More Reliable Software Faster and Cheaper, 2nd ed. McGraw-Hill, 2004.
  3. R. Hamlet,
    “Random Testing,” in Encyclopedia of Software Engineering, Wiley, 1994.
  4. H. Koziolek,
    “Operational Profiles for Software Reliability.” Seminar paper, Seminar Dependability Engineering, Carl von Ossietzky University of Oldenburg, DFG Graduate School TrustSoft, July 2005.

Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.

This site uses Just the Docs, a documentation theme for Jekyll.