The knee's predicted range: fit and check (in-sample)
Generated by ./gradlew :archsim-benchmarks:run --args="report --dev" from benchmarks/knee-range.json, which ./gradlew :archsim-benchmarks:run --args="range-fit" writes. The range is built as docs/validation.md describes (Protocol, step 3). Its check here uses only the four in-sample experiments, with today's harness and engine re-deriving each run's inputs and knee from its calibration (as the development checks do): the knees are not the predictions the runs made before measuring, and the model was tuned on these runs. The held-out experiments test the range out of sample.
Fitted 2026-11-01T15:31:14Z with engine 0.16.0 (commit c4bb64f579c1), on 25 calibrations and 12 counted runs of 4 experiments.
Coverage#
How many of the runs' measured knees fell inside the range, at three nominal coverages:
| Nominal coverage | In-sample | Leaving out the run | Leaving out its experiment |
|---|---|---|---|
| 50% | 5 of 12 | 5 of 12 | 5 of 12 |
| 80% | 9 of 12 | 9 of 12 | 9 of 12 |
| 90% | 10 of 12 | 10 of 12 | 10 of 12 |
Before W28 the same runs' ranges (their replicate calibration sets' knees, nominal 50%) held 1 of 12.
- In-sample: fitted on every run, then checked on each.
- Leaving out the run: each run checked with a fit that left out its calibration and its error.
- Leaving out its experiment: each run checked with a fit that left out every calibration and error of its experiment.
Each input's run-to-run variation#
Between calibrations of the same experiment, pooled over the experiments; the band is one standard deviation.
| Input | Log standard deviation | One standard deviation | Calibrations | Experiments |
|---|---|---|---|---|
serviceTimeMs |
0.167 | x0.85 to x1.18 | 25 | 4 |
clientLinkLatencyMs |
0.360 | x0.70 to x1.43 | 12 | 2 |
readServiceTimeMs |
0.175 | x0.84 to x1.19 | 13 | 2 |
writeServiceTimeMs |
0.178 | x0.84 to x1.19 | 20 | 3 |
commitFlushMs |
0.091 | x0.91 to x1.10 | 20 | 3 |
dbLinkLatencyMs |
0.218 | x0.80 to x1.24 | 25 | 4 |
cacheServiceTimeMs |
0.116 | x0.89 to x1.12 | 5 | 1 |
cacheLinkLatencyMs |
0.041 | x0.96 to x1.04 | 5 | 1 |
publishTimeMs |
0.105 | x0.90 to x1.11 | 6 | 1 |
consumerTimeMs |
0.104 | x0.90 to x1.11 | 6 | 1 |
An input never seen varying takes the median, 0.141. Common correlation between two inputs of the same calibration: 0.49.
The model's own error#
Log standard deviation 0.149 (x0.86 to x1.16 for one standard deviation, x0.83 to x1.21 for the 80% range alone), by maximum likelihood over 12 runs of 4 experiments after each run's calibration spread.
Each run#
Knees in req/s; error = ln(measured / predicted); the ranges are the 80% ones, inside (yes) or not (no).
| Experiment | Results | Knee measured | Knee predicted | Error | Calibration spread | In-sample | Leaving out the run | Leaving out its experiment |
|---|---|---|---|---|---|---|---|---|
| pg-mixed | pg-mixed/run-8/ |
4,160 | 5,000 | -0.184 | 0.167 | 3,751 to 6,665 (yes) | 3,737 to 6,690 (yes) | 3,748 to 6,671 (yes) |
| pg-mixed | pg-mixed/run-9/ |
3,783 | 4,725 | -0.222 | 0.167 | 3,544 to 6,299 (yes) | 3,547 to 6,295 (yes) | 3,541 to 6,305 (yes) |
| pg-mixed | pg-mixed/ |
4,006 | 4,950 | -0.212 | 0.167 | 3,713 to 6,598 (yes) | 3,710 to 6,604 (yes) | 3,710 to 6,604 (yes) |
| pg-hot-row | pg-hot-row/run-6/ |
819 | 744 | +0.096 | 0.107 | 588 to 941 (yes) | 571 to 969 (yes) | 542 to 1,022 (yes) |
| pg-hot-row | pg-hot-row/run-7/ |
960 | 1,062 | -0.101 | 0.100 | 843 to 1,337 (yes) | 823 to 1,370 (yes) | 775 to 1,455 (yes) |
| pg-hot-row | pg-hot-row/ |
926 | 868 | +0.065 | 0.105 | 687 to 1,097 (yes) | 675 to 1,117 (yes) | 632 to 1,191 (yes) |
| cache-aside | cache-aside/run-3/ |
7,138 | 5,300 | +0.298 | 0.174 | 3,952 to 7,108 (no) | 3,984 to 7,051 (no) | 4,198 to 6,691 (no) |
| cache-aside | cache-aside/run-4/ |
6,432 | 4,250 | +0.414 | 0.170 | 3,181 to 5,679 (no) | 3,283 to 5,502 (no) | 3,384 to 5,338 (no) |
| cache-aside | cache-aside/ |
7,324 | 4,850 | +0.412 | 0.169 | 3,634 to 6,473 (no) | 3,752 to 6,269 (no) | 3,868 to 6,082 (no) |
| kafka | kafka/run-3/ |
2,132 | 2,825 | -0.281 | 0.168 | 2,119 to 3,767 (yes) | 2,130 to 3,746 (yes) | 2,076 to 3,844 (yes) |
| kafka | kafka/run-4/ |
2,087 | 2,138 | -0.024 | 0.167 | 1,604 to 2,850 (yes) | 1,585 to 2,884 (yes) | 1,572 to 2,909 (yes) |
| kafka | kafka/ |
2,028 | 2,100 | -0.035 | 0.167 | 1,575 to 2,800 (yes) | 1,557 to 2,832 (yes) | 1,544 to 2,857 (yes) |