Skip to content
Browse the docs

The knee's predicted range: fit and check (in-sample)

View on GitHubReport a problem with this page

Generated by ./gradlew :archsim-benchmarks:run --args="report --dev" from benchmarks/knee-range.json, which ./gradlew :archsim-benchmarks:run --args="range-fit" writes. The range is built as docs/validation.md describes (Protocol, step 3). Its check here uses only the four in-sample experiments, with today's harness and engine re-deriving each run's inputs and knee from its calibration (as the development checks do): the knees are not the predictions the runs made before measuring, and the model was tuned on these runs. The held-out experiments test the range out of sample.

Fitted 2026-11-01T15:31:14Z with engine 0.16.0 (commit c4bb64f579c1), on 25 calibrations and 12 counted runs of 4 experiments.

Coverage#

How many of the runs' measured knees fell inside the range, at three nominal coverages:

Nominal coverage In-sample Leaving out the run Leaving out its experiment
50% 5 of 12 5 of 12 5 of 12
80% 9 of 12 9 of 12 9 of 12
90% 10 of 12 10 of 12 10 of 12

Before W28 the same runs' ranges (their replicate calibration sets' knees, nominal 50%) held 1 of 12.

  • In-sample: fitted on every run, then checked on each.
  • Leaving out the run: each run checked with a fit that left out its calibration and its error.
  • Leaving out its experiment: each run checked with a fit that left out every calibration and error of its experiment.

Each input's run-to-run variation#

Between calibrations of the same experiment, pooled over the experiments; the band is one standard deviation.

Input Log standard deviation One standard deviation Calibrations Experiments
serviceTimeMs 0.167 x0.85 to x1.18 25 4
clientLinkLatencyMs 0.360 x0.70 to x1.43 12 2
readServiceTimeMs 0.175 x0.84 to x1.19 13 2
writeServiceTimeMs 0.178 x0.84 to x1.19 20 3
commitFlushMs 0.091 x0.91 to x1.10 20 3
dbLinkLatencyMs 0.218 x0.80 to x1.24 25 4
cacheServiceTimeMs 0.116 x0.89 to x1.12 5 1
cacheLinkLatencyMs 0.041 x0.96 to x1.04 5 1
publishTimeMs 0.105 x0.90 to x1.11 6 1
consumerTimeMs 0.104 x0.90 to x1.11 6 1

An input never seen varying takes the median, 0.141. Common correlation between two inputs of the same calibration: 0.49.

The model's own error#

Log standard deviation 0.149 (x0.86 to x1.16 for one standard deviation, x0.83 to x1.21 for the 80% range alone), by maximum likelihood over 12 runs of 4 experiments after each run's calibration spread.

Each run#

Knees in req/s; error = ln(measured / predicted); the ranges are the 80% ones, inside (yes) or not (no).

Experiment Results Knee measured Knee predicted Error Calibration spread In-sample Leaving out the run Leaving out its experiment
pg-mixed pg-mixed/run-8/ 4,160 5,000 -0.184 0.167 3,751 to 6,665 (yes) 3,737 to 6,690 (yes) 3,748 to 6,671 (yes)
pg-mixed pg-mixed/run-9/ 3,783 4,725 -0.222 0.167 3,544 to 6,299 (yes) 3,547 to 6,295 (yes) 3,541 to 6,305 (yes)
pg-mixed pg-mixed/ 4,006 4,950 -0.212 0.167 3,713 to 6,598 (yes) 3,710 to 6,604 (yes) 3,710 to 6,604 (yes)
pg-hot-row pg-hot-row/run-6/ 819 744 +0.096 0.107 588 to 941 (yes) 571 to 969 (yes) 542 to 1,022 (yes)
pg-hot-row pg-hot-row/run-7/ 960 1,062 -0.101 0.100 843 to 1,337 (yes) 823 to 1,370 (yes) 775 to 1,455 (yes)
pg-hot-row pg-hot-row/ 926 868 +0.065 0.105 687 to 1,097 (yes) 675 to 1,117 (yes) 632 to 1,191 (yes)
cache-aside cache-aside/run-3/ 7,138 5,300 +0.298 0.174 3,952 to 7,108 (no) 3,984 to 7,051 (no) 4,198 to 6,691 (no)
cache-aside cache-aside/run-4/ 6,432 4,250 +0.414 0.170 3,181 to 5,679 (no) 3,283 to 5,502 (no) 3,384 to 5,338 (no)
cache-aside cache-aside/ 7,324 4,850 +0.412 0.169 3,634 to 6,473 (no) 3,752 to 6,269 (no) 3,868 to 6,082 (no)
kafka kafka/run-3/ 2,132 2,825 -0.281 0.168 2,119 to 3,767 (yes) 2,130 to 3,746 (yes) 2,076 to 3,844 (yes)
kafka kafka/run-4/ 2,087 2,138 -0.024 0.167 1,604 to 2,850 (yes) 1,585 to 2,884 (yes) 1,572 to 2,909 (yes)
kafka kafka/ 2,028 2,100 -0.035 0.167 1,575 to 2,800 (yes) 1,557 to 2,832 (yes) 1,544 to 2,857 (yes)