Development checks (in-sample)
Generated by ./gradlew :archsim-benchmarks:run --args="report --dev" from benchmarks/results/dev/, which ./gradlew :archsim-benchmarks:run --args="resimulate" writes. Each row is a kept benchmark run: its calibration passes are turned into model inputs again with today's harness, and its measured load steps are simulated with the engine in each column. These checks are designed after the measurements were seen, so they show what a model change does on data it may have been tuned on. They are never predictions: docs/validation.md and the /accuracy page show only one sentence from them, labelled re-simulated, on how many of the frozen engine's knees the Java engine reproduces.
| Experiment | Results | Calibrated (UTC) | Inputs as calibrated | Knee measured | Counted | Engine 0.14.0: knee (error) | Engine 0.15.0-w21: knee (error) | Engine 0.15.0-w22: knee (error) | Engine 0.15.0-w23: knee (error) | Engine 0.15.0-w24: knee (error) | Engine 0.15.0: knee (error) | Engine 0.16.0-java: knee (error) | Engine 0.16.0-port: knee (error) | Engine 0.16.0-w26: knee (error) | Engine 0.16.0-w27 with its host machine: knee (error) | Engine 0.16.0-w29: knee (error) | Engine 0.16.0: knee (error) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| java-mixed | java-mixed/run-1/ |
2026-11-03T21:34:40Z | yes | 4,198 | yes | - | - | - | - | - | - | 7,915 (+89%) | 7,915 (+89%) | - | - | - | - |
| java-mixed | java-mixed/run-2/ |
2026-11-04T00:33:23Z | yes | 4,799 | yes | - | - | - | - | - | - | 7,857 (+64%) | 7,857 (+64%) | - | - | - | - |
| java-mixed | java-mixed/ |
2026-11-04T03:38:18Z | yes | 4,650 | yes | - | - | - | - | - | - | 7,907 (+70%) | 7,907 (+70%) | - | - | - | - |
| go-mixed | go-mixed/run-1/ |
2026-11-03T22:16:39Z | yes | 5,508 | yes | - | - | - | - | - | - | 9,517 (+73%) | 9,517 (+73%) | - | - | - | - |
| go-mixed | go-mixed/run-2/ |
2026-11-04T01:13:30Z | yes | 6,204 | yes | - | - | - | - | - | - | 9,802 (+58%) | 9,802 (+58%) | - | - | - | - |
| go-mixed | go-mixed/ |
2026-11-04T04:19:22Z | yes | 5,950 | yes | - | - | - | - | - | - | 9,916 (+67%) | 9,916 (+67%) | - | - | - | - |
| python-mixed | python-mixed/run-1/ |
2026-11-03T22:58:02Z | yes | 1,428 | yes | - | - | - | - | - | - | 1,850 (+30%) | 1,850 (+30%) | - | - | - | - |
| python-mixed | python-mixed/run-2/ |
2026-11-04T01:55:30Z | yes | 1,385 | yes | - | - | - | - | - | - | 1,788 (+29%) | 1,788 (+29%) | - | - | - | - |
| python-mixed | python-mixed/ |
2026-11-04T05:00:48Z | yes | 1,388 | yes | - | - | - | - | - | - | 1,762 (+27%) | 1,762 (+27%) | - | - | - | - |
| chain | chain/run-1/ |
2026-11-03T23:27:46Z | yes | 3,503 | yes | - | - | - | - | - | - | 4,353 (+24%) | 4,353 (+24%) | - | - | - | - |
| chain | chain/run-2/ |
2026-11-04T02:28:58Z | yes | 3,475 | yes | - | - | - | - | - | - | 4,531 (+30%) | 4,531 (+30%) | - | - | - | - |
| chain | chain/ |
2026-11-04T05:34:15Z | yes | 3,388 | yes | - | - | - | - | - | - | 4,455 (+32%) | 4,455 (+32%) | - | - | - | - |
| cache-ttl | cache-ttl/run-1/ |
2026-11-04T00:02:42Z | yes | 5,747 | yes | - | - | - | - | - | - | 5,303 (-8%) | 5,303 (-8%) | - | - | - | - |
| cache-ttl | cache-ttl/run-2/ |
2026-11-04T03:05:38Z | yes | 5,336 | yes | - | - | - | - | - | - | 5,401 (+1%) | 5,401 (+1%) | - | - | - | - |
| cache-ttl | cache-ttl/ |
2026-11-04T06:11:13Z | yes | 5,420 | yes | - | - | - | - | - | - | 5,104 (-6%) | 5,104 (-6%) | - | - | - | - |
| pg-mixed | pg-mixed/run-2/ |
2026-10-26T08:23:18Z | no | 1,895 | no: disturbed | 2,314 (+22%) | 2,314 (+22%) | 2,314 (+22%) | - | 2,314 (+22%) | 2,314 (+22%) | 2,314 (+22%) | 2,314 (+22%) | 2,314 (+22%) | 2,086 (+10%) | 2,314 (+22%) | 2,314 (+22%) |
| pg-mixed | pg-mixed/run-3/ |
2026-10-26T09:46:26Z | yes | 3,162 | yes | 808 (-74%) | 808 (-74%) | 873 (-72%) | - | 808 (-74%) | 873 (-72%) | 709 (-78%) | 709 (-78%) | 873 (-72%) | 706 (-78%) | 709 (-78%) | 873 (-72%) |
| pg-mixed | pg-mixed/run-4/ |
2026-10-26T11:13:41Z | yes | 2,291 | yes | 2,064 (-10%) | 2,065 (-10%) | 2,064 (-10%) | - | 2,064 (-10%) | 2,065 (-10%) | 2,065 (-10%) | 2,065 (-10%) | 2,065 (-10%) | 1,860 (-19%) | 2,065 (-10%) | 2,065 (-10%) |
| pg-mixed | pg-mixed/run-5/ |
2026-10-28T23:21:36Z | yes | 2,430 | yes | 2,238 (-8%) | 2,239 (-8%) | 2,238 (-8%) | - | 2,238 (-8%) | 2,239 (-8%) | 2,239 (-8%) | 2,239 (-8%) | 2,239 (-8%) | 2,018 (-17%) | 2,239 (-8%) | 2,239 (-8%) |
| pg-mixed | pg-mixed/run-6/ |
2026-10-30T21:30:44Z | yes | 1,089 | no: disturbed | 4,230 (+288%) | 4,229 (+288%) | 4,230 (+288%) | - | 4,230 (+288%) | 4,229 (+288%) | 4,229 (+288%) | 4,229 (+288%) | 4,229 (+288%) | 3,810 (+250%) | 4,229 (+288%) | 4,229 (+288%) |
| pg-mixed | pg-mixed/run-7/ |
2026-10-30T23:44:23Z | yes | 1,449 | no: disturbed | 5,108 (+253%) | 5,106 (+252%) | 5,108 (+253%) | - | 5,108 (+253%) | 5,106 (+252%) | 5,106 (+252%) | 5,106 (+252%) | 5,106 (+252%) | 4,602 (+218%) | 5,106 (+252%) | 5,106 (+252%) |
| pg-mixed | pg-mixed/run-8/ |
2026-10-31T01:52:33Z | yes | 4,160 | yes | 5,003 (+20%) | 5,003 (+20%) | 5,003 (+20%) | - | 5,003 (+20%) | 5,003 (+20%) | 5,003 (+20%) | 5,003 (+20%) | 5,003 (+20%) | 4,506 (+8%) | 5,003 (+20%) | 5,003 (+20%) |
| pg-mixed | pg-mixed/run-9/ |
2026-10-31T04:02:33Z | yes | 3,783 | yes | 4,729 (+25%) | 4,728 (+25%) | 4,729 (+25%) | - | 4,729 (+25%) | 4,728 (+25%) | 4,728 (+25%) | 4,728 (+25%) | 4,728 (+25%) | 4,257 (+13%) | 4,728 (+25%) | 4,728 (+25%) |
| pg-mixed | pg-mixed/ |
2026-10-31T04:34:12Z | yes | 4,006 | yes | 4,956 (+24%) | 4,955 (+24%) | 4,956 (+24%) | - | 4,956 (+24%) | 4,955 (+24%) | 4,955 (+24%) | 4,955 (+24%) | 4,955 (+24%) | 4,464 (+11%) | 4,955 (+24%) | 4,955 (+24%) |
| pg-mixed | pg-mixed/run-11/ |
2026-11-01T22:53:59Z | yes | 4,495 | yes | - | - | - | - | - | - | 4,780 (+6%) | 4,780 (+6%) | - | - | 4,780 (+6%) | 4,780 (+6%) |
| pg-mixed | pg-mixed/run-12/ |
2026-11-02T01:02:03Z | yes | 4,364 | yes | - | - | - | - | - | - | 4,856 (+11%) | 4,856 (+11%) | - | - | 4,856 (+11%) | 4,856 (+11%) |
| pg-mixed | pg-mixed/run-13/ |
2026-11-02T03:15:28Z | no | 4,237 | yes | - | - | - | - | - | - | 5,055 (+19%) | 5,055 (+19%) | - | - | 5,055 (+19%) | 5,055 (+19%) |
| pg-mixed | pg-mixed/run-14/ |
2026-11-04T07:15:57Z | yes | 2,875 | yes | - | - | - | - | - | - | 5,406 (+88%) | 5,406 (+88%) | - | - | - | - |
| pg-mixed | pg-mixed/run-15/ |
2026-11-04T09:35:52Z | yes | 4,371 | yes | - | - | - | - | - | - | 5,059 (+16%) | 5,059 (+16%) | - | - | - | - |
| pg-mixed | pg-mixed/ |
2026-11-04T11:52:16Z | yes | 4,483 | yes | - | - | - | - | - | - | 5,203 (+16%) | 5,203 (+16%) | - | - | - | - |
| pg-hot-row | pg-hot-row/run-2/ |
2026-10-26T08:42:10Z | no | 819 | yes | 418 (-49%) | 418 (-49%) | 419 (-49%) | - | 418 (-49%) | 419 (-49%) | 419 (-49%) | 419 (-49%) | 419 (-49%) | 418 (-49%) | 419 (-49%) | 419 (-49%) |
| pg-hot-row | pg-hot-row/run-3/ |
2026-10-26T10:11:09Z | yes | 132 | no: disturbed | 504 (+281%) | 504 (+281%) | 504 (+281%) | - | 504 (+281%) | 504 (+281%) | 504 (+281%) | 504 (+281%) | 504 (+281%) | 503 (+281%) | 504 (+281%) | 504 (+281%) |
| pg-hot-row | pg-hot-row/run-4/ |
2026-10-26T11:33:38Z | yes | 225 | no: disturbed | 444 (+97%) | 444 (+97%) | 445 (+97%) | - | 444 (+97%) | 445 (+97%) | 445 (+97%) | 445 (+97%) | 445 (+97%) | 444 (+97%) | 445 (+97%) | 445 (+97%) |
| pg-hot-row | pg-hot-row/run-5/ |
2026-10-30T10:32:56Z | yes | 814 | yes | 519 (-36%) | 519 (-36%) | 519 (-36%) | - | 519 (-36%) | 519 (-36%) | 519 (-36%) | 519 (-36%) | 519 (-36%) | 519 (-36%) | 519 (-36%) | 519 (-36%) |
| pg-hot-row | pg-hot-row/run-6/ |
2026-10-30T21:58:50Z | yes | 819 | yes | 663 (-19%) | 663 (-19%) | 730 (-11%) | - | 663 (-19%) | 730 (-11%) | 730 (-11%) | 730 (-11%) | 730 (-11%) | 730 (-11%) | 730 (-11%) | 730 (-11%) |
| pg-hot-row | pg-hot-row/run-7/ |
2026-10-31T00:14:43Z | yes | 960 | yes | 958 (0%) | 958 (0%) | 1,055 (+10%) | - | 958 (0%) | 1,055 (+10%) | 1,055 (+10%) | 1,055 (+10%) | 1,055 (+10%) | 1,055 (+10%) | 1,055 (+10%) | 1,055 (+10%) |
| pg-hot-row | pg-hot-row/ |
2026-10-31T02:21:39Z | yes | 926 | yes | 783 (-16%) | 783 (-16%) | 862 (-7%) | - | 783 (-16%) | 862 (-7%) | 862 (-7%) | 862 (-7%) | 862 (-7%) | 861 (-7%) | 862 (-7%) | 862 (-7%) |
| pg-hot-row | pg-hot-row/run-9/ |
2026-11-01T23:24:56Z | no | 953 | yes | - | - | - | - | - | - | 862 (-10%) | 862 (-10%) | - | - | 862 (-10%) | 862 (-10%) |
| pg-hot-row | pg-hot-row/run-10/ |
2026-11-02T01:33:51Z | yes | 939 | yes | - | - | - | - | - | - | 851 (-9%) | 851 (-9%) | - | - | 851 (-9%) | 851 (-9%) |
| pg-hot-row | pg-hot-row/run-11/ |
2026-11-02T03:46:41Z | yes | 942 | yes | - | - | - | - | - | - | 851 (-10%) | 851 (-10%) | - | - | 851 (-10%) | 851 (-10%) |
| pg-hot-row | pg-hot-row/run-12/ |
2026-11-04T07:49:19Z | yes | 697 | no: disturbed | - | - | - | - | - | - | 826 (+19%) | 826 (+19%) | - | - | - | - |
| pg-hot-row | pg-hot-row/run-13/ |
2026-11-04T10:07:36Z | yes | 947 | yes | - | - | - | - | - | - | 863 (-9%) | 863 (-9%) | - | - | - | - |
| pg-hot-row | pg-hot-row/ |
2026-11-04T12:24:30Z | yes | 947 | yes | - | - | - | - | - | - | 857 (-10%) | 857 (-10%) | - | - | - | - |
| cache-aside | cache-aside/run-1/ |
2026-10-26T11:53:13Z | yes | 4,016 | no: disturbed | 3,457 (-14%) | 3,456 (-14%) | 3,457 (-14%) | - | 3,457 (-14%) | 3,456 (-14%) | 3,456 (-14%) | 3,456 (-14%) | 3,456 (-14%) | 3,118 (-22%) | 3,456 (-14%) | 3,456 (-14%) |
| cache-aside | cache-aside/run-2/ |
2026-10-30T10:50:27Z | yes | 4,848 | yes | 2,352 (-51%) | 2,351 (-52%) | 2,352 (-51%) | - | 2,352 (-51%) | 2,351 (-52%) | 2,351 (-52%) | 2,351 (-52%) | 2,351 (-52%) | 2,125 (-56%) | 2,351 (-52%) | 2,351 (-52%) |
| cache-aside | cache-aside/run-3/ |
2026-10-30T22:22:04Z | yes | 7,138 | yes | 5,303 (-26%) | 5,304 (-26%) | 5,303 (-26%) | - | 5,303 (-26%) | 5,304 (-26%) | 5,307 (-26%) | 5,307 (-26%) | 5,304 (-26%) | 4,772 (-33%) | 5,307 (-26%) | 5,304 (-26%) |
| cache-aside | cache-aside/run-4/ |
2026-10-31T00:38:29Z | yes | 6,432 | yes | 4,227 (-34%) | 4,227 (-34%) | 4,227 (-34%) | - | 4,227 (-34%) | 4,227 (-34%) | 4,230 (-34%) | 4,230 (-34%) | 4,227 (-34%) | 3,807 (-41%) | 4,230 (-34%) | 4,227 (-34%) |
| cache-aside | cache-aside/ |
2026-10-31T02:44:32Z | yes | 7,324 | yes | 4,851 (-34%) | 4,851 (-34%) | 4,851 (-34%) | - | 4,851 (-34%) | 4,851 (-34%) | 4,474 (-39%) | 4,474 (-39%) | 4,851 (-34%) | 4,028 (-45%) | 4,474 (-39%) | 4,851 (-34%) |
| cache-aside | cache-aside/run-6/ |
2026-11-01T23:49:28Z | no | 6,702 | yes | - | - | - | - | - | - | 4,932 (-26%) | 4,932 (-26%) | - | - | 4,932 (-26%) | 4,925 (-27%) |
| cache-aside | cache-aside/run-7/ |
2026-11-02T01:58:24Z | no | 7,010 | yes | - | - | - | - | - | - | 5,327 (-24%) | 5,327 (-24%) | - | - | 5,327 (-24%) | 5,302 (-24%) |
| cache-aside | cache-aside/run-8/ |
2026-11-02T04:11:08Z | no | 6,918 | yes | - | - | - | - | - | - | 5,220 (-25%) | 5,220 (-25%) | - | - | 5,220 (-25%) | 5,212 (-25%) |
| cache-aside | cache-aside/run-9/ |
2026-11-04T08:18:36Z | yes | 6,086 | yes | - | - | - | - | - | - | 6,007 (-1%) | 6,007 (-1%) | - | - | - | - |
| cache-aside | cache-aside/run-10/ |
2026-11-04T10:33:51Z | yes | 6,874 | yes | - | - | - | - | - | - | 5,452 (-21%) | 5,452 (-21%) | - | - | - | - |
| cache-aside | cache-aside/ |
2026-11-04T12:50:54Z | yes | 6,752 | yes | - | - | - | - | - | - | 5,652 (-16%) | 5,652 (-16%) | - | - | - | - |
| kafka | kafka/run-1/ |
2026-10-26T12:22:50Z | yes | 945 | no: disturbed | 1,006 (+6%) | 996 (+5%) | 1,006 (+6%) | 1,006 (+6%) | 1,006 (+6%) | 996 (+5%) | 996 (+5%) | 996 (+5%) | 996 (+5%) | 503 (-47%) | 996 (+5%) | 996 (+5%) |
| kafka | kafka/run-2/ |
2026-10-30T11:20:01Z | yes | 2,040 | yes | 1,602 (-21%) | 1,601 (-22%) | 1,602 (-21%) | 1,602 (-21%) | 1,602 (-21%) | 1,601 (-22%) | 1,601 (-22%) | 1,601 (-22%) | 1,601 (-22%) | 800 (-61%) | 1,601 (-22%) | 1,601 (-22%) |
| kafka | kafka/run-3/ |
2026-10-30T23:00:02Z | yes | 2,132 | yes | 2,829 (+33%) | 2,828 (+33%) | 2,829 (+33%) | 2,829 (+33%) | 2,829 (+33%) | 2,828 (+33%) | 2,828 (+33%) | 2,828 (+33%) | 2,828 (+33%) | 2,124 (0%) | 2,828 (+33%) | 2,828 (+33%) |
| kafka | kafka/run-4/ |
2026-10-31T01:19:14Z | yes | 2,087 | yes | 2,140 (+3%) | 2,139 (+3%) | 2,140 (+3%) | 2,140 (+3%) | 2,140 (+3%) | 2,139 (+3%) | 2,139 (+3%) | 2,139 (+3%) | 2,139 (+3%) | 1,609 (-23%) | 2,139 (+3%) | 2,139 (+3%) |
| kafka | kafka/ |
2026-10-31T03:26:27Z | yes | 2,028 | yes | 2,102 (+4%) | 2,101 (+4%) | 2,102 (+4%) | 2,102 (+4%) | 2,102 (+4%) | 2,101 (+4%) | 2,101 (+4%) | 2,101 (+4%) | 2,101 (+4%) | 1,580 (-22%) | 2,101 (+4%) | 2,101 (+4%) |
| kafka | kafka/run-6/ |
2026-11-02T00:26:56Z | yes | 2,614 | yes | - | - | - | - | - | - | 2,551 (-2%) | 2,551 (-2%) | - | - | 2,551 (-2%) | 2,551 (-2%) |
| kafka | kafka/run-7/ |
2026-11-02T02:36:13Z | yes | 2,659 | yes | - | - | - | - | - | - | 2,578 (-3%) | 2,578 (-3%) | - | - | 2,578 (-3%) | 2,578 (-3%) |
| kafka | kafka/run-8/ |
2026-11-02T04:48:49Z | yes | 2,802 | yes | - | - | - | - | - | - | 2,528 (-10%) | 2,528 (-10%) | - | - | 2,528 (-10%) | 2,528 (-10%) |
| kafka | kafka/run-9/ |
2026-11-04T08:59:13Z | yes | 2,266 | no: disturbed | - | - | - | - | - | - | 2,677 (+18%) | 2,677 (+18%) | - | - | - | - |
| kafka | kafka/run-10/ |
2026-11-04T11:12:22Z | yes | 2,487 | yes | - | - | - | - | - | - | 2,728 (+10%) | 2,728 (+10%) | - | - | - | - |
| kafka | kafka/ |
2026-11-04T13:27:44Z | yes | 2,387 | yes | - | - | - | - | - | - | 2,090 (-12%) | 2,090 (-12%) | - | - | - | - |
Knees in req/s; error = (simulated - measured) / measured.
-
Engine 0.14.0: median absolute knee error 20% over the latest run it checked of each experiment, 24% over all 19 counted runs.
-
Engine 0.15.0-w21: median absolute knee error 20% over the latest run it checked of each experiment, 24% over all 19 counted runs.
-
Engine 0.15.0-w22: median absolute knee error 15% over the latest run it checked of each experiment, 24% over all 19 counted runs.
-
Engine 0.15.0-w23: median absolute knee error 4% over the latest run it checked of each experiment, 13% over all 4 counted runs.
-
Engine 0.15.0-w24: median absolute knee error 20% over the latest run it checked of each experiment, 24% over all 19 counted runs.
-
Engine 0.15.0: median absolute knee error 15% over the latest run it checked of each experiment, 24% over all 19 counted runs.
-
Engine 0.16.0-java: median absolute knee error 16% over the latest run it checked of each experiment, 20% over all 56 counted runs.
-
Engine 0.16.0-port: median absolute knee error 16% over the latest run it checked of each experiment, 20% over all 56 counted runs.
-
Engine 0.16.0-w26: median absolute knee error 15% over the latest run it checked of each experiment, 24% over all 19 counted runs.
-
Engine 0.16.0-w27: median absolute knee error 17% over the latest run it checked of each experiment, 22% over all 19 counted runs.
-
Engine 0.16.0-w29: median absolute knee error 15% over the latest run it checked of each experiment, 19% over all 31 counted runs.
-
Engine 0.16.0: median absolute knee error 15% over the latest run it checked of each experiment, 19% over all 31 counted runs.
-
Engine 0.16.0-w27 ran the designs on a host machine (
BENCH_HOST=on,run.py machine_inputs): contention check 2026-11-01T00:48:59Z: 4 physical cores of 8 threads (the VM's topology); clock 4,050 MHz with 1 busy thread, 2,951 with 8 (a line through 24 probe conditions) = 0.729; a thread sharing its core at 0.48 of its speed alone (the probe's CPU per unit of work against the busy threads, 20 conditions, rms 5.5% in log); ASSUMED, not measured (the quiet service check never ran: other sessions' jobs and a throttled laptop): 0.30 threads busy at rest and k6 0.25 ms of CPU per request; other load: the background, k6.
"Inputs as calibrated: no" marks a run whose protocol derived its inputs differently from today's code (v2 had no anchoring of the client latency), so its row is today's reading of the same passes. Runs calibrated from one pass (run 1) are left out. A disturbed run missed the objective at one load step and met it at a higher one (docs/validation.md, protocol step 5): it is listed, but left out of the medians.