Skip to content
Browse the docs

Development checks (in-sample)

View on GitHubReport a problem with this page

Generated by ./gradlew :archsim-benchmarks:run --args="report --dev" from benchmarks/results/dev/, which ./gradlew :archsim-benchmarks:run --args="resimulate" writes. Each row is a kept benchmark run: its calibration passes are turned into model inputs again with today's harness, and its measured load steps are simulated with the engine in each column. These checks are designed after the measurements were seen, so they show what a model change does on data it may have been tuned on. They are never predictions: docs/validation.md and the /accuracy page show only one sentence from them, labelled re-simulated, on how many of the frozen engine's knees the Java engine reproduces.

Experiment Results Calibrated (UTC) Inputs as calibrated Knee measured Counted Engine 0.14.0: knee (error) Engine 0.15.0-w21: knee (error) Engine 0.15.0-w22: knee (error) Engine 0.15.0-w23: knee (error) Engine 0.15.0-w24: knee (error) Engine 0.15.0: knee (error) Engine 0.16.0-java: knee (error) Engine 0.16.0-port: knee (error) Engine 0.16.0-w26: knee (error) Engine 0.16.0-w27 with its host machine: knee (error) Engine 0.16.0-w29: knee (error) Engine 0.16.0: knee (error)
java-mixed java-mixed/run-1/ 2026-11-03T21:34:40Z yes 4,198 yes - - - - - - 7,915 (+89%) 7,915 (+89%) - - - -
java-mixed java-mixed/run-2/ 2026-11-04T00:33:23Z yes 4,799 yes - - - - - - 7,857 (+64%) 7,857 (+64%) - - - -
java-mixed java-mixed/ 2026-11-04T03:38:18Z yes 4,650 yes - - - - - - 7,907 (+70%) 7,907 (+70%) - - - -
go-mixed go-mixed/run-1/ 2026-11-03T22:16:39Z yes 5,508 yes - - - - - - 9,517 (+73%) 9,517 (+73%) - - - -
go-mixed go-mixed/run-2/ 2026-11-04T01:13:30Z yes 6,204 yes - - - - - - 9,802 (+58%) 9,802 (+58%) - - - -
go-mixed go-mixed/ 2026-11-04T04:19:22Z yes 5,950 yes - - - - - - 9,916 (+67%) 9,916 (+67%) - - - -
python-mixed python-mixed/run-1/ 2026-11-03T22:58:02Z yes 1,428 yes - - - - - - 1,850 (+30%) 1,850 (+30%) - - - -
python-mixed python-mixed/run-2/ 2026-11-04T01:55:30Z yes 1,385 yes - - - - - - 1,788 (+29%) 1,788 (+29%) - - - -
python-mixed python-mixed/ 2026-11-04T05:00:48Z yes 1,388 yes - - - - - - 1,762 (+27%) 1,762 (+27%) - - - -
chain chain/run-1/ 2026-11-03T23:27:46Z yes 3,503 yes - - - - - - 4,353 (+24%) 4,353 (+24%) - - - -
chain chain/run-2/ 2026-11-04T02:28:58Z yes 3,475 yes - - - - - - 4,531 (+30%) 4,531 (+30%) - - - -
chain chain/ 2026-11-04T05:34:15Z yes 3,388 yes - - - - - - 4,455 (+32%) 4,455 (+32%) - - - -
cache-ttl cache-ttl/run-1/ 2026-11-04T00:02:42Z yes 5,747 yes - - - - - - 5,303 (-8%) 5,303 (-8%) - - - -
cache-ttl cache-ttl/run-2/ 2026-11-04T03:05:38Z yes 5,336 yes - - - - - - 5,401 (+1%) 5,401 (+1%) - - - -
cache-ttl cache-ttl/ 2026-11-04T06:11:13Z yes 5,420 yes - - - - - - 5,104 (-6%) 5,104 (-6%) - - - -
pg-mixed pg-mixed/run-2/ 2026-10-26T08:23:18Z no 1,895 no: disturbed 2,314 (+22%) 2,314 (+22%) 2,314 (+22%) - 2,314 (+22%) 2,314 (+22%) 2,314 (+22%) 2,314 (+22%) 2,314 (+22%) 2,086 (+10%) 2,314 (+22%) 2,314 (+22%)
pg-mixed pg-mixed/run-3/ 2026-10-26T09:46:26Z yes 3,162 yes 808 (-74%) 808 (-74%) 873 (-72%) - 808 (-74%) 873 (-72%) 709 (-78%) 709 (-78%) 873 (-72%) 706 (-78%) 709 (-78%) 873 (-72%)
pg-mixed pg-mixed/run-4/ 2026-10-26T11:13:41Z yes 2,291 yes 2,064 (-10%) 2,065 (-10%) 2,064 (-10%) - 2,064 (-10%) 2,065 (-10%) 2,065 (-10%) 2,065 (-10%) 2,065 (-10%) 1,860 (-19%) 2,065 (-10%) 2,065 (-10%)
pg-mixed pg-mixed/run-5/ 2026-10-28T23:21:36Z yes 2,430 yes 2,238 (-8%) 2,239 (-8%) 2,238 (-8%) - 2,238 (-8%) 2,239 (-8%) 2,239 (-8%) 2,239 (-8%) 2,239 (-8%) 2,018 (-17%) 2,239 (-8%) 2,239 (-8%)
pg-mixed pg-mixed/run-6/ 2026-10-30T21:30:44Z yes 1,089 no: disturbed 4,230 (+288%) 4,229 (+288%) 4,230 (+288%) - 4,230 (+288%) 4,229 (+288%) 4,229 (+288%) 4,229 (+288%) 4,229 (+288%) 3,810 (+250%) 4,229 (+288%) 4,229 (+288%)
pg-mixed pg-mixed/run-7/ 2026-10-30T23:44:23Z yes 1,449 no: disturbed 5,108 (+253%) 5,106 (+252%) 5,108 (+253%) - 5,108 (+253%) 5,106 (+252%) 5,106 (+252%) 5,106 (+252%) 5,106 (+252%) 4,602 (+218%) 5,106 (+252%) 5,106 (+252%)
pg-mixed pg-mixed/run-8/ 2026-10-31T01:52:33Z yes 4,160 yes 5,003 (+20%) 5,003 (+20%) 5,003 (+20%) - 5,003 (+20%) 5,003 (+20%) 5,003 (+20%) 5,003 (+20%) 5,003 (+20%) 4,506 (+8%) 5,003 (+20%) 5,003 (+20%)
pg-mixed pg-mixed/run-9/ 2026-10-31T04:02:33Z yes 3,783 yes 4,729 (+25%) 4,728 (+25%) 4,729 (+25%) - 4,729 (+25%) 4,728 (+25%) 4,728 (+25%) 4,728 (+25%) 4,728 (+25%) 4,257 (+13%) 4,728 (+25%) 4,728 (+25%)
pg-mixed pg-mixed/ 2026-10-31T04:34:12Z yes 4,006 yes 4,956 (+24%) 4,955 (+24%) 4,956 (+24%) - 4,956 (+24%) 4,955 (+24%) 4,955 (+24%) 4,955 (+24%) 4,955 (+24%) 4,464 (+11%) 4,955 (+24%) 4,955 (+24%)
pg-mixed pg-mixed/run-11/ 2026-11-01T22:53:59Z yes 4,495 yes - - - - - - 4,780 (+6%) 4,780 (+6%) - - 4,780 (+6%) 4,780 (+6%)
pg-mixed pg-mixed/run-12/ 2026-11-02T01:02:03Z yes 4,364 yes - - - - - - 4,856 (+11%) 4,856 (+11%) - - 4,856 (+11%) 4,856 (+11%)
pg-mixed pg-mixed/run-13/ 2026-11-02T03:15:28Z no 4,237 yes - - - - - - 5,055 (+19%) 5,055 (+19%) - - 5,055 (+19%) 5,055 (+19%)
pg-mixed pg-mixed/run-14/ 2026-11-04T07:15:57Z yes 2,875 yes - - - - - - 5,406 (+88%) 5,406 (+88%) - - - -
pg-mixed pg-mixed/run-15/ 2026-11-04T09:35:52Z yes 4,371 yes - - - - - - 5,059 (+16%) 5,059 (+16%) - - - -
pg-mixed pg-mixed/ 2026-11-04T11:52:16Z yes 4,483 yes - - - - - - 5,203 (+16%) 5,203 (+16%) - - - -
pg-hot-row pg-hot-row/run-2/ 2026-10-26T08:42:10Z no 819 yes 418 (-49%) 418 (-49%) 419 (-49%) - 418 (-49%) 419 (-49%) 419 (-49%) 419 (-49%) 419 (-49%) 418 (-49%) 419 (-49%) 419 (-49%)
pg-hot-row pg-hot-row/run-3/ 2026-10-26T10:11:09Z yes 132 no: disturbed 504 (+281%) 504 (+281%) 504 (+281%) - 504 (+281%) 504 (+281%) 504 (+281%) 504 (+281%) 504 (+281%) 503 (+281%) 504 (+281%) 504 (+281%)
pg-hot-row pg-hot-row/run-4/ 2026-10-26T11:33:38Z yes 225 no: disturbed 444 (+97%) 444 (+97%) 445 (+97%) - 444 (+97%) 445 (+97%) 445 (+97%) 445 (+97%) 445 (+97%) 444 (+97%) 445 (+97%) 445 (+97%)
pg-hot-row pg-hot-row/run-5/ 2026-10-30T10:32:56Z yes 814 yes 519 (-36%) 519 (-36%) 519 (-36%) - 519 (-36%) 519 (-36%) 519 (-36%) 519 (-36%) 519 (-36%) 519 (-36%) 519 (-36%) 519 (-36%)
pg-hot-row pg-hot-row/run-6/ 2026-10-30T21:58:50Z yes 819 yes 663 (-19%) 663 (-19%) 730 (-11%) - 663 (-19%) 730 (-11%) 730 (-11%) 730 (-11%) 730 (-11%) 730 (-11%) 730 (-11%) 730 (-11%)
pg-hot-row pg-hot-row/run-7/ 2026-10-31T00:14:43Z yes 960 yes 958 (0%) 958 (0%) 1,055 (+10%) - 958 (0%) 1,055 (+10%) 1,055 (+10%) 1,055 (+10%) 1,055 (+10%) 1,055 (+10%) 1,055 (+10%) 1,055 (+10%)
pg-hot-row pg-hot-row/ 2026-10-31T02:21:39Z yes 926 yes 783 (-16%) 783 (-16%) 862 (-7%) - 783 (-16%) 862 (-7%) 862 (-7%) 862 (-7%) 862 (-7%) 861 (-7%) 862 (-7%) 862 (-7%)
pg-hot-row pg-hot-row/run-9/ 2026-11-01T23:24:56Z no 953 yes - - - - - - 862 (-10%) 862 (-10%) - - 862 (-10%) 862 (-10%)
pg-hot-row pg-hot-row/run-10/ 2026-11-02T01:33:51Z yes 939 yes - - - - - - 851 (-9%) 851 (-9%) - - 851 (-9%) 851 (-9%)
pg-hot-row pg-hot-row/run-11/ 2026-11-02T03:46:41Z yes 942 yes - - - - - - 851 (-10%) 851 (-10%) - - 851 (-10%) 851 (-10%)
pg-hot-row pg-hot-row/run-12/ 2026-11-04T07:49:19Z yes 697 no: disturbed - - - - - - 826 (+19%) 826 (+19%) - - - -
pg-hot-row pg-hot-row/run-13/ 2026-11-04T10:07:36Z yes 947 yes - - - - - - 863 (-9%) 863 (-9%) - - - -
pg-hot-row pg-hot-row/ 2026-11-04T12:24:30Z yes 947 yes - - - - - - 857 (-10%) 857 (-10%) - - - -
cache-aside cache-aside/run-1/ 2026-10-26T11:53:13Z yes 4,016 no: disturbed 3,457 (-14%) 3,456 (-14%) 3,457 (-14%) - 3,457 (-14%) 3,456 (-14%) 3,456 (-14%) 3,456 (-14%) 3,456 (-14%) 3,118 (-22%) 3,456 (-14%) 3,456 (-14%)
cache-aside cache-aside/run-2/ 2026-10-30T10:50:27Z yes 4,848 yes 2,352 (-51%) 2,351 (-52%) 2,352 (-51%) - 2,352 (-51%) 2,351 (-52%) 2,351 (-52%) 2,351 (-52%) 2,351 (-52%) 2,125 (-56%) 2,351 (-52%) 2,351 (-52%)
cache-aside cache-aside/run-3/ 2026-10-30T22:22:04Z yes 7,138 yes 5,303 (-26%) 5,304 (-26%) 5,303 (-26%) - 5,303 (-26%) 5,304 (-26%) 5,307 (-26%) 5,307 (-26%) 5,304 (-26%) 4,772 (-33%) 5,307 (-26%) 5,304 (-26%)
cache-aside cache-aside/run-4/ 2026-10-31T00:38:29Z yes 6,432 yes 4,227 (-34%) 4,227 (-34%) 4,227 (-34%) - 4,227 (-34%) 4,227 (-34%) 4,230 (-34%) 4,230 (-34%) 4,227 (-34%) 3,807 (-41%) 4,230 (-34%) 4,227 (-34%)
cache-aside cache-aside/ 2026-10-31T02:44:32Z yes 7,324 yes 4,851 (-34%) 4,851 (-34%) 4,851 (-34%) - 4,851 (-34%) 4,851 (-34%) 4,474 (-39%) 4,474 (-39%) 4,851 (-34%) 4,028 (-45%) 4,474 (-39%) 4,851 (-34%)
cache-aside cache-aside/run-6/ 2026-11-01T23:49:28Z no 6,702 yes - - - - - - 4,932 (-26%) 4,932 (-26%) - - 4,932 (-26%) 4,925 (-27%)
cache-aside cache-aside/run-7/ 2026-11-02T01:58:24Z no 7,010 yes - - - - - - 5,327 (-24%) 5,327 (-24%) - - 5,327 (-24%) 5,302 (-24%)
cache-aside cache-aside/run-8/ 2026-11-02T04:11:08Z no 6,918 yes - - - - - - 5,220 (-25%) 5,220 (-25%) - - 5,220 (-25%) 5,212 (-25%)
cache-aside cache-aside/run-9/ 2026-11-04T08:18:36Z yes 6,086 yes - - - - - - 6,007 (-1%) 6,007 (-1%) - - - -
cache-aside cache-aside/run-10/ 2026-11-04T10:33:51Z yes 6,874 yes - - - - - - 5,452 (-21%) 5,452 (-21%) - - - -
cache-aside cache-aside/ 2026-11-04T12:50:54Z yes 6,752 yes - - - - - - 5,652 (-16%) 5,652 (-16%) - - - -
kafka kafka/run-1/ 2026-10-26T12:22:50Z yes 945 no: disturbed 1,006 (+6%) 996 (+5%) 1,006 (+6%) 1,006 (+6%) 1,006 (+6%) 996 (+5%) 996 (+5%) 996 (+5%) 996 (+5%) 503 (-47%) 996 (+5%) 996 (+5%)
kafka kafka/run-2/ 2026-10-30T11:20:01Z yes 2,040 yes 1,602 (-21%) 1,601 (-22%) 1,602 (-21%) 1,602 (-21%) 1,602 (-21%) 1,601 (-22%) 1,601 (-22%) 1,601 (-22%) 1,601 (-22%) 800 (-61%) 1,601 (-22%) 1,601 (-22%)
kafka kafka/run-3/ 2026-10-30T23:00:02Z yes 2,132 yes 2,829 (+33%) 2,828 (+33%) 2,829 (+33%) 2,829 (+33%) 2,829 (+33%) 2,828 (+33%) 2,828 (+33%) 2,828 (+33%) 2,828 (+33%) 2,124 (0%) 2,828 (+33%) 2,828 (+33%)
kafka kafka/run-4/ 2026-10-31T01:19:14Z yes 2,087 yes 2,140 (+3%) 2,139 (+3%) 2,140 (+3%) 2,140 (+3%) 2,140 (+3%) 2,139 (+3%) 2,139 (+3%) 2,139 (+3%) 2,139 (+3%) 1,609 (-23%) 2,139 (+3%) 2,139 (+3%)
kafka kafka/ 2026-10-31T03:26:27Z yes 2,028 yes 2,102 (+4%) 2,101 (+4%) 2,102 (+4%) 2,102 (+4%) 2,102 (+4%) 2,101 (+4%) 2,101 (+4%) 2,101 (+4%) 2,101 (+4%) 1,580 (-22%) 2,101 (+4%) 2,101 (+4%)
kafka kafka/run-6/ 2026-11-02T00:26:56Z yes 2,614 yes - - - - - - 2,551 (-2%) 2,551 (-2%) - - 2,551 (-2%) 2,551 (-2%)
kafka kafka/run-7/ 2026-11-02T02:36:13Z yes 2,659 yes - - - - - - 2,578 (-3%) 2,578 (-3%) - - 2,578 (-3%) 2,578 (-3%)
kafka kafka/run-8/ 2026-11-02T04:48:49Z yes 2,802 yes - - - - - - 2,528 (-10%) 2,528 (-10%) - - 2,528 (-10%) 2,528 (-10%)
kafka kafka/run-9/ 2026-11-04T08:59:13Z yes 2,266 no: disturbed - - - - - - 2,677 (+18%) 2,677 (+18%) - - - -
kafka kafka/run-10/ 2026-11-04T11:12:22Z yes 2,487 yes - - - - - - 2,728 (+10%) 2,728 (+10%) - - - -
kafka kafka/ 2026-11-04T13:27:44Z yes 2,387 yes - - - - - - 2,090 (-12%) 2,090 (-12%) - - - -

Knees in req/s; error = (simulated - measured) / measured.

  • Engine 0.14.0: median absolute knee error 20% over the latest run it checked of each experiment, 24% over all 19 counted runs.

  • Engine 0.15.0-w21: median absolute knee error 20% over the latest run it checked of each experiment, 24% over all 19 counted runs.

  • Engine 0.15.0-w22: median absolute knee error 15% over the latest run it checked of each experiment, 24% over all 19 counted runs.

  • Engine 0.15.0-w23: median absolute knee error 4% over the latest run it checked of each experiment, 13% over all 4 counted runs.

  • Engine 0.15.0-w24: median absolute knee error 20% over the latest run it checked of each experiment, 24% over all 19 counted runs.

  • Engine 0.15.0: median absolute knee error 15% over the latest run it checked of each experiment, 24% over all 19 counted runs.

  • Engine 0.16.0-java: median absolute knee error 16% over the latest run it checked of each experiment, 20% over all 56 counted runs.

  • Engine 0.16.0-port: median absolute knee error 16% over the latest run it checked of each experiment, 20% over all 56 counted runs.

  • Engine 0.16.0-w26: median absolute knee error 15% over the latest run it checked of each experiment, 24% over all 19 counted runs.

  • Engine 0.16.0-w27: median absolute knee error 17% over the latest run it checked of each experiment, 22% over all 19 counted runs.

  • Engine 0.16.0-w29: median absolute knee error 15% over the latest run it checked of each experiment, 19% over all 31 counted runs.

  • Engine 0.16.0: median absolute knee error 15% over the latest run it checked of each experiment, 19% over all 31 counted runs.

  • Engine 0.16.0-w27 ran the designs on a host machine (BENCH_HOST=on, run.py machine_inputs): contention check 2026-11-01T00:48:59Z: 4 physical cores of 8 threads (the VM's topology); clock 4,050 MHz with 1 busy thread, 2,951 with 8 (a line through 24 probe conditions) = 0.729; a thread sharing its core at 0.48 of its speed alone (the probe's CPU per unit of work against the busy threads, 20 conditions, rms 5.5% in log); ASSUMED, not measured (the quiet service check never ran: other sessions' jobs and a throttled laptop): 0.30 threads busy at rest and k6 0.25 ms of CPU per request; other load: the background, k6.

"Inputs as calibrated: no" marks a run whose protocol derived its inputs differently from today's code (v2 had no anchoring of the client latency), so its row is today's reading of the same passes. Runs calibrated from one pass (run 1) are left out. A disturbed run missed the objective at one load step and met it at a higher one (docs/validation.md, protocol step 5): it is listed, but left out of the medians.