Skip to content

Accuracy

How accurate is ArchSim?

We run small real systems, ask ArchSim to predict how they behave under load before measuring them, and then measure. Here is every result, the good and the bad, and how far to trust them.

held-out knees predicted within ±25%
1 of 5
range of the knee errors
−16% to +70%
experiments meet every proposed threshold
0 of 9
latest run, on one machine
4 Nov 2026

The short version

  • Redis cache with a 5 s TTL in front of PostgreSQL (held out): Predicted the knee within 6% (5,104 req/s predicted against 5,420 req/s measured), on the low side, which is the safe side.
  • Flask on gunicorn and PostgreSQL, mixed reads and writes (held out): Predicted the knee 27% too high: 1,762 req/s where the system managed 1,388 req/s.
  • A gateway in front of an API, then PostgreSQL (held out): Predicted the knee 32% too high: 4,455 req/s where the system managed 3,388 req/s.
  • Go and PostgreSQL, mixed reads and writes (held out): Predicted the knee 67% too high: 9,916 req/s where the system managed 5,950 req/s.
  • Spring Boot and PostgreSQL, mixed reads and writes (held out): Predicted the knee 70% too high: 7,907 req/s where the system managed 4,650 req/s.
  • PostgreSQL, every request updating one row (tuned on): Predicted the knee within 10% (857 req/s predicted against 947 req/s measured), on the low side, which is the safe side.
  • Kafka queue feeding PostgreSQL (tuned on): Predicted the knee within 12% (2,090 req/s predicted against 2,387 req/s measured), on the low side, which is the safe side.
  • PostgreSQL, mixed reads and writes (tuned on): Predicted the knee within 16% (5,203 req/s predicted against 4,483 req/s measured), on the high side.
  • Redis cache in front of PostgreSQL (tuned on): Predicted the knee within 16% (5,652 req/s predicted against 6,752 req/s measured), on the low side, which is the safe side.
  • At light load (at most half the predicted knee), ArchSim's p95 was off by −2.91 ms to +3.82 ms, or −54% to +164%.
  • ArchSim's predicted range for the knee held the measured one in 29 of 37 runs, where about 80% was expected. On the held-out experiments alone, 9 of 15.
  • ArchSim's CPU predictions (busy cores) were off by a median of 1% (PostgreSQL in “Kafka queue feeding PostgreSQL”) to 22% (Node.js producer in “Kafka queue feeding PostgreSQL”) at the load steps up to both knees. See CPU and memory.
  • None of the 9 experiments meets all the proposed thresholds yet, so no technology counts as validated.

Knees move between runs

The measured knee is not one fixed number: run the same experiment again on the same machine and it moves, sometimes by more than ArchSim's error.

  • Spring Boot and PostgreSQL, mixed reads and writes: The measured knee ranged from 4,198 to 4,799 req/s over 3 runs on this machine (±7% around the middle): the +70% error is larger than that.
  • Go and PostgreSQL, mixed reads and writes: The measured knee ranged from 5,508 to 6,204 req/s over 3 runs on this machine (±6% around the middle): the +67% error is larger than that.
  • Flask on gunicorn and PostgreSQL, mixed reads and writes: The measured knee ranged from 1,385 to 1,428 req/s over 3 runs on this machine (±2% around the middle): the +27% error is larger than that.
  • A gateway in front of an API, then PostgreSQL: The measured knee ranged from 3,388 to 3,503 req/s over 3 runs on this machine (±2% around the middle): the +32% error is larger than that.
  • Redis cache with a 5 s TTL in front of PostgreSQL: The measured knee ranged from 5,336 to 5,747 req/s over 3 runs on this machine (±4% around the middle): the −6% error is larger than that.
  • PostgreSQL, mixed reads and writes: The measured knee ranged from 2,875 to 4,495 req/s over 9 runs on this machine (±22% around the middle, leaving out 2 disturbed runs): wider than the +16% error, which is inside run-to-run noise.
  • PostgreSQL, every request updating one row: The measured knee ranged from 819 to 960 req/s over 8 runs on this machine (±8% around the middle, leaving out 1 disturbed run): the −10% error is larger than that.
  • Redis cache in front of PostgreSQL: The measured knee ranged from 6,086 to 7,324 req/s over 9 runs on this machine (±9% around the middle): the −16% error is larger than that.
  • Kafka queue feeding PostgreSQL: The measured knee ranged from 2,387 to 2,802 req/s over 5 runs on this machine (±8% around the middle, leaving out 1 disturbed run): the −12% error is larger than that.

Read these errors as results from one machine, not as ArchSim's accuracy in general.

Each experiment

The latest run of each experiment. The objective is p95 at most 50 ms with at least 99% of requests succeeding; the knee is the highest load that still meets it.
  • Measured knee
  • ArchSim's prediction, made before measuring
  • Its predicted range
  • Measured knee in each run on the same machine

Spring Boot and PostgreSQL, mixed reads and writes

Spring Boot (Tomcat, HikariCP) + PostgreSQL: 90% reads, 10% writes over 100,000 rows

Held outNot validated
offered load, req/s
Capacity knee · error +70%
Predicted the knee 70% too high: 7,907 req/s where the system managed 4,650 req/s.
Run to run
The measured knee ranged from 4,198 to 4,799 req/s over 3 runs on this machine (±7% around the middle): the +70% error is larger than that. Over 3 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of +70% (+64% to +89%). The measured knee fell inside ArchSim's predicted range in 0 of 3 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 2 of 2 load steps up to the measured knee; the typical miss was 22%.
Light load
At light load (1,975 and 3,950 req/s), predicted p95 was 0.04 to 1.19 ms below the measured one.

Proposed thresholds for “validated”

  • Not met: Median knee error within ±25% over 3+ runs (3 so far): no
  • Met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: yes
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (3 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 03:38 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). Held out: predicted by a frozen engine that was not tuned on it, before it was first measured. The model inputs and how each was measured are in docs/validation.md (section java-mixed) in the repository.

Every load step (8)
Spring Boot and PostgreSQL, mixed reads and writes: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
1,9751.671.63−2%100% / 100%
3,9502.881.70−41%100% / 100%
5,92556.81.87−97%97% / 100%
7,11078.32.19−97%92% / 100%
7,900unstable99.733.7−66%86% / 100%
8,6901491,916+1188%80% / 75%
9,8751711,954+1044%70% / 15%
11,850194--57% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

Go and PostgreSQL, mixed reads and writes

Go (net/http, pgx) + PostgreSQL: 90% reads, 10% writes over 100,000 rows

Held outNot validated
offered load, req/s
Capacity knee · error +67%
Predicted the knee 67% too high: 9,916 req/s where the system managed 5,950 req/s.
Run to run
The measured knee ranged from 5,508 to 6,204 req/s over 3 runs on this machine (±6% around the middle): the +67% error is larger than that. Over 3 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of +67% (+58% to +73%). The measured knee fell inside ArchSim's predicted range in 0 of 3 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 2 of 2 load steps up to the measured knee; the typical miss was 26%.
Light load
At light load (2,475 and 4,950 req/s), predicted p95 was 0.09 to 1.39 ms below the measured one.

Proposed thresholds for “validated”

  • Not met: Median knee error within ±25% over 3+ runs (3 so far): no
  • Met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: yes
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (3 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 04:19 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). Held out: predicted by a frozen engine that was not tuned on it, before it was first measured. The model inputs and how each was measured are in docs/validation.md (section go-mixed) in the repository.

Every load step (8)
Go and PostgreSQL, mixed reads and writes: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
2,4751.541.45−6%100% / 100%
4,9502.951.57−47%100% / 100%
7,42545.71.80−96%98% / 100%
8,91066.22.11−97%92% / 100%
9,90076.520.2−74%84% / 100%
10,89090.91,916+2007%79% / 77%
12,3751141,954+1620%69% / 16%
14,850141--54% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

Flask on gunicorn and PostgreSQL, mixed reads and writes

Flask on gunicorn (10 sync workers) + PostgreSQL: 90% reads, 10% writes over 100,000 rows

Held outNot validated
offered load, req/s
Capacity knee · error +27%
Predicted the knee 27% too high: 1,762 req/s where the system managed 1,388 req/s.
Run to run
The measured knee ranged from 1,385 to 1,428 req/s over 3 runs on this machine (±2% around the middle): the +27% error is larger than that. Over 3 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of +29% (+27% to +30%). The measured knee fell inside ArchSim's predicted range in 3 of 3 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 2 of 3 load steps up to the measured knee. Worst: at 440 req/s it predicted 6.14 ms where the system measured 2.33 ms.
Light load
At light load (440 and 881 req/s), predicted p95 was 2.77 to 3.82 ms above the measured one.

Proposed thresholds for “validated”

  • Not met: Median knee error within ±25% over 3+ runs (3 so far): no
  • Not met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: no
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (3 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 05:00 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). Held out: predicted by a frozen engine that was not tuned on it, before it was first measured. The model inputs and how each was measured are in docs/validation.md (section python-mixed) in the repository.

Every load step (8)
Flask on gunicorn and PostgreSQL, mixed reads and writes: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
4402.336.14+164%100% / 100%
8813.135.90+88%100% / 100%
1,3227.606.78−11%100% / 100%
1,58682.510.9−87%96% / 100%
1,76282.645.4−45%86% / 100%
1,9381621,916+1086%87% / 91%
2,2022921,954+569%74% / 16%
2,643316--62% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

A gateway in front of an API, then PostgreSQL

Node.js gateway, Node.js API, PostgreSQL in a chain: 90% reads, 10% writes over 100,000 rows

Held outNot validated
offered load, req/s
Capacity knee · error +32%
Predicted the knee 32% too high: 4,455 req/s where the system managed 3,388 req/s.
Run to run
The measured knee ranged from 3,388 to 3,503 req/s over 3 runs on this machine (±2% around the middle): the +32% error is larger than that. Over 3 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of +30% (+24% to +32%). The measured knee fell inside ArchSim's predicted range in 3 of 3 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 2 of 3 load steps up to the measured knee. Worst: at 3,338 req/s it predicted 2.67 ms where the system measured 30.9 ms.
Light load
At light load (1,112 and 2,225 req/s), predicted p95 was 0.16 to 2.91 ms below the measured one.

Proposed thresholds for “validated”

  • Not met: Median knee error within ±25% over 3+ runs (3 so far): no
  • Not met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: no
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (3 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 05:34 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). Held out: predicted by a frozen engine that was not tuned on it, before it was first measured. The model inputs and how each was measured are in docs/validation.md (section chain) in the repository.

Every load step (8)
A gateway in front of an API, then PostgreSQL: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
1,1122.582.42−6%100% / 100%
2,2255.382.47−54%100% / 100%
3,33830.92.67−91%99% / 100%
4,00562.63.20−95%97% / 100%
4,45090.627.7−69%91% / 100%
4,8951131,916+1595%87% / 77%
5,5621491,954+1209%76% / 15%
6,675211--59% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

Redis cache with a 5 s TTL in front of PostgreSQL

Cache-aside: Node.js, Redis (5 s TTL) and PostgreSQL; Zipf 1.1 over 1,000,000 keys

Held outNot validated
offered load, req/s
Capacity knee · error −6%
Predicted the knee within 6% (5,104 req/s predicted against 5,420 req/s measured), on the low side, which is the safe side.
Run to run
The measured knee ranged from 5,336 to 5,747 req/s over 3 runs on this machine (±4% around the middle): the −6% error is larger than that. Over 3 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of −6% (−8% to +1%). The measured knee fell inside ArchSim's predicted range in 3 of 3 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 5 of 5 load steps up to the measured knee; the typical miss was 37%.
Light load
At light load (1,275 and 2,550 req/s), predicted p95 was 0.64 to 0.67 ms above the measured one.

Proposed thresholds for “validated”

  • Met: Median knee error within ±25% over 3+ runs (3 so far): yes
  • Met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: yes
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (3 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 06:11 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). Held out: predicted by a frozen engine that was not tuned on it, before it was first measured. The model inputs and how each was measured are in docs/validation.md (section cache-ttl) in the repository.

Every load step (8)
Redis cache with a 5 s TTL in front of PostgreSQL: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
1,2751.732.37+37%100% / 100%
2,5502.112.78+32%100% / 100%
3,8255.965.04−15%100% / 100%
4,59015.19.50−37%100% / 100%
5,10022.635.8+58%100% / 100%
5,61026.11,806+6822%99% / 98%
6,37551.81,954+3670%95% / 21%
7,65071.6--92% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

PostgreSQL, mixed reads and writes

Node.js + PostgreSQL: 90% reads, 10% writes over 100,000 rows

Tuned onNot validated
offered load, req/s
Capacity knee · error +16%
Predicted the knee within 16% (5,203 req/s predicted against 4,483 req/s measured), on the high side.
Run to run
The measured knee ranged from 2,875 to 4,495 req/s over 9 runs on this machine (±22% around the middle, leaving out 2 disturbed runs): wider than the +16% error, which is inside run-to-run noise. Over 6 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of +16% (+6% to +88%). The measured knee fell inside ArchSim's predicted range in 5 of 6 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 2 of 3 load steps up to the measured knee. Worst: at 3,900 req/s it predicted 1.87 ms where the system measured 9.78 ms.
Light load
At light load (1,300 and 2,600 req/s), predicted p95 was 0.05 to 0.33 ms below the measured one.

Proposed thresholds for “validated”

  • Met: Median knee error within ±25% over 3+ runs (6 so far): yes
  • Not met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: no
  • Not met: No unstable step below the knee, in 2 of 3 runs: no
  • Not met: Reproduced on a second machine (6 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 11:52 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). In-sample: the calibration protocol was revised after this experiment's earlier results had been seen. The model inputs and how each was measured are in docs/validation.md (section pg-mixed) in the repository.

Every load step (8)
PostgreSQL, mixed reads and writes: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
1,3001.711.66−3%100% / 100%
2,6002.091.76−16%100% / 100%
3,9009.781.87−81%100% / 100%
4,68063.62.28−96%99% / 100%
5,20014538.0−74%86% / 100%
5,7201651,916+1062%77% / 75%
6,5001891,954+934%67% / 15%
7,800219--55% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

PostgreSQL, every request updating one row

Node.js + PostgreSQL: every request updates the same row

Tuned onNot validated
offered load, req/s
Capacity knee · error −10%
Predicted the knee within 10% (857 req/s predicted against 947 req/s measured), on the low side, which is the safe side.
Run to run
The measured knee ranged from 819 to 960 req/s over 8 runs on this machine (±8% around the middle, leaving out 1 disturbed run): the −10% error is larger than that. Over 5 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of −10% (−10% to −9%). 1 disturbed run is not counted: it missed the objective at one load step and met it at a higher one. The measured knee fell inside ArchSim's predicted range in 5 of 5 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 4 of 6 load steps up to the measured knee. Worst: at 942 req/s it predicted 1,916 ms where the system measured 19.0 ms.
Light load
At light load (214 and 428 req/s), predicted p95 was 0.21 to 0.56 ms below the measured one.

Proposed thresholds for “validated”

  • Met: Median knee error within ±25% over 3+ runs (5 so far): yes
  • Not met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: no
  • Not met: No unstable step below the knee, in 2 of 3 runs: no
  • Not met: Reproduced on a second machine (6 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 12:24 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). In-sample: the calibration protocol was revised after this experiment's earlier results had been seen. The model inputs and how each was measured are in docs/validation.md (section pg-hot-row) in the repository.

Every load step (8)
PostgreSQL, every request updating one row: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
2143.082.52−18%100% / 100%
4282.632.42−8%100% / 100%
6422.662.62−1%100% / 100%
7703.263.32+2%100% / 100%
85612.237.2+206%100% / 100%
942unstable19.01,916+10003%100% / 78%
1,070unstable8451,954+131%96% / 16%
1,2841,930--49% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

Redis cache in front of PostgreSQL

Cache-aside: Node.js, Redis (30 s TTL) and PostgreSQL; Zipf 0.99 over 100,000 keys

Tuned onNot validated
offered load, req/s
Capacity knee · error −16%
Predicted the knee within 16% (5,652 req/s predicted against 6,752 req/s measured), on the low side, which is the safe side.
Run to run
The measured knee ranged from 6,086 to 7,324 req/s over 9 runs on this machine (±9% around the middle): the −16% error is larger than that. Over 6 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of −23% (−27% to −1%). The measured knee fell inside ArchSim's predicted range in 5 of 6 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 4 of 6 load steps up to the measured knee. Worst: at 6,215 req/s it predicted 1,916 ms where the system measured 10.7 ms.
Light load
At light load (1,412 and 2,825 req/s), predicted p95 was 0.00 to 0.09 ms below the measured one.

Proposed thresholds for “validated”

  • Met: Median knee error within ±25% over 3+ runs (6 so far): yes
  • Not met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: no
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (6 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 12:50 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). In-sample: the calibration protocol was revised after this experiment's earlier results had been seen. The model inputs and how each was measured are in docs/validation.md (section cache-aside) in the repository.

Every load step (8)
Redis cache in front of PostgreSQL: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
1,4121.391.390%100% / 100%
2,8251.541.45−6%100% / 100%
4,2382.382.370%100% / 100%
5,0857.034.30−39%100% / 100%
5,6505.8741.9+614%100% / 100%
6,21510.71,916+17808%100% / 86%
7,06227.61,954+6979%98% / 17%
8,47548.4--96% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

Kafka queue feeding PostgreSQL

Kafka: Node.js producer, 6 partitions, 3 consumers inserting into PostgreSQL

Tuned onNot validated
offered load, req/s
Capacity knee · error −12%
Predicted the knee within 12% (2,090 req/s predicted against 2,387 req/s measured), on the low side, which is the safe side.
Run to run
The measured knee ranged from 2,387 to 2,802 req/s over 5 runs on this machine (±8% around the middle, leaving out 1 disturbed run): the −12% error is larger than that. Over 5 runs under these conditions (protocol v6, engine 0.16.0) the knee error had a median of −3% (−12% to +10%). 1 disturbed run is not counted: it missed the objective at one load step and met it at a higher one. The measured knee fell inside ArchSim's predicted range in 5 of 5 runs (expected about 80%).
Latency below the knee
Latency (p95) within a factor of 2 or 3 ms at 3 of 6 load steps up to the measured knee. Worst: at 2,297 req/s it predicted 1,954 ms where the system measured 36.4 ms.
Light load
At light load (522 and 1,044 req/s), predicted p95 was 0.36 to 1.45 ms below the measured one.

Proposed thresholds for “validated”

  • Met: Median knee error within ±25% over 3+ runs (5 so far): yes
  • Not met: p95 within a factor of 2 or 3 ms below the knee, in 2 of 3 runs: no
  • Met: No unstable step below the knee, in 2 of 3 runs: yes
  • Not met: Reproduced on a second machine (6 runs under v6): no

Run of 4 Nov 2026 (calibrated 4 Nov, 13:27 UTC), protocol v6, engine 0.16.0. 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz; Docker 29.1.3 (linux/amd64) with 8 CPUs and 7.6 GiB; Windows-11-10.0.26300-SP0 (Best performance). In-sample: the calibration protocol was revised after this experiment's earlier results had been seen. The model inputs and how each was measured are in docs/validation.md (section kafka) in the repository.

Every load step (8)
Kafka queue feeding PostgreSQL: measured and predicted p95 latency and success at each offered load
req/sp95 meas. (ms)p95 pred. (ms)ErrorSuccess
5222.351.99−15%100% / 100%
1,0443.562.11−41%100% / 100%
1,56612.92.62−80%100% / 100%
1,87918.55.04−73%100% / 100%
2,08826.029.4+13%100% / 100%
2,29736.41,954+5268%99% / 84%
2,61045.81,954+4167%98% / 16%
3,13276.9--97% / 0%

Greyed rows are above the measured knee. Success is measured / predicted; “-” means no request succeeded.

CPU and memory

ArchSim also predicts how busy each part's CPU is and how much memory it holds, and picks instance sizes from those figures. Here they are against each container's own counters. ArchSim's CPU predictions (busy cores) were off by a median of 1% (PostgreSQL in “Kafka queue feeding PostgreSQL”) to 22% (Node.js producer in “Kafka queue feeding PostgreSQL”) at the load steps up to both knees.

Spring Boot and PostgreSQL, mixed reads and writes

Busy cores at 3,950 req/s, the highest load step at or below both knees

Spring Boot and PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Spring Boot service1.041.00−4%6%874 MiB / 925 MiB
PostgreSQL0.630.59−6%4%102 MiB / -
Every load step (8)
Spring Boot and PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part at each offered load
req/sSpring Boot servicePostgreSQL
1,9750.52 / 0.560.29 / 0.30
3,9501.04 / 1.000.63 / 0.59
5,9251.56 / 1.500.97 / 0.89
7,1101.72 / 1.801.11 / 1.07
7,9001.75 / 2.001.13 / 1.18
8,6901.81 / 2.001.20 / 1.18
9,8751.82 / 2.001.21 / 1.18
11,8501.78 / 2.001.20 / 1.18

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

Go and PostgreSQL, mixed reads and writes

Busy cores at 4,950 req/s, the highest load step at or below both knees

Go and PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Go service0.971.00+2%5%42 MiB / 33 MiB
PostgreSQL0.700.64−8%5%110 MiB / -
Every load step (8)
Go and PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part at each offered load
req/sGo servicePostgreSQL
2,4750.57 / 0.520.33 / 0.32
4,9500.97 / 1.000.70 / 0.64
7,4251.13 / 1.501.05 / 0.97
8,9101.16 / 1.801.16 / 1.16
9,9001.15 / 2.001.19 / 1.28
10,8901.22 / 2.001.24 / 1.29
12,3751.24 / 2.001.25 / 1.29
14,8501.24 / 2.001.23 / 1.29

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

Flask on gunicorn and PostgreSQL, mixed reads and writes

Busy cores at 1,322 req/s, the highest load step at or below both knees

Flask on gunicorn and PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Flask service (gunicorn)1.491.48−1%6%340 MiB / 329 MiB
PostgreSQL0.250.23−6%6%124 MiB / -
Every load step (8)
Flask on gunicorn and PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part at each offered load
req/sFlask service (gunicorn)PostgreSQL
4400.45 / 0.550.09 / 0.08
8810.94 / 0.990.16 / 0.16
1,3221.49 / 1.480.25 / 0.23
1,5861.80 / 1.770.30 / 0.28
1,7621.75 / 1.970.30 / 0.31
1,9381.99 / 2.000.35 / 0.32
2,2021.99 / 2.000.35 / 0.32
2,6431.99 / 2.000.35 / 0.32

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

A gateway in front of an API, then PostgreSQL

Busy cores at 3,338 req/s, the highest load step at or below both knees

A gateway in front of an API, then PostgreSQL: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Node.js gateway0.780.75−5%5%170 MiB / 116 MiB
Node.js API0.660.61−7%13%150 MiB / 145 MiB
PostgreSQL0.680.66−3%3%104 MiB / -
Every load step (8)
A gateway in front of an API, then PostgreSQL: busy cores measured and predicted for each part at each offered load
req/sNode.js gatewayNode.js APIPostgreSQL
1,1120.31 / 0.310.26 / 0.200.23 / 0.22
2,2250.55 / 0.520.47 / 0.410.45 / 0.44
3,3380.78 / 0.750.66 / 0.610.68 / 0.66
4,0050.89 / 0.900.75 / 0.740.80 / 0.79
4,4500.92 / 1.000.79 / 0.820.86 / 0.88
4,8950.94 / 1.000.81 / 0.820.88 / 0.88
5,5620.93 / 1.000.82 / 0.820.88 / 0.88
6,6750.96 / 1.000.82 / 0.820.86 / 0.88

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

Redis cache with a 5 s TTL in front of PostgreSQL

Busy cores at 5,100 req/s, the highest load step at or below both knees

Redis cache with a 5 s TTL in front of PostgreSQL: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Node.js service0.800.97+22%6%179 MiB / 190 MiB
PostgreSQL0.290.29+3%3%386 MiB / -
Redis0.180.23+25%16%9 MiB / 28 MiB
Every load step (8)
Redis cache with a 5 s TTL in front of PostgreSQL: busy cores measured and predicted for each part at each offered load
req/sNode.js servicePostgreSQLRedis
1,2750.32 / 0.340.12 / 0.090.08 / 0.06
2,5500.51 / 0.520.18 / 0.170.13 / 0.12
3,8250.68 / 0.730.23 / 0.230.17 / 0.17
4,5900.75 / 0.870.26 / 0.270.18 / 0.21
5,1000.80 / 0.970.29 / 0.290.18 / 0.23
5,6100.83 / 1.000.30 / 0.300.18 / 0.23
6,3750.86 / 1.000.34 / 0.300.17 / 0.23
7,6500.86 / 1.000.36 / 0.300.16 / 0.23

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

PostgreSQL, mixed reads and writes

Busy cores at 3,900 req/s, the highest load step at or below both knees

PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Node.js service0.790.75−5%4%148 MiB / 155 MiB
PostgreSQL0.770.71−7%1%103 MiB / -
Every load step (8)
PostgreSQL, mixed reads and writes: busy cores measured and predicted for each part at each offered load
req/sNode.js servicePostgreSQL
1,3000.29 / 0.300.24 / 0.24
2,6000.53 / 0.520.47 / 0.48
3,9000.79 / 0.750.77 / 0.71
4,6800.92 / 0.900.91 / 0.86
5,2001.00 / 1.001.00 / 0.95
5,7200.99 / 1.001.00 / 0.95
6,5000.99 / 1.001.00 / 0.95
7,8000.99 / 1.001.01 / 0.95

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

PostgreSQL, every request updating one row

Busy cores at 856 req/s, the highest load step at or below both knees

PostgreSQL, every request updating one row: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Node.js service0.220.28+27%21%58 MiB / 74 MiB
PostgreSQL0.270.24−10%10%106 MiB / -
Every load step (8)
PostgreSQL, every request updating one row: busy cores measured and predicted for each part at each offered load
req/sNode.js servicePostgreSQL
2140.09 / 0.090.09 / 0.06
4280.13 / 0.140.13 / 0.12
6420.17 / 0.210.18 / 0.18
7700.20 / 0.250.22 / 0.21
8560.22 / 0.280.27 / 0.24
9420.24 / 0.310.31 / 0.24
1,0700.27 / 0.350.58 / 0.24
1,2840.39 / 0.420.61 / 0.24

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

Redis cache in front of PostgreSQL

Busy cores at 5,650 req/s, the highest load step at or below both knees

Redis cache in front of PostgreSQL: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Node.js service0.810.99+22%7%181 MiB / 158 MiB
PostgreSQL0.240.22−6%9%99 MiB / -
Redis0.190.23+23%15%11 MiB / 31 MiB
Every load step (8)
Redis cache in front of PostgreSQL: busy cores measured and predicted for each part at each offered load
req/sNode.js servicePostgreSQLRedis
1,4120.31 / 0.320.11 / 0.090.08 / 0.06
2,8250.55 / 0.510.16 / 0.150.14 / 0.12
4,2380.70 / 0.740.20 / 0.190.17 / 0.18
5,0850.79 / 0.890.23 / 0.210.18 / 0.21
5,6500.81 / 0.990.24 / 0.220.19 / 0.23
6,2150.85 / 1.000.24 / 0.220.20 / 0.23
7,0620.90 / 1.000.27 / 0.220.19 / 0.23
8,4750.92 / 1.000.28 / 0.220.19 / 0.23

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

Kafka queue feeding PostgreSQL

Busy cores at 2,088 req/s, the highest load step at or below both knees

Kafka queue feeding PostgreSQL: busy cores measured and predicted for each part
PartMeas.Pred.ErrorTypical error (median up to both knees)Memory meas. / pred.
Node.js producer0.720.99+38%22%175 MiB / 81 MiB
Node.js consumer0.30not modelled--182 MiB / -
PostgreSQL0.450.44−2%1%132 MiB / -
Kafka broker0.54not modelled--989 MiB / -
Every load step (8)
Kafka queue feeding PostgreSQL: busy cores measured and predicted for each part at each offered load
req/sNode.js producerNode.js consumerPostgreSQLKafka broker
5220.28 / 0.250.32 / -0.17 / 0.160.40 / -
1,0440.45 / 0.500.34 / -0.33 / 0.320.48 / -
1,5660.61 / 0.740.29 / -0.43 / 0.440.45 / -
1,8790.68 / 0.890.29 / -0.45 / 0.440.51 / -
2,0880.72 / 0.990.30 / -0.45 / 0.440.54 / -
2,2970.78 / 1.000.31 / -0.47 / 0.440.58 / -
2,6100.81 / 1.000.30 / -0.47 / 0.440.63 / -
3,1320.88 / 1.000.30 / -0.47 / 0.450.70 / -

Busy cores measured / predicted. Greyed rows are above the measured knee or the model's.

Busy cores are CPU seconds per second: 0.5 is half of one core busy on average. Error is (predicted − measured) ÷ measured; the typical error is the median over the load steps up to both knees, the measured one and the model's. Past the model's own knee its queue holds every request the real service would time out, so its CPU reads fully busy and its memory grows without bound; the knee error already counts that.

Measured: each container's CPU and working set (memory less reclaimable page cache), read every 5 s inside the window k6 measures.

Predicted: ArchSim's predictions, saved before the measurements.

Not modelled: Node.js consumer (inside the model's queue node, which has no CPU or memory); Kafka broker (the model's queue node has no CPU or memory).

What the gaps mean and where they come from is under “Resources: CPU and memory” in docs/validation.md in the repository.

How to read these numbers

Knee
The highest load that still meets the objective (p95 at most 50 ms, at least 99% success). k6 measures a ladder of load steps, three repeats each; the knee is interpolated between the last step that meets the objective and the first that does not. The predicted knee uses the same steps.
Disturbed run
A run that missed the objective at one load step and met it at a higher one. A stable system does not get faster with more load, so its knee is not the system's: it is listed, but not counted in the medians, the spread or the thresholds.
Knee error
(predicted − measured) ÷ measured. Negative means ArchSim promised less capacity than the system had, which is the safe side; positive means it promised more.
Latency below the knee
At each load step up to the measured knee, a prediction counts as close when it is within a factor of 2 of the measured p95 (from half to double) or within 3 ms of it, whichever is wider. The test is the same both ways: predicting half the measured latency misses as much as predicting double. The 3 ms allowance is there because latencies of a millisecond or two make factors look large.
Unstable step
A load step where some repeats met the objective and others missed it, with p95s more than 2× apart. Its median is less trustworthy than other steps'.
Validated
A proposed bar, not applied anywhere yet: over at least three counted runs, a median knee error within ±25%; latency close at every step below the knee, and no unstable step there, in two of three runs; and a run on a second machine. A technology is validated when every experiment that uses it is.
In-sample
PostgreSQL, mixed reads and writes, PostgreSQL, every request updating one row, Redis cache in front of PostgreSQL and Kafka queue feeding PostgreSQL are in-sample: the calibration protocol was revised after their earlier results had been seen, so those numbers may flatter the model. Spring Boot and PostgreSQL, mixed reads and writes, Go and PostgreSQL, mixed reads and writes, Flask on gunicorn and PostgreSQL, mixed reads and writes, A gateway in front of an API, then PostgreSQL and Redis cache with a 5 s TTL in front of PostgreSQL are held out: registered before the engine was frozen, sealed until it had predicted them, and never used to tune it. They are the test of ArchSim on stacks it was not tuned on.
Predicted first
Each experiment calibrates the model from a staircase of runs well below the knee, then ArchSim predicts every higher step and the predictions are saved before the first step is measured. Nothing measured afterwards feeds back into the model.
Java engine
Re-simulated, in-sample: engine 1.1.1 (Java) re-simulates every kept calibration with exactly 0.16.0's knees (65 of 65); the latest predictions shown are 0.16.0's, which 1.1.1 reproduces. The comparison is in docs/validation/java-parity.md in the repository.

Every run so far

67 runs between 26 Oct 2026 and 4 Nov 2026, under calibration protocols v1, v2, v3, v4, v5 and v6; 10 of them were disturbed and are not counted. What changed between protocols, and why, is under “Protocol history” in docs/validation.md in the repository. Knees are in req/s.

Spring Boot and PostgreSQL, mixed reads and writes

Spring Boot and PostgreSQL, mixed reads and writes: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v63 Nov, 21:34 UTC4,1987,915+89%
run-2 · v64 Nov, 00:33 UTC4,7997,857+64%
latest · v64 Nov, 03:38 UTC4,6507,907+70%

Go and PostgreSQL, mixed reads and writes

Go and PostgreSQL, mixed reads and writes: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v63 Nov, 22:16 UTC5,5089,517+73%
run-2 · v64 Nov, 01:13 UTC6,2049,802+58%
latest · v64 Nov, 04:19 UTC5,9509,916+67%

Flask on gunicorn and PostgreSQL, mixed reads and writes

Flask on gunicorn and PostgreSQL, mixed reads and writes: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v63 Nov, 22:58 UTC1,4281,850+30%
run-2 · v64 Nov, 01:55 UTC1,3851,788+29%
latest · v64 Nov, 05:00 UTC1,3881,762+27%

A gateway in front of an API, then PostgreSQL

A gateway in front of an API, then PostgreSQL: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v63 Nov, 23:27 UTC3,5034,353+24%
run-2 · v64 Nov, 02:28 UTC3,4754,531+30%
latest · v64 Nov, 05:34 UTC3,3884,455+32%

Redis cache with a 5 s TTL in front of PostgreSQL

Redis cache with a 5 s TTL in front of PostgreSQL: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v64 Nov, 00:02 UTC5,7475,303−8%
run-2 · v64 Nov, 03:05 UTC5,3365,401+1%
latest · v64 Nov, 06:11 UTC5,4205,104−6%

PostgreSQL, mixed reads and writes

PostgreSQL, mixed reads and writes: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v126 Oct, 08:00 UTC> 1,359906-
run-2 · v226 Oct, 08:23 UTC1,8952,314+22% (not counted)
run-3 · v326 Oct, 09:46 UTC3,162808−74%
run-4 · v426 Oct, 11:13 UTC2,2912,064−10%
run-5 · v428 Oct, 23:21 UTC2,4302,238−8%
run-6 · v530 Oct, 21:30 UTC1,0894,230+288% (not counted)
run-7 · v530 Oct, 23:44 UTC1,4495,108+253% (not counted)
run-8 · v531 Oct, 01:52 UTC4,1605,003+20%
run-9 · v531 Oct, 04:02 UTC3,7834,729+25%
run-10 · v531 Oct, 04:34 UTC4,0064,956+24%
run-11 · v61 Nov, 22:53 UTC4,4954,780+6%
run-12 · v62 Nov, 01:02 UTC4,3644,856+11%
run-13 · v62 Nov, 03:15 UTC4,2375,055+19%
run-14 · v64 Nov, 07:15 UTC2,8755,406+88%
run-15 · v64 Nov, 09:35 UTC4,3715,059+16%
latest · v64 Nov, 11:52 UTC4,4835,203+16%

Not counted: a disturbed run missed the objective at one load step and met it at a higher one. A stable stack does not get faster with more load, so its knee is not the stack’s.

PostgreSQL, every request updating one row

PostgreSQL, every request updating one row: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v126 Oct, 08:11 UTC303403+33% (not counted)
run-2 · v226 Oct, 08:42 UTC819418−49%
run-3 · v326 Oct, 10:11 UTC132504+281% (not counted)
run-4 · v426 Oct, 11:33 UTC225444+97% (not counted)
run-5 · v430 Oct, 10:32 UTC814519−36%
run-6 · v530 Oct, 21:58 UTC819663−19%
run-7 · v531 Oct, 00:14 UTC9609580%
run-8 · v531 Oct, 02:21 UTC926783−16%
run-9 · v61 Nov, 23:24 UTC953862−10%
run-10 · v62 Nov, 01:33 UTC939851−9%
run-11 · v62 Nov, 03:46 UTC942851−10%
run-12 · v64 Nov, 07:49 UTC697826+19% (not counted)
run-13 · v64 Nov, 10:07 UTC947863−9%
latest · v64 Nov, 12:24 UTC947857−10%

Not counted: a disturbed run missed the objective at one load step and met it at a higher one. A stable stack does not get faster with more load, so its knee is not the stack’s.

Redis cache in front of PostgreSQL

Redis cache in front of PostgreSQL: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v426 Oct, 11:53 UTC4,0163,457−14% (not counted)
run-2 · v430 Oct, 10:50 UTC4,8482,352−51%
run-3 · v530 Oct, 22:22 UTC7,1385,303−26%
run-4 · v531 Oct, 00:38 UTC6,4324,227−34%
run-5 · v531 Oct, 02:44 UTC7,3244,851−34%
run-6 · v61 Nov, 23:49 UTC6,7024,925−27%
run-7 · v62 Nov, 01:58 UTC7,0105,302−24%
run-8 · v62 Nov, 04:11 UTC6,9185,201−25%
run-9 · v64 Nov, 08:18 UTC6,0866,007−1%
run-10 · v64 Nov, 10:33 UTC6,8745,452−21%
latest · v64 Nov, 12:50 UTC6,7525,652−16%

Not counted: a disturbed run missed the objective at one load step and met it at a higher one. A stable stack does not get faster with more load, so its knee is not the stack’s.

Kafka queue feeding PostgreSQL

Kafka queue feeding PostgreSQL: every kept run
RunCalibratedMeasuredPredictedError
run-1 · v426 Oct, 12:22 UTC9451,006+6% (not counted)
run-2 · v430 Oct, 11:20 UTC2,0401,602−21%
run-3 · v530 Oct, 23:00 UTC2,1322,829+33%
run-4 · v531 Oct, 01:19 UTC2,0872,140+3%
run-5 · v531 Oct, 03:26 UTC2,0282,102+4%
run-6 · v62 Nov, 00:26 UTC2,6142,551−2%
run-7 · v62 Nov, 02:36 UTC2,6592,578−3%
run-8 · v62 Nov, 04:48 UTC2,8022,528−10%
run-9 · v64 Nov, 08:59 UTC2,2662,677+18% (not counted)
run-10 · v64 Nov, 11:12 UTC2,4872,728+10%
latest · v64 Nov, 13:27 UTC2,3872,090−12%

Not counted: a disturbed run missed the objective at one load step and met it at a higher one. A stable stack does not get faster with more load, so its knee is not the stack’s.

Machine (all 67 runs)

  • 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz

Each experiment runs on Docker with its containers pinned to their own CPUs; the latest run's full environment is under its experiment above. The model inputs and every measured repeat are in docs/validation.md and benchmarks/results/ in the repository. A benchmark workflow (.github/workflows/benchmark.yml) is set to re-run the experiments every week and for every release and to open a pull request with the new numbers; this page changes only when a person merges one.

How to calibrate your own design

These experiments are small synthetic services on one machine. A new design in ArchSim starts from illustrative defaults, not from your system. Before you trust its numbers for a decision:
  1. 1Calibrate from your telemetry. Feed ArchSim your Prometheus metrics or an OpenTelemetry trace export (in the editor: Ctrl/⌘ K → Calibrate from telemetry, or archsim calibrate). Each fitted input is marked measured, with its source and fit error.
  2. 2Check the replay. Calibration replays the traffic it saw through the model and reports how close it comes to the latency and throughput your system actually had. A large replay error means the model is missing something about your system.
  3. 3Look at what is still assumed. Results list every input as measured, assumed or illustrative. Numbers that rest on illustrative inputs are estimates of a system like yours, not of yours.
  4. 4Measure your own knee. Load-test one step past the predicted knee in a staging environment, or run this page's benchmarks on your hardware (./gradlew :archsim-benchmarks:run --args="all").

In the repository, docs/calibration.md is the calibration guide (Prometheus and OpenTelemetry, what each fitted input means, the replay check), and docs/validation.md has the full method behind this page, the model inputs of every experiment and the raw data.