Accepted27 Oct 2026
0003. Run simulations from a PostgreSQL queue with leases
Recorded retroactively on 2026-11-02. The decision dates from commits b025f82 and 318145d
(27 Oct 2026), which added the queue (migration 0003) and the worker that uses it.
Context and Problem Statement#
A simulation takes from seconds to minutes. The API has to answer at once, keep runs through restarts, work with more than one API instance, and accept a cancel request on any of them. A simulation that crashes should fail only its own run.
The first version ran simulations inside the API process (ThreadJobRunner, ProcessJobRunner
with a shared process pool). On start-up it failed every run that was still active, whichever
process had owned it.
How should runs be queued and executed?
Considered Options#
- In-process runners inside the API, as in the first version
- The run row in the database is the job: workers claim queued runs under a lease and renew it
Decision Outcome#
Chosen option: "The run row in the database is the job", because the database is already shared by every instance. The queue adds no new infrastructure, and a lease says who owns a run at any moment.
How it works (services/api/src/archsim_api/jobs/):
- Claim:
claimtakes the oldest queued runs. On PostgreSQL it uses aSELECT … FOR UPDATE SKIP LOCKEDCTE, thenUPDATE … RETURNING. On SQLite it usesBEGIN IMMEDIATE, which is safe on one host only. - Lease: a run's
lease_ownerandlease_expires_atare renewed with each heartbeat (every 2 s), and cancel requests are picked up at the same time. - Ownership: progress and outcomes are written only while the worker holds the lease, so a worker that lost a run can't overwrite the one that took it over.
- Expired leases: they are reaped every 10 s. The run is requeued after a first attempt, failed after a second, and cancelled if a cancel was requested.
- Isolation: each run simulates in a process of its own (
jobs/child.py), so a crash fails only that run.
Status (10 Nov 2026): the queue works as decided here, but its worker is archsim-app worker,
in Java, since stage 4 of the port (ADR-0053),
and jobs/ went with the Python worker. A run simulates on a thread of its worker, not in a process
of its own, so a run that exhausts the worker's heap stops its other runs too; the reaper settles
them like the runs of any worker that stops: requeued once, then failed. The claim above is its
first version (b025f82). The same day W8's workspace caps (d892b19) replaced SKIP LOCKED:
to count each workspace's running runs and pick consistently, claims take turns under a
transaction advisory lock (pg_advisory_xact_lock), and a row_number pick takes every
workspace's oldest queued run before any workspace's second. The Java worker ports that claim.
Consequences#
- Good, because API instances and workers scale independently, with nothing to run but the database.
- Good, because any instance can report progress or cancel a run, since both go through the database.
- Bad, because queue throughput is bounded by the database. That's fine for runs that take seconds, but not for a high-volume job stream.
- Bad, because SQLite needs a second claim path that is limited to one host (ADR-0016 removes it from the Java version).
More Information#
docs/architecture.md, "Run lifecycle"- The Java port keeps the queue as plain SQL (ADR-0008), served by
the
workerrole of one application (ADR-0014).