Skip to content
Browse the docs

Running ArchSim for a team

View on GitHubReport a problem with this page

By default ArchSim is a local, single-user tool: no sign-in, every request is the owner of one default workspace, and every port is bound to 127.0.0.1. This page covers what changes when more people, more machines or more runs are involved, and Hosting a public server takes one server from nothing to a site on the internet with the repository's hosted kit, deploy/. Read SECURITY.md, the threat model, before exposing a server.

The images#

Each release publishes two images to GitHub's container registry, for linux/amd64 and linux/arm64, tagged with its version (latest follows the newest release that is not a pre-release):

Image What it runs
ghcr.io/dante0747/archsim-app:<version> The API, the simulation workers and the database migration: one image, started as api, worker, migrate or seed
ghcr.io/dante0747/archsim-web:<version> The web app, which calls the API at ARCHSIM_API_URL (default http://api:8000)

docker-compose.yml and the hosted kit's deploy/compose.yaml run both at the version their checkout names. Pin a version wherever you run them, so that every API instance and worker runs the same code. .github/workflows/release.yml builds them from the release's tag (ADR-0077).

Several API instances and workers#

Runs wait in a queue in the database, and workers simulate them. To share the work, start more workers against the same PostgreSQL:

  • with Compose: docker compose up --scale worker=3;
  • elsewhere: java -jar archsim-app.jar worker on any machine that reaches the database, with the heap archsim-app's README asks for and -XX:+ExitOnOutOfMemoryError.

A run survives a restart (it is queued again), and cancel works from any instance. A worker's runs share its heap: a run that exhausts it stops the worker, whose runs are then requeued. Any number of API instances can share the database too; run the migrate role once (java -jar archsim-app.jar migrate, the Compose migrate service) and set ARCHSIM_AUTO_MIGRATE=false on the API instances. Hosts need synchronised clocks (NTP), because leases compare their times. Details and limits: api.md.

Sign-in, teams and API tokens#

To let other people in, run the API with ARCHSIM_AUTH_MODE=oauth, a secret key, the public address and a GitHub or Google OAuth app:

Variable What it is
ARCHSIM_AUTH_MODE oauth (the default, none, is local use only)
ARCHSIM_SECRET_KEY 32 or more random characters, the same on every API instance
ARCHSIM_PUBLIC_URL The address people open, for example https://archsim.example.com
ARCHSIM_GITHUB_CLIENT_ID, ARCHSIM_GITHUB_CLIENT_SECRET A GitHub OAuth app whose callback URL is <public url>/api/v1/auth/github/callback
ARCHSIM_GOOGLE_CLIENT_ID, ARCHSIM_GOOGLE_CLIENT_SECRET A Google OAuth client whose callback URL is <public url>/api/v1/auth/google/callback
ARCHSIM_ADMIN_EMAILS Verified emails whose first sign-in makes them an owner of default
ARCHSIM_SANDBOXES true lets visitors try ArchSim without signing in, each in a sandbox deleted 24 hours later (default false; SECURITY.md says what limits them)

Every variable, with the optional ones (GitHub Enterprise, session length, trusted proxies, the outbound policy, rate limits), is described in .env.example. With Docker Compose, put them in .env.docker (git ignores it): the API and worker containers read it. Then:

  • people sign in with GitHub or Google and get a personal workspace; whoever's verified email is in ARCHSIM_ADMIN_EMAILS also owns default;
  • the person menu at the top right switches workspaces, and Settings creates team workspaces, invites people as viewers, editors or owners, and makes API tokens for the CLI, CI and MCP (Authorization: Bearer asim_pat_... to https://<host>/api/v1);
  • runs per hour and runs at once are limited per workspace (429 with Retry-After), and the server calls a Prometheus only on an allowlist of public hosts.

The endpoints are in api.md.

Before you expose a server#

  • Use PostgreSQL with a password of your own: the Compose file's archsim / archsim is a local development default.
  • Serve HTTPS from a reverse proxy in front of the web server, and keep the API's port private: browsers, the CLI and CI reach the API through the web server at https://<host>/api/v1.
  • Tell each side about the proxy: ARCHSIM_TRUSTED_HOPS on the web server (1 behind one reverse proxy) and ARCHSIM_TRUSTED_PROXIES on the API (the web server's address), so rate limits see each visitor's own address.
  • Keep the database reachable only by the API instances and workers: whoever can write to it can queue, cancel or change runs.
  • Leave the outbound policy closed unless calibration needs a Prometheus server, and then list its host in ARCHSIM_PROMETHEUS_ALLOWED_HOSTS.

SECURITY.md explains each of these, and what remains unprotected.

Hosting a public server#

The repository's deploy/ folder is a kit for one server on the internet (ADR-0080):

File What it does
compose.yaml Runs a release's images with sign-in: PostgreSQL, the migration, the API, a worker, the web app, the way in and a nightly backup
hosted.env.example The settings: the way in, the domain, the secrets, the OAuth app, and sizes and limits you may change
Caddyfile With Caddy as the way in: HTTPS for your domain, with a certificate Caddy gets from Let's Encrypt and renews itself
backup.sh The backup service's nightly pg_dump, the newest seven kept
update.sh Moves the server to another release

It does each step of Before you expose a server: only the way in takes connections from outside, browsers reach the API through the web server only, the database sits on a network with no way out that only the API, the workers and the backup reach, and calibration never calls out. Runs are smaller than a local install allows, for a public demo: at most 1,000,000 expected requests and 120 s (ARCHSIM_MAX_REQUESTS, ARCHSIM_MAX_WALL_SECONDS), more than any template needs.

1. A server#

Linux with 2 cores and 4 GB of memory or more, amd64 or arm64 (every image is built for both), and 20 GB of disk; Ubuntu 24.04 LTS is a good choice. The defaults fit that smallest size: the memory limits add up to 3.3 GB, and one worker simulates two runs at once, with a 1 GB heap. On a bigger server, give the JVMs more in hosted.env (a container's memory limit is its heap and 384 MB):

Setting 2 cores, 4 GB (default) 4 cores, 8 GB or more
ARCHSIM_MAX_CONCURRENT_RUNS (the worker's runs at once) 2 3, leaving a core to the rest
ARCHSIM_WORKER_HEAP 1g 2g
ARCHSIM_WORKER_MEMORY 1408m 2432m
ARCHSIM_API_HEAP 512m 1g
ARCHSIM_API_MEMORY 896m 1408m

The worker's heap on a big server is the one archsim-app's README asks: 512 MB a run and 256 MB besides. The default gives each run less, about 384 MB, because runs here stop at 1,000,000 requests instead of 2,000,000, and a run's heap shrinks by about 128 MB a million. More workers (--scale worker=2) need more cores still, each with its own heap.

2. Docker#

Docker Engine 27 or later, with its Compose plugin, from Docker's own packages (docs.docker.com describes the apt repository; its convenience script does the same):

curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker "$USER"   # sign out and in again; the docker group is as good as root
docker compose version

3. The way in: Caddy or a Cloudflare Tunnel#

COMPOSE_PROFILES in hosted.env starts one of two:

caddy tunnel
What runs Caddy, on ports 80 and 443 cloudflared, which connects out to Cloudflare; no port is open
HTTPS Caddy, with a certificate from Let's Encrypt Cloudflare's edge
DNS Anywhere: an A record to the server The domain on Cloudflare; the tunnel makes its record
The visitor's address Caddy writes the address it saw into X-Forwarded-For Cloudflare appends the visitor's address to X-Forwarded-For

Either way the web server takes the last entry of X-Forwarded-For (ARCHSIM_TRUSTED_HOPS=1) and passes it to the API, which believes only the web server (ARCHSIM_TRUSTED_PROXIES=web), so rate limits count each visitor, not the proxy. Whatever a visitor writes into the header themselves stays to the left and is ignored. cloudflared passes the header on as Cloudflare wrote it, and Cloudflare also sends the address alone as CF-Connecting-IP.

4. A firewall#

Let in TCP 22 and, with Caddy, TCP 80 and 443 and UDP 443 (HTTP/3); nothing else. With the tunnel, 22 alone: cloudflared only connects out, to Cloudflare's port 7844, which most servers allow. Set this in your provider's firewall, and on the server with ufw:

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp   # with Caddy
sudo ufw allow 443      # with Caddy: TCP and UDP
sudo ufw enable

Docker opens the ports it publishes with rules of its own, ahead of ufw's; only Caddy publishes any.

On Oracle Cloud's Ubuntu images, leave ufw off: the image's own iptables rules (restored from /etc/iptables/rules.v4 by netfilter-persistent) reject everything but SSH, so insert ACCEPT rules for 80 and 443 before their REJECT line (sudo iptables -L INPUT --line-numbers shows it) and save them with sudo netfilter-persistent save; the VCN's security list needs the same ports. Oracle also reclaims Always Free instances that stay idle, unless the account is pay-as-you-go.

5. DNS and HTTPS#

With Caddy, point the name people will open, say app.example.com, at the server: an A record for its IPv4 address, and an AAAA record only if it has a public IPv6 address. Caddy asks for the certificate on the first start, so the records must resolve to the server by then (dig +short app.example.com). On Cloudflare's DNS, leave the record DNS only: behind Cloudflare's proxy, the SSL/TLS mode must be Full (strict) (Flexible loops between HTTP and HTTPS), and Caddy would see Cloudflare's addresses instead of each visitor's. Use the tunnel instead.

With a Cloudflare Tunnel, the domain's DNS is on Cloudflare:

  1. Delete any record the name already has (say app.example.com): the tunnel makes its own.
  2. Cloudflare Zero Trust > Networks > Tunnels > Create a tunnel, of the Cloudflared kind; name it (archsim). Of the install command it shows, copy the token, the long string after --token, into hosted.env as TUNNEL_TOKEN. The kit runs cloudflared itself; install nothing.
  3. Add a public hostname: subdomain app, domain example.com, service type HTTP, URL web:3000. The tunnel then sends https://app.example.com to the web server.
  4. In the zone's SSL/TLS > Edge Certificates, turn on Always Use HTTPS and HSTS (Caddy sends HSTS itself; through the tunnel Cloudflare does). Leave the managed transform Remove visitor IP headers off: without X-Forwarded-For, every visitor would count as one.

6. A GitHub OAuth app#

GitHub > Settings > Developer settings > OAuth Apps > New OAuth App, with:

  • Homepage URL: https://app.example.com
  • Authorization callback URL: https://app.example.com/api/v1/auth/github/callback

Then generate a client secret. A Google OAuth client works as well, or instead; its redirect URI is https://app.example.com/api/v1/auth/google/callback.

7. The kit and its settings#

Clone the kit alone (a partial clone with the deploy folder checked out) as the user that runs it, then fill in hosted.env:

git clone --filter=blob:none --sparse https://github.com/dante0747/archsim.git ~/archsim
cd ~/archsim && git sparse-checkout set deploy
cd deploy
cp hosted.env.example hosted.env && chmod 600 hosted.env
openssl rand -hex 32      # POSTGRES_PASSWORD: hex, since it goes into a URL
openssl rand -base64 48   # ARCHSIM_SECRET_KEY
nano hosted.env

main's kit runs the newest release's images; ./update.sh v<version> (step 10) moves it to a release's own kit. hosted.env needs the way in (COMPOSE_PROFILES, with TUNNEL_TOKEN for the tunnel), the domain, the two secrets, the OAuth app's client id and secret, and your email in ARCHSIM_ADMIN_EMAILS, so that your first sign-in owns the default workspace. Its optional settings are the sizes above, the run limits, and the backups' hour and count. Git ignores hosted.env; keep it on the server only.

8. Start#

docker compose --env-file hosted.env up -d
docker compose --env-file hosted.env ps

Every Compose command for the kit takes --env-file hosted.env, which holds the settings Compose fills in and the profile it starts. The first start pulls the images, migrates the database, and starts the rest once the API answers; Caddy has its certificate, or the tunnel its connections, within a minute. Each service starts again after a crash or a reboot.

To change the way in later, remove the old one, change COMPOSE_PROFILES (and the DNS), and start again: docker compose --env-file hosted.env rm --stop --force caddy (or cloudflared), then up -d.

9. Check#

curl -sI https://app.example.com | head -1                 # HTTP/2 200
curl -s https://app.example.com/api/v1/health              # {"status":"ok","database":"ok"}
curl -s https://app.example.com/api/v1/auth/config         # {"mode":"oauth","providers":["github"]}
nc -zv -w 5 <server address> 5432                          # from elsewhere: refused or timed out

Then sign in with GitHub with the email in ARCHSIM_ADMIN_EMAILS, open a template and simulate it. To see that rate limits count visitors and not the proxy, list the addresses the sign-in limit has counted, a few minutes after signing in at most (ended windows are cleared); yours should be there, not a 172. address of Docker's:

docker compose --env-file hosted.env exec backup psql -Atc "select key from rate_limits where key like 'ip:%'"

10. Upgrade to a new release#

Take a backup (below), then move the kit to the release's tag:

cd ~/archsim/deploy
docker compose --env-file hosted.env exec backup archsim-backup now
./update.sh v1.2.0

update.sh checks the kit out at the tag, pulls its images, migrates the database and restarts what changed, Caddy and the backup service too when the release changed their files. If the images cannot be pulled, because the release's are not published yet, it goes back to the commit it was on and restarts nothing. Going back to an older release works only if the newer one brought no migration (its worker refuses a database past its own schema): otherwise restore the backup taken before the upgrade.

The project's own demo deploys from GitHub Actions instead (.github/workflows/deploy-demo.yml): on each published release, or by hand, it connects over SSH with a key whose only command is update.sh. To set it up, make a key pair, put its public half in the server user's ~/.ssh/authorized_keys with a forced command, and give the workflow the rest:

restrict,command="/home/<user>/archsim/deploy/update.sh" ssh-ed25519 AAAA... deploy-demo
In GitHub What it holds
Environment demo, secret DEMO_SSH_KEY The key pair's private half (ssh-keygen -t ed25519 -N '' -f deploy-demo)
Environment demo, secret DEMO_HOST <user>@<server address> (the address, not a name behind Cloudflare)
Environment demo, secret DEMO_KNOWN_HOSTS ssh-keyscan -t ed25519 <server address>, checked against ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the server
Environment demo, deployment branches and tags main and v* only, so that a run from another branch never gets the key
Repository variable DEMO_ENABLED true; until then the workflow is skipped

Publish a GitHub release once Release images has passed for its tag: the server pulls those images.

11. Backups and restore#

The backup service dumps the database every night at 03:00 UTC into the backups volume and keeps the newest seven (ARCHSIM_BACKUP_HOUR, ARCHSIM_BACKUP_KEEP). The volume is on the server's disk, so copy dumps elsewhere too, or use your provider's volume backups:

docker compose --env-file hosted.env exec backup archsim-backup now                # one now
docker compose --env-file hosted.env exec backup ls -l /backups
docker compose --env-file hosted.env cp backup:/backups/archsim-2026-11-16T030000Z.dump .

To restore one, stop what writes to the database, restore, and start it again (a dump from elsewhere goes in first with docker compose --env-file hosted.env cp <file> backup:/backups/):

docker compose --env-file hosted.env stop api worker web
docker compose --env-file hosted.env exec backup pg_restore --clean --if-exists --single-transaction \
  --dbname archsim /backups/archsim-2026-11-16T030000Z.dump
docker compose --env-file hosted.env start api worker web

12. Logs#

docker compose --env-file hosted.env logs -f api worker   # or web, caddy, cloudflared, postgres, backup

Docker keeps five files of 10 MB for each container. Caddy logs its certificates, not each request; a log line in the Caddyfile adds an access log, which holds visitors' addresses. Through the tunnel, Cloudflare's dashboard has the traffic.