Alberto VasquezOne write lock: measuring SpacetimeDB's subscription cost at 1,000 tables
A 1,000-table poker deployment on SpacetimeDB is not a compute problem. One core deals roughly 3,400 hands a minute across 1,000 bot tables, and the hot reducers do not get slower as the table count grows. What stops you is subscription design: a view declared per-sender is materialized once per connected identity and re-evaluated on every write to the relation underneath, inside the transaction that holds the database’s only write lock. At 1,000 listed rooms with 100 people connected, one beat of one hand costs 400,101 index seeks and 3.3 seconds of that lock.
We run a six-seat online poker app whose entire server is SpacetimeDB: 95 tables, 92 reducers, and the poker rules executing as TypeScript inside the database, with the iOS client as a subscriber that sends commands. Our poker engine runs inside the database is the four-month ledger of that. This is the one question I most wanted answered, measured properly in September, because a multi-table tournament is the same question wearing a hat.
The tables are the easy part
One transaction that creates N tables, seats the field and deals N first hands costs about 0.3 ms of transaction CPU per table, with exactly 38 rows inserted, 10.5 KB written and 31 index seeks per table. A 128-table start with 1,024 bot seats took 35 to 42 ms, and none of 57 such starts failed.
Hand loops are linear too: each table deals 3.1 to 3.5 hands a minute at a 20-second clock whatever N is, the hot reducers do not get slower as N grows, and the instance spends about 0.001 cores per table. 128 tables dealt 432 hands a minute on 0.11 cores.
Extrapolate that arithmetic and 1,000 bot tables is about one core and roughly 3,400 hands a minute. The server is not what stops you.
The clients are
Two field-wide views in our base subscription were declared per-sender although they never read the sender, which means SpacetimeDB materializes one copy per connected identity and re-evaluates every copy on every write to the underlying table, inside the transaction that holds the database’s only write lock. Measured on 2.10.2, one write to the tournament entry table cost:
| Connected clients | Field 64 | Field 128 | Field 512 | Field 1,024 |
|---|---|---|---|---|
| 0 | 0.12 ms | |||
| 50 | 12.1 ms | 14.5 ms | 26.4 ms | |
| 100 | 34 ms | 61 ms | 99 ms |
That is the shape of the ceiling: subscribers times rows, paid per write, serialized on one lock. A fit on those medians is about -1.2 ms + S x (0.23 ms + 0.75 us x F) up to 100 clients, and past 100 every number is an extrapolation whose model forms disagree by 2 to 4 times.
Two details make it worse than the table suggests. A disconnected client’s view instances keep being re-evaluated until a cleanup pass drops them, which we watched take 16.5 minutes both times we measured it. And the cost is paid by every connected client, not just the ones in that tournament.
Then we measured the lobby, and it is worse
The tournament views at least only matter during a tournament. public_active_game is the relation every phone subscribes to for its whole session, and it does four index seeks per listed room plus one lookup for the branch that shows a player their own practice room, repeated per subscriber, inside whatever write triggered it. The triggering write is the most ordinary one there is: settling a street touches public_table, so this is the cost of a heartbeat of poker.
| Connected clients | 8 rooms | 64 rooms | 256 rooms | 1,000 rooms |
|---|---|---|---|---|
| 0 | 0.26 ms | 0.26 ms | 0.26 ms | 0.26 ms |
| 10 | 7.0 ms | 16.8 ms | 46 ms | 255 ms |
| 50 | 18.1 ms | 62.8 ms | 222 ms | 1,391 ms |
| 100 | 30.2 ms | 110 ms | 667 ms | 3,328 ms |
Measured on 2.12.0, the version Maincloud runs, on a busy machine, so read the ratios rather than the absolutes. The seek counts are the part that no load can distort: S x (4G + 1) + 1, which comes to 400,101 index seeks for one beat of one hand at 1,000 rooms with 100 people connected. Put it against a lock budget of 50 ms per write and the lobby view alone allows about 80 connected clients at 8 rooms, 33 at 64, 11 at 256, and 3 at 1,000. We list six rooms in beta today. That is the only reason this is a growth problem and not a live one.
Declaring that one view anonymous removes the client dimension completely: 1.4 ms at 8 rooms, 4.0 at 64, 12 at 256, 26 at 1,000, and the same whether 10 or 100 clients are connected. In the worst cell that is 128 times cheaper. What it does not remove is the room dimension, because the body still walks every room on every write, about 1.2 ms plus 24 microseconds per room. At 1,000 rooms that is 25 ms of write lock for every beat of every hand, so a lobby that big needs a projection that is not recomputed from scratch. That is a different and harder problem, and we have not solved it.
The three things I would do
A 1,000-table deployment is not a compute problem, it is a subscription-design problem.
- Make anything field-wide or lobby-wide an anonymous view. We measured the remedy on one of them: declaring it anonymous instead of per-sender cut the same write from 12.1 ms to 3.8 ms at a 64-entrant field, and from 17.3 to 6.7 at 256. Our lobby view is still per-sender today, and with 1,000 tables it would materialize a 1,000-row copy per connected phone, re-evaluated on every street of every hand at every table. It is the first thing that breaks. The flip is not always one word, though: the lobby’s body genuinely does read the caller, because a practice room is listed only for its owner, and switching a view from sender-scoped to anonymous is itself planned as a disconnect-everyone change. So the real move is to publish an anonymous projection and a small per-sender view for the one personal branch beside the shipped one, then move the subscription in an app update. Adding views is compatible, and a view nobody subscribes to costs nothing.
- Do not expect a narrower subscription to save you. This was my own wrong guess, and the measurement killed it: with every client asking for one room instead of all 256, the same write did the same 51,251 index seeks and cost no less. A per-sender view is materialized per identity first, and the query filters its result afterwards. Paging the lobby is good for egress and does nothing for the write lock. The relation itself has to stop being per-sender.
- Make the retention caps global, not per table. Ours keep 2,000 hands of audit per table, which is exactly right at 200 tables and 2 million rows of ratcheting storage at 1,000.
And the dashboard the question started from
The one that watches 1,000 tables and all their transactions: do not build it on subscriptions. Build it on the two things that are already cheap. The server exposes a Prometheus endpoint per database (transaction CPU per reducer, rows inserted, bytes written, index seeks, bytes sent) which is how we got every number here. For game state, read narrow SQL over HTTP, remembering that each call is its own transaction, and have reducers maintain the aggregates you want to display, since the SQL dialect will not sum them for you.
A dashboard that subscribes to everything is just another connected client multiplying the write lock.