The watcher ate the meter: a fraud monitor that outspent the idle database it watched

I built a monitor to catch collusion and chip fraud on my live poker database. It worked. It also quietly ate almost an entire month of hosting energy while nobody was playing. The billing email arrived with 225.4911 of 2,500 TeV left, and the database it belonged to had been completely idle for sixteen hours: no hands dealt, no players seated, no timers running.

Why I watch production at all

The game is play money, which sounds like it lowers the stakes. It does not. Chips are an economy: every buy-in, cash-out, grant and purchase is a ledger row, and the whole point is that no chip is ever created or destroyed outside those rows. Players can still collude, soft-play, dump chips to a friend, or find a seam in the betting code that mints value out of nothing.

You cannot find those things in a test suite, because the suite only knows the cases you thought of. So I run a watcher against the production database: it reads every committed transaction, enriches it with context (who was in the seat, what the hand paid out, what the table’s chips did), and asks a model a fixed set of yes-or-no questions. Do these chips add up? Do these two players look like they are cooperating? Is this bot acting like no sound bot would? Is this table stuck?

Anything scored above the bar becomes a concern the operator sees, and most concerns turn out to be wrong. That is the useful part. Every false alarm I close teaches the system something exact: a rule gets sharper, a question gets rewritten, a check grows a test. The screening is not there to be right on day one, it is there to be wrong in a way I can learn from.

All of which is to say: turning the watcher off is not a cost saving. It is a safety regression.

What actually went wrong

SpacetimeDB’s hosted tier bills in energy units. Compute, bandwidth and storage all draw from one monthly credit. On the free plan that credit is 2,500 TeV, which the pricing page translates to roughly 3 million function calls, 12.5 GB of egress, or 1 GB of stored tables.

The watcher reads the database in four loops. Three of them read on a fixed timer, and that is where the whole incident lives.

Loop	What it reads	Was	Now
Counts behind the status row	11 whole-table reads per round	every 5 s	every 60 s
Transaction feed	5 cursor reads per round	every 2 s	every 30 s
Player reports	3 reads, one of them whole-table	every 60 s	every 5 min
Chip conservation sweep	The entire chip ledger, 9 reads	every 5 min	every 5 min

Measured against an idle database, that came to roughly 411,000 read transactions and 530 MB of egress per day. Using the host’s own conversion, half a gigabyte a day is about 106 TeV, which over a month is more than the entire free credit. The monitor was outspending the game it was monitoring, by a wide margin, while the game sat empty.

An idle database should cost almost nothing. Mine cost the same whether people were playing or not, because the thing watching it never changed its mind about how often to look.

Three decisions that got me here

I picked the poll rates for the demo, not for the year. Two seconds and five seconds are live-dashboard numbers. They feel instant when you are building the thing and watching a table play out. Nobody revisited them once the dashboard became a service that runs at all hours, and the rates were constants in the source, not settings anyone would think to tune.

I treated a hosted database like a local one. During development the same code pointed at a local instance, where reads are free and a tight loop is just a convenience. The production target was added later as another entry in a list, inheriting every rate. Nothing in the code and nothing in my head marked one of those databases as metered.

I never measured what I was spending. The host returns the server-side cost of every query it answers. I parsed the rows out of those responses and threw the cost field away. So when the warning email arrived, there was no answer to “which loop is expensive”, only arithmetic over interval constants. The first real number in this entire incident was measured by hand, after the fact.

The fix I nearly shipped

The obvious move was to stop watching production and keep watching the local database. One setting, no code, burn goes to zero. I did exactly that for about an hour.

It was the wrong call, and what caught it was the operator’s question rather than the engineer’s: watching only the local database means the collusion screening, the player report watch and the chip conservation checks all run against a database where nobody plays. The cost went away because the value went away. That is not a tradeoff, it is a retreat dressed as one.

The better question is not whether to watch production but how often you need to look. I went through the loops one at a time and found that the answer differed sharply.

The transaction feed loses nothing when slowed. Every read is a cursor read: give me everything after the last row I saw. A poll thirty seconds late returns the same rows as a poll two seconds late, just in a bigger batch. The only thing that changes is how quickly a verdict lands, and no fraud judgement in this system needs to land in two seconds.

The live subscription was never the problem. A subscription pushes changes as they happen, so an idle database sends nothing at all. The expensive thing was polling, not listening. I kept the subscription exactly as it was. One check genuinely needs to be connected while people play: matching a table’s chip movement against the ledger entries written in the same transaction can only be done as it happens. That check rides the subscription, so it survived untouched.

The status counts are the odd one out. They are a snapshot, not a cursor, so slowing them does lose information: anything that changes and changes back inside the window is simply never seen. That is a real cost of the fix, and it needs naming rather than hiding.

So the database got marked as metered, and the metered rates went to 60 seconds and 30 seconds, with the report watch dropped to 5 minutes. About 31,000 reads a day instead of 411,000. A test now pins the whole daily budget, every loop counted, so the next person who adds a poller has to answer to that number.

Slowing a loop breaks things you did not expect

This is the part worth copying, because the rate change itself is easy and the fallout is not. A review of the change found four places where code had quietly assumed the fast cadence.

A blind spot grew by a factor of twelve. The chip check skipped any table it had not seen in the last status read, on the grounds that it could not tell yet what kind of table it was. At five seconds that blind window was invisible. At sixty seconds, a brand new table’s first buy-ins would never be checked at all. It now asks for the status immediately when it meets a table it does not recognise.

I started attributing actions to the wrong player. Feed rows were labelled with whoever was sitting in the seat at the moment the poll ran. Read thirty seconds after the action, one player can leave and another sit down in the same seat. Blaming the second player for the first player’s bet is precisely the error that ruins a collusion judgement. The fix was to take names from the hand itself, which freezes them when the cards are dealt.

The health indicator started crying wolf. It warned when the last successful read was more than thirty seconds old, a threshold derived from the old rate. The new rate put normal, healthy operation about one second inside the warning band. Thresholds that encode a cadence have to read that cadence rather than hardcode it.

A freshness number was being measured from the wrong clock. The module’s own heartbeat was dated from when the page last updated rather than from when that heartbeat was actually read, which at the slower rate understated its age by up to a minute.

None of these were caught by the test suite, because each was a sound assumption until the day it was not.

One wrong diagnosis, for the record

While investigating, the service log showed dozens of timeouts on the slowest poller. The tempting story wrote itself: the fast loops are saturating the HTTP connection pool, so the slow poller starves. It was a satisfying explanation, it fit the evidence, and it was wrong.

A second pass checked it instead of accepting it. The runtime has no per-origin connection cap here, nineteen parallel requests completed in under six tenths of a second with no failures, and the one poller that logged failures was simply the only poller that logged at all. The timeouts were network stalls, and a separate alarm blamed on the same cause had actually fired on an earlier run entirely.

A monitor that cannot say why it failed will get blamed for whatever happened near it. Visibility into your own tooling is not a nice-to-have; it is what keeps your postmortems honest.

If you are building on SpacetimeDB

Mark your metered databases in code. Not in a runbook, not in your memory. If a hosted target and a local target can be configured identically, someone will eventually point production at a development cadence, and nothing will stop them.

Prefer subscriptions to polls. Subscriptions cost in proportion to activity. Polls cost in proportion to time, which means an empty database bills exactly like a busy one. If you are polling what you could be subscribed to, you are paying for silence.

Keep the cost the server already tells you. Every SQL response carries the duration the host spent on it. Record it per query and per caller. It costs nothing, and without it you cannot answer the only question that matters during an incident.

Check which queries actually use an index. One of my cursor reads was scanning thirty-seven thousand rows on every poll and costing more than twenty times the next most expensive query. It had grown that way gradually and silently.

Make your monitor’s failures loud. If a poller can fail silently, it will, and the first sign will be a mystery you cannot reconstruct afterwards.

Rates are a feature, not a constant. The right rate is a function of how busy the system is. I am moving toward exactly that: the subscription already knows when a hand is in play, so it can drive a fast rate when something is happening and a slow one when nothing is.

What I still do not know

Honestly: whether reads are billed the way I assume. The arithmetic lines up well with the month’s consumption, and reducing reads is obviously right either way, but I matched a measured egress figure to a published conversion rate rather than to a line item. The fix stands on its own merits. The attribution is still a hypothesis, and it says so on the ticket rather than being closed as understood.

The credit resets at the start of the month. Then I find out.