The missing post: debugging a $4 AT Protocol firehose
I run my own corner of the fediverse on AWS. serverless.social is a personal data server for the AT Protocol, the network under Bluesky, and it also speaks ActivityPub, so the same posts reach Mastodon. Almost all of it is Lambda, DynamoDB, S3 and CloudFront. It costs very little, it scales to zero when nobody is reading, and for most of this year it had one problem I could not see: some of my posts never reached Bluesky.
This is the story of finding out why, fixing it, and then noticing that the bill for the one always-on piece was three times what I thought.
The symptom
An audit of the site at the end of October turned up four blog posts that had been published, syndicated to Bluesky from the server's point of view, and never appeared to anyone on Bluesky. The posts existed on my server. The server said it had sent them. My followers had never seen them.
Nothing was alarming. No errors, no failed deploys, no alarms. The site was quietly lying to me about being federated.
How a post is supposed to leave
When I publish, a Lambda adds the post to my account's data tree and signs a new commit to it. A quick word on vocabulary, because these are not git terms: in AT Protocol every account's data, posts, likes, profile, articles, lives in one signed Merkle tree the protocol calls a repository, and every change to it is a commit, a small object holding the new root hash, a pointer to the previous commit and a signature from the account's key. Nothing about my code is involved. The Lambda then writes two things to DynamoDB: it increments a sequence counter, and it stores the signed event under that sequence number.
From there the two networks differ completely.
ActivityPub is push. A DynamoDB stream triggers a Lambda, which signs an Article and POSTs it to every follower's inbox. Each delivery is one HTTP request. If Mastodon is slow, the Lambda retries later. Nothing is running when nothing is happening.
AT Protocol is pull. A relay run by Bluesky opens a WebSocket to my server at com.atproto.sync.subscribeRepos and keeps it open, forever, waiting for binary CBOR frames. When a frame arrives the relay verifies the signature and hands it to the AppView, which is what the apps read. If that WebSocket is not open, or the frame does not arrive on it, my post does not exist on Bluesky.
That WebSocket is the reason there is one server in an otherwise serverless system. API Gateway's WebSocket product cannot send binary frames, and the relay will not accept anything else. So the firehose lives on a t4g.nano, the smallest instance Amazon sells, with 418 MB of memory, running Caddy for TLS and a small Node program that polls the sequence counter once a second and streams any new events to whoever is connected. CloudFront sits in front of it so the public hostname is the same as the rest of the site.
It worked. The audit proved it worked: a test post reached the AppView in two seconds. And yet, four posts.
Cause one: the diff was too big
Three of the four had the same signature. My server hosts more than one identity in one table, and a commit's "since" pointer, which tells the relay what the previous state was, was pointing at another account's commit. The relay computed a diff against the wrong base, found it was the entire repository, flagged the event as too big, and moved on. That was a real bug in the sequencer and it was fixed in the same week as the audit.
That left one post, and no theory.
Cause two: the box was dead
To prove the fix I published a throwaway post and watched. It reached my server. It never reached the relay. The relay's view of my hostname said it was active, but its latest sequence number was weeks old.
The firehose box had been replaced by CloudFormation during the deploy, as it is whenever its bootstrap script changes. The new box had booted, run cloud-init, and started installing Node with dnf install. Loading the package repository metadata on a 418 MB machine was enough to get the installer killed by the kernel for running out of memory. That killed cloud-init. Nothing was ever installed, nothing listened on port 443, and the instance sat there healthy by every EC2 measure, serving nothing. It had happened to at least one earlier replacement too.
The fix was to stop using the package manager. The bootstrap now downloads the Node tarball from nodejs.org and checks its hash, unpacks the server with Python's zipfile module, and pins the machine image so a routine deploy no longer replaces the box when Amazon publishes a new AMI. I also learned that CloudFormation will happily update an instance's user data in place without running it, since cloud-init runs a script once per instance, so a user-data change now forces a replacement.
With a working box, the throwaway posts reached the AppView in two seconds. The rehearsal post, a full blog article with code, tables, syndication to Bluesky and an ActivityPub Article, went out and came back down cleanly. I was ready to call it done.
Cause three: the last post before a quiet period
Then the rehearsal's Bluesky post got stuck. The relay was exactly one event behind my server. When I deleted the post, the deletion got stuck too, and the earlier event moved. Whatever was happening, it only affected the most recent event, and only until the next one arrived.
That pattern pointed straight at the box's polling loop.
The sequencer does its work in steps. Step one increments the counter and reserves a number. Step six, several seconds later for a large repository, writes the signed event row under that number. The box polled the counter, saw it had moved from 261 to 262, queried for rows 262 through 262, found nothing, and advanced its cursor to 262 anyway. When the row finally landed, the counter still said 262, the cursor said 262, and there was nothing to do. The relay never received frame 262 until it happened to reconnect with its old cursor and replay, which it does rarely.
Every post that was the last thing I published before a quiet period had been going through this. It is why the fourth missing post had no company: most of the time another event came along and dragged the stuck one with it.
The fix is a rule about cursors that I now think applies to every poll-based tail over a counter and a table written in separate steps: only advance through contiguous events you actually delivered. A gap gets a grace period, fifteen seconds here, for the row to show up. Only after that is it treated as a real hole, which can happen if the sequencer dies between step one and step six, and logged.
for (const { seq, frame } of frames) {
if (seq > next) { // a hole
if (conn.gapSince == null) conn.gapSince = Date.now()
if (Date.now() - conn.gapSince < gapGraceMs) break // wait for the row
skipHole(seq) // a real hole, log it and move on
}
conn.safeSend(frame)
next = seq + 1
conn.gapSince = null
}
conn.lastSeq = next - 1
Two tests reproduce the race against a local DynamoDB: one bumps the counter without writing the row and checks the row is still delivered when it arrives, the other leaves a permanent hole and checks it is skipped after the grace period. In production, a single post with nothing after it now reaches the AppView in about ten seconds, and so does its deletion. The relay's head matches my server's head after each one.
Then I looked at the bill
The firehose documentation said the box cost about four dollars a month. Cost Explorer, broken down by usage type, disagreed.
The largest line on the whole account was DynamoDB reads on the repository table: forty-six million read units in October. The box polled the sequence counter once per second per connection. Several relays follow this server, not one, and each holds its own connection, so every tick cost a handful of reads for a number that is identical for all of them. The fix is one line of intent: read the counter once per table per tick and share it. That takes the line from about $5.80 to about $0.70.
The second line was the public IPv4 address. Since February 2024 every public IPv4 address costs half a cent an hour, $3.71 a month, more than the instance it is attached to. The documentation still said "free while attached," which was true when it was written.
CloudFront added IPv6 support for custom origins in September 2025, which made it possible to remove the address entirely. The box now lives in a dual-stack VPC with no public IPv4 at all: a dedicated network interface with one IPv6 address, an AAAA record for it in Route 53, and CloudFront told to connect to the origin over IPv6. Everything the box talks to had to be reachable over IPv6 as well: nodejs.org is, S3, DynamoDB and SSM are through their dual-stack endpoints, Let's Encrypt is, npm is. GitHub is not, which is where the Caddy binary used to come from; it now comes from caddyserver.com. The instance metadata service still works because it lives on a link-local address that needs no public IPv4.
The first IPv6 deploy failed, which is worth recording. The attribute I used to get the interface's address is only populated when the interface designates a primary IPv6 address, so Route 53 was handed an empty string and refused it. The rollback then stuck, because CloudFormation cannot remove an IPv6 block from a subnet while a half-created instance still holds an address in it. Terminating the orphan by hand and resuming the rollback took about an hour, most of it waiting. The fix itself was one flag. The second deploy went through, and the first post through the IPv6-only box reached the AppView in five seconds.
The box itself, the thing I had been thinking of as the expensive part, is $3.11 a month and stays. With the reads fixed and the address gone, the firehose costs about $4.50 a month instead of about $14.
What I would tell myself in May
Check the relay's head against your own. Both getLatestCommit calls are public and take a second; if the relay is exactly one event behind, something between your table and the wire is wrong, and it is not the relay.
Reproduce with a lone event. Every earlier test published a post and then did something else, which masked the bug. A single post followed by silence is the test.
An instance can be healthy and empty. Status checks say the kernel is up, not that your service exists. Health is the application answering on the port.
Read the bill by usage type, not by service. "DynamoDB $7.80" is a shrug. "Forty-six million read units on one table" is a bug report.
Cost notes rot. Pricing changed under the documentation and nobody re-read it, because the number was small enough not to question. Small numbers are exactly the ones nobody questions.
If you want to see whether the fixes held, subscribe to this account on Bluesky or follow it from Mastodon. The next post will tell you.
Comments
Reply on Serverless Social