In our Devex Goals for Spaces post last month, we asked whether Spaces could still support Jetstream client semantics, and showed a terminal screenshot(!) of a Jetstream client reading from a space. Since then, we’ve written about Open Social Groups and the role of the Group Host, the forthcoming background service that administers and serves a group’s spaces as a whole.
This week we want to show how Spaces handle one of our core primitives: sync. And the answer may surprise you. Jetstream will be fully compatible with Spaces. This is great news, and directly addresses one of our core goals with Spaces — it means that, for most developers, there are not going to be two dramatically different ways of working with the protocol for public and non-public data. If you’ve built an app that consumes public atproto data from Jetstream, consuming space data should look very familiar.
Why Spaces Need Their Own Sync
Public atproto data flows through relays. Every PDS broadcasts its commits, a relay aggregates them into the firehose, and services like Jetstream turn that into JSON you can filter by collection. Because Spaces are assumed to be non-public due to the sync boundary of a spce, they can’t work that way. In other words, we don’t take for granted the ability to rebroadcast this data. There’s no relay carrying space records across the network. We want data shared directly with the parties who have access.
To broadcast public atproto data, there are many parts of the protocol that, from a developer point of view, “just work.” This is how we typically position Jetstream against the Relays — if you’re not doing full zero-trust validation, if you just want to be an atproto consumer, that’s what Jetstream is for. Some of those internal mechanisms are different when talking about Spaces data, but the principle here is the same. We haven’t talked too much about each of those mechanisms in our docs before finalizing Spaces (stay tuned), but there’s a lot there that you won’t have to solve for on your own.
We have talked a lot lately about the ability to self-host Jetstream. This is important to our broader ecosystem goals; we provide a high-scale Jetstream for most developers that serves data from our own Relays, but the new replay functionality in Jetstream can be computationally expensive at scale, and we absolutely don’t want to be the sole arbiter of those costs.
Soon, there will be an even better reason to host your own Jetstream — as a secure endpoint for Spaces data. As with the entire atproto stack, Jetstream can scale down to a point where it consumes very few resources; you don’t see this with the public Jetstream rebroadcasting every record on the network, but a Spaces-only Jetstream will start out much more manageable. If you have 10 users, you only have access to those 10 users’ relevant spaces. At that point, running your own Jetstream sidecar service should feel more like “filling a gap you already have” than “adding another dependency.”
Consuming the Stream
Your apps will be able to read from the sync service with any up-to-date Jetstream client: e.g., the TypeScript SDK, or a raw websocket or anything in between. A Spaces-compatible Jetstream would use the same replay-then-live model: you can start from a saved cursor and read through the archive, and eventually cut over to live events — or just start there.
For example, here’s an indexer for Statusphere status records, written with the TypeScript SDK’s LexIndexer, which calls your handlers for each record write and delete and tracks the cursor for you. The highlighted lines are the only ones that differ from the same indexer reading public Jetstream:
import { Jetstream, LexIndexer } from '@bsky/jetstream'
import { xyz } from './lexicons'
import { db, cursor } from './db'
const js = new Jetstream({ service: process.env.SYNC_URL, mode: 'spaces' })
const indexer = new LexIndexer({ mode: 'spaces' })
.commit(xyz.statusphere.status, {
put: async (e) => {
await db.upsertStatus({
space: e.space,
uri: e.uri,
status: e.record.status,
createdAt: e.record.createdAt,
})
},
del: async (e) => {
await db.deleteStatus(e.space, e.uri)
},
})
await js.runner(indexer).replay({ cursor })
The client and the indexer are now both constructed in spaces mode and pointed at your own sync service instead of a public Jetstream host. Then, each event carries a space field identifying the space the record belongs to. One stream provides every space the service follows, and the same author can write a record with the same collection and record key into two different spaces. e.did is still the record’s author.
The stream is meant for your app’s backend; i.e., not for browsers. It contains records from every space the service follows, so your app still decides which viewers can see what.
Identity on the Server Side
A Jetstream for public data doesn’t need anyone’s permission to read the firehose. A sync service for Spaces does, and the way it gets that permission is what makes it possible to run one without changing the trust model of the network.
The sync service doesn’t have an account or a password of its own, and it never holds users’ OAuth tokens. When it needs to read a space, it asks for a short-lived delegation from a user who already has read access to that space. It exchanges that delegation with the space’s authority for a credential, and the authority determines whether that user is a member. If the user signs out or loses access, the sync service has to find another eligible reader or stop syncing that space.
Write notifications work through identity as well. The service publishes its own service identity in a DID document, and space authorities send signed notifications to it when new records are written. Those signatures are checked against the authority’s DID keys, so the service only acts on notifications that actually came from the authority for that space. Account-level changes, like deactivations, come from the public Jetstream, so when a member deactivates their account, the stream tells your app to hide their records in every space.
Jetstream Spaces Prototypes
In the current prototype, the sync service runs alongside an app and borrows read access from that app’s signed-in users.
For example, here’s a screenshot of trails, a demo Strava-like app for sharing runs and rides, which is currently built on Spaces, using an adapted version of Jetstream sync. Each user shares their runs and rides as atproto records in a space shared with their friends. When Bob sends Alice a “cheer” on one of her rides, the cheer is written to the same space as her ride, and references her ride in its Lexicon, just like a like references a post. Alice’s ride and Bob’s cheer are still stored in their own repos, on whichever PDS each of them uses.
The app doesn’t fetch from either PDS directly. The sync service brings both records into the shared space, and the app joins the ride and the cheer from a single stream, like for public records in our Statusphere example:
We’ll be rolling up all of these additions to the atproto stack as we approach the Spaces launch over the next several weeks. If you haven’t already checked out the new Jetstream SDK or APIs, now is a great time. New tutorials and docs for working with Spaces, including a full example of a Jetstream sync service, are coming soon. Look forward to more Space exploration next week!