at://

Streaming Data

Working with the Firehose and Jetstream

Streaming data

One of the core primitives of the AT Protocol is the firehose. It is an authenticated stream of events used to efficiently sync user updates (posts, likes, follows, handle changes, etc).

Many AT applications that need to stream incoming data will utilize the firehose — from feed generators to labelers, to bots and search engines. In the AT ecosystem, there are many different endpoints that seed these firehose APIs. Each PDS serves a stream of all of the activity on the repos it is responsible for. From there, Relays aggregate the streams of each participating PDS into a single unified stream — the firehose.

Firehose

Anyone can connect to the firehose without authentication — this is a core feature of the protocol. To get started, open a WebSocket connection to any provider of the com.atproto.sync.subscribeRepos endpoint:

$ websocat wss://bsky.network/xrpc/com.atproto.sync.subscribeRepos

From here, you would need to read each message as it comes in, and decode the associated data. Our Go SDK is currently the most feature-complete for interacting with the firehose directly.

Bear in mind that firehose output format is one of the more complex parts of atproto, involving decoding binary CBOR data and CAR files. Additionally, the volume of data has increased rapidly as the network has grown. The full synchronization firehose is core network infrastructure, but for end users such as feed developers, we provide an alternative streaming solution called Jetstream.

Jetstream

Jetstream is the easiest way to get data off the network at scale. Filter the records you want (likes, posts, a single account), and Jetstream streams them as plain JSON over one WebSocket, the moment they happen.

Jetstream delivers data live, or replayed from history, or both at once, through the same interface. The same code works whether you're building a realtime bot or replaying a month of history into a larger application.

There are three ways to consume it:

  • Live tail — a single WebSocket that streams events the moment they happen.
  • Replay — the WebSocket plus a few HTTP calls to pull history, then cut over to live.
  • Snapshotting — a point-in-time copy of the archive over HTTP, with no live tail.

Jetstream is open source, implemented in Go, and cheap to self-host. Refer to Bluesky Protocol Services for guides on working with Jetstream.

Further Reading and Resources