Why a New Protocol? The History and Future of AT Protocol

Jim Ray: Hey everybody, this is Jim from the Bluesky Developer Relations team, and I’m excited to welcome you to the very first episode of what we’re now calling Off Protocol. So at the start of the year, we started live streaming and we called those office hours and we had folks from the team, but then we also were inviting special guests on and we were streaming on Streamplace, of course. And one of the things that we kind of quickly discovered was that the episodes that had special guests on them were they were some of our favorite episodes and they were also some of your favorite episodes. But that live stream format didn’t really give us a chance to dig in with our guests.

And so we’ve kind of retooled a little bit. So we’re going to be continuing these bi-weekly live streams, and we’re going to call those off protocol live. And then we’re also going to periodically record some more in-depth conversations. And so these conversations are going to include folks from the ecosystem, of course, folks that have been building on the protocol. But we’re also going to be working to find folks that have been building on the open web, that work in media, researchers, critics, other folks who are just building a more vibrant internet. And so for our first conversation, we brought back Paul Frazee, who’s the CTO of Bluesky, and Daniel Holmgren, who is the head of the protocol.

And we wanted to take a look at the history of atproto. And then we also just wanted to talk a little bit about where things are going. And so as with everything on the protocol, this works best with everyone’s participation. So do reach out, let us know who we should talk to next. I’m excited to have the opportunity to go a little bit deeper with our guests and to continue to see all of you on the live stream. So now I hope you enjoy this conversation with Paul and Danny. Thanks so much. Paul Frazee, Daniel Holmgren, welcome to the inaugural first episode of Off Protocol. And to get us started, I’d love if you can tell us what is AT Protocol.

Paul, maybe we’ll start with you.

Paul Frazee: Sure thing. AT Protocol is a high-scale open social network. It is built from the ground up by the Bluesky team, but it is making its way to becoming an IETF standard. We just recently got a working group going. And it is built so that people can essentially create applications on one shared backbone, one shared network that kind of splurs the line between being inside of a company and outside of the company so that if somebody decides to build an application, same infrastructure, same kind of data space, same users. So that you’re not locked into one particular application, one particular provider. It’s just like I said, open network that is available to everybody to build against.

Jim Ray: So I think protocols are these things that exist in the world. We’re familiar with them. We use them all the time. What is it that AT Protocol allows us to do today that maybe we couldn’t have done before the protocol existed? or in the world of social media as we came to know about it. I think we’re gonna have we’re gonna have plenty of time to talk a little bit about centralized protocols, but what is it we can do now that we couldn’t do three or four years ago?

Paul Frazee: Did you did you start that by saying that everybody knows what protocols are? Who is our audience here? Yeah, well so what does this do? I so I think probably I guess we’ll just say that everybody’s pretty familiar with the idea of the web or just the internet in general. And the thing that we’re doing with that protocol that’s a little bit different than what’s come before is the kind of stepping up a stack, a st a stepping up a step in this in the kind of the layer of what’s done at the protocol level. A lot of the protocols are very oriented towards getting computers connected to each other and accomplishing some kind of transactional task with each other.

Like you know, the web uses HTTP. They’re connecting to each other to get web pages or to get the kind of the assets that drive the application or just to communicate with the with the service that you’re using. And it’s pretty unopinionated beyond that. It provides nice guardrails for how that works. And then, you know, on top of that, there’s the idea of links. And so since everybody understands HTML in the web, you’re able to jump around between these services and they’re able to display the interfaces and that’s kind of what the web does. But the AT Protocol instead it sort of gets into, well, okay, those services, they have all this data inside of them.

They have all these behaviors. Could we get that into the shared protocol so that there’s kind of a shared understanding and a shared connective tissue for the actual mechanics, the business logic and the actual data of the applications, as opposed to being kind of dumb at the protocol level and not understanding that. So rather than thinking in terms of web pages that you’re going connecting to or thinking in terms of connections that you’re making to other computers, you’re thinking in terms of users and you’re thinking in terms of records and posts and profiles and stuff like that. That’s the it’s it steps us up in the abstraction layer one step and starts to think in terms of the actual application.

Jim Ray: Amazing. Thanks for that that introduction there. Daniel Holmgren, our head of protocol, maybe you can help us walk through a little bit more about the actual pieces that go into making the protocol work. And so, you know, we got a little bit of a high level there from Paul about what the protocol is and why we’re excited to build things on it, starting with Bluesky. But can you add a little bit of additional detail? Like what are the moving pieces of atproto?

Daniel Holmgren: Yeah, totally. So as Paul said, we’re kind of like moving up a layer in the in the stack. And so we actually build on top of a lot of web protocols like DNS and HTTP, things like that. And atproto is really kind of centered around establishing this universal data space for identities and then the data that comes out of them so I kind of call out atproto as being like three pieces to atproto. You have the identity piece, which we use DIDs or decentralized identifiers. That’s a W3C spec and it’s kind of like a universal identifier that users themselves can actually own and have control over. And then repositories, which are the structure that we use to publish data into the network.

So you publish these basically JSON records that describe something going on. They’re actually encoded as CBOR, but for all intents and purposes, you can think of them as JSON that describe some social interaction that you doing whether it’s a post or a like or a follow or a profile, something like that. You publish those. Now they’re in this universal data space and applications can grab stuff out of it. recompose it into kind of like the rich application views that you expect from a modern social app. And then the so the second piece is those repositories where that structured data is stored. And then the third piece is lexicon, which is our schema system for describing what those pieces of data actually are.

It’s kind of history comes out of JSON schema. It’s very inspired by JSON screen schema, but just like a kind of a few small things layered on top that make it work well. In this kind of like decentralized network where there’s not a clear authority for stuff, it makes them extend it. You can basically provide extension points that other applications can tap into. It kind of gives this namespacing mechanisms so that developers across different app applications can collaborate on schemas. And it’s really a coordination tool. Because it’s like, okay, we have this universal data space. A bunch of applications are going to be building in it. How do we keep each other from stepping on each other?

Like we can’t just have something that’s like, oh, this is where your bookmarks are. It’s like bookmarks are different across all these different apps. And so how do you define what a bookmark is? How do you define what it is in one app versus the other one? And Lexicon gives a mechanism for developers kind of without actually coordinating with each other, like without you know, giving each other a ring and saying, hey, what are you using to coordinate around the evolution of these applications, semantics in the web. So those are kind of the three big pieces, DIDs for identity, repositories for publishing data, and lexicon, which describes the shape of that data and what it actually looks like.

And then I don’t know, should I get into kind of like network topology a little bit or the or the role? It’s dangerous.

Paul Frazee: How fast do we want to move? So the like the lexicons are kind of like file extensions or like mime types, right? Like it’s just a really simple way to establish the structure of the data. And then, as you said, like the all the data it’s basically JSON. And a lot of the JSON actually links like HTML does, right?

Daniel Holmgren: Yes, exactly. Yep. They the those records link to each other with AT URIs, which is a URI format for the protocol. And it forms this massive web of interconnected data in the same way that HTTP documents link out to other HTTP documents. and form this interconnected web. AT records link out to other AT records and also form an interconnected web, this time of social structured data rather than document data.

Paul Frazee: Yeah, I imagine I imagine some people are getting flashbacks if they’re been around since the nineties about like the semantic web because in many ways there’s a lot of similar ideas. There’s the description layer which is lexicon right for describing all the data but we just chose not to use the semantic web tech which the old heads would know were was RDF. We chose to create lexicon instead. But then and then it’s linked to data and that’s it’s in many ways it’s Bringing back those old ideas, but with a with a couple of tweaks up top.

Daniel Holmgren: Yeah.

Jim Ray: You know, this was one of my first questions when I started really getting into the protocol. And I think it’s Paul, I think it’s one that I asked you. And it’s one that still kind of lingers a little bit for me. But you know, maybe it’s something that the old webheads have in mind as well. You know, folks that do remember some of these nascent S efforts like the semantic web. So it kind of begs maybe an interesting question, why do or why did we need a new protocol when we had all this structured data? I mean obviously, you know, a lot of that got centralized into what we think of now as the big platforms, but what is what is kind of structurally the problem that we’re aiming to solve here that maybe the semantic web either didn’t solve or couldn’t solve or, you know, we just didn’t have the technology 25 years ago.

Paul Frazee: Well, I mean of course the first thing is it wasn’t invented here. So we had to fix that problem. Yeah, so there’s there are a couple of reasons why we didn’t go with semantic web tech or some of the other options that were out there. And it’s a you know a lot of near-miss aspects of it. But one of them is that so early on actually I did some pretty deep evaluations into using RDF, came close, but the complexity of RDF is quite difficult for a lot of developers to wrangle. And I know enough of the people from that community that they’re gonna be frustrated to hear me say that, but it’s just the truth if I had another six months, maybe I would have been able to build the entire thing on RDF.

But RDF is like the assembly language of schemas or of semantics. It’s very low level in how it’s designed. And so to get all the way into the kind of affordances that developers expect, something as simple as like JSON schema. You have to do a lot of work to get all the way up to that point. And it introduced a lot of boilerplate. So we just went more opinionated with it based on how we knew the system would be used and would what we knew developers would understand. And you know, that j that was kind of a major divergence point. The other was that a lot of the existing efforts in the semantic web stuff like Solid or you know arguably I guess you could say that ActivityPub is in the semantic web category because they do use RDF they were built around fairly point-to-point transactional thinking, which doesn’t scale terribly.

well. I led by talking about it being in a high-scale open social network. And that really was one of the biggest things that caused us to diverge from how everybody else does things. We knew from the start that we needed to be building services that could be decentralized and distributed, but also handle millions upon millions of users. And the only open technology that’s really accomplished that so far is the actually the traditional web with search engines crawling all that data into these low high scale kind of indexes that are capable of aggregating all that. So we wanted to take a similar model of okay the applications are going to handle that high scale by aggregating across the entire network instead of trying to do point-to-point exchanges for everything.

We knew that that could scale and then we had to introduce more sort of real-time synchronization guarantees to make that work. And so essentially what we did is we took how a traditional data center would work with a bunch of microservices and said we’re gonna turn the entire internet into that where you have the primary data stores, that’s where all the those repositories of JSON live. And they’re going to get actively replicated. They’re going to be like almost like Kafka streams going into the big high scale indexes to make sure that the high scale aggregations of all the data can actually be produced. And you look at how the other systems are built, they’re just not built for that.

In particular, you have to have a pr a very specific signing scheme to maintain the make sure that the data is authentic at rest whenever it’s moving through the system. And without building for that in the entire kind of data model of the of the protocol, you end up running into trust problems when you’re trying to aggregate all that data. So that’s why we built the that’s why it’s AT Protocol actually stands for authenticated transfer, because that was the first thing that we knew that we had to fix was providing these high-scale aggregations. We’re going to need at-rest signing of the data to make that architecture work.

Jim Ray: And I think that aggregation is another piece that I didn’t quite understand when I first kind of came into this. And you know, as you originally explained it to me, Paul, the idea that the entire network works in some ways like a distributed open index similar to the way that we think of like a search engine index.

Paul Frazee: Yeah, yeah, that’s right. I mean, I suppose we you can’t share the actual created index. We don’t do that at the mobile, but what you do is Indexing is core to the model. Everybody sort of publishes in their own repositories, and you could almost think of their repositories with all the JSON in it as being like websites that use JSON instead of HTML. And so then the applications are going and looking at your repository and you’ve got a collection of posts and you’ve got your profile record and you’ve got your collection of likes and they all just kind of live on their own inside the repository. But the applications then pull them in like a search engine would crawl them and then looks at those links and goes like oh okay so this like was a like on a post over here so I know how to count that like correctly, or this w this post was actually a reply.

And here’s the link to what it was replying to so we can assemble the thread correctly. So.

Daniel Holmgren: the indexes themselves aren’t necessarily open. Any anybody can crawl the data and build an index and they might make it open to like they might provide an open API for it, but it doesn’t necessarily have to be. But the network that allows for the creation of those indexes is the thing that is sort of locked open by the protocol. And as Paul said, it really does work pretty similarly to crawling the web for documents and building a search engine, but that was kind of grafted onto the web a little bit after the fact in a sense. And like we provide what AT does is it gives use some things that make basically make the data hypercrawlable.

It makes it very easy to locate. It makes it very easy to get real-time updates of it so that you don’t have to keep running back to the same server. And it’s also all signed, so it can be rebroadcastable. So this crawling function can be pulled out into, again, as Paul said, like its own microservice that then different applications or different indexes can subscribe to get the to get the content. So it’s like it functions similarly to web crawling, but with extra facilities that make it easier, more complete, more consistent, faster, all of those things.

Paul Frazee: Yeah, I mean like websites, HTTP doesn’t have a notion of like a change log that you could subscribe to for a website. And that’s kind of what we introduced.

Daniel Holmgren: Yeah, yeah.

Paul Frazee: Yeah, yeah. So what do you end up doing is like kind of polling over and over, like the crawlers have to go and you know, that’s essentially like a poll model and you know it’s worked out, but it also limits what it could do. Like you just use it for a search stuff and then like RSS was trying to kind of graph that on. Still poll based, and still pretty limited. If you created a version of RSS that allowed for active sync and which was a little bit more rich in the kind of objects it can describe for what’s on a website, you’d be maybe 50% there for what AT Protocol does.

Jim Ray: And I think that actually brings us to kind of good point. So before we go too far down where things are headed, I’m interested in trying to figure out how we got here. And so you know, one thing that’s pretty interesting about the protocol is I’ve always been really attracted to the way that a lot of the decisions were really practical. You were you were kind of talking a little bit earlier about the decision not to use RDF, for instance, and kind of work a little bit higher up. And I think this often comes up in ideas of decentralization or you know what’s real decentralization. I think peer-to-peer often gets stood up as kind of an example of true decentralization.

And atproto is obviously not the first decentralized protocol. And Paul, it’s not even the first decentralized protocol that you’ve even worked on. And so I know you’ve got a little bit of history working on a project called Beaker and building other peer-to-peer networks. So tell us a little bit about how it is that you came to working on this decentralization tech.

Paul Frazee: Yeah. And well and before Beaker it was Secure Scuttlebutt, but the I’ll kind of broaden it even to a full-on history lesson. I think 2010 to 2020, I kind of peg as a decade of a lot of decentralized protocol sort of exploration that was happening in the community. There were three main camps that were attacking kind of different branches of technology. There was the peer-to-peer camp, the blockchain camp, and the federation camp. And the Federation camp went to ActivityPub, the blockchain camp. I don’t think I need to describe what they all did. And the peer-to-peer world was like IPFS and dat, which later became hypercore and Secure Scuttlebutt stuff like that. Probably the peer-to-peer one is less known by many people because the other ones have gotten kind of proper coverage or you know, federation is an old premise, so they need an explanation.

But yeah, the peer-to-peer, the whole peer-to-peer category was a bunch of people that looked a bitTorrent and you know and the fact that we had elliptic curved keys, so it was now possible to have more compact public keys that could actually fit as identifiers. They were looking at some of the data structures that the cryptographic data structures that like blockchains were using and went, well, okay, maybe we could make a more sophisticated BitTorrent that’s actually capable of running application networks. Going beyond just file syncing. BitTorrent original there are some kind of updates that made it possible to like modify the files inside of a Torrent once it gets secured. But the original premise was it was kind of frozen.

enumerate a set of files, you would calculate the root hash and the that was of the Merkle tree and that was it. So for IPFS, for data, for SSB, they were they were kind of like creating a linked list that was signed was Secure Scuttlebutt. That was kind of a similarly it was actually a weird Merkle tree that functioned like a like a log. IPFS was all about a kind of a Merkle forest, which is a giant interconnected web of hashed objects. And so we were creating these BitTorrent variants, seeing how far we could how much computing at scale, networked computing at scale we could do with purely edge devices, purely people’s computers and no servers.

I’ve you know, the reality is that the there are limits to what you can do. There’s a lot of challenges that come along. And we were able to make some pretty good forward strides with what was possible. But you know, peer-to-peer is challenging. And the thing that you run into key sync trouble, you run into connection establishment punching holes through NATs. The DHT for peer discovery is an entire category of really challenging problem domains that nobody has actually, in my opinion, really solved with the level of resilience that you would need for a kind of a production applications network. The biggest problem though, the thing that you ultimately we kind of a number of us started to realize you just can’t work around is the need for high-scale indexes.

You are just not going to be able to store the ultimately terabytes of data that an edge device needs or any kind of application needs to run something that feels like a social network. These are, you know, they’re high-scale systems. Even if they everybody did have that kind of disk, there’s a sync time, you need to have the thing persistently running and keeping up to date. And we were hoping that we would find some kind of experiential compromise that like hid away the fact that you weren’t working on all the data. But the reality is that’s just not how social networks work. Social networks, you’re browsing around very large public networks, even if your concept of it is a small group.

Chat office productivity, those things you could probably pull off in a peer-to-peer setting, but large-scale multiplayer, no. So with that realization, we started to think, okay. Well we like some of the properties of what peer-to-peer systems give. In particular, we really like the ability to treat the services or the individual devices involved in keeping the network online as interchangeable. All of the data inside of a peer-to-peer network is designed to kind of think in terms of the data, not in terms of the server. And so we thought, well look, you know, and Daniel was on a similar train. And in fact, I would say that the kind of the initial Bluesky team, one of the things that brought us all together was we all had this parallel realization of like, oh, take the peer-to-peer tech, move it up on the servers.

So that you have the servers acting as almost a peer-to-peer mesh with each other. And then you’re going to get that kind of account portability. You’re able to move accounts across servers. You’re able to have an open data sharing network. But without the downsides of the edge devices trying to actually provide the services to the network. And that’s how we got to building atproto.

Jim Ray: And so you mentioned those kind of three categories of the 2010s. You know, you’ve got P2P, you’ve got crypto and blockchain, and you’ve got federation. Where does atproto ultimately land in between those three things? Because, you know, if you squint, you can kind of see pieces. And I think, you know, there’s often some discourse about like where we fall in between which ones, but you yeah, I mean you can kind of see parts of all three of those.

Paul Frazee: A secret fourth thing? Yeah. That’s right. Yeah. I tried to invent a term for it at one point because I was yeah, we were so stymied by try to describe it to people. And yeah, I tried the term internection, which is a little bit, I think, too silly for some people. But it is a real word, actually. If you look at the dictionary, it’s listed archaic, which this is what you want on your technology.

Jim Ray: Absolutely.

Daniel Holmgren: I do think that we have our feet most firmly planted in like the peer-to-peer space. Like I consider that sort of like the intellectual history.

Paul Frazee: But if you said that to somebody, they would be so confused.

Daniel Holmgren: The technology is the same. And there’s this there’s the thing that drew me to like peer-to-peer technology is that like data in those networks cord sort of has this you get abstracted away from the actual network topology and the machines. And so data sort of has this like essential quality. It’s like I can reach into this network this fog, this like whatever, and say I want this piece of data and pull it out of there. And it’s like the details of how I actually got that are all abstracted away and handled by the protocol or whatever. And I do think that we hang on to that piece. Instead of basing it off of like content, we root it in identity.

instead, but we hang on to that piece of kind of creating this universal universally addressed data network that’s abstracted above the actual underlying topology so that was always my favorite piece about peer-to-peer stuff. I think that we retain that and then we use a lot of the similar technology. Although there are things from kind of the federated networks and blockchain networks that do pop up in atproto. I think it’s and as Paul said it’s probably a secret fourth thing. But I think most closely our intellectual roots are in or are in peer-to-peer.

Jim Ray: In peer-to-peer. Yeah.

Paul Frazee: Yeah. So I mean I one of the things I’ve been trying out is calling it atmospheric computing, which is just kind of a pun that we haven’t even properly introduced. We talk about the network as the atmosphere. But one of the reasons I actually kind of like that is that it you one way you could conceptualize it is like peer-to-peer for servers, so you could say it’s cloud to cloud. Might be asking too much of people, but it is you just a second ago, Dan, you were talking about like the mesh or the fog of or whatever. Well it’s the atmosphere. It’s the atmosphere of devices. You know I don’t know. We’ll see if that how well that catches on.

Jim Ray: And Daniel, you wrote a pretty similar document that I think gets into a lot of the things that we’ve discussed, you wrote this AT Protocol ethos document. You actually presented it at the first Atmosphere Conf. We just wrapped the second AtmosphereConf about a month ago. And so you know we’ve got two on the books. And so but I you know that’s a document that I actually returned to a lot, you know, especially when I was really kind of getting into this. And maybe walk us through some of the kind of foundational pieces of what you think of as the ethos and maybe how you’re thinking is updated in the year plus that that’s transpired since then.

Daniel Holmgren: Yeah, totally. And I actually go back and reference it some too. I’m working on our current like permissioned data and went back and reread parts of atproto ethos because I’m like I want to make sure I’m holding it.

Jim Ray: That’s great.

Daniel Holmgren: But yeah, I kind of try to trace our like the maybe the intellectual history and the movements that influenced proto and then the things that we added on top of it. And I like sort of like locate atproto as the synthesis of three different movements. One is the web, which we all know and love, and there’s kind of like the original document-based web 1.0, but then also the modern social web two point oh it’s like it we didn’t go into this like doughy eyed nostalgic and being like we need to go back to web 1.0 we like the modern whip. We want to retain the good things about it, but we also lost something in that transition and we kind of want to recover that thing that we lost and then the two other movements that kind of influence atproto are the peer-to-peer movement, which I think that we just spent a good amount of time talking about, which was sort of a backlash against this evolution from Web 1.0 to Web 2.0 and seeing the things that we lost in there and the shifting power dynamics in the network.

And then the sort of like large-scale distributed data-intensive systems that powered the emergence of like Web 2.0. We learned all of these new techniques for it’s like okay, you got petabytes of data now and users are expecting to get their like next view in thirty milliseconds how are you going to make that work? Right.

Jim Ray: Right.

Daniel Holmgren: And there’s entire books written about this. The most famous one is designing data intensive applications, which I think is right there by Martin Kleppmann but there’s entire like books written about this. There’s a ton of new database technologies that emerged over the last couple of decades that try to facilitate this, a bunch of design patterns. people came up with for how to do this. And we were looking at all of those and we said, okay, those are clearly going to be important if we want to create a high-scale modern social network. And so atproto is kind of trying to take the take both technical and philosophical pieces from the web peer-to-peer movement and then these large-scale data intensive applications and the design patterns that emerged around that.

And then if I had to call out two things that we add on top of it are, and I kind of mentioned this earlier, but like the web has location based addressing you go to a website and you load something from that website, right? And you know that that’s the thing that you got from the website because you asked the website what it you asked the location that you were looking for what it was and it sent it to you. And so that’s how you know that that’s the thing there these peer-to-peer networks kind of abstracted away the network topology and locations and gave you that what I was talking about earlier, that like essential quality to the data that you’re asking this mesh for this piece of data, and you can kind of pull it out of there.

And the details of how you actually get that data to your machine are abstracted away by the protocol and we wanted to keep that essential quality of the data, but instead of having it come from like being content addressed or something like that, it’s identity addressed. So every AT URI, the authority section of it, is the DID, decentralized identifier. And so it’s always the data is always flowing out of identities. It’s not flowing out of a location. It’s not just a mishmash of a bunch of objects that you have to scoop up. It’s identities and data flowing out of those. And so yeah, identity based authority. And then the other kind of like characteristic thing of atproto is we have this like we have the segmentation between this generic data layer and then the rich application layers on top of it.

And that’s in contrast to something like ActivityPub or Mastodon, where you kind of have these full stack social servers that provide the account, the data, but then they also give you your timeline and you also log into them and they give you your web views. It’s like a full stack social media experience. And what atproto does instead is it draws the line horizontally and it says, okay, we have generic data hosting. It’s like commoditized. The places that you host your data, it’s cheap, it’s easy. My personal data server has no idea what I’m doing on it. It doesn’t, it doesn’t know that these are likes, it doesn’t know that these are posts. It goes, these are dumb JSON records, I’m signing it, I’m making it available.

And then having that generic data layer, that’s really where the protocol is, then a bunch of applications and these full network indexes can build on top of it and imbue that generic JSON data with the rich application semantics that people are familiar with. From modern social apps. And not just applications, but algorithms and moderation services and developer tools and search engines and you know, whatever you can think of can scoop up all of this data and then imbue it with the application semantics that they actually know how to process and imbue the data with.

Jim Ray: Yeah. I think that composability breaking down what we think of when you when you typically think of a web server, for instance, you know, a web server has a bunch of different component parts. A web service might have, you might have database servers, you have all these different kind of microservices or things like that, but they tend to sit in a in a stack and the composability of all of those things and the breaking down is something that I think is really unique to what we’re doing. You were mentioning that your PDS is, you know, it does it doesn’t have any awareness of what else is going on, but it’s also, you know, my PDS is on a little Raspberry Pi that sits on my desk.

And there are there’s actually a number of bits of infrastructure across the network that are able to run on pretty modest consumer hardware. And I that that part I personally find pretty compelling and I know it’s something that has attracted like certainly a certain kind of developer, folks that have home labs and folks that are kind of interested in and really running their infrastructure themselves, not just having something that’s on somebody else’s cloud somewhere.

Daniel Holmgren: Yeah, totally. And we wanted, you know, the inher the openness of the network is inherited from the open hosting layer, which is the personal data servers, the PDSs. And so we wanted that to be cheap and easy and commoditized to participate in. And so similarly, I think my PDS runs on like a $4 a month. VPS or something like that. You can totally run it on a Raspberry Pi. They’re very cheap and easy to run. We didn’t want it to cost like $100 a month to host your host your data online because then nobody could do it and nobody could have that that sense of ownership of their data.

Jim Ray: Right.

Paul Frazee: Yeah. You know, a good way to think about the PDS. It’s worth trying to make sure that people really understand what a PDS is. But an analogy might be like imagine if like Dropbox could run applications for you. Like if you when you signed into Twitter, it was like, all right, give us your Dropbox so that we know where to store your tweets and then, you know, that’s now where they’re gonna live. And then if you switched over to, I don’t know, threads or Bluesky or something like that, you would say they would also be like, okay, give me your Dropbox so I can find your content. It’s the exact same idea. It’s like a little personal cloud.

And that all the applications connect to and that’s where they’re gonna be storing your data, that’s where they’re gonna be reading your data. And because all the applications are connecting to your PDS, your personal data server, they’re all working off the same data. And so that’s how you have this account getting shared across all the applications. So we really kind of broke that this is the s the simplest way to simple you know to talk about the AT Protocol is that you got the PDSs and the applications. And we really snap those two in a half. Those are the pieces that that have to kind of be independent of each other. It’s the most essential topology of the system.

And then from there we did subdivide everything quite a bit, which is that composability you’re talking about. It’s quite modular and so a lot of the subdivision from there is actually more on the application side, like the part that goes around pulling all the data from the PDSs and turning them into an event stream, kind of like the like a Kafka for the entire network. That’s called a relay. That’s something that we made a service that anybody could also tap into. So you know you could have just said as part of the application, but we decided to go ahead and share that. because the data aside, you don’t really lose anything. It’s just a nice convenience for everybody.

Jim Ray: Yeah. Well, before we get into kind of where things stand right now, I do want to spend a little bit more time on the on the history of things. And one thing that is interesting about how the team has approached things is while y’all were building out the protocol layer. It was also important to build out a kind of canonical example application that could live on top of that. And that is Bluesky, which if you know folks that are on end of the Protocol have almost certainly heard of Bluesky. Paul, I’d love to just kind of get a little bit of maybe a high-level overview of what that was like in the early days to one, you know, make the decision like, hey, we’re gonna build a microblogging platform on top of this protocol.

It didn’t need to be a microblog platform, it could have been, you know, other things, and we’ve seen other types of applications built on the protocol but you know what was that decision process and then what were some of the challenges as you started building out this the network that we think of as Bluesky on top of the protocol on top of the protocol that is atproto?

Paul Frazee: Yeah, this is we have a lot of deep lore on this one. So for anybody who doesn’t know, Bluesky was created as essentially a consultancy to Twitter to provide a protocol that Twitter could adopt. And that drove a whole lot of the thinking in the early days. In fact, you know, I mentioned before, you gotta build for scale. We had a very specific customer in mind about and the scale profile in mind. Like, well, if Twitter’s adopting this, it really has to scale well. So we were somewhat locked into that from the from the start in terms of our thinking. And we began working on the Bluesky Client as a reference client to build against because we have a pretty firm philosophy after doing this for quite a long time that protocol development always needs to happen in conjunction with actual application development.

You just don’t want the technology to drift from the specific needs of the application developers. We’ve seen too many times how much that can cause problems. So it was really a part of the protocol engineering process to put together the reference application. which eventually became Bluesky. We started at the beginning of twenty-two and so if you remember your timelines, Elon put in his bid for Twitter gosh, was it March or April, about four months into that year?

Jim Ray: Sure.

Paul Frazee: And so we ended up deciding to launch the application after I think around like October was when we started the private beta. Or at least we decided to lock in on it. Maybe it was maybe a little bit still fuzzy there. I suppose it was kind of at the turn of the year, roughly, that we started really taking in people. And then I think really the biggest surprise about all that relative to like what our plans you know our plans were to improve the technology and you know get into this the kind of evangelizing the protocol thing and the big surprise for us is that we had users that actually stuck around. That w that was the thing.

I had worked on so many different you know, protocol projects before many of them had social network front ends. And so we started to have users come in and I was telling Jay and Daniel, you know well, people will come in, they’ll kick the tires, then they won’t come back, so don’t get too excited. And then they did keep coming back and I that was a very surprising feeling for me.

Jim Ray: One of my overriding rubrics of kind of evaluating the tech industry everybody’s so focused on how not to fail. There’s so few people that that are that try to figure out like, well, what if we do what happens when we succeed? Informs a lot of those early decisions.

Paul Frazee: Yeah, it really does. I you know, I don’t know. The it was surprising. I think I think what I was just expecting was it to be much more of a slog to sell it. But the market opportunity just really presented itself in on unique ways. And so the adoption of the Bluesky application kind of flew out of our hands. It went much faster. The demand was much higher than we expected. And that was really quite a challenge because we still needed to wrap a lot of the protocol. We were not prepared to scale the system. We were not prepared. We you there are two things you gotta scale whenever you’re running a social network, and that’s the actual technical infrastructure and with the trust and safety team and you know we’re doing everything we can to get that done while also staying true to the protocol goals.

And so we made the call not to open up we did a an invite-based system initially and there was just one rubric that we had for opening it up. Well, two, I suppose. One of them was the scaling, but the other one was is the is it actually open? Are we you know, staying true to the mission. And we chose to wait, and that took us over a little over a year, relative to or about a year. You know, it was the following year and the February, I want to say that we finally opened things up because we had the actual open sync the I guess what we call federation in place. So that was a tough call because people really wanted in.

There was a market opportunity that we wanted to be sensitive to, but we just we knew this was a line in the in the sand that we didn’t want to cross. We wanted to make sure that we stay true to what we were doing.

Jim Ray: Yeah.

Paul Frazee: And I don’t regret that call, even if we did end up dropping some possible upside.

Jim Ray: And Daniel, you know, one of the one of the many tensions that I’ve noticed within the company and within the work that we’re doing is there is the kind of typical need of a technology company, especially a social media company, to move quickly and to adapt. But the work that you’re doing on the protocol side actually kind of demands some of the opposite of that. You know, you want to be thoughtful. You and Bryan have been working on pushing the protocol actually through the IETF standardization process, which is it certainly doesn’t operate at kind of Silicon Valley scale and speed. And so, you know, especially during, you know, that Paul that that time that Paul was talking about with the scaling up and with the users who were actually sticking around.

What were some of the tensions that came up for you as you were working on getting the fundamentals of the protocol in place, making sure, as Paul mentioned, that you were actually sticking to kind of the fundamentals of keeping it open?

Daniel Holmgren: Yeah. Oh man, it was tough. Yeah, I like there’s so many like little things that you want to nail down in a protocol because if you don’t nail it down, then it’s just like you’re just stuck with the corrupt. It’s like these things are so hard to upgrade because they’re deployed across thousands of different machines. And specifically with atproto, it’s a stateful data protocol. So if you write something into a record, it’s like really hard to update. Those and so decisions that you made maybe on a whim, you might have to live with for years and years and years. And so we’re trying to nail down these like low-level details and stuff, but then at the same time we have, you know, hundreds of thousands of users onboarding and we have our PagerDuty going off and screaming at us and we’re just like, we got to keep this site online.

I don’t care about the AT URI syntax right now, you know. And so yeah, it was it was a tough it was a tough juggle, but we tried to carve out time to like had kind of do the deliberate work on the protocol. As you said, it’s a different type of thinking than like normal startup lifecycle because like with startups you’re trying to move you’re trying to move quickly and small mistakes are fine and you can correct it and it’s all good. With protocols, as I said, you really have to be slow and deliberate about it. So what how we’re handling that now is, well, one, we’re taking the we’ve established a working group at the IETF for the data protocol, data side of atproto, basically the repository structure and the URI format and the records.

And so hopefully they’ll help us fix up all the dumb things that we did along the way. And then we also carved out this protocol department inside of Bluesky, which I currently lead and I work pretty closely with Bryan over there that where we focus on protocol design and development and also on some of these governance things in the standardization process, stuff like that. And we work really closely with everybody else from the team, but kind of the goal of that is to carve out a space inside of Bluesky where you can do kind of the slower, more deliberate work that doesn’t get caught up in the you know, whatever the fire of the day is, or the very fast-paced like startup way of doing stuff.

So.

Jim Ray: yeah.

Paul Frazee: If you will recall, I said we had we spent about 10 months before we decided to start a private beta. That included designing the protocol as well as writing the application. And I don’t know if people remember this, but in those days you wrote code by hand using a keyboard. And I would say that we have remarkably few regrets. In terms of things that we locked into. And you know, now we’re you know, as we look at it I’m really pretty happy with how rarely I go.

Daniel Holmgren: I don’t I don’t have any I don’t have any huge regrets. The m the order of my regrets are like ATURIs, I wish we had said they have to be you can’t have a handle in it. Like that’s the example of like a regret, a regret that I have. It’s just kind of like it’s a wart, but it’s not like I don’t know, it doesn’t it doesn’t.

Paul Frazee: Yeah, yeah. Yeah.

Jim Ray: Pretty manageable.

Daniel Holmgren: Yeah.

Paul Frazee: Yeah. And the also the did syntax inside the at URIs is biting us now because it’s yeah, it’s actually a nice appliant. So yeah, but.

Jim Ray: yeah.

Paul Frazee: Yeah, yeah. Which is what the IETF standards process will help us iron out.

Jim Ray: Amazing. Well, kind of to that effect, are there things that you know, it’s it sounds like maybe like against all historical measures that this protocol is actually, you know, you baked it pretty well. Are there things that are kind of like working better than you expected, or you know, anything that has turned out to be a little bit harder than you thought it would be at this point?

Paul Frazee: The biggest pain point for us has been OAuth. Which is not even.

Jim Ray: every developer who has touched OAuth is nodding alone right now.

Paul Frazee: Yeah. Which is you know, wasn’t even us.

Daniel Holmgren: I do think that there’s sort of inherent complexity in that space. The really hard thing about OAuth is that devs just run into it’s the first thing that you run into. to and.

Paul Frazee: the gate.

Daniel Holmgren: It’s the most complex part of the protocol, which is just pretty harsh. I’m like, I wish that you could create a post and then have to do login.

Jim Ray: Yeah, exactly.

Paul Frazee: Yeah.

Daniel Holmgren: But yeah, there’s complexity around OAuth. One thing that I think has gone better than I expected it to I expected to hit scaling limits on our firehose fast like before we did. And I think like the current iteration of it will probably scale up to I like it is a single point or it is like a single pipe and eventually the network will too be too big for a single pipe. So we know that we’ll hit a scaling limit there, but I expect the current shape of it to scale up to at least 10,000 events a second. Or something like that. So.

Paul Frazee: we’ll just shard. Because okay, well maybe that face makes me think of walking into a train.

Daniel Holmgren: Yeah, but that that’s gone better than I expected. I don’t know what’s gone worse than I expected. Oh, another thing that went better than I expected, like Lexicon, it like worked pretty well. For us, but I was like, are people gonna like the or are people just gonna be like that’s what gone like really well. People have really taken to it. Our publishing lexicons are coordinating on it are collaborating on lexicons, like three different companies coming together to work on the standard.site lexicon. It’s just like really cool. It’s like all of the coordination things that we were kind of hoping for with Lexicon are actually happening. And I didn’t know if that was like a crazy pipe dream or if that was yeah.

So it’s just it’s just cool to see that type of coordination happening.

Paul Frazee: Yeah, same on that. Actually, in a lot of ways the I had concerns in general about the overall complexity of the technology and whether or not people would have trouble understanding it. And if the early days people were a little overwhelmed but it pretty quickly people started to really get their arms around it and now it that is just not my chief concern at all. It seems like the community has really done a good job of kind of getting up to speed and working with it. So and Lexicon is just an example of that.

Jim Ray: Yeah. Yeah, I’ve been incredibly impressed with like you said, some of that coordination around I think standard.site is the one that we point to often. It’s certainly one of the more successful ones. But you know the way that people are coming together with some of these shared resources very much intrinsic to kind of the nature of the protocol. And if I and I if I can just make a shout out to the ecosystem about how just an incredible group of people. You know, a month ago we all got together up in Vancouver for the Atmosphere Conf. They’ve all been welcoming to me personally. I’ve been at this job for six months now. And you know, just felt a really it’s a r it’s a pretty incredible small and growing ecosystem of folks.

So I want to spend just a couple minutes on kind of where things are going. And Paul, I want to kill off something you said, you know, back in the day we had to write all of this code by hand, and that has certainly changed in the last six to twelve months or so and you know, one thing that we’ve noticed, and I feel like this is one of these happy accidents, but it really does seem like the protocol is pretty well designed for working with AI coding assistants. You know, lexicons like I can throw lexicon at Claude and just because it’s almost self-documenting in a way, like I don’t have to do a lot of explanation.

And so I’m kind of curious like how you’re thinking, you’ve been working with some of these AI coding tools. We’ve been building things internally and even you know there’s projects like Attie. And so I guess I’m curious like how are you seeing how ai is working with these with the tools of the protocol itself, is it is it true to say that the protocol is actually kind of uniquely suited for building on AI?

Paul Frazee: I would say so and not just because I want that to be the talking point that people no, it does actually work pretty well because like agents there’s a couple of things that they like. They like clear kind of I don’t know how to put it you want to reduce ambiguity for a coding agent, especially if you’re as the technical like knowledge of the operator reduces so you have like a whole class of like prosumers essentially that are starting to write software that they traditionally don’t, right? The less that they know to guide the agent the more trouble they’re going to run into if there’s ambiguity about the options, if there’s more than one way to do things.

So having opinionation in the entire system is such a really big strength that that’s one of the ways that you know, atproto really helps is that it’s actually introduces even more opinion relative to you know other similar technologies. The other thing agents really like is being able to look up clear documentation. You know, if they can see a reference and then go, okay, so what is that? and then get the answer really quickly, that really helps. And so Lexicon really set up the you can for a given identifier for like a data type, you can just fetch off the network the schema, the lexicon schema. And so the agent is able to resolve documentation reliably when it’s looking at code.

And I expect we will just keep on pushing on that, continuing to put push technical documentation into the protocol so that by the end of it, if it works the way that I think it can, you should be able to point it at the ID of like the Bluesky application and it fetch all the tech technical documentation necessary to just re-implement the Bluesky application. And for that I would even include like business logic descriptions and things like that. So we’re gonna need to push into like essentially getting README’s and technical specs published in the lexicons. But yeah, it that part of it is just a really compelling angle to be pressing on now that this technology is here to see how much of this we can get sort of automated.

Jim Ray: And.

Daniel Holmgren: it also makes the OAuth thing easier.

Paul Frazee: Yeah, exactly.

Jim Ray: Solving OAuth. with AI agents. And I think one thing that we’re very cognizant of is that we don’t want the network, anything built on the protocol really to be kind of flooded with the AI bots. And I think that that’s something that we’re seeing in a lot of kind of the more traditional platforms, even to the degree that there are kind of the owners of those platforms are sort of leaning into that. And so what are some of the things that we can do to make sure that AI becomes a force for good for building new novel experiences and not just getting flooded with slop.

Paul Frazee: One of the hardest problems of technology, I’m sure we got a good answer for that.

Jim Ray: always.

Paul Frazee: Yeah. You know, Tangled is messing around with a couple of interesting premises on that with the Web of Trust. They just recently so Tangled is a Git code hosting platform that’s on AT Protocol. And they just recently introduced the ability to vouch or denounce between users to build a web of trust, which they’re introducing so that they can begin to help to filter down this the potential you know profusion of nonsense i expect that you’re gonna need something like that because you know, we’re already pretty close to it. There’s some giveaways still when it’s an agent talking, yeah, which is almost like mercifully true, right? But that just will not continue to be I mean I would probably make the case that it’s already not reliably true.

We just don’t know the extent to which it’s hard to tell. So there’s gonna have to be some means by which people are capable of distinguishing between slop and non-slop, you know?

Jim Ray: Yeah.

Paul Frazee: I do think having the system be open like we have it is great so that people can begin to try different things. It’s not incumbent on us. I’d also say that to some degree having an agent to yourself to help you parse through things and help to you know, run fact checks on your behalf and stuff like that is a pretty promising angle.

Jim Ray: Yeah.

Daniel Holmgren: Yeah. I do think that the Web of Trust stuff is pretty promising there. And it’s just like Web of Trust has a different sort of like feeling to it whenever it’s in an open network. You know, it’s like I like people talk about the everything app or something like that. And people a lot of people are very averse to it because it’s like I don’t want to have an everything app that’s owned by one company but if it’s deployed on top of an open network that it would like a multi-stakeholder resilient open network where like you’re kind of the one in the driver’s seat and you have an app that can actually tap into everything, then you get all of like the kind of usability upside of it without the like creepy dystopian downside of it.

And I feel pretty similarly about web of trust stuff where like maybe past iterations didn’t quite work because we didn’t have this we didn’t have this layer that you could do it in like a open manner. So.

Jim Ray: Daniel, maybe you can walk us through as kind of a final bit here. Some of the work that you have been doing pretty much all this year. You presented a little bit at AtmosphereConf, but we call it permissions data. And really it’s a way of setting different levels of access because right now everything published on the network is open. And so what we want to do next is allow people to create spaces for that data to live, right?

Daniel Holmgren: Yep. Yeah, that’s right. So we’re currently working on a permissioned data protocol. Everything that we’ve talked about. So far and everything that happens on atproto today is public data. And it’s sort of manifestly public data. You know, it’s like you post and it gets shot out on the global firehose to the entire world. So it’s like very public data. And that’s like a very important mode for social. And it was kind of core to our original mission was to facilitate public conversation in an open man. And a lot of the most important things that happen on the internet are done in public like that and should operate on a public data protocol. But a lot of like people’s experience with social media is not they don’t want to blast it out to the entire world.

They wanna, you know, they want to talk to their friends, they want to talk to their loved ones, they want to talk to their community. And right now doesn’t have the facilities. So we’re working on a data protocol that does support that. I’ve been writing a series of Permissioned Data Diaries that kind of like walks through where our heads are at as we’re working on this. I think we have a pretty good idea of the rough shape of this I started sketching out like an initial implementation of it and it’s kind of coming together. I’m not going to give any promises as to timeline on it, but you know, sometime this year, hopefully in the hopefully in the next few months we’ll have a pretty substantive something substantive that people can sink their teeth into on it.

And yeah, it just like kind of evolves around or it revolves around this concept of carving out a space that us that that certain people can see. And the people that can see it can scale all the way down to you, you know, if it’s my bookmarks or my drafts and only I want to see it and maybe the apps that I use need to see it. It can scale all the way down to that, all the way up to larger communities with 10 million people in them. And it’s like, you know, maybe you give up on any notion of like OPSEC whenever you do that, but at least it isn’t getting blasted out on the global firehose.

And we kind of think that we can capture that whole spectrum of use cases with this permissioned data protocol.

Paul Frazee: Yeah. This is really the last major piece that the protocol needs. And you know, from there I’ll consider it kind of feature complete at its at its core.

Daniel Holmgren: I agree. This is this is the big piece. Oh, I there was one other thing I wanted to say on it, which is that it should look and feel very similar to what people expect from atproto. It revolves around a lot of the same abstractions, a lot of the same functions. It also revolves around this like universal data space, data flows out of identities, all of that. The technical details are different under the hood, but whenever you’re holding it, it should feel pretty similar.

Paul Frazee: Yeah, it’s like carving out little subnetworks in a way.

Daniel Holmgren: Yeah.

Jim Ray: Amazing. Well, I really appreciate you taking the time. Paul Frazee, you’re pfrazee.com on Bluesky, and Daniel, you’re dholms.at, is that right?

Daniel Holmgren: That’s right. Yeah.

Jim Ray: Amazing. Excellent. So go if I’m certain everyone listening now is following, but if you’re not.

Daniel Holmgren: I did that because the next IETF is in Austria.

Jim Ray: Oh nice. Excellent. Excellent.

Paul Frazee: Pandering.

Jim Ray: Well, I really appreciate you taking the time. I hope this has provided a pretty good introduction to the protocol to everyone listening. And we’re excited to follow up with more of these. So thanks again.

Daniel Holmgren: Sweet. Yeah, thanks for having us, Jim.

Paul Frazee: Yeah, it was a good time.