“Policy Without Tools Is Just Poetry”

Jim Ray: Hi, and welcome to another Off Protocol conversation where we talk with the people who are building a better open web. So I feel like trust and safety is one of those simultaneously completely overlooked and vitally important parts of building any kind of social application. And so much of what I would identify as wrong with today’s internet starts with this refusal to take trust and safety with any kind of seriousness. So today’s conversation is with Juliet Shen, who is the head of product at ROOST, which builds robust, open, online safety tools. And they’re an organization that is committed to not just building really deeply serious safety tools, but making them available to everyone. And it’s a vital mission that they have just released version 1.0 of Coop, which is their open source review and moderation platform.

So the trust and safety tools that ROOST builds, they’re not actually atproto-specific. Anyone can use them, but they are heavily used within the ecosystem, including right here at Bluesky. And I think a lot of what makes ROOST so compelling is that they’re building these really first class tools in a way that scales from the smallest teams. You might be a one-person shop that is just getting started, all the way up to the biggest online platform. Today’s conversation is hosted by my colleague Alex, who you all know from our live streams. And Alex has actually really gone deep on the ROOST tools. And he’s written a bunch of guides for atproto.com, and we’ve got more on the way.

I personally got a lot out of his conversation with Juliet, and I hope you do too.

Alex Garnett: We’re super excited about Coop 1.0. We’re super excited to talk to you and feature your work here. And so let’s just take it from the top. It’s great to talk to you today. Congratulations on 1.0.

Juliet Shen: Thank you. Thank you. It’s a true collaborative effort. Like the community came through and we Managed to pull off a what I think is a pretty amazing 1.0.

Alex Garnett: Yeah, I was following pretty closely back around our conference and then I kind of just lost track of stuff between all my travel I was doing in April and May and I was like, when have I got a hit 1.0 after all? And it turns out it was this week, which is really, really great. I know our audience knows you as the head of product at ROOST, and I know how excited you personally are about open online safety tools. Can you tell us a little bit about why this is such a great time to be doing this? this work? Like what is cool about June 2026? What is the momentum that you have at ROOST?

Juliet Shen: Yeah, I think it’s like we’re in such a time where everyone is building something. I think like more people just because there’s more tools more technology available, like more people are building things than I think than ever before. You know, I’ve definitely been tinkering on a few side projects of my own. And I think, you know as these new exciting projects come into being, they all need safety tools because we all want to protect our users. We want people to experience the amazing experience that we have kind of like design and chart it out for them and ideally they don’t encounter anything that makes them hate the internet.

Alex Garnett: Yeah, that’s a good way to put it. There’s also obviously a bunch of mandates coming into place that we want to be able to contest, like we want to be able to be there with the larger and better funded social networks, most of whom just blow past these mandates because they expect to be given a grace period and oftentimes smaller players who have to actually abide by the letter of the law, whether or not they’re in favor of or oppose these policies coming in, like we need something to just actually get from the policy stage to the implementation stage, right? And I think that’s why what you’re doing is so exciting for us.

Juliet Shen: Yeah, you know, I think it was my co-founder Camille Francois who says that, you know, policy without tools is just poetry. You know, we just want to give people more options so that way, you know, they can make their own version of our tools, you know, do whatever they need to make the users on their community feel protected and feel safe and feel free to express themselves. And the more that we can kind of make these tools easier to build on and just like easy to deploy, the better.

Alex Garnett: I did know some public servants in Canada who would have rather been poets. So I think there is something to that, just sort of like, you know, where your mindset goes. But it’s true. It’s nice to be able to set policy. Like that is very valuable if you can just close the loop on it.

Juliet Shen: Well helps people who make policy as well, because like until ROOST came along, like to my knowledge, there were no moderation or safety tools that were open source or just like public. So if you don’t understand how platforms are able to do things, I think that limits the amount of kind of like you know robust policy that people can make as well.

Alex Garnett: Yeah, I’m always wearing my commercial open source hat and being like, all right, what part of this is like ready for folks to run and like where is the mind share aspect of like yes, our platform is open source versus this is the part that’s really hard to do that pays the bills that we know is not trivial for somebody else to do, right and those incentives are sometimes healthy and sometimes incidental and sometimes not so healthy, right? Or like the classical commercial open source story is like hey, you can run this, but it’s kind of hard to host, so be wary, right? And that’s often the case with like DevOpsy stuff where it’s like you can run our stack, but not at scale.

For moderation for trust and safety policies, being able to actually say like we’re encoding our policies in an open and reusable way. Like that is not the part that belongs in a private repo suddenly. Like that’s really, really cool.

Juliet Shen: Yeah, and I think that’s where we’ve seen the community come through. You know, I think like we see ourselves as like tools for your rules. And so we’ve seen some really cool groups and like even individuals and like civil society groups. Like we got Christchurch Call Foundation offering up policies on how to handle terrorism violent extremism content. And I think that’s where the policies need to come from is like people who are actually operating platforms or people who have this domain expertise who can say, hey, you know, like if you’re gonna use ROOST tools, here’s a bunch of policies that we think work really well with it.

Alex Garnett: Yeah. It’s really nice to create more avenues for actually meaningfully contributing. That’s also like a constant problem in open source where it’s like We’d love to have your contributions, but unfortunately, surface A, B, C, 3D, you know, none of these are easy things to contribute to. And therefore, it winds up being a case of like we want to not block each other, but like how do we actually make this constructive and how do we move fast together? So any new ways of doing that where you can genuinely create space for contributions that are mutually beneficial, it like touches my heart.

Juliet Shen: Yeah, and it’s been fun too, because it’s like, you know, I think trust and safety is such a cross-disciplinary space. You know, I’m a sociologist, I study political economics and sociology, and like yeah, I’m in a tech role now, but I think I’ve worked with so many people in the anti-abuse safety space who, you know, like their strength is writing policy. And that’s also something that they can contribute. And I think we want to make it easy for like anyone to contribute anything. Because like every piece makes it a little bit better.

Alex Garnett: Yeah I was at universities for a while and trying to figure out like where the real avenues for working with tech have like changed so much in the last decade. Like I don’t even want to get into it because it’s like good faith and not so good faith and who has some money and you know, but it’s just it’s nice to really be doing the work as you feel like it can move forward right now, which gets me to my next question. What’s on your roadmap now that you’re at 1.0? Like what are the next few weeks and months look like for Coop.

Juliet Shen: Well, I think late last night Cassidy helped cut the 1.0.1 release, which is a bunch of fixes that we found immediately after releasing. But you know, our roadmap is public. We’ve got some big ideas that we want to go for. And actually, you know, one of the open discussions that’s happening in the Coop project right now is we actually got this amazing set of PRs introducing this agentic framework on like how can you kind of like use agents for safety and use Coop as that kind of like hub. So I think that’s something we’re like, I want to get more perspectives on like, you know, is this something that would be really helpful? You know, like w does it belong in Coop?

Does it belong in the side repo? Like what does the future of side cars for side cars for Coop look like and I think that’s definitely one direction because I think like, you know, everyone’s always thinking about like, oh, like how can AI actually be used in safety, at least in the trust and safety spaces. And we wrote a blog post about this. I think, you know, human judgment is always going to be necessary. It’s always the most important part. It’s what makes trust and safety, trust and safety. But also AI is a tool and we can use these tools to help people, especially when this kind of work is difficult. So really excited for that.

Really excited to just like continue building more things. I after years of you know what witnessing q a processes done by safety and moderation teams I am committed to building in tool qa into Coop and we also want to build in like kind of like an Easter egg kind of game, kind of like a like Tetris style, but not Tetris for copyright reasons. But Tetra style game where it’s like, you know, we know that playing little games like that actually can help reduce the kind of like mental health impact of doing content moderation. So building that directly into the tool, I think won I want it to just pop out and like people can just like immediately spend like, you know, five minutes playing this game and just reset their brain a little bit and then go back to some of the work or take a break entirely.

Alex Garnett: Nice. There’s a few things there that I wanna touch on actually. The first of them is the AI piece. So there’s about three different talks with variations I’ve been giving on behalf of Bluesky at conferences throughout this year. You know, one of them is about my role as a docs person because I love docs and that aspect of DevRel. One is more about the atmosphere and the ecosystem more broadly and how we’re trying to juice developers and help them bootstrap on their social network. already exists in Bluesky. The third one is more AI-centric, and it’s actually articulating a positive vision for how we use AI in development and with Osprey. which is ROOST’s other platform that is very effective for writing automatic moderation tools and how we use that in tandem with Ozone, which is the moderation interface that our human moderators actually look at all day long when they’re taking actions.

And I think that together there’s a really powerful narrative here for, you know, the AI is not actually making decisions about who should get banned or anything. like that, right? Like the AI is not in charge of making content decisions. But for smaller shops to be able to scale, for us even to be able to bootstrap to the point that we got to, that would have been impossible without having an auto mod layer supplementing the human work and also helps take off a lot of the burden and the exhaustion that’s really like well characterized and like by like very widely described by human models. And so I think there’s a real positive vision there in terms of, you know, use AI for this, not for that.

Use AI for this for this reason. Here’s where you can actually demonstrate gains in this space. And so I don’t get tired of articulating that vision. I think it’s really important to be able to draw lines that people can see and appreciate there.

Juliet Shen: Yeah, totally. You know, I really think that like it can be so helpful, especially if you’re a small team, especially if you’re just like resource strap, like I think it could it’s really good at, you know, AI is fantastic at analyzing large amounts of data. If you’re doing any kind of analyst work, investigation work, trying to figure out like the scale of a campaign. You know, you can use it as just pure automod, but you can also use it just like kind of like track some stuff and try to figure out like there’s something going on, we’re not quite sure what it is yet, but Osprey can actually help kind of like identify those patterns so actually our next release is going to be coming out in July, which is going to be the next major Osprey release.

And we’re also excited because I think that there’s going to be some introduction of some AI powered features in there that make it easier to write queries. Yeah, I had to learn how to write SML to use Osprey, and it took a second. And wouldn’t it be nice to just like ask maybe like an LLM and be like, hey. Find me all the accounts that have this specific pattern and then it can do that analysis. And again, just provide recommendations or just information and context for people to make the ultimate decisions.

Alex Garnett: Yeah, I’m kind of surprised there hasn’t been an FML markup language yet. One of us should make one just for kicks. I have some other notes of mine that I wrote down this morning while I was actually running Coop myself. I deployed 1.0. I wanted to check it out and kick the tires a bit, and I was reading your release announcements. So do you mind if I go through like some of my hits right here? I just I have some like yeah, okay, awesome. So first of all I am so excited that you got the memory footprint down to like eight gigabytes. I think that I was really annoying about that. I know like pre-release I was like.

Juliet Shen: we needed that. We needed to hear that. Like that was a huge like citation for us. Like, we need to get this memory footprint down.

Alex Garnett: No, but like I was like, all right, I want everyone to know like this is your starting point for most people getting started with moderation and it takes 20 gigs of memory. What? And I was like, so way to go on that. I was reading all about your release notes, like how you pruned out dependencies, and like this actually feels doable. That’s like $10 a month on OVH now. That is a big achievement. That means devs can run locally on their own machines for the most part. And they can afford to add a production instance to their stack without really like breaking the bank. Like that is a really, really especially with memory prices being what they are right now.

Like that is a really important audience to capture. Just brand new bootstrappers, new social experience. I think I’ve told you that like whenever people come to us and say, we want integration in Bluesky, the app. And we’re like, hey, that’s great. We’re happy to do that. We don’t want everything to be about Bluesky the app. We care about the ecosystem, but like we understand visibility is really important. The thing is, though, you have to have a moderation system in place for it to be in front of all these people all of a sudden. What’s your moderation solution? And consistently, people who are working with the existing Bluesky lexicons, like follow relationships posts, they’re like, oh, we’re just depending on your moderation because we moderate those lexicons.

accounts but if they’re producing their own like standard.site leaflet whatever any other kind of data they’re like yeah we haven’t done that yet which is fair enough right because if you’re just brand new project like we make it really easy we’re proud of how easy we make it to bootstrap all these new data types and modalities, but you don’t get to moderation first thing as a two-person team. You just don’t. And now that Coop is like deployable in a pretty small footprint, I feel like that’s really clearing the way for that to happen.

Juliet Shen: Yeah, and that’s that was our goal from the beginning. It’s like we knew that it was, it was just, you know, when we first dropped the source code, it wasn’t designed for this kind of use case. You know, Coop started off as like a startup source code. that we then acquired and we have been refactoring ever since. We are going to make Scylla optional as well. We should also make it a lot easier to deploy, just require fewer resources to get things up and running. Because yeah,.

Alex Garnett: with the Odyssey movie, like Scylla is now optional, right? Like that would be awesome. Yeah, yeah. Get like Matt Damon just steering around it being like Scylla’s optional. You can use that.

Juliet Shen: I’m definitely doing that actually.

Alex Garnett: Sorry I jumped in there.

Juliet Shen: No, I love it. But yeah, you know, it’s like yeah, a one-person startup, two person startup. It’s like, yeah, like we want to make this super easy to deploy. And we’ve gotten some nice validation. Like we know Kyoto is running this at like less than two hundred dollars a month and they’ve got like millions of messages going across and like you know I think it’s a nice demonstration. We get such direct feedback too that it’s like we’re a small team, we can move pretty quickly, and like, yeah, we’re gonna try to fix all the bugs that people raise and add in all the features people want and still try to keep that memory footprint super small.

Alex Garnett: Yeah. So you actually also just got into my next point, which is messages coming across Coop. So I looked at your docs, I’ve got my demo instance. Everything works super great for the deployment. And like that’s very exciting to me. The out-of-the-box experience is really well. The documentation is really comprehensive, and I can tell how it works with one quasi-exception that you probably know I’m getting to, which is what do we do to make it easier to actually send messages to Coop and to like wire that into a moderation workflow it’s like you have everything in Coop. You have all these connectors to different third-party services that you can use to actually process content once it’s in Coop to do TNS checks.

You’ve got your submit items API well documented. It’s like post content here. That is how the items get into Coop. But there’s not right now like an end-to-end. for actually sending anything into that API that I’m aware of. There’s no out-of-the-box where it’s like, here’s how Coop gets some data populated from one of these sources that you might want to use it with. That to me seems like a pretty exciting thing to work on next.

Juliet Shen: Oh yes. And I can I can I feel like Cassidy’s ears are gonna be burning because this is something we’ve talked about a lot. You know, I think we know that like this is like that last missing piece of our documentation, our deployment guide of like how do we make it, how do we make it all fit together in a really easily understandable way? Because yeah, we’ve got the submit items endpoint, we’ve got the submit reports endpoint, items goes to automate enforcement, reports goes to manual review. And you know, we want to make it really, really, really easy to see and to experience what that process looks like of like getting data into Coop.

Alex Garnett: Yeah. I think, I mean, for the atproto use case, obviously, which just speaking for myself, you know, like I think the ego aside, it seems like a pretty compelling one. We would want to build out like a round trip, even, right? So you’ve got like a firehose consumer that goes into Coop and you’ve got some data filtering and scaling there, I assume. That’s one part that’s still a question mark to me, is like you probably don’t want to post the entire firehose directly into Coop. So there’s a middleware piece in there somewhere. And then Coop in turn would be able to just emit labels, right? It would be wired up to a labeler that runs on the atproto network.

Like I can see how it works end to end. And I think, especially for non-Bluesky lexicons, which are the ones we’re talking about, who’d be starting out on their moderation journey pretty green field, like you don’t need as much middleware filtering. You can just kind of send everything from the firehose that matches a certain lexicon directly in there. And I’m assuming, you know. Coop is pretty good. It’s just like no action, no action, no action, no action until it wants to flag something. So probably you can scale pretty effectively out of the box like that. We’ve just got to build out the end-to-end example of what it looks like.

Juliet Shen: Yeah, exactly. I couldn’t have put it better.

Alex Garnett: Awesome. Well, I mean, that sounds like a pretty exciting thing to work together on, too. Like I was mentally holding some space this summer for like let’s see when Coop gets to 1.0 because I will want to make sure that we can ship like a little integration or a small tutorial and I am newly excited to do just that. So I think I’ll follow up with you probably next week or the week after when I have a minute to see if we can build something like that.

Juliet Shen: Yes, please. And I also say, you know, like I use the AT Protocol for any demos because public social media posts like make it super easy to demonstrate all the functionalities of Coop. So I do have a branch right now. I connected it to tap instead of Jetstream or the direct firehose. So there’s some rewiring needed. But it’s, you know, I would love to work together. I think this would be such a fun summer project. And you know, I think it is Bluesky’s summer of shipping. So let’s do it.

Alex Garnett: It is the atmosphere. And we have coming very soon later on this summer, I can a sneak preview Jetstream V2, which is going to offer a lot better server-side slicing. So exactly for the use case of like, I only want to get these lexicons out into a different stack. So I’m not just sending data everywhere and having to add filtering. That’ll be a part of Jetstream, hopefully by the end of June.

Juliet Shen: Oh, that’s awesome. I can’t wait for that yeah, I’m really excited to work with like everyone in the atmosphere and just figure out like who wants to be part of this journey. And we can do an example, we can figure out a pilot together. We’re here to help out with, you know, integrating and you know answering questions and everything too.

Alex Garnett: Awesome. Well, thank you so much for coming out.