“Nothing Is Ever Over”

Jim Ray: Welcome to Off Protocol, a show about building a better internet. I’m Jim Ray, your host, and part of the developer relations team at Bluesky, where we are working to build that better internet with the AT Protocol. One of the things I’m hoping to accomplish with these conversational episodes is to bring in folks who get at the why of building online networks. And I can’t think of a better person to help unpack that than Erin Kissane. Erin will be familiar to anyone who’s been working in atproto land as she’s given the keynote at both of the Atmosphere conferences. This year’s conference in Vancouver or last year’s in Seattle. If you haven’t had a chance to see those, I highly recommend checking out the recordings.

I’ll throw a link into the description. So Erin’s interest in the internet goes back much further than the current decentralized social networks that we know today. She’s someone who’s done a lot of work in and around the online space of all shapes and sizes, trying to figure out how we build networks that, as she put it in Vancouver, treat ordinary people as subjects with agency, not objects to be manipulated. Now, if ever there was a guiding ethos for building online communities today, that has got to be it. We also talked about trying to unflood the zone with the COVID tracking project and the work she’s doing right now with unbreaking.org her writing with pieces like Meta in Myanmar, and how we can build better networks by listening to people.

Erin’s writing and thinking for going on two decades now has been hugely influential to my own, and it was a real joy to get to talk to her I really hope you enjoy my conversation with Erin Kissane. Erin Kissane, welcome to Off Protocol. It’s lovely to have you here.

Erin Kissane: Thank you for having me on. I am so psyched.

Jim Ray: So Erin, I have this suspicion that we all kind of have a sense of like when things started going wrong. But I think that you probably know like the actual day that the internet broke. And is that like is that like forbidden knowledge you can actually share with us? Or.

Erin Kissane: I wish I had that. God, I feel like that’s what I’ve been fruitlessly trying to assemble for at least.

Jim Ray: January 9th. 2014.

Erin Kissane: I was waiting for which year. Like I definitely think sometime in 2013 some things took place.

Jim Ray: Okay.

Erin Kissane: But you know I you can’t overestimate how much we’re working from the vibes in the corner of the internet that we were each seeing. So obviously like the internet apocalypse like the future is unevenly distributed so it hit us all in different places at different times but you know by the time the twenty twelve US presidential election cycle wrapped, we were already, I think, in a different and weirder world and just hadn’t quite realized it yet. That’s my hypothesis for which I have zero empirical data, but it was that like that was the question that pushed me down some rabbit holes.

Jim Ray: You know, certainly in 2008, we were like, isn’t this great? Obama, and we did it, and the internet helped us. And then a quick four years later, things had started to turn a little bit.

Erin Kissane: I mean I think every new technology has this liberatory potential as well as the other side. And it’s I don’t know, maybe, maybe we won’t make that exact mistake again, but I don’t have a ton of faith in it. I think there are reasons to be optimistic about new sort of sociotechnical phenomena and also reasons to assume that, for instance people who expressly do not have our best interests in mind are looking at the same things we are. So the idea that only we special people can understand how this will help us and they will never figure it out I hope we got over that one, because in retrospect it was an appalling lapse failure to plan.

Jim Ray: Failure to plan, exactly. There’s a line from Zeynep Tufekci’s Twitter and Tear Gas, which I don’t know is completely held up, but it’s it continues to stick with me. It’s that power learns. And that’s if that’s one thing for us to take away, it’s that power always learns.

Erin Kissane: Absolutely. Yeah, and just because the first fringe implementations of something are feel intrinsically liberatory does not mean that power won’t find its ways. Maybe that’s the corollary to the Gibsonian street finds its uses, right? There always could be the subversive uses of Hegemonic technologies as well.

Jim Ray: So we kind of jumped right in there, and maybe we can backtrack just a little bit. As I was kind of trying to think about how to frame this up a little bit, I was looking for some through lines. Through your work. And I think the one that stuck with me the most is that we live in this world that is just increasingly mediated and controlled by all of these ever-increasing complex systems. If you listen to the AI people, they all will tell you they don’t even understand how these things work on the inside. And I think one of the things that you keep working on and keep trying to point out and keep trying to remind us of…

is that when these systems are designed without people in mind, we get ourselves into trouble. Do you think that’s kind of a fair through line?

Erin Kissane: It’s always a f Mortifying and fun exercise for me to pull try to pull out the three lines. So I appreciate you having done that work on my behalf.

Jim Ray: Sure. Yeah.

Erin Kissane: Yeah, I do think that’s one. And specific people. It’s hard to talk about people as a group. We are.

Jim Ray: Right. Well, so when you say specific people, you are you’re talking about like the ones who are building these systems and not kind of taking into account what the effects downstream will start to look like.

Erin Kissane: Yeah, I mean I think so. We almost always are building for ourselves first. And that’s actually fine, but if we’re also building for everyone, I think it’s tremendously helpful to break down what everyone actually is and who are all the publics and communities that make up everyone. Who is most likely to take the most damage from new systems, which inequities or liberatory potentials are likely to get boosted, but also like that sort of sounds like we’re making decisions at the beginning of a new age. And if we talk to enough people, we’ll get it right this time, which I think is also just a huge trap because that’s not how any of this ever plays out.

And I want to go like full cybernetics, but I do think having some kind of feedback mechanism that also involves talking to a lot of people is tremendously helpful for staying on track or coming back on track. So it nothing is ever over. No new thing is ever actually a genuine reset. This is like the horrible lesson of adulthood for me.

Jim Ray: Right. Yeah, you know, it’s interesting, like speaking of like the lessons of adulthood. I you know, one thing that I got to kind of immerse myself in some of your work the past week. And you know, one thing that I kept reminding myself of is that oftentimes we are we were the person that these things were designed for. And then increasingly as we move on, we move out of that phase. And that also very much applies to, I mean, you and I are in very similar demographics and so this idea of systems being designed for folks who look and think and act and live in places like we do. And then they start getting used by places, by, by people in very different places, people of very different backgrounds, very different demographics, that seems like the first part where we start to get ourselves into trouble and where we stop listening to the folks who might have things to tell us.

Erin Kissane: Well, and like l listening sounds easy and it’s incredibly d difficult. Conflating listening with just attending to the models people sort of the folk models people express, you know, for instance, we are among other things talking about social internet platforms, right? So there’s like listening as in what’s in the trending topics and what people are mad at, and then there’s listening in a series of increasingly complex ways that I think mostly happen. away from those platforms and offline and are like unfortunately like complicated, expensive, multi-stakeholder. That there are all these institutions in you know what we at least used to call civil society, working on this stuff and they try at least to know some things but one of the like the huge frustrations for me is how little contact there often is between these different institutional layers.

So there’s the knowledge that comes out of many different academic disciplines, which themselves don’t always like to talk to each other. And then there’s like the work that does come out of civil society and, you know, people working on political governance and community governance. And then there are people making technologies, which it turns out I like The Barlow model, alas, does not hold. We are also building civilization online. It’s not a get out of civilization. free card, unfortunately. So I do feel like we haven’t really caught up to that reality and we don’t actually have a lot of interchange between those bodies of knowledge and people who know how to listen to people who are inaccessible to those of us.

so far in under the experiences of other people in really different situations and places. There are people who know how to do that, but we don’t really have a framework for moving knowledge around through those layers, which if we are ultimately having to sort out like the foundational questions of living together. online as well as offline. Like we don’t have a tremendously successful track record of doing that offline, but we have a lot more experimental data. I think my hope is that we see perhaps not in my lifetime. And understanding that we do need interchange across those zones if we’re gonna do civilization as like a species that also lives in big networks online. Which I don’t it’s hard for me to imagine the situation of complete surrender of that mode of human connection entirely.

So if we assume that we do have this dimension now for as long as we have things like electricity, then we should probably try to solve for like the full level of complexity. But it falls to a lot of people who are doing what they thought was going to be network architecture. Which does not necessarily prepare you for that kind of appallingly complex problem.

Jim Ray: Exactly. It turns out treating people like nodes on a network. Great for creating that that that proof of concept, but doesn’t and it scales in a very technical sense, but not certainly not in a social sense. You know, you said something there about kind of the institutions, and I want to talk about the institutions a little bit. And I think we associate gatekeepers with institutions a lot. as you and I were coming up at the as the internet was becoming this thing from this toy thing to something that was like really remaking societies and governments and politics and entertainment, all the things. The notion of taking down gatekeepers was at one point seen as just an inherent good and I think I look back on this now, it’s like, man, we could use a couple of gatekeepers right now.

And I feel like during this conversation, we’re going to talk a lot about like the inherent tensions of all of these things. And that feels like kind of an early one, that like removing some of the gatekeepers was an inherently good thing, particularly when we think about the diversity of voices that allowed into again, all of these cultural institutions, but it’s also allowed for things that like obviously not so great. And I guess I’m curious how you think about that tension.

Erin Kissane: That’s a very large question. Maybe it’s useful to sort of untangle like monolithic lead institutions from the much broader slate of like I use institution maybe in an idiosyncratic way, but to refer to I think a much larger slate including a lot of relatively informal groups that that coalesce to try to do some kind of governance or have some other so it doesn’t have to be governance. But so at the very local level a lot of us I think many of us at least, I feel very fortunate to have to live in a place where there are some quite vibrant, although contentious, you know, sort of networks of local institutions sort of thrashing out. you know what we do as a local populace and community in our particular like biome with its weird characteristics and so there’s a there’s just there’s a tremendous richness a few levels down from like gigantic institution, and I think that’s where things get really interesting.

So thinking about it maybe it’s easier to sort of estrange ourselves and say like thinking about the groups that sprang up around the world in that 2011, 2012, 2013, back to maybe 2009 zone of local institutions that were bringing the internet into communities, teaching people how to use them, getting students acclimated to what the internet was and could do for them. And I am thinking of specif some specifics in certain sort of semi-authoritarian situations. Those kinds of institutions have tremendous listening ability. They know a lot of things on the ground. So for me, like those kinds of institutions include things like here where I am the both governmental and non-governmental organizations trying to put together an effective support system for children who are and youth who are in foster care, right?

Just take that as a slice. So there’s like four different institutions working together to patch something together and different pieces of you know different institutions there are no different things about their public. So like that’s the level of institution where I think things get are enduringly interesting and fruitful even when The big institutions theoretically at the top of our society are just either lying flat or being actively demolished or doing awful things. You.

Jim Ray: you were talking about how you know some of these big centralized networks one thing that’s always annoyed me is when somebody like Mark Zuckerberg comes around and talks about his com like the Facebook community as if it’s a singular thing. And it’s like it’s always just kind of grated at me. And it’s like, one, it’s just so clear you obviously have no idea what you’re building, but two, it doesn’t even make sense to talk about like a community of one, two, four billion people.

Erin Kissane: Yeah, I don’t I don’t even think those people believe that though. Come on. I try to stay away from like the intent and the internal states of others, but who even believes that? Yeah, I mean I don’t obviously this is I’m not rated professionally to sort out some of these things, but I will say it’s felt like I am just sort of being forced by the tides of history further and further into sort of a more radical like municipal municipalist syndicalist ultimately like very orderly anarchist position about this I do have a lot of faith in highly local institutions that doesn’t make them all good. Some of them are extremely bad, but a lot of them are in touch with the material realities of the humans that they allegedly represent or serve in ways that it’s just really hard to do if you are a bigger and more central institution and it’s funny like this is true even in like I live in Oregon and I lived in Portland for quite some time and felt that on the whole our state government, although sometimes incompetent, sort of had our needs in mind, and then I moved to a very rural area with very different needs.

And it’s not just a conspiracy theory that rural communities are not well represented. Like we need different things. We need, for instance, more vets. But the governor vetoed a thing that would increase vet access for like small farms out in the country because there’s not actually a lot of knowledge being passed back and forth through those levels again, it’s something that’s super hard offline, but I’m afraid we have to probably address it online and maybe we get someday to the full cybernetic dream without it also being the authoritarian nightmare. I’m quite nervous about that. But I do think there’s a lot to be gained by just talking to people who are working on small stuff.

Sometimes clumping together small things into bigger things. Yeah. It’s hard to believe in the top down right now at the scales that our networks are working on because we don’t actually have like world government. We have the UN which does its quite marginal things and we nominally have some international courts which our country opts out of opts out of. But we do have something much closer to world government online, which should concern us. Right. I think. So yeah, I would love to see a fresh generation of well-intentioned poli-sci students take on these but we are doing politics, so I guess it’s on us to try to find the best way we possibly can to do them in our moment and I do tend to come back to, I think we should talk across layers more, but it’s very difficult.

And it’s more difficult when we do it as a form of public performance on these big online systems, I would like us to have more of those conversations in places that are safer for relationship building and all of that stuff. Which is hard to do while you’re also tap dancing for a potential unknown audience of maybe trolls advocates and maybe future employers. And you know, it’s just like no. Not ideal.

Jim Ray: So speaking of the One World online government, I kind of want to start in what I’m pegging as the middle of maybe your more recent career stuff, which is Meta in Myanmar. I guess I’m curious one where that kind of came from and how is it that you decided to spend so much time and write so many words about what Meta was doing in this far and kind of not well known about certainly in the Western world part of the world, but ultimately like but really hard to come away from that and not like see like if not full-blown intention than just like a complete dereliction of duty there. So I guess maybe we’ll start at the beginning with Meta in Myanmar.

It’s like, how is it that you kind of came to have an interest in that and like spend so much time and effort on it.

Erin Kissane: Yeah, I mean part of it is just having sort of a fundamentally bad personality about certain things. I don’t like not being able to find information that I think I need. So I will latch on. And in this case, I was trying to write more generally about some of the things that you and I have just been discussing. And I’ve had a sense, because I’m, you know, an interested reader and had been reading most of the reporting from, you know, the BBC The Guardian, the Times, the various places that were covering this in the most generalist mainstream press. Covering by this, I mean like w what relationship Facebook specifically had to a genocide in Myanmar and also thereafter to you know the coup and the ongoing rule of Myanmar by the military.

So I kinda thought I knew what had happened, but I didn’t think I should build on it without a better understanding and I went looking for a better understanding, something that stitched it all together at a level of detail that I felt was like rigorous enough to build an argument on. And I found a lot of really scattered stuff and some of the really scattered stuff also went very deep on, you know, one piece of this. But I thought, well, I’ll just put together for my own self a timeline of what happened when, when Facebook entered the market, what role they were playing. ‘Cause there was a Myanmar’s a special case because the internet came to Myanmar quite late because they had been under very tight authoritarian rule that had kept the internet officially out of Myanmar.

There were people who had you know phones from China and other places, but it was very underground. So Myanmar was a weird case because the internet came to Myanmar in the form of mobile phones with Facebook on them Facebook became the internet connection for Myanmar. So it was weird in those ways. And also the more I read, the more in e even in some cases narratives from people I had known online who were there at the time and who were around for the super exciting moment when Myanmar began to open up. and the extraordinary liberatory potential of all of this information flowing into the country and you know the young activists and people doing civil society work being able to use this to do like big education stuff and farmers getting weather forecasts.

for the first time and being able to really dramatically change. Like they’re like very low level agrarian practice, people who didn’t have wired electricity, running mobile phones on a car battery. in the house so that they could get the internet so they had these things. But it was coming through Facebook. So in addition to the weather forecasts, they began to get what turned into the groundwork for building public support through misinformation for a genocide. And the further I got in, the more I felt like it was really hard for people to understand, people in the West, people working on tech, to actually get any. So it was super hard for me and I was like all in on the after the first week, I was just like, well, clear my schedule.

Super hard to understand what had happened. And I thought what exactly had happened was important, not just because there was a level of callousness on Facebook’s part and on the part of specific people at Facebook. That I like a level of callousness that perhaps rises to malice whether or not there was explicit intent and all these warnings from people in Myanmar and you know, from people like Susan Benesch, who was working in the US as a scholar of human rights, who was also trying to get into Facebook and say you’re gonna get so many people killed and Facebook saying, nah. So the further I got in, the more the full picture. It was a very it was like a very horror movie experience because like the genre of horror dep like is dependent on like as soon as you think you’re oriented, it gets way worse.

The floor f another floor falls out from under you. And that was really the process that I went through as a researcher trying to piece together what had happened. And I had the really good fortune of being able to speak to some of the people who were active on the ground in Myanmar at the time who spoke with me later, I couldn’t credit them or name them in the work that I did for safety reasons. But who helped me kind of orient on this. And by that time I was like, well, everyone needs to know. So it wasn’t it wasn’t ever something that I did strategically, but once I knew the shape of what had happened, it felt like such an extraordinary cautionary tale that those of us building new networks should know about.

There were many people at Facebook, but also who were doing you know technology work in Myanmar who were not malicious. And this thing happened anyway, and a lot of it happened because Myanmar didn’t matter to Facebook, just in the way that, you know, the majority world largely doesn’t matter to large US tech companies. But it was not my impression that people building new networks felt that way about, you know it this is in quotes who listeners won’t but the rest of the world, right?

Jim Ray: Yeah.

Erin Kissane: So then I was I don’t know two months deep into the research and just had to write my way back out. What was interesting to me though was I wrote it as a series of blog posts on my personal website because I was like, who would take this? But it did wind up getting taught in an algorithmic justice course at Cornell in a maybe CS ethics class at Cooper Union. I did some class visits and saw it just like kind of going around places. I had the exceptionally great experience of getting to do a class visit virtually with Susan Benesch, the human rights scholar. It’s just that the information wasn’t largely available. No one had been able to do the work of piecing all that stuff together because it’s not actually a job.

Like no one the journalists who worked on this stuff are the only reason I had a lot of the data and the other reason I had a lot of the information is because the UN did a ton of reporting and then Amnesty Tech did but there’s not really like a role in our society. There’s not like a national transportation safety board for the internet. There should be. We should have that. But we don’t have that. So we don’t have investigations that pull together like what went wrong, which is I think probably a dangerous way to proceed. I’m not volunteering to be that again, not rated.

Jim Ray: One thing that I noticed, and I’m as guilty of this as anybody, as a longtime Facebook hater, like I feel like Myanmar became this shorthand. for people on either sides of this debate that we’ve been having for a while, but that really spiked 2016 and onwards. And you would just see these just little snipes back and forth where it’s just like, oh, Facebook did a genocide. No, you can’t hold Facebook responsible for all of human behavior, can you? And it and it would so it just became this other talking point for talking past each other and so one, I was just personally just very appreciative that you actually did do the work because as you said, there’s it’s not anybody’s job.

So you mentioned that there were some actual ways to learn from this and ways for us to kind of build. You know, one thing that I’m kind of curious to know a little bit more about is were there any like were there any changes on the ground that you’re aware of after this in Myanmar and in any other places? Or is this just kind of another horrible thing happens, nobody’s really held accountable for it, and we all just kind of move on.

Erin Kissane: Well, I mean the situation in Myanmar is still just extremely bad. Many, many of the survivors, the Rohingya survivors of the genocide w which I will say The UN Human Rights Council’s investigation actually did say that Facebook was instrumental, which is something that I felt like I could call on as a non-expert. And they, you know, they published hundreds and hundreds of pages of evidence for why they thought that. But for the folks who won’t have read this extremely long report. But essentially what happened is when Facebook became the internet for Myanmar it came in to a naive population. This is one of the very early common explanations of sort of what happened on the internet side of this.

People who hadn’t encountered you know anything but essentially government controlled and approved information before and they were getting undifferentiated kinds of information. So it’s the weather report and also a fake story about a three-headed cow, right? And it’s all coming at the same level in the same ways. The part that I think Western audiences saw is that Facebook didn’t moderate there are very complicated things actually that happened that go all the way back to the fight around character encoding.

Jim Ray: Okay.

Erin Kissane: dissertations have been written. Yeah. We’ll skip over that. It’s but it was there was an encoding problem where there were two different ways of encoding Burmese. And so as a result of the sort of ripple effects of those very low-level technology decisions. Facebook had a lot of trouble doing any kind of automated moderation of Burmese language material. And again, like the narrative that came out for the first few years in the West was there’s all this bad stuff happening because Facebook didn’t moderate. And it’s true that they like they had for a while one or two Burmese-speaking moderators, and I think they were in Dublin, if I remember correctly. So they were not doing that work.

But that was just a tiny piece of what was happening because also Facebook as the internet came in with I don’t know if you remember instant articles, you come from journals.

Jim Ray: Oh I do, I absolutely do.

Erin Kissane: didn’t have an enormous lasting effect here in the US, but it wiped out Myanmar’s media landscape. Just destroyed it, replaced it with fake content, click farms, all of that that was making some money from Facebook and because of currency arbitrage made, you know, significant money for more local operators. So we, you know, we lost the media internet access being piped entirely through this system of undifferentiated factual and obviously not factual information. But also there were elements of Burmese government and society, religious extremists went in positions of power, and the military government themselves, who had sort of receded in Myanmar’s opening, but were interested in coming back. And they did quite a society-wide, vast and sophisticated Russian backed military operation to just produce industrial scale genocidal material and seeded it throughout the Burmese language internet, much of which was on Facebook.

And were able to create a public understanding that made the subsequent genocide and ethnic cleansing more acceptable to more people. So that’s what happened at a very, very high level. The things about specific decisions that Facebook made are super gruesome and I won’t I won’t get into them. That’s the that’s the overall chronology of what happened and then after that the military just gained more and more power and were able to sort of use the tactics they had tested out on this demonized Muslim minority. and roll them out across the rest of Myanmar, which is an incredibly complex political landscape. Like you could spend an entire graduate degree understanding the operations of many, many ethnic minorities that had been fighting back against this government for generations, but to dramatically oversimplify the military government.

took what they learned from doing a genocide without much accountability and then used it to take the country. And they’re engaged in active warfare with people throughout and with armies. Mostly associated with specific other ethnic minorities across the country and also against Burmese-speaking majority people who many of them learned too late that they had been taken in and were able to understand what had happened. Meanwhile, the survivors of the genocide are mostly in giant refugee camps that have been defunded by our government here. So it’s very hard to think about accountability for something that is I mean it’s a it’s an ongoing disaster. Facebook itself was condemned by a number of international bodies. Amnesty did exemplary work, picking apart what happened, the result has been almost nothing.

There was a big there have been a number of lawsuits. Nothing has really happened. I gather that there were some people at Facebook who felt bad. That’s kind of where we ended.

Jim Ray: Yeah, and it feels like you could, if not see repetitions, you could see how that pattern has repeated itself. Certainly not to the same degree, certainly not to there I think there’s probably few places where you know, as you were mentioning, the kind of unique set of circumstances that existed in Myanmar before where they didn’t have widespread internet and Facebook came in and kind of became the internet, but you can certainly see how that playbook has existed even in places that we’re like people listening are much more familiar with, whether it’s 2016 in the UK, in the United States, you mentioned Russia, which of course is pretty masterful at figuring out how to use these I don’t know if you want to call them corrupted social networks or just centrally controlled.

And then obviously there are everything that’s happening with we’ve got thirty years of central control in places like China of the internet. You kind of wrote this for people building new networks. And so obviously we would have all loved that Erin Kissane’s piece results in people going to jail. That clearly hasn’t happened yet, but you know, for the folks that are interested in this conversation, people that are building new networks, trying to figure out ways to certainly not make these same mistakes, are there are there are there lessons to be kind of taken away that apply to The decentralized protocols that we work on, other types of technology.

Erin Kissane: The wrap-up of that series was really hard to put together because how do you how do you even propose ways of going forward that address something of this magnitude and I should say thank you for the sort of larger connection. Karen Hao did some incredible reporting with whistleblower Sophie Zhang. That was at the MIT technology review. I highly recommend network makers take a look at that because it does make clear that these patterns, even just Facebook was really a global ongoing known problem inside Facebook that Facebook was being weaponized by authoritarian governments and by other, you know, sort of just like almost cartoonishly villainous entities openly against citizens against vulnerable populations, against people. And Facebook knew it and didn’t act.

So Karen’s reporting before the AI pivot, w which I think her work there is better known, is foundational and really helpful for getting a sense of that. What was so shocking to me is how much either was known or could easily have been known about how badly things were going. and how Facebook had structured itself to push that kind of information away from decisions in the organization. In a number of ways, including by just writing off that what they were doing could possibly have that kind of effect. And that’s I mean that kind of maybe loops back to what you were saying about the way that Myanmar turns into sort of just like people talking across each other online.

Well, you can’t look at Facebook for everything. And of course not. the people doing the genocide didn’t have to have Facebook to do it, but Facebook sure helped. And shouldn’t have, I think we should say. There are always going to be people who are trying to do enormous societal level harm, so we shouldn’t help them. Just some basics.

Jim Ray: It’s so there are so many things like this. You just say the thing out loud and you’re like, I feel like an idiot that I have to say this.

Erin Kissane: Maybe we should say it though. We should we shouldn’t help. If there was something that I think could potentially be framed as encouraging for people working on new networks is that this was not actually unpredictable. This could have been like a lot of this could have been mitigated. All it would have taken is an institution that could have heard the warnings. that individuals that like there were Westerners specifically trying to say like look at me I am an American and I’m telling you this that will you listen to me if I bring you the testimony of all of these Burmese you know activists and officials who are worried? The information is gettable. It’s harder without centralized telemetry Facebook could know a lot of things because it was just one Facebook.

and they had a lot of data and newer network structures don’t necessarily produce that kind of information, which is why we need organizations or perhaps vernacular institutions to do a dumb self-site. We need some institutions. small is fine, that can work across decentralized and federated systems to pull together some information about how these systems are being weaponized because you know that they are being. It may not be as obvious, but we should assume that they always are but cultivating listening relationships with vulnerable communities and making a way for people in trouble to tell you that what you’re doing is causing harm could actually prevent a lot of it could have prevented so much. I don’t know what would have happened.

in Myanmar. There’s no way to unpick that not. But we should have tried. And new networks should try not to make themselves tools in the hands of people doing tremendous evil. That’s why I talk so much about listening. The Myanmar work really radicalized me on the idea that if you don’t have the capacity for people to tell you when something is going horribly wrong, you will enable at some point harms that you never intended and that could be extremely bad it’s not a recommendation that is commensurate with the damage, but it’s the one thing that I came out really believing in.

Jim Ray: And you know, there is one aspect to this that I think is a little bit different than the place where we find ourselves now. And it’s like just trying to remember what the world was like. There were there were still corners of the earth that didn’t have the internet. So as you were saying, you know, it’s like Facebook moves in and they effectively were the internet. And I think that aspect of it is a is probably a little bit different. And so this idea of like coming in and being able to be the monolith that becomes effectively the internet for people is like that exact mistake is not one that hopefully we’re going to be repeating anytime soon, but like this idea of figuring out ways to let people effectively give you feedback and then kind of take action on that feedback is that is that kind of like where we are right now and as we’re moving forward, hopefully figuring out better ways to be and to build?

Erin Kissane: I think it would be a good thing to try.

Jim Ray: Yeah.

Erin Kissane: And I agree with you that it’s very unlikely that we see a repeat of the way that Facebook was able to really sort of elbow its way into becoming the primary infrastructure, communication infrastructure in a whole country. I mean e that was rare even then for it to be that that monolithic, although Facebook worked really hard to internet for a whole lot of people and still, you know, Meta’s still doing that. That’s still happening. But yeah, for a new generation of network builders, I think you can probably avoid a lot of that by listening to people ideally, I think we should hope for a situation where it doesn’t always take the form of the bountiful United States technology interest industry rolling out tools for those other less sophisticated world.

I do, of course, have some concerns about how frontier AI models controlled by US technology corporations plays out as a recentralizing force for the distribution and control of knowledge, but that’s another podcast.

Jim Ray: And that’s a whole other podcast. Maybe one theme there. And this is something that it’s something I really have to check myself on because I have this tendency to be a little bit moralizing. And so One thing I’ve always really appreciated about your work is that it feels very confirming of the things that I believe, but it but it feels also as if it is not moralizing in a way that I want it to be always, if that makes sense. And I think that that is actually very helpful. And I like I’m curious how you think about, I guess what I would call like the moral valence of technology and with the exception of maybe Ticketmaster, I don’t think I’ve hated any of these companies longer.

And oftentimes for me, that ends up resulting in like maybe not very helpful or certainly not very healthy. Ways of expressing that onto the people who are using these technologies. And it’s it’s really difficult for me to separate because at the end of the day, I do believe that people have agency. And I do believe that like we all understand what these things are now, and you are continuing to keep them engaged. But at the same time, like people very near and dear to me continue to send me Instagram posts. And so One, I guess I’m just curious, like how do you personally think about that? Like, are there effective strategies for, you know, dealing with this in a way that isn’t belittling, isn’t moralizing, and is also effective.

You know, and ultimately at the end of the day, I want the number of people using any meta property to be zero. I know that is not practical, but like that is if I could push that button, I absolutely would.

Erin Kissane: Well, there’s a bunch of questions in that question. I think I have a few answers. Well one of them is that what I’ve read and also what I heard directly from people who were who had a history of working in Myanmar was taking Facebook away would actually be worse. It’s been very, very bad. And also destroying it would be worse. And I will say I know that there were people in Facebook who then worked quite dip quite hard to build some government surveillance evasion. tools for people in Myanmar. There are people with good intentions working on this stuff, but we if you if you dig around through sort of the testimonial information coming from people all over the world, including in communities here in the US who are doing you know political opposition work just getting rid of Facebook now would m make some things quite a lot worse.

So there’s just a lot of complexity because these are super smart people. They are figuring out how to use bad systems to do good and important work so it’s all actually quite complicated. Like I think we should assume that the sophistication of people using these systems in ways that are perhaps subversive, but that that that do the things they need to have done. That’s very real. The Engine Room is, I think, a London-based NGO that did this great report that came out in, I want to say, end of 2024. that was based on a like a year-long series of interviews with, you know, Global South majority world activists about the transition to better alternative networks.

And it turned into being, in large part, activist explaining their strategies for using the bad centralized networks to do their work and what they would need from decentralized systems to make that work. So one answer is just the good side of the street is finding its uses for this stuff. So it’s not the case that you can just rip out the, you know the evil infrastructure all at once, even though obviously I’m with you. I wish I wish it hadn’t happened this way is what it is.

Jim Ray: Right.

Erin Kissane: But I think the other thing I now I’m going to recommend as I do continuously my current favorite philosopher of networked epistemics of C. Thi Nguyen, but his work on it in epistemics right now, there’s a really interesting debate about sort of epistemic vices, basically like the bad things. we do as people that lead us to believe wrong things. It’s a very ungracious summary.

Jim Ray: Yeah.

Erin Kissane: But Quassim Cassam’s work on, you know, epistemic vice basically. Like we’re just like not very good at this and we should we should reform. And what I love about C. Thi Nguyen’s work is that he is taking the position that I find most fruitful myself, which is to speak to like the sort of problems with the moralistic register his position is that there are hostile epistemic structures and that it is more fruitful to focus on this the systems level, the structure level, the way that we build structures and systems that take advantage of our innate weaknesses as entities as human beings and communities to produce bad results. Nguyen’s interest is in how do we how do we get a handle on like what the systemic and the structural problems are rather than taking on the sort of moral question of like, but why are you doing this?

Why are you know, why are you in the why are you maybe why are you on Instagram? But in particular, just like, if we just were better, this would be better. Which true, but we’re not gonna just magically get better as a species. Yeah, exactly. So rather than working on the sort of moral question, it has been most fruitful for me to try to think about the structural side. Because again, I just like I would love to believe that we will all experience some kind of transformation of the soul as a species someday, but I don’t really believe in it. I don’t so I think our best bet is to try to understand the structures that can reinforce our better natures and build some communally agreed upon guardrails for trying to stay on the on the path of righteousness, which does mean that I have to set aside like I’ve been doing this is a confession.

I think I mentioned it once in public, but like as a personal project for the last couple of years, I’ve been trying to shove myself out of the register of contempt. Not because I think that there aren’t people and entities toward whom we are justified in feeling contempt, but because contempt sort of forecloses curiosity about how it’s not just some people are bad. There’s some people who are bad, but there’s a whole lot of systems around those people that put them in power that make us vulnerable to them. So like refusing contempt in favor of inquiry has been hard. And I don’t always, you know it’s hard. It’s hard. You know, because in some of my other work I’m looking directly down the maw of the Trump administration’s choices, and it’s hard not to feel contempt.

But just trying to like shove myself out of that lane as much as possible. But a lot of our online conversations are really rooted in contempt and impurity and just a different moral register, which I think is a hard problem. I have no plan to fix it. I don’t know how we do it.

Jim Ray: That sounds like an almost spiritual practice and one that I think I would certainly benefit from. So I’m gonna take that one with me. I do want to I want to ask you a little bit about we were just talking about the work that you mentioned against the Trump administration and I feel like there were two projects that kind of bookend the Myanmar project in some ways, which are the COVID tracking project and then the Unbreaking Project. And in some ways they’re very different. The COVID tracking project, it starts off as this project to just like we were in this sea of like no information, misinformation, disinformation, intentionally bad, people that just didn’t know what was going on.

COVID tracking just felt like an effort to just kind of pull everything together. The kind of thing that like journalism 101 is really all about. And then there’s the Unbreaking Project, which is has like some similar goals, I think, to just take this massive slurry of information that we’re getting every single minute of every single day and try to make some sense of it. But I think The big difference to me is one of, I guess, intentionality and where so much of what was coming out of COVID was like we just had no idea what was going on, even though we were not currently being served by terribly competent people at that point. There was just there was just so much that we just didn’t know.

And I feel like the Unbreaking project is almost kind of an inverse of that in some ways, where it’s like there’s a there’s a huge amount of intentionality that’s coming down. There’s just so much of it that we can’t keep track of it. Is that like is that kind of a fair characterization? And do they feel similar to you?

Erin Kissane: Yeah, I mean I do think that’s I think that’s a fair read. They feel similar in structure in that they’re both sort of large scale volunteer collectives working on you know, building good information for our communities in a roughly mutual aid shaped way. I think you’re right. I mean what we what Robinson Meyer and Alexis Madrigal were trying to do originally was just find one number for COVID tracking, which was, okay, but how many tests? Like you’re saying we have this many positive cases, but how many tests did you do? And that was not known, not findable as a single number that did turn into dozens of numbers across fifty-six jurisdictions, states and territories. And we encountered really Lovecraftian levels of complexity in the data definitions and all of this stuff.

So even that really bounded thing sort of fractalized on us and turned into something where we could produce certain numbers, but the work underneath those numbers of trying to make sure we were, you know using the data from different places in correct ways that we understood the composition of all of that. The complexity in COVID tracking was mostly under the hood. And we had essentially like whole research initiatives happening that no one ever saw outside the project that were just us trying to track down where does this number come from and what does it mean? What does it actually represent? are these people who are in the hospital, are these people who have you know, so the so the complexity there was on the sort of the underside and with unbreaking, I think you’re right.

The co the complexity is what we’re all swimming through, just this overwhelming amount of narrative. In the way that journalism, th in particular like newspaper journalism and broadcast journalism are shaped, there’s a little bit of Groundhog Day. There’s like we do a little context at the beginning of the story and then we say that the thing that happened and that story’s gone. You know, so to build a durable understanding of what’s actually happened and is happening over time requires a superhuman cognitive effort. And I think it’s just like breaking us. I mean people who are really trying to stay like right like we used to call it current affairs. Yeah right. We had we had that in schools.

We’re s those of us who are still trying to stay abreast of what’s going on in even a few areas of you know American life right now. are struggling, I think, at a minimum. So the Unbreaking Project is more about taking all of that complexity and trying to pin down some verifiable facts that we can line up in order. We wound up grounding our work in literal timelines, in chronologies. So that we can organize our own understanding of what’s going on and send out, you know, periodic briefings that say, this appears to have moved in this way. Those are very similar to the weekly data analysis that I was the lead writer on at COVID tracking for weeks and weeks and weeks and weeks.

So yeah, I think what we’re trying to do is still just give people a sense of what is happening. but from two different really like the wide end of the funnel has inverted. Yeah so it does I think mean that unbreaking for a lot of us feels like something we are in large part making for the future. We know that there are people who are following it now, but I think if we have this fully footnoted, cited, rigorous as possible record with as many primary documents as we can pull of what has happened when, it is at least my hope. that we will have something to ground our collective future memory of this event in because I saw what happened in 2022, 2023 with the rewriting of sort of public memory of what happened with COVID.

And it was harder to do that kind of memory reshaping. for the areas where we had the data public. You know, it’s hard to go back and say people didn’t die in these numbers and in these places at these times because that’s a durable record. So that’s probably that’s probably one of the things for me. There’s a lot of overlap with COVID tracking. Like we have a lot of a lot a lot of the same people have both projects, but I want us to be able to look back and say, actually, no, this happened on this date, and then the next day this other thing happened. if we’re gonna do a reconstruction, I think we’re gonna need to have that record.

I think we don’t think it’s exclusively a future of value because there is, I think, some sort of subset of people who find it helpful to be able to kind of check in and be like so sorry, what do we do to food safety?

Jim Ray: Yeah, right.

Erin Kissane: And get that, and to be able to be like, oh, okay, it’s basically these three things that are happening. There’s some utility for folks who aren’t actually face down in everything all the time. To be able to look across issues and we’re always adding more things. You know, we have more issues coming all the time. No human can keep track of everything. So even if you’re like really following, you know what’s going on in say immigration or like the attacks on trans health care, you might not be also following the dismantling of international of funding for international infectious disease control and prevention. For myself, I think I would hope that people at many levels of society, including all of the work of I mean we’re losing we’re losing labs.

We’re losing university capacity in a way that I think if you’re not in that world is very difficult to understand the scale of. It’s qu it’s quite bad. We’re losing a lot that is not necessarily in intelligible in the headlines. So my hope is that people who approach that rebuilding work inside and outside of governments will find it helpful to be able to say what we these are the losses. They happen this way. When we choose to rebuild, we can at least look back and say, like, this was the shape that we had. These are the things that we decided not to do anymore as a country. So there’s that. I mean it’s tricky because like the COVID data for CTP, you know, we were we were using official government data sources from states and territories and compiling and contextualizing it.

And what we’re using now is sort of built from the journalistic record from like the best good faith analysis we can find and from prime documentation, a lot of stuff from courts. We’re tracking a ton of court cases. So I don’t you know, it’s not completely clear to me. I think that there probably are congressional committees that will pull it in. We got so like so much of our work made it into congressional reports and was read into the record and used by federal agencies at COVID tracking both during that year we were active and later. It’s hard for me to understand who will wind up using this stuff. I don’t think there are people on the project who are just doing that because they have a specific future need in mind.

I think most of us also just have a need to do this as humans. To be like, well, there is something here. There is a shape. There’s a structure that’s happening in all this noise. That’s not just don’t get me wrong, there’s a lot of great analysis that’s going out in the form of newsletters and helping people understand the framing of this stuff, but we want to work on something different that is like here is the emergent structure, here’s what we were bringing. They did all these things in all these places here it is. So I think the project has self-selected for people who are that kind of woodpecker. We’re just going to keep smashing our little pointy skulls into the trees until bugs fall out, and then we’re going to put them in a line.

So we’ll see, right? It’s a weirder project, especially because it’s not the case that a ton of people are sort of suddenly stuck at home. We’re all still doing our work and our family stuff and our everything else. But we feel compelled to try.

Jim Ray: Something that I think about a lot when I see these amazing projects come into being is that there are real people making them happen. And we’ve talked about a lot of not very fun things. And a, how are you holding up? And like I mean this is we’re talking about years and years of like really immersing yourselves in like some of the most not fun parts of the human condition. And you seem like a very well-adjusted person and you’ve got family and like how do you keep going?

Erin Kissane: You need a peer group. You need people with you. I think that is the thing that I come back to all the time and the thing that really characterized COVID tracking and I hope unbreaking, you know, something that we heard from people at COVID tracking all the time was that they it fulfilled a necessary function for them psychologically. as they were s sort of struggling with the unfixable uncertainty of the moment and all the chaos. Having a community of people that they could come back to every day or twice a week or whatever their commitment was. And whatever else was going on, they were gonna sit down at, you know, for the people who do the main data run, they were gonna sit down at Noon and they were gonna join sixty other people and go out and try to do this work for their communities.

So when we when we built Unbreaking, some of us COVID tracking alumni and some of us not, we all we wanted to make sure that it was healthy rather than destructive for people. I will say like as someone who was helping sort of guide and lead COVID tracking, I had a huge health crash afterwards and got knocked off the board for almost a year. It was just beyond stressful and we built Unbreaking to try to be more humane for all of us, for everybody, including people who were running or guiding or leading any given part of the project. But The difference between having people you can work through these ideas with and work alongside and doing it alone.

Which I was doing with the Myanmar stuff. Like there were people I was talking to, but I was mostly just me like hanging out, looking at atrocity reports alone in my office, and I cannot recommend that as a way to live.

Jim Ray: Right.

Erin Kissane: It was really hard. And what we’re doing now and what we were doing at like our immigration team at Unbreaking is I mean they are also face down in atrocity reports. I was with that team for like many, many months and it is so, so rough stringing all of that stuff together, but doing it with other people. who have some shared hope for our future. It makes all the difference. So I’m I am a huge fan of you cannot go alone. Do not go alone into any of these bad zones. Take people with you, stick together, take care of each other. One of our practices is that people have to rotate out and go rest.

And if we have to lock you out of the slack, we will lock you out of the slack. I mean communal care. Like that’s what we’re talking about with the networks too, right? Like the listening and the like it all comes back to communal care. And if we don’t have that, we don’t have anything.

Jim Ray: Kind of on that note, we’re in the midst of trying to figure out how to take these big open networks that we have and then let people kind of create communities within them. So you wrote a really interesting piece called Against the Dark Forest, and it helped me actually kind of realign some of my thinking. Just real quick, can you kind of like give us a high-level summary of what we talk about when we talk about the dark forest.

Erin Kissane: Yeah, yeah, for sure. So the lineage here is some posts that Yancy Strickler did, and I think Maggie Appleton. as just like a really wonderful encapsulation of how the original dark forest ideas which come out of science fiction m mash up together with sort of thinking about vernacular thinking about the cozy web and I would try to encapsulate the theory as the big public internet is damaging and hostile and overtaken by surveillant systems and bots and also bad faith human actors. So the way to go is underground. to make safer places and private spots essentially like the retreat into the group chat. And so I did write a long piece and give a talk at to the last XOXO festival about not doing that.

But I should say it’s there’s nothing wrong with that. It’s super important to have your group chats. Like I am pro having the group chats and I like a lot of my current work is focused on like people doing knowledge work, epistemic work of any kind, collective sense making need some of their work to happen and their conversations to happen in shelter. There needs to be some kind of boundaries and access filtering. f less visibility, less amplification, some kind of controls so that you can have a sheltered place to have the vulnerable conversations that you need to have when you’re working with new ideas. And this is something that I saw really come up in the open science track at the Atmosphere Conf in Vancouver this spring.

But the you see across various places and people who do epistemic work like really get this right away. Like yes, obviously, there’s work that we need to do in private and there’s work that we would like to be able to do in public. My concern with we should assume that the internet is basically hosed and give up on the public internet is that there is no way to preserve the that liberatory potential the internet was supposed to give us where we could find new ideas, find new people, reach communities find our counterparts doing this, you know, whatever work we do in other places, in other demographics, you lose all of that possibility. I think it is perhaps particularly attractive to people who have already found their people and found their places and, you know, they can kind of do that retreat, but where does that leave the people who are coming along after us?

If the if the public internet is hosed and they’ve just got their group chats of people they already know. Like how do we maintain a fruitful movement of ideas and people and connections? So I and additionally, like how do you keep from just entrenching existing power structures where the way you reach people is through these elite channels. Like it’s just a capitulation of the things that I think were exciting and good about the social internet. And I don’t think we can really afford to just seed that to like Shitty companies and awful AI bots and like I don’t think we can give it up. I think it’s too important. People who are trying to do for instance activist communication now have to grapple with how many you know fragmented platforms, each with their own policies, some of which are very transparent, some of which are super opaque.

It’s a mess. But I don’t think we can sort of culturally afford to just give up on that ideal. So my interest is in how do we you know, what do we make then? What kind of privacies and gradients of intimacy and shelter can we build into and alongside of more open systems? Which is how the whole offline world is structured, the natural world, the built world, the whole thing. It’s not just like the cloud. and like the closet, right? There’s a lot of different spaces.

Jim Ray: And so maybe to kind of help us finish up here, what’s what are your recommendations for the folks who are building these things. This actually came up in a conversation we were having with Dan Holmgren last week is we don’t want Bluesky to just become a ghost town that’s nothing but like lightly interconnected. communities. So as we work on these permissioned data project and the communities projects, how do we make sure that we kind of keep the those interconnected threads between the big public spaces that do have a lot of benefits. I think you and I both owe our entire career, I owe a lot of my personal life to having been online at a certain point.

And making those available for other people without pulling up the ladder or.

Erin Kissane: Yeah, I mean I wish I had a great answer for you. I think this is the most exciting question. to me right now and one of the reasons that I’ve been doing a lot of like dipping into philosophers who specifically work on epistemics and democracy and things like that is like they’re actually engaging with these questions. To go back to C. Thi Nguyen, what is the difference between an epistemic bubble, which is permeable, and an echo chamber, which is armored a lot of the answers are cultural rather than technological, but maybe there’s some bearing. So I mean I do think that there’s something probably to be gained by not having privacies happen in one place and our more public interactions happen entirely in another one.

Like the if it’s if it’s just like to rewind back many years, if it was just public Twitter, no private accounts, everything in the big undifferentiated stream plus group chat there’s not there’s not a lot of productive movement, but I do think if you had communication systems, ideally interoperable ones, that permitted groups that do some of their work together to send other parts of their work out into ideally a thriving more you know into the faster waters, into the places where there’s more people and more stuff moving around. I think that is worth a lot of experimentation. I don’t know I don’t know what will happen, but it seems to me like a really good zone to do a set of experiments on.

And I think that’s what we’re gonna see. I think like Bluesky PBC y’all are gonna do the one set of things and like Blacksky is gonna do another one and I don’t know what we’ll see from various other big projects. On the protocol, I watch closely Bonfire Networks. work on the Fediverse, which may eventually also, you know, overlap with atproto. They’re specifically building on what I still think of as like the LiveJournal model, where you have a thing moving broadly and then sort of not concentric but overlapping spaces, it’s hard to find ways to do that without like a power user interface.

Jim Ray: Right.

Erin Kissane: You know, this is kind of a hand wavy answer, but I think we’re gonna we’re gonna have to find out. Like this is fuck around and find out, but maybe listen in the process so we don’t hurt people is sort of my that’s like my big hope for all of this stuff, right? Like let’s make new mistakes. And then try not to hurt people while we do them. Let’s like, yes, we can do things, but let’s do them with an openness to the needs of others to try to do them in ways that produce good things and not hurt people.

Jim Ray: I love that. That’s that really does feel like a perfect place to end. So thank you for that, Erin.