How Decentralized Is Bluesky Really? dustycloud.org/blog/how-decent…
A technical deep-dive, since people have been asking me for my thoughts. I'll expand a bit on some of the key points here in a thread. 🧵
teilten dies erneut
How Decentralized Is Bluesky Really? dustycloud.org/blog/how-decent…
A technical deep-dive, since people have been asking me for my thoughts. I'll expand a bit on some of the key points here in a thread. 🧵
teilten dies erneut
⚝ Mirko ⚝
Als Antwort auf Christine Lemmer-Webber • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •First of all, before I say anything else, my goal here is NOT to be mean to Bluesky's devs. I know there's a lot of fediverse-Bluesky rivalry, but I have enormous respect for Jay Graber and her team and I know they believe in their vision!
This started because I got some very kind encouragement by @bnewbold to write something. I'm trying to be technical in my analysis, not unkind. I hope that can be recognized, really and truly.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •That said, let's get to the summary: Bluesky / ATProto are not decentralized or federated, according to my analysis.
However, the "credible exit" goal is worth perusing, and does use decentralization techniques! But it is not decentralization/federation without moving the goalposts on those terms.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Furthermore, I think Bluesky is providing something valuable: a lot of people are trying to leave X-Twitter *right now* because it has become a completely toxic place.
The fact that Bluesky's team has managed to scale to receive such users is incredible, nearly feeling miraculous.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •On the fediverse we also see a lot of accusations of Bluesky being owned by Jack Dorsey, and this isn't true. My understanding is that Jay performed an impressive amount of negotiation to allow Bluesky to receive funding independently.
These days Jack Dorsey is instead focusing on Nostr, which I can only describe as "a sequel to Secure Scuttlebutt with extremely bad vibes where bitcoin people talk about bitcoin"
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •I participated a bit in the process of when Bluesky was Jack Dorsey and Parag Agrawal's personal project. I also believe Jack and Parag were sincere about Bluesky as a decentralized social network protocol that Twitter would adopt, which is the directive that Bluesky was given as an organization.
When Jay Graber was awarded the position to lead Bluesky, I was not surprised. To me, Jay was the obvious choice to deliver what Bluesky was being directed, and I do think Jay is an excellent leader
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •There is also something which Bluesky gets right which the fediverse does not. I mentioned that Bluesky uses decentralization *techniques*, and the most important of those is content-addressing. This allows content to exist even when a server goes down.
This is a great decision and I have advocated that the fediverse do so as well. In fact several years ago I wrote a demo in @spritely's early days showing off how one could build a content-addressed ActivityPub in a spec-compatible way.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •So I have opened here with the things that Bluesky does well. As you may guess, we are about to move into critiques territory, and it's a lot of critiques from a *decentralization*/*federation* perspective. It doesn't erase the "credible exit" goals, which I think are good still.
Let's dive in...
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •A frequent way of describing Bluesky's decentralization, including by Bluesky's team, is "it's like a bunch of blogs (Personal Data Stores), and then the relay/appview/etc pieces are like search engines"
This is a reasonable starting point for thinking about things, so let's run with it.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •In fact ATProto's own tutorial even says "Think of our app like a Google": atproto.com/guides/application…
And indeed this is a good way to think about things. But it doesn't seem so bad, because we have Personal Data Stores like blogs, so probably things are fine, right?
Quick start guide to building applications on AT Protocol - AT Protocol
AT ProtocolChristine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •While most people would argue that blogs and websites are open, few would argue that *Google* is open. So this is a curious place to begin thinking, and yet structually, it is actually quite apt.
PDS'es are like blogs, the rest is like Google. But relays/appviews/etc do a lot *more* than Google.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •But let's stay on this blog/search engine analogy for a while before we unpack what it means on a *technical* level, which is interesting. Let's analyze for the moment from a power dynamics level.
Building a web search engine is actually pretty easy these days, you can do so with off-the-shelf tools. And yet there are only a couple of search engines *really*, Google and Bing (DDG mostly uses Bing). And yet the information is right there. *Anyone* could run their own engine. Why don't they?
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Furthermore there is an interesting connection between blogs and social media: the death of blogs + feed aggregation directly aligns with the death of social media.
How many of you were around for the birth and awkward death of blog engine feeds? Because I was! Oh, remember Google Reader?
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Feed readers are also simple, and in fact they were even easy to self host, even on the desktop! But Google Reader came in and was such a good design that everyone used it.
When it went away, blogs were still *there*. But blogging as a *syndication medium* died. One big player left, and it's gone.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •This was sad for me especially; my favorite medium on the internet ever was webcomics. Webcomics still exist, sort of, but the loss of independent publishing and aggregation meant that they had to change to survive.
The shape of webcomics started to get shaped to the shape of Twitter's image box.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •So. It isn't enough to self-host your own PDS. Whether or not people can run their own relays/appviews/etc actually matters *a lot* if we want this stuff to survive.
So, can we? How hard is it to run your own AppView/Relay/etc?
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Today, there is only one real organization running a Relay that really matters or an AppView that people use for anything other than fun aggregation of statistics. Nothing that resembles meaningful decentralization of the network. It's all run by one company: Bluesky.
But could we change that?
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •People are trying; most notably alice has done some great work recently: alice.bsky.sh/post/3laega7icmi…
So now someone *can* run their own Relay (not the AppView yet, but maybe soon), and we're getting a sense of the cost and scale. This is good news; we didn't know before.
How to self-host all of Bluesky except the AppView (for now) — alice.bsky.sh
alice.bsky.shChristine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •In fact we also have an idea of the rate of growth. Approximately 4 months prior, @bnewbold.net posted an article detailing how to run a Bluesky relay: whtwnd.com/bnewbold.net/entrie…
This is great. We need more people trying to do so to get a sense of how decentralized things can be.
Notes on Running a Full-Network atproto Relay (July 2024) | bryan newbold
whtwnd.comChristine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Just focusing on storage, in July @bnewbold.net estimated the amount of storage expected to run a Bluesky relay is approx 1 terabyte. In just 4 months at start of this month (November), alice estimates nearly 5 terabytes.
This is a fast growth rate and this is *before* the big post-election influx.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •I tried estimating how much this would cost; as a lazy approximation I dumped a 5 terabyte machine into seeing what Linode would cost to self-host, and it was approximately 55k a year: bsky.app/profile/dustyweb.bsky…
That's a lazy estimate, but that's also what many people make in the US every year
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •In fact @bnewbold and alice had gotten the server down to just close to $200/month in their estimate, much much cheaper than I had, by choosing a dedicated server plan. Much cheaper!
But there's a problem though; that's cheap because you've got a server that has a dedicated disk...
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •I have spent a lot of time focusing on the cost of storage, but storage is only one cost required. These estimates have been done so far against servers that *nobody is actually using*. The cost of servers that people are using will be much higher, because more needs to happen than just store things.
And that is not even to mention the challenges with administrating, dealing with takedown requests, illegal content, etc, which are probably much more serious.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Let's take a break, the analysis of server costs is boring and I don't like doing it, and I'm sure people will throw numbers at me of the absolute race-to-the-bottom hosting numbers they can find to store and run all this stuff, but really that's not interesting to me.
Let's do a comparison.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •As said, tens of thousands of people are self-hosting *today*. Fediverse software doesn't just scale up, it scales *down*.
GotoSocial is cheap enough on resources where you can run it for family and friends on a raspberry pi or spare laptop you have sitting around.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Now you're hitting the point in this thread where some of you may be thinking "aha! this is where Christine is saying that the fediverse/activitypub are awesome and atproto is terrible!"
you have NO IDEA HOW MUCH I CRITICIZE THE FEDIVERSE ALL THE TIME, I do it all the time, and will later here
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •The fediverse has a lot of flaws. Oh trust me, we're gonna get to that.
But comparison-wise: what I mean to say is that architectural decisions matter, and scaling up isn't the only thing that's important, *scaling down matters too*.
If you care about decentralization, anyway.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Now look, we're about 1/3 of the way done here, there's a lot more to say, and a lot more said in my article, it's about 24 pages long if you print it out.
This is because in the age of TikTok I somehow have decided to model myself after David Foster Wallace, sorry
"Consider the Fediverse" I guess
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •But now, I will break for lunch. Enjoy your intermission because I will be back. We still have to get through the remaining 2/3 of the analysis, after all.
======= LUNCH BREAK HERE =======
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Okay I am back from lunch, time to resume my analysis thread for "How decentralized is bluesky really?" dustycloud.org/blog/how-decent…
I have been receiving a lot of notifications, I am not reading any of them until I finish with this so bear with me, BEAR WITH ME, we're gonna make it through
How decentralized is Bluesky really? -- Dustycloud Brainstorms
dustycloud.orgChristine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •And before we make it any further can I say that I watched a nice medley of David Bowie and Cher singing, and it was so lovely youtube.com/watch?v=KPlN8RBP-W…
@mlemweb said "of course it's very heteronormative despite having two queer coded icons on the stage and ISN'T THAT THE WAY I guess
- YouTube
www.youtube.comChristine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •But where was I? Oh yes. We had talked about why PDS'es aren't enough (blog/google analogy), relative costs of hosting things on ATProto vs ActivityPub, etc etc
But we haven't gotten into the really interesting parts which are the structural analysis stuff, so let's move onto that
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Now you may be saying, "Christine, this is really unfair, because you're looking at ActivityPub servers which are only dealing with a small amount of the network, what if it were an ActivityPub mega-node? What are the costs THEN huh?" and "What if we hosted just PART of ATProto?"
What then INDEED
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •ATProto is not designed for the Relay and AppViews to only hold part of the network, not *really*, and ActivityPub is. We'll get to this in a moment.
But Bluesky actually has good justification for this! I will defend it insofar as Bluesky was making a serious *design decision*
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Remember the directive that Bluesky was given: develop a decentralized protocol which Twitter can adopt. That informs a lot of things, and has meant that Bluesky was really very ready for this moment!
If you're an ex-X-Twitter user then by god, you're going to be amazed! It's just like Twitter!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •- Bluesky's gotta scale BIG and do so FAST (scaling down: not a priority at all)
- It has to be something Twitter can adopt (of course, not anymore, but initially)
- Everything on ATProto is public (yes, everything, including your blocks btw, we'll get to that)
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •But here's the other thing. People have trouble with the fediverse! All those decentralization decisions get in the way, my god, you've got to choose a server, search doesn't work well (actually it could but it's a cultural thing, different topic), and worst of all:
Sometimes you DON'T SEE REPLIES!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •The thing you often get seen thrown around is "it's amazing, I had no idea a decentralized protocol could just work like that! How on earth did they solve that in a decentralized system and so FAST too!"
It's simple: all those things "just work" because Bluesky is centralized.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Now yes, they are using decentralized techniques. Remember when I said content-addressed storage is a good idea and the fediverse should do it too? IT IS! (And as I also said, it's actually fully possible for the fediverse to do, more on that later.)
But the reality is, it's still *centralized*
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •In every meaningful way from a power dynamics perspective *EXCEPT* the category of "credible exit" (which I am saying and agreeing is a good idea!) Bluesky is centralized.
MAYBE another big corporation could come along and host all this stuff but that's adding a Bing to our Google
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Yes, you can host your own PDS. You can also host your own blog. But try hosting your own PDS and NOT hosting a relay or AppView and you can't do much.
Blogs are decentralized, Google is not.
PDS'es are decentralized, Bluesky is not.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •We're getting to the point where we get to why I'm so damn frustrated about this and have been biting my tongue until it nearly comes detached from my mouth: users THINK Bluesky is decentralized because they're TOLD Bluesky is decentralized
AUGH! *That's* what drives me nuts.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Here's an example of this problem in action
fry69: "The working search box was the second thing that impressed me on Bluesky, I thought that was not possible with a decentralized model"
Sorry fry sixty-nine I regret to inform you the reason search works so well is that it's centralized! THAT'S WHY
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •So hold on, let me set some terms for "decentralization" and "federation" that I think are reasonable.
> Decentralization: the result of a system that diffuses power throughout its structure, so that no node holds particular power at the center.
Pretty reasonable. Do you agree? I hope so!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Okay how about "federation" now because this is a *technical term* that the *fediverse has established* and I'm kinda PO'ed about the goalposts being moved on this one.
A lot of people coming to Bluesky have never heard of "federation" before in a social network so listen up this is important!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Here's my definition of federation:
> Federation: a technical approach to communication architecture which achieves decentralization by many independent nodes cooperating and communicating to be a unified whole, with no node holding more power than the responsibility or communication of its parts.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Maybe there is another way to achieve federation, but it's about the power dynamics. It's a technical immersion of power dynamics, the flow and interchange of cooperation between many parts.
So you may say, well, doesn't ATProto have that? After all, messages flow through the different parts!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •I'll take one more note about federation which is that often time the message passing mechanism of the fediverse is often called "federation", but theoretically another mechanism could exist, but I'm actually not so sure of that.
There's a reason the actor model and the lambda calculus are undying
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Oh god Christine said "the lambda calculus" did you know she's into lisp and functional programming, what's she going to talk about next monads?!?!
I am not going to talk about monads. Not TODAY
But we do need to get a better architectural idea of how these systems work because it matters a lot!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •So let me introduce two models of communication which we can use to analyze these two systems. It's important!
- Fediverse/ActivityPub: "message passing"
- Bluesky/ATProto: "shared heap"
Okay, cool, terms established, let's talk about them and why they matter because they matter A LOT
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •"Message passing" is what ActivityPub uses. It's "like email", people say, and that's true.
Actually it's even a lot like physical mail. You write a letter, you say where it should go, it gets delivered to your house.
Message passing. The world runs on it.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Now I can use message passing to send a message to you *directly* and indeed, that's "like email". For one-to-one correspondence, that's enough.
But it's not enough for a followers/following type mechanism. But we can build it on top! Thank *you* computational abstractions!
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •On top of "message passing" we will build "publish-subscribe" as a second-layer abstraction
"Your ideas are interesting and I'd like to subscribe to your newsletter."
You send me a letter saying you'd like to hear the things I have to say, okay, you're part of the reader list. That's how it works.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •On top of that we can build even more abstractions and the net result is that this is how federation works in pretty much every "federated" system I know.
ActivityPub does some extra work to help you see replies on a thread, think "letters to the editor". This is a bit lossy sometimes though
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •I actually think this is improvable but I mostly don't care because this isn't as big a complaint as people tend to think it is on the fediverse, the other concerns like "what instance do I pick" tend to be bigger and "oh no my server went down"
That can be improved, we'll talk about that later
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •So okay, the federation is "message passing" and like email, or physical mail. You have an idea how it works.
Now we need to get to that other thing, a "shared heap" architecture. What on earth does that mean?
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •If "message passing" is like "mail comes to your house", a "shared heap" system works differently
In a "shared heap" system, all the mail gets dumped at the post office, and in the most naive version, you go over there and read through every single piece of mail to see which one is relevant to you
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •The Bluesky approach to the "shared heap" is that *everything* goes into the big, centralized shared heap. Bluesky takes a "god's eye" view: it knows everything, and so knows what all your replies are, and can give you perfect search.
Secure Scuttlebutt / Nostr... well long story. Lossier, I'll say
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •You can imagine the physical world version of "message passing" already because you already live in this world. Messages come to your house or apartment building or whatever
For Bluesky's "shared heap" architecture, you'd have to build a whole addition to your house for everyone's mail
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •That's exactly why running a Relay or AppView is expensive: you're building an addition to your house for all the world's mail.
Eeep! That ain't cheap. That's why I'm saying: decentralization also means the ability to *scale down*.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Look, I know that I've been hitting this nail on the head for a while but: the web is open, blogs are open, but Google isn't open
But you could run your own Google, in theory. You could index the web. So why aren't you?
Ah yeah. Same thing here. That's what I mean, that's why it's centralized
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •Now as I have said, this is a *design decision*. And remember: most users of Bluesky really *don't care*. Decentralization is not their focus, they're trying to get the hell off the nazi hellscape that Musk's toxic reign of Twitter has become.
Bluesky's architecture, actually, is great for them.
Christine Lemmer-Webber
Als Antwort auf Christine Lemmer-Webber • • •If what your *goal* is to get off Twitter, then Bluesky has solved it. They solved it by building another Twitter, and this time it's open source, which is cool! And it might have this "credible exit" thing.
But god damnit it's not decentralized and it's not federated stop TELLING people that