Show more
Sore feet, fore feet. Whucha gonna do, owhucha gonna do...
I don't engage in mental combat with the unarmed.
:hacker_f::hacker_s::hacker_e:
:hacker_p::hacker_e::hacker_e::hacker_p::hacker_s:

The downtime yesterday was caused, it appears, by a confluence of a few things, none of which appear to have been done to intentionally disrupt FSE.

First, there were repeated attempts to load the current Hellthread, which at this point contains a 17MB JSON payload that will take your browser an excessive amount of time to render if your browser even loads it. Surprisingly, though this tied up some resources, this was not the biggest deal for the DB, but it did occasionally crash some of the processes in the Pleroma app server. BEAM was using an inordinate amount of RAM and it appears that Ecto has trouble with unexpected crashes. (That thread started some time in 2019, though it has since been orphaned and the oldest post was started on the first. Maybe make a new thread and tag the people you were talking with in that subthread.)

We also got hit by some weird scrapers. That's background noise, someone is always scraping the site. This time, someone was going through every account, even long-dead accounts, and following/followers lists, that's weird.

There was a lot of traffic from Gab, but not the way you might expect. Too many posts were coming in from them still, but the unusual part was a bunch of people arriving at FSE and landing directly on the local mirrors of Gab users' pages and then leaving the browser on that tab. I don't know why, maybe someone was checking if their posts had federated.

So already the traffic patterns were a little odd, FSE was creaking a bit, but the error rate was still low. Then, the tipping point: a few people decided to jack off. Different IPs, different searches, all porn-related, none of them logged in, just clicked around a bunch of porn-related accounts and hashtags. The patterns looked human. I wonder if we somehow started getting SEO juice for some porn terms, but you guys were e-circle-jerking so I hope you feel appropriately grossed out. Once all that came together, DB connections started timing out.

So, it'd have to be someone pretty clever if it was an intentional DoS, as they'd have to be able to make it look like just an unlucky confluence of some unique events and Ecto doing kind of a shit job deciding when to drop a DB connection and reconnect.

Fun fact: on average, four times a second, someone hits the notifications endpoint. The most requested endpoint is /objects/$object_id (used by other instances to get posts from FSE), then notifications, then /inbox. (Over the course of the 24-hour period before the server fell over, the only minute during which these lost the top slot was 22:12 UTC, when someone using Tor started hammering the shit out of the public timeline, but that doesn't actually break the server; that endpoint is pretty fast and it is rate-limited.)
hellthread2020ofthefuture.png
๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•

๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŽฉ๐ŸŒ•๐ŸŒ•

๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ˜๐ŸŒ‘๐ŸŒ’๐ŸŒ•

๐ŸŒ•๐ŸŒ•๐ŸŒ•๐ŸŒ˜๐ŸŒ‘๐ŸŒ‘๐ŸŒ‘๐ŸŒ“

๐ŸŒ•๐ŸŒ•๐ŸŒ–๐ŸŒ‘๐Ÿ‘๐ŸŒ‘๐Ÿ‘๐ŸŒ“

๐ŸŒ•๐ŸŒ•๐ŸŒ—๐ŸŒ‘๐ŸŒ‘๐Ÿ‘„๐ŸŒ‘๐ŸŒ”

๐ŸŒ•๐ŸŒ•๐ŸŒ˜๐ŸŒ‘๐ŸŒ‘๐ŸŒ‘๐ŸŒ’๐ŸŒ•

๐ŸŒ•๐ŸŒ•๐ŸŒ˜๐ŸŒ‘๐ŸŒ‘๐ŸŽ€๐ŸŒ“๐ŸŒ•

๐ŸŒ•๐ŸŒ•๐ŸŒ˜๐ŸŒ‘๐ŸŒ‘๐ŸŒ‘๐ŸŒ”๐ŸŒ•

๐ŸŒ•๐ŸŒ•๐ŸŒ˜๐ŸŒ”๐ŸŒ—๐ŸŒ‘๐ŸŒ•๐ŸŒ•

๐ŸŒ•๐ŸŒ–๐ŸŒ’๐ŸŒ•๐ŸŒ—๐ŸŒ’๐ŸŒ•๐ŸŒ•

๐ŸŒ•๐ŸŒ—๐ŸŒ“๐ŸŒ•๐ŸŒ—๐ŸŒ“๐ŸŒ•๐ŸŒ•

๐ŸŒ•๐ŸŒ˜๐ŸŒ”๐ŸŒ•๐ŸŒ—๐ŸŒ“๐ŸŒ•๐ŸŒ•

๐ŸŒ•๐Ÿ‘ ๐ŸŒ•๐ŸŒ•๐ŸŒ•๐Ÿ‘ ๐ŸŒ•๐ŸŒ•
The saddest part is that Gab mongs will buy his gaslighting that somehow even though Gab has had these features in place for a long while Gab is somehow more of a free speech haven:
7564CEBE-1D16-4CD9-8211-7B432A2โ€ฆ
Show more

Also Named Bort ๐Ÿ‡จ๐Ÿ‡ฆ's choices:

Librem Social

Librem Social is an opt-in public network. Messages are shared under Creative Commons BY-SA 4.0 license terms. Policy.

Stay safe. Please abide by our code of conduct.

(Source code)

image/svg+xml Librem Chat image/svg+xml