#Pleromatan stealing gigabytes of your storage for its neverending database hunger ​:pleroma-tan_corndog:​

#Pleroma

RE:
https://mai.waifuism.life/notes/9pag98xnswua00a4

@kirby@lab.nyanide.com I heard that Pleroma's postgres db is usually larger than Misskey's ​:satrithink:​

Like Pleroma is already using 1gb for just a few users while a Misskey that has plenty of users (like 50) is at around 400-500mb
​:sagume_think:​

@mima @kirby pleroma stores the whole objects/activities separately as jsonb and then indexes them in too many ways to make up for it

example, for 30.6 GB of activities, there's an 17.8 GB to go with it!1!
same for objects, 19.0 GB of data needs an index of 7.0 GB to go with it
Follow

@i
How would parsing activities right away affect the performance though?
Pleroma has very mild hardware requirements for a fully featured instance software, Miski on they other hand… I don't know really, I don't have the hardware to host it 🀭
@kirby @mima

@m0xee @kirby @mima there's a time/space tradeoff, mastodon/misskey is architectured to run on many servers, so they seem daunting at first, while pleroma can not, even though erlang/elixir is known for scaling to thousand node clusters, as it was not written to

so people have to keep buying ever bigger servers with ever bigger drives, instead of being able to split up the responsibilities in any way that's not handling user uploads

or continuously pruning the database, which means facing hours of performance issues while postgresql repacks all the messy data/indexes all over again
@i
Good point!
Mine was that throwing JSON in original form right into DB has some advantages too. TBH I was surprised too when I discovered that, but now I thing I can see some reasoning behind it.
As for scalability β€” you're right of course, but ActivityPub itself isn't very scalable β€” if I designed Fedi from the ground up, I'd probably do a lot of things differently and implement it as sort of bus with guaranteed delivery and all that β€” probably won't work well in real world, with the way some run their instances, they might go offline all of a sudden, then come back up unpredictably. Sometimes I'm surprised Fedi works at all πŸ˜‚
I'm not well-versed in how Misskey works, but it seems to me it tries to work this around as much as it's possible β€” and that makes the queues grow quite big and weaker hardware starts choking on it.
@kirby @mima @m0xee
@m0xEE @kirby @mima @m0xee >sort of bus with guaranteed delivery
we already have oban as a job queue, as do misskey/mastodon, with multiple retries on an exponential delay

we could even run oban on multiple machines, if the code wasn't an absolute mess
Sign in to participate in the conversation
Librem Social

Librem Social is an opt-in public network. Messages are shared under Creative Commons BY-SA 4.0 license terms. Policy.

Stay safe. Please abide by our code of conduct.

(Source code)

image/svg+xml Librem Chat image/svg+xml