So I spent the whole day optimizing bird.makeup and got about a 5x in performance. Things are starting to run better, seems we are iterating on all accounts within 4-5 hours right now. I have other ideas that will probably get another 3x that I want to implement tonight, so performance should be right on target.
Until more people sign up of course! 🙃
@vincent What are the system requirements, if any, for running your own instance?
My cloud server has 2 GB of RAM and it keeps running out of memory. I had to add a memory limit to the birdmakeup container in the docker-compose.yml file to keep the system stable (but now the container is killed/restarted every half hour or so).
When it works, performance is great though 😁 Thanks so much for building this!
@smallsco Yeah, there is a memory leak somewhere... I'm hunting for it!
I just took a quick look at it (by the way: nice code). Three things potential things I found (no guarantees):
1. The used version of Newtonsoft has a potential memory leak: https://github.com/advisories/GHSA-5crp-9r3c-p9vr
2. MagicKey has a property `_rsa` which needs to be disposed. (there might be more)
3. Quite a few regex that might never complete.
I would need to actually run the application for my tools to find something, but maybe I'm lucky with the above.
@DevWouter @smallsco Thanks for the compliment! But the credit goes to @BirdsiteLIVE .
I think I figured it out. I took a memory dump, and noticed large allocation related to SQL queries. When I removed Dapper and used vanilla npgsql the problem when away. It's really not obvious to me what was happening there, but it might be related to your number 1
@BirdsiteLIVE @DevWouter @smallsco Okay so the npgsql thing was not it at all. It's actually that TransformBlock will actually buffer data if you don't bound them.
See the commit that fixed it for me : https://git.sr.ht/~cloutier/bird.makeup/commit/5c75e79abc3cf377da4dbbe369a156ae9c3b890a