So I've been optimizing the number of calls I make to twitter this week ( https://git.sr.ht/~cloutier/bird.makeup/log ) and the error rate has been going down a lot.
It seems the user endpoint is the only one rated limited, so all the errors left are coming from that. You'll see that when looking at a profile or when trying to follow.
Still have many ideas on how I could optimize this further