@vincent I am working on a twitter-to-bird.makeup importer and I have some questions:
1) Are you okay with that? Would it be too bad if you suddenly have some hundreds of users, each of them importing hundreds to thousands of twitter profiles?
2) What would be a reasonable request rate? Is something like Mastodon's built-in rate limits (300 requests / 5min window) ok?
3) it seems that I get an error for any profile your server hasn't seen before. I tried checking the user page and expected a 404, but that seems to trigger an error as well. Is there any workaround and/or how can I help you fix it?
@raphael I get rated limited on profiles lookup right now. I'll have to figure out an alternative way to get the information I want, or add more servers to spread the crawling on more IPs. I have some ideas for profiles especially, but people figuring things out about the twitter internal api and sending me them is always useful
Demand is very high for bird.makeup, I optimize things almost every weekend, but by the end of the week, more optimizations are needed! 🙃
@raphael Exactly! I am not considering that kind of architecture right now, as there are a lot of simpler options to scale this.
Scaling it is mostly just time I have to put in the project, but time is limited since I also have a programming job during the day
@vincent
Are you using nitter to get the data from Twitter?
I have setup a nitter instance (https://nitter.communick.com) and I'd gladly run the lookups on my side and send the info to you, but I guess that would require you to trust the client and/or having some quorum from different people before accepting it?