To fix the crawling, I'm going to need to start using Twitter accounts to fetch new posts. This is getting complicated for the time I can put on this as a side project, so I'm going to rely on Nitter implementing this, then I'm going to use that. You can track progress here: https://github.com/zedeus/nitter/issues/983
There is another thing I can fix natively in bird.makeup though. I noticed pinned tweets are there in user lookups. I can forward this, and the top 100 limitation doesn't apply to that.
2/2
I'm getting a lot of question about what's currently working with bird.makeup, so here it is:
The system is mostly back online, but there are some major limitation when crawling new posts.
- To be forwarded, a post has to be in the top 100 of an account as measured by likes
- There are no retweets being forwarded right now
1/2
@troy except hitting that triggers Nitter to do a request, consuming some of the precious quota. I was thinking of having a way to make another use of requests that are being done anyway
One really useful thing Nitter could do is to log the id of new tweets (and their author) as it encounters them during normal operation. Bird.makeup could simply hook into that byproduct and it can already figure out the rest on its own.
I’ll sleep on it and if I still like it I’ll propose it to the nitter dev
@troy I figured it out, it was because I hand rolled my nginx config, and I didn't pass the remote IP like I should: https://github.com/mastodon/mastodon/blob/main/dist/nginx.conf#L146
So mastodon added an artificial rate limit
@troy it’s an empty server without federation yet. All request are exactly 20.2 to 20.5 seconds which makes me think there is an artificial delay of 20s added somewhere
Open source developer. Wikidata, IPFS, Linux, Ethereum. /r/fuckcars enthusiast. I tend to boost funny stuff.