Follow
I have a #prometheus question. Suppose I have many targets and there is a performance problem if all of them are scraped at the same time.
I can set scrape_interval to decide how often they are scraped, but prometheus seems to scrape them all at once, e.g. if scrape_interval is 2 minutes then it scrapes all of them at 20:00, then all again at 20:22, then again at 20:24 and so on.
What I would like to do is to tell prometheus to smear them out over time. Is it possible and if so, how?
@eliasr there's a number of agents that can do the scraping with random offsets and then use remote_write to push the metrics to Prometheus. Off the top of my head #Telegraf and #VictoriaMetrics agent can do this. #OTelCollector probably as well, though I'd need to consult the documentation to verify that.