问题1:vmagent负责采集数据,prometheus负责计算数据,会偶现如下问题
prometheus会偶现报错:
level=error ts=2021-11-05T02:35:42.386Z caller=write_handler.go:53 component=web msg="Out of order sample from remote write" err="out of order sample"
2021-11-04T02:18:11.380Z error app/vmagent/remotewrite/client.go:242 couldn't send a block with size 28985 bytes to "1:secret-url": Post "
http://cluster-monitor:9090/api/v1/write": dial tcp4: i/o timeout; re-sending the block in 2.000 seconds
prometheus出现问题可能是因为vmagent在发送数据时,prometheus在接受新数据的同时也接收到了时间戳更早的旧数据
经常查询文档,可能是如下原因(
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149 ):
同时,vmagent会出现如下报错: