Client_golang: Consider allowing setting a timestamp for ConstMetrics.

8

With a lot of caveats….

beorn7 picture beorn7  ·  7 Dec 2015

Most helpful comment

14

With staleness handling complete and timestamp abuse mitigation in place, I believe it's now okay to expose timestamp support for the handful of advanced use cases that require it.

brian-brazil picture brian-brazil  ·  9 Jun 2017

All comments

0

Use-case: Monitored client cares about an exact timestamp, and the timestamp is close to the time of the scrape (e.g. it is between beginning and end of a scrape, which implies no problem with staleness).

beorn7 picture beorn7  ·  1 Jan 2016
0

That's a highly advanced use case that should generally be discouraged, as having different timestamps for metrics from the same scrape will cause oddness and may prevent some features from working correctly.

brian-brazil picture brian-brazil  ·  1 Jan 2016
7

Another use case case is for hardware devices that generate counters with their own time stamps. These are retrieved and pushed to some server that is then scraped by Prometheus. It helps to use the source timestamp to get an accurate rate.
Unlike the comment by @beorn7 above, this timestamp cannot be guaranteed to be between start and end of a scrape, it sounds like that may be a problem.

bits01 picture bits01  ·  24 Feb 2016
5

@bits01 above is applicable to network devices too. Would I be right in saying, that, without this enhancement, a gateway implemented in go and terminating telemetry streams from network devices, and wishing to persist to prometheus while preserving original timestamps must push via pushgw to do so. It would be great if the telemetry gateway could be scraped directly and network device timestamps could be preserved.

ccassar picture ccassar  ·  21 Aug 2016
0

Those are the okay use cases (presuming you aren't talking about SNMP).

However offering any functionality like this before https://github.com/prometheus/prometheus/issues/398 is complete would be reckless, as the vast majority of users looking to use timestamps it turned out that they should either have used normal pull or the pushgateway without timetamps. Had those users used timestamps, they would have run into a number of problems (as the few users that figured out how discovered).

brian-brazil picture brian-brazil  ·  21 Aug 2016
2

Indeed. The interesting case is streaming telemetry not SNMP. The context is a network pushing timestamped content. I would like to push the timestamped content into prometheus. We can accommodate collapsing multiple updates in the same series to the latest within a scrape period, but we would like to preserve the original timestamp. We are currently achieving something close using the pushgateway, but that has its own downsides.

ccassar picture ccassar  ·  25 Aug 2016
0

Coming from the discussion about the collectd_exporter, an exporter that does something equivalent to federation, i.e. it exposes regularly updated timestamped metrics from another monitoring system, would be a good example for a use case.

I'll mark this with the v0.10 milestone because the API to create const metrics will be reworked in v0.10, and it is the right time to think about how the timestamp would be passed in. At least I want to avoid a breaking change for this feature.

beorn7 picture beorn7  ·  3 Nov 2016
0

My stance remains that this shouldn't be exposed before https://github.com/prometheus/prometheus/issues/398 is resolved. We continue to see users abusing and misusing timestamps, which'd only get far worse if the official client libraries actually supported it.

brian-brazil picture brian-brazil  ·  3 Nov 2016
0

Any progress being made on #398? It has been open for a long time.

bits01 picture bits01  ·  3 Nov 2016
6

I just wanted to voice an additional position on the relevance of this for device telemetry. Prometheus makes a great tool for analyzing IoT device telemetry. It is relatively easy to make a telemetry extractor that is scraped instead of using the push gateway - but the problem is time skew between event time (on the device) and processing time (on the extractor/target) can impact the interpretation of the resulting visualized data (in grafana in this case)

ptone picture ptone  ·  12 May 2017
14

With staleness handling complete and timestamp abuse mitigation in place, I believe it's now okay to expose timestamp support for the handful of advanced use cases that require it.

brian-brazil picture brian-brazil  ·  9 Jun 2017
11

Due to public demand, this was fixed for v0.9 after all.

beorn7 picture beorn7  ·  21 Aug 2018