Prometheus - Collector

About

A collector is a metrics collector and is embedded in the prometheus server.

Management

Conf

The scrape_configs block of the prometheus.yml file controls what resources Prometheus monitors. The below example shows a single job, called prometheus, which scrapes the metrics (time series data) exposed by an exporter (below this is the internal Prometheus server exporter)

Example:

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
    scheme: 'https'

where:

Prometheus - Job

Ref Conf Doc

Relabeling

Relabeling permits:

to manipulate the label
to filter out time serie

Relabeling Configuration:

relabel_config - This relabeling step happens before a connection to an exporter with connection data available (Prometheus Service Discovery)
- No connection means there is no data metrics or labels yet, so you can't drop metrics.
- You can only manipulate the internal labels Prometheus set.
- Why ? You can use relabel_config to change address, port or to make a decision whether this instance should be scraped at all.
metric_relabel_configs
- Metric relabeling is applied to samples as the last step before ingestion.
- Usage:
  - exclude time series that are too expensive to ingest.
  - drop/modify internal URI labelling such as instance and scrape_uri

Extra labels:

Discovered label that you can see on the target page by clicking on a target
Internal Labels

Internal labels begin with two underscores and are removed after all relabeling steps are applied

Label name	Description
__name__	The scraped metric’s name
__address__	host:port of the scrape target
__scheme__	URI scheme of the scrape target
__metrics_path__	Metrics endpoint of the scrape target
__param_<name>	is the value of the first URL parameter passed to the target
__scrape_interval__	The target’s scrape interval (experimental)
__scrape_timeout__	The target’s timeout (experimental)
__meta_	Special labels set set by the Service Discovery mechanism
__tmp	Special prefix used to temporarily store label values before discarding them

After relabeling, the instance label is set to the value of __address__ by default if it was not set during relabeling.

How and when to drop internal labels?

Don't delete the instance label on relabel_config or you get an error because the collector don't know where to connect:

msg="Creating target failed" err="instance 0 in group endpoints/com-datacadamia/com-datacadamia: no address"

If you want to drop it, drop it on metric_relabel_configs