Configure Prometheus scrape for gnmic #45

New Issue

2026-06-11T10:35:32Z

Damien commented

2026-06-11 10:35:32 +00:00

Add gnmic scrape job to existing Prometheus configuration
Set scrape interval (5s for lab)
Define relabeling rules to expose device, interface, vni as Prometheus labels
Validate metrics land correctly in Prometheus with expected label dimensions
Confirm metric names match what will be referenced in the weathermap-ng panel config

Architecture change

Prometheus is currently a dedicated external instance. Scope of this issue is extended to deploy a dedicated Prometheus instance inside the Containerlab topology (self-contained, reproducible with clab deploy/clab destroy), alongside the existing gnmic node (see #43/#44).

The existing dedicated Prometheus instance stays untouched in parallel for now — no cutover until the containerized instance is validated.

Grafana is out of scope for this issue. The existing Grafana instance is used as-is; no new datasource provisioning here.

- Add gnmic scrape job to existing Prometheus configuration - Set scrape interval (5s for lab) - Define relabeling rules to expose `device`, `interface`, `vni` as Prometheus labels - Validate metrics land correctly in Prometheus with expected label dimensions - Confirm metric names match what will be referenced in the weathermap-ng panel config ## Architecture change Prometheus is currently a dedicated external instance. Scope of this issue is extended to deploy a dedicated Prometheus instance **inside the Containerlab topology** (self-contained, reproducible with `clab deploy`/`clab destroy`), alongside the existing gnmic node (see #43/#44). The existing dedicated Prometheus instance stays untouched in parallel for now — no cutover until the containerized instance is validated. **Grafana is out of scope for this issue.** The existing Grafana instance is used as-is; no new datasource provisioning here.

Damien added the telemetry label 2026-06-11 10:35:32 +00:00

Damien commented

2026-07-09 14:28:18 +00:00

Containerized Prometheus deployed and validated

Node: prometheus (prom/prometheus:v2.55.1), added to evpn-lab.clab.yml
Address: 172.16.0.71:9090 (mgmt network), self-contained with clab deploy/clab destroy
Config: configs/prometheus/prometheus.yml — single scrape job gnmic targeting 172.16.0.70:9273, scrape_interval: 5s
Existing external Prometheus instance untouched, running in parallel

Validation (live against the deployed lab)

Target health: up (/api/v1/targets).

BGP session-state and system-memory metrics validated in #44 are queryable with clean labels:

network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{
  device="dc-leaf5", neighbor_address="10.0.1.8", network_instance_name="default", ...} 1

system_memory_state_used{device="dc-access3", subscription_name="eos-system"} ...

core1/core2 BGP sessions present (20 series across underlay + gold VRF)
gold VRF (EVPN Type-5 stretch) present in network_instance_name label (18 series)

Relabeling applied (`metric_relabel_configs`)

Raw label	Clean label
`source`	`device`
`interface_name`	`interface`
`neighbor_neighbor_address`	`neighbor_address`
`vlan_to_vni_vlan` / `entry_vlan`	`vlan`
`entry_mac_address`	`mac_address`
`afi_safi_afi_safi_name`	`afi_safi`

Gap: no native `vni` label

vni is not exposed as a label by any gnmic metric. It only appears as the sample value of interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni, keyed by (device, interface, vlan), e.g.:

interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni{
  device="dc-leaf5", interface="Vxlan1", vlan="40"} 110040

Relabeling can only rename existing scrape-time labels, not promote a sample value to a label. Per-VNI MAC counts (join of vlan_to_vni_state_vni with FDB entry counts) need a PromQL join on vlan, not a native vni label — flagging this as a known gap vs. the original ask, not something fixable at the relabeling stage. Left for a future recording-rule pass if needed.

Grafana/dashboard work out of scope here per issue description.

## Containerized Prometheus deployed and validated - Node: `prometheus` (`prom/prometheus:v2.55.1`), added to `evpn-lab.clab.yml` - Address: `172.16.0.71:9090` (mgmt network), self-contained with `clab deploy`/`clab destroy` - Config: `configs/prometheus/prometheus.yml` — single scrape job `gnmic` targeting `172.16.0.70:9273`, `scrape_interval: 5s` - Existing external Prometheus instance untouched, running in parallel ### Validation (live against the deployed lab) Target health: `up` (`/api/v1/targets`). BGP session-state and system-memory metrics validated in #44 are queryable with clean labels: ``` network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{ device="dc-leaf5", neighbor_address="10.0.1.8", network_instance_name="default", ...} 1 system_memory_state_used{device="dc-access3", subscription_name="eos-system"} ... ``` - `core1`/`core2` BGP sessions present (20 series across underlay + `gold` VRF) - `gold` VRF (EVPN Type-5 stretch) present in `network_instance_name` label (18 series) ### Relabeling applied (`metric_relabel_configs`) | Raw label | Clean label | | ---------------------------------- | -------------------- | | `source` | `device` | | `interface_name` | `interface` | | `neighbor_neighbor_address` | `neighbor_address` | | `vlan_to_vni_vlan` / `entry_vlan` | `vlan` | | `entry_mac_address` | `mac_address` | | `afi_safi_afi_safi_name` | `afi_safi` | ### Gap: no native `vni` label `vni` is **not** exposed as a label by any gnmic metric. It only appears as the sample **value** of `interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni`, keyed by `(device, interface, vlan)`, e.g.: ``` interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni{ device="dc-leaf5", interface="Vxlan1", vlan="40"} 110040 ``` Relabeling can only rename existing scrape-time labels, not promote a sample value to a label. Per-VNI MAC counts (join of `vlan_to_vni_state_vni` with FDB entry counts) need a PromQL join on `vlan`, not a native `vni` label — flagging this as a known gap vs. the original ask, not something fixable at the relabeling stage. Left for a future recording-rule pass if needed. Grafana/dashboard work out of scope here per issue description.

Damien referenced this issue from a commit

2026-07-09 17:45:49 +00:00

Deploy containerized Prometheus for gnmic telemetry

Damien referenced this issue

2026-07-10 09:15:41 +00:00

Build full telemetry dashboard around the weathermap panel #49

Damien added reference feat/telemetry

2026-07-10 13:01:35 +00:00

Damien changed reference from ~~feat/telemetry~~ to feat/telemetry

2026-07-10 13:01:38 +00:00

Damien referenced a pull request that will close this issue

2026-07-20 16:58:24 +00:00