Damien/arista-evpn-vxlan-clab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Configure Prometheus scrape for gnmic #45

New Issue
Closed
opened 2026-06-11 10:35:32 +00:00 by Damien · 1 comment
Damien commented 2026-06-11 10:35:32 +00:00
Owner
Copy Link
  • Add gnmic scrape job to existing Prometheus configuration
  • Set scrape interval (5s for lab)
  • Define relabeling rules to expose device, interface, vni as Prometheus labels
  • Validate metrics land correctly in Prometheus with expected label dimensions
  • Confirm metric names match what will be referenced in the weathermap-ng panel config

Architecture change

Prometheus is currently a dedicated external instance. Scope of this issue is extended to deploy a dedicated Prometheus instance inside the Containerlab topology (self-contained, reproducible with clab deploy/clab destroy), alongside the existing gnmic node (see #43/#44).

The existing dedicated Prometheus instance stays untouched in parallel for now — no cutover until the containerized instance is validated.

Grafana is out of scope for this issue. The existing Grafana instance is used as-is; no new datasource provisioning here.

- Add gnmic scrape job to existing Prometheus configuration - Set scrape interval (5s for lab) - Define relabeling rules to expose `device`, `interface`, `vni` as Prometheus labels - Validate metrics land correctly in Prometheus with expected label dimensions - Confirm metric names match what will be referenced in the weathermap-ng panel config ## Architecture change Prometheus is currently a dedicated external instance. Scope of this issue is extended to deploy a dedicated Prometheus instance **inside the Containerlab topology** (self-contained, reproducible with `clab deploy`/`clab destroy`), alongside the existing gnmic node (see #43/#44). The existing dedicated Prometheus instance stays untouched in parallel for now — no cutover until the containerized instance is validated. **Grafana is out of scope for this issue.** The existing Grafana instance is used as-is; no new datasource provisioning here.
Damien added the telemetry label 2026-06-11 10:35:32 +00:00
Damien commented 2026-07-09 14:28:18 +00:00
Author
Owner
Copy Link

Containerized Prometheus deployed and validated

  • Node: prometheus (prom/prometheus:v2.55.1), added to evpn-lab.clab.yml
  • Address: 172.16.0.71:9090 (mgmt network), self-contained with clab deploy/clab destroy
  • Config: configs/prometheus/prometheus.yml — single scrape job gnmic targeting 172.16.0.70:9273, scrape_interval: 5s
  • Existing external Prometheus instance untouched, running in parallel

Validation (live against the deployed lab)

Target health: up (/api/v1/targets).

BGP session-state and system-memory metrics validated in #44 are queryable with clean labels:

network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{
  device="dc-leaf5", neighbor_address="10.0.1.8", network_instance_name="default", ...} 1

system_memory_state_used{device="dc-access3", subscription_name="eos-system"} ...
  • core1/core2 BGP sessions present (20 series across underlay + gold VRF)
  • gold VRF (EVPN Type-5 stretch) present in network_instance_name label (18 series)

Relabeling applied (metric_relabel_configs)

Raw label Clean label
source device
interface_name interface
neighbor_neighbor_address neighbor_address
vlan_to_vni_vlan / entry_vlan vlan
entry_mac_address mac_address
afi_safi_afi_safi_name afi_safi

Gap: no native vni label

vni is not exposed as a label by any gnmic metric. It only appears as the sample value of interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni, keyed by (device, interface, vlan), e.g.:

interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni{
  device="dc-leaf5", interface="Vxlan1", vlan="40"} 110040

Relabeling can only rename existing scrape-time labels, not promote a sample value to a label. Per-VNI MAC counts (join of vlan_to_vni_state_vni with FDB entry counts) need a PromQL join on vlan, not a native vni label — flagging this as a known gap vs. the original ask, not something fixable at the relabeling stage. Left for a future recording-rule pass if needed.

Grafana/dashboard work out of scope here per issue description.

## Containerized Prometheus deployed and validated - Node: `prometheus` (`prom/prometheus:v2.55.1`), added to `evpn-lab.clab.yml` - Address: `172.16.0.71:9090` (mgmt network), self-contained with `clab deploy`/`clab destroy` - Config: `configs/prometheus/prometheus.yml` — single scrape job `gnmic` targeting `172.16.0.70:9273`, `scrape_interval: 5s` - Existing external Prometheus instance untouched, running in parallel ### Validation (live against the deployed lab) Target health: `up` (`/api/v1/targets`). BGP session-state and system-memory metrics validated in #44 are queryable with clean labels: ``` network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{ device="dc-leaf5", neighbor_address="10.0.1.8", network_instance_name="default", ...} 1 system_memory_state_used{device="dc-access3", subscription_name="eos-system"} ... ``` - `core1`/`core2` BGP sessions present (20 series across underlay + `gold` VRF) - `gold` VRF (EVPN Type-5 stretch) present in `network_instance_name` label (18 series) ### Relabeling applied (`metric_relabel_configs`) | Raw label | Clean label | | ---------------------------------- | -------------------- | | `source` | `device` | | `interface_name` | `interface` | | `neighbor_neighbor_address` | `neighbor_address` | | `vlan_to_vni_vlan` / `entry_vlan` | `vlan` | | `entry_mac_address` | `mac_address` | | `afi_safi_afi_safi_name` | `afi_safi` | ### Gap: no native `vni` label `vni` is **not** exposed as a label by any gnmic metric. It only appears as the sample **value** of `interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni`, keyed by `(device, interface, vlan)`, e.g.: ``` interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni{ device="dc-leaf5", interface="Vxlan1", vlan="40"} 110040 ``` Relabeling can only rename existing scrape-time labels, not promote a sample value to a label. Per-VNI MAC counts (join of `vlan_to_vni_state_vni` with FDB entry counts) need a PromQL join on `vlan`, not a native `vni` label — flagging this as a known gap vs. the original ask, not something fixable at the relabeling stage. Left for a future recording-rule pass if needed. Grafana/dashboard work out of scope here per issue description.
Damien added reference feat/telemetry 2026-07-10 13:01:35 +00:00
Damien changed reference from feat/telemetry to feat/telemetry 2026-07-10 13:01:38 +00:00
Sign in to join this conversation.
feat/telemetry
Branches Tags
Labels
Clear labels
containerlab
Containerlab topology et configs
 epic
Epic / Phase de projet
 generator
Infrahub Generator Python code
 resource-manager
Resource Manager / IPAM pools
 schema
Modifications de schéma Infrahub
 telemetry
monitoring configuration with prometheus and grafana
No Label telemetry
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Damien
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Damien/arista-evpn-vxlan-clab#45
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?