Damien/arista-evpn-vxlan-clab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build full telemetry dashboard around the weathermap panel #49

New Issue
Closed
opened 2026-07-10 09:15:41 +00:00 by Damien · 1 comment
Damien commented 2026-07-10 09:15:41 +00:00
Owner
Copy Link

Extend the weathermap panel (#48) into a complete operational dashboard, with the weathermap as the top overview panel and supporting panels for detail below.

Depends on: #48 (weathermap panel, done + visually validated), #44 (metrics, validated), #45 (Prometheus datasource NetLab).

Panels

  1. Weathermap (existing from #48) — top, full width. Node status logic to be reworked, see below.
  2. BGP sessions table — device, neighbor address, network-instance (VRF), session state (0/1 mapped to Down/Up with color), using the metric validated in #44
  3. Ports/interfaces table — device, interface, oper-status (Down/Up colored), current in/out rate — joined via Grafana transformations (merge), not a PromQL join
  4. Throughput graphs — aggregated per site (campus/core/dc), not per-link (61 links is too many to plot individually) — sum of interface rate() grouped by site

Weathermap node status rework (new requirement)

Current node.statusQuery from #48 is "BGP {{device}}", one legend per device, sourced from the raw per-neighbor BGP session-state series. Bug: if a device has more than one BGP neighbor, multiple series collide on the same legend string and only one arbitrarily wins in Grafana's display-name map. Not previously caught because it wasn't visually obvious.

New requirement — node is green only if:

  • Device is up: proposed as min by (device) (interfaces_interface_state_oper_status) (at least one interface active) — confirm this is the right proxy, device-level reachability has no direct metric
  • AND all BGP sessions OK, if applicable: min by (device) (bgp_session_state) — must reflect the worst session, not just any one. Devices with zero BGP sessions (e.g. access switches) must NOT be penalized by missing data — needs an explicit or fallback so "no BGP configured" reads as neutral/OK, not down.

Combine both into a single per-device status series feeding node.statusQuery, replacing the current one-neighbor-wins query. Validate live via /api/ds/query (same method as #48) against at least one device with multiple BGP neighbors and one device with none, to confirm both edge cases resolve correctly.

Template variables

  • site — campus/core/dc, derived from device naming convention or IPFabric site grouping
  • device — filtered by selected site
  • Both apply globally across all panels (table row filtering + throughput graph scope)

Notes

  • Site is not an existing Prometheus label today — confirm whether it needs to be added via gnmic/Prometheus relabeling (derivable from hostname prefix: campus-*/core*/dc-*) or computed at query time
  • Reuse the NetLab datasource (uid cfrlusac89se8a) already validated in #48
  • Dashboard-as-code in Gitea, same pattern as the weathermap panel JSON
Extend the weathermap panel (#48) into a complete operational dashboard, with the weathermap as the top overview panel and supporting panels for detail below. Depends on: #48 (weathermap panel, done + visually validated), #44 (metrics, validated), #45 (Prometheus datasource `NetLab`). ## Panels 1. **Weathermap** (existing from #48) — top, full width. **Node status logic to be reworked, see below.** 2. **BGP sessions table** — device, neighbor address, network-instance (VRF), session state (0/1 mapped to Down/Up with color), using the metric validated in #44 3. **Ports/interfaces table** — device, interface, oper-status (Down/Up colored), current in/out rate — joined via Grafana transformations (merge), not a PromQL join 4. **Throughput graphs — aggregated per site** (campus/core/dc), not per-link (61 links is too many to plot individually) — sum of interface rate() grouped by site ## Weathermap node status rework (new requirement) Current `node.statusQuery` from #48 is `"BGP {{device}}"`, one legend per device, sourced from the raw per-neighbor BGP session-state series. **Bug**: if a device has more than one BGP neighbor, multiple series collide on the same legend string and only one arbitrarily wins in Grafana's display-name map. Not previously caught because it wasn't visually obvious. New requirement — node is green only if: - **Device is up**: proposed as `min by (device) (interfaces_interface_state_oper_status)` (at least one interface active) — confirm this is the right proxy, device-level reachability has no direct metric - **AND all BGP sessions OK, if applicable**: `min by (device) (bgp_session_state)` — must reflect the worst session, not just any one. Devices with zero BGP sessions (e.g. access switches) must NOT be penalized by missing data — needs an explicit `or` fallback so "no BGP configured" reads as neutral/OK, not down. Combine both into a single per-device status series feeding `node.statusQuery`, replacing the current one-neighbor-wins query. Validate live via `/api/ds/query` (same method as #48) against at least one device with multiple BGP neighbors and one device with none, to confirm both edge cases resolve correctly. ## Template variables - `site` — campus/core/dc, derived from device naming convention or IPFabric site grouping - `device` — filtered by selected site - Both apply globally across all panels (table row filtering + throughput graph scope) ## Notes - Site is not an existing Prometheus label today — confirm whether it needs to be added via gnmic/Prometheus relabeling (derivable from hostname prefix: `campus-*`/`core*`/`dc-*`) or computed at query time - Reuse the `NetLab` datasource (uid `cfrlusac89se8a`) already validated in #48 - Dashboard-as-code in Gitea, same pattern as the weathermap panel JSON
Damien added the telemetry label 2026-07-10 09:15:41 +00:00
Damien commented 2026-07-10 09:46:35 +00:00
Author
Owner
Copy Link

Implemented in b03464d (branch feat/telemetry). Dashboard regenerated via scripts/generate_weathermap.py (extended, not hand-edited) → configs/grafana/weathermap-dashboard.json, re-provisioned to Grafana (evpn-vxlan-fabric-weathermap, now dashboard version 4).

Part 1 — node status fix

Final PromQL (per device, refId: A, legendFormat: "STATUS {{device}}"):

min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"})
*
(
  min by (device) (network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{network_instance_name="default", device=~"<host_regex>"})
  or
  (min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"}) * 0 + 1)
)

node.statusQuery on all 28 nodes now points at "STATUS {{device}}".

Edge-case validation via /api/ds/query against the live in-topology Prometheus:

  • Multi-neighbor device (dc-spine1, 20 BGP neighbors): confirmed 20 raw series previously collided under the single legend "BGP dc-spine1" (the bug). New combined query returns exactly one series STATUS dc-spine1 = 1, correctly reflecting min() over all 20 sessions (worst wins, not first-match).
  • Zero-BGP device (dc-access1, access switch, 0 BGP sessions): raw BGP query returns empty for this device; new combined query still returns STATUS dc-access1 = 1 via the or fallback — not penalized as down.
  • All 28 devices resolved to a value (spot-checked full set, all healthy = 1 in current lab state, as expected).

Part 2 — site label

Resolved via Prometheus metric_relabel_configs (not label_replace() per query) — configs/prometheus/prometheus.yml:

- source_labels: [device]
  target_label: site
  regex: (campus|core|dc)-?.*
  replacement: $1
  action: replace

Chosen over per-query label_replace() since site is needed by every new panel (BGP table filter, ports table filter, throughput aggregation) — one relabel rule beats repeating the regex 5+ times. Applied by restarting the Prometheus container (no --web.enable-lifecycle, so /-/reload returned 403). Validated: count by (site) (interfaces_interface_state_oper_status) → campus=66, core=12, dc=120.

Surprise: right after restart, /api/v1/query briefly returned both the old (no-site) and new (site-tagged) series for the same device/interface, both with near-current timestamps — looked like a live duplicate-write bug. It was actually the pre-restart series sitting in Prometheus's 5-minute staleness/lookback window (frozen at its last real sample) alongside the new series. Resolved itself once >5 min had passed since restart. Worth remembering if this comes up again after any future relabel change.

Template variables

  • site: query variable, label_values(interfaces_interface_state_oper_status, site), multi + include-all
  • device: query variable chained on site, label_values(interfaces_interface_state_oper_status{site=~"$site"}, device), multi + include-all

New panels

  • BGP Sessions (table, instant): network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{device=~"$device"}, columns device/neighbor_address/network_instance_name(VRF)/state, 0/1 → Down(red)/Up(green) value mapping. Validated live: 180 series resolve with all needed labels present.
  • Ports/Interfaces (table): 3 instant queries (oper-status, in-rate, out-rate on interfaces_interface_state_counters_{in,out}_octets) combined with Grafana's merge transformation (matches on shared device/interface/site fields) — deliberately not a PromQL join, per the VXLAN join failure already logged in #44/#48. Validated all 3 queries independently resolve (198 series each) with matching label sets.
  • Throughput per Site (time series): sum by (site) (rate(interfaces_interface_state_counters_{in,out}_octets{site=~"$site"}[5m]) * 8), signed graph (out negated) rather than two separate panels — clearer at a glance which direction dominates per site. Validated live: campus/core/dc all return non-zero bps.

Naming surprises

None beyond the staleness-window one above — metric/label names matched #44/#45 exactly as documented, no new aliasing needed.

Outstanding

Manual visual check in the Grafana UI (node colors, table rendering, variable filtering) still needs a human pass — this Grafana instance has no image-renderer plugin installed and I have no browser tool available, so I could not screenshot it myself. Dashboard: $GRAFANA_URL/d/evpn-vxlan-fabric-weathermap.

Implemented in `b03464d` (branch `feat/telemetry`). Dashboard regenerated via `scripts/generate_weathermap.py` (extended, not hand-edited) → `configs/grafana/weathermap-dashboard.json`, re-provisioned to Grafana (`evpn-vxlan-fabric-weathermap`, now dashboard version 4). ## Part 1 — node status fix Final PromQL (per device, `refId: A`, `legendFormat: "STATUS {{device}}"`): ``` min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"}) * ( min by (device) (network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{network_instance_name="default", device=~"<host_regex>"}) or (min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"}) * 0 + 1) ) ``` `node.statusQuery` on all 28 nodes now points at `"STATUS {{device}}"`. Edge-case validation via `/api/ds/query` against the live in-topology Prometheus: - **Multi-neighbor device** (`dc-spine1`, 20 BGP neighbors): confirmed 20 raw series previously collided under the single legend `"BGP dc-spine1"` (the bug). New combined query returns exactly one series `STATUS dc-spine1 = 1`, correctly reflecting `min()` over all 20 sessions (worst wins, not first-match). - **Zero-BGP device** (`dc-access1`, access switch, 0 BGP sessions): raw BGP query returns empty for this device; new combined query still returns `STATUS dc-access1 = 1` via the `or` fallback — not penalized as down. - All 28 devices resolved to a value (spot-checked full set, all healthy = `1` in current lab state, as expected). ## Part 2 — site label Resolved via Prometheus `metric_relabel_configs` (not `label_replace()` per query) — `configs/prometheus/prometheus.yml`: ```yaml - source_labels: [device] target_label: site regex: (campus|core|dc)-?.* replacement: $1 action: replace ``` Chosen over per-query `label_replace()` since `site` is needed by every new panel (BGP table filter, ports table filter, throughput aggregation) — one relabel rule beats repeating the regex 5+ times. Applied by restarting the Prometheus container (no `--web.enable-lifecycle`, so `/-/reload` returned 403). Validated: `count by (site) (interfaces_interface_state_oper_status)` → `campus=66, core=12, dc=120`. **Surprise**: right after restart, `/api/v1/query` briefly returned *both* the old (no-`site`) and new (`site`-tagged) series for the same device/interface, both with near-current timestamps — looked like a live duplicate-write bug. It was actually the pre-restart series sitting in Prometheus's 5-minute staleness/lookback window (frozen at its last real sample) alongside the new series. Resolved itself once >5 min had passed since restart. Worth remembering if this comes up again after any future relabel change. ## Template variables - `site`: query variable, `label_values(interfaces_interface_state_oper_status, site)`, multi + include-all - `device`: query variable chained on `site`, `label_values(interfaces_interface_state_oper_status{site=~"$site"}, device)`, multi + include-all ## New panels - **BGP Sessions** (table, instant): `network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{device=~"$device"}`, columns device/neighbor_address/network_instance_name(VRF)/state, 0/1 → Down(red)/Up(green) value mapping. Validated live: 180 series resolve with all needed labels present. - **Ports/Interfaces** (table): 3 instant queries (oper-status, in-rate, out-rate on `interfaces_interface_state_counters_{in,out}_octets`) combined with Grafana's **merge** transformation (matches on shared `device`/`interface`/`site` fields) — deliberately not a PromQL join, per the VXLAN join failure already logged in #44/#48. Validated all 3 queries independently resolve (198 series each) with matching label sets. - **Throughput per Site** (time series): `sum by (site) (rate(interfaces_interface_state_counters_{in,out}_octets{site=~"$site"}[5m]) * 8)`, signed graph (out negated) rather than two separate panels — clearer at a glance which direction dominates per site. Validated live: campus/core/dc all return non-zero bps. ## Naming surprises None beyond the staleness-window one above — metric/label names matched #44/#45 exactly as documented, no new aliasing needed. ## Outstanding Manual visual check in the Grafana UI (node colors, table rendering, variable filtering) still needs a human pass — this Grafana instance has no image-renderer plugin installed and I have no browser tool available, so I could not screenshot it myself. Dashboard: `$GRAFANA_URL/d/evpn-vxlan-fabric-weathermap`.
Damien added reference feat/telemetry 2026-07-10 13:01:22 +00:00
Sign in to join this conversation.
feat/telemetry
Branches Tags
Labels
Clear labels
containerlab
Containerlab topology et configs
 epic
Epic / Phase de projet
 generator
Infrahub Generator Python code
 resource-manager
Resource Manager / IPAM pools
 schema
Modifications de schéma Infrahub
 telemetry
monitoring configuration with prometheus and grafana
No Label telemetry
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Damien
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Damien/arista-evpn-vxlan-clab#49
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?