Build full telemetry dashboard around the weathermap panel #49

New Issue

2026-07-10T09:15:41Z

Damien commented

2026-07-10 09:15:41 +00:00

Extend the weathermap panel (#48) into a complete operational dashboard, with the weathermap as the top overview panel and supporting panels for detail below.

Depends on: #48 (weathermap panel, done + visually validated), #44 (metrics, validated), #45 (Prometheus datasource NetLab).

Panels

Weathermap (existing from #48) — top, full width. Node status logic to be reworked, see below.
BGP sessions table — device, neighbor address, network-instance (VRF), session state (0/1 mapped to Down/Up with color), using the metric validated in #44
Ports/interfaces table — device, interface, oper-status (Down/Up colored), current in/out rate — joined via Grafana transformations (merge), not a PromQL join
Throughput graphs — aggregated per site (campus/core/dc), not per-link (61 links is too many to plot individually) — sum of interface rate() grouped by site

Weathermap node status rework (new requirement)

Current node.statusQuery from #48 is "BGP {{device}}", one legend per device, sourced from the raw per-neighbor BGP session-state series. Bug: if a device has more than one BGP neighbor, multiple series collide on the same legend string and only one arbitrarily wins in Grafana's display-name map. Not previously caught because it wasn't visually obvious.

New requirement — node is green only if:

Device is up: proposed as min by (device) (interfaces_interface_state_oper_status) (at least one interface active) — confirm this is the right proxy, device-level reachability has no direct metric
AND all BGP sessions OK, if applicable: min by (device) (bgp_session_state) — must reflect the worst session, not just any one. Devices with zero BGP sessions (e.g. access switches) must NOT be penalized by missing data — needs an explicit or fallback so "no BGP configured" reads as neutral/OK, not down.

Combine both into a single per-device status series feeding node.statusQuery, replacing the current one-neighbor-wins query. Validate live via /api/ds/query (same method as #48) against at least one device with multiple BGP neighbors and one device with none, to confirm both edge cases resolve correctly.

Template variables

site — campus/core/dc, derived from device naming convention or IPFabric site grouping
device — filtered by selected site
Both apply globally across all panels (table row filtering + throughput graph scope)

Notes

Site is not an existing Prometheus label today — confirm whether it needs to be added via gnmic/Prometheus relabeling (derivable from hostname prefix: campus-*/core*/dc-*) or computed at query time
Reuse the NetLab datasource (uid cfrlusac89se8a) already validated in #48
Dashboard-as-code in Gitea, same pattern as the weathermap panel JSON

Extend the weathermap panel (#48) into a complete operational dashboard, with the weathermap as the top overview panel and supporting panels for detail below. Depends on: #48 (weathermap panel, done + visually validated), #44 (metrics, validated), #45 (Prometheus datasource `NetLab`). ## Panels 1. **Weathermap** (existing from #48) — top, full width. **Node status logic to be reworked, see below.** 2. **BGP sessions table** — device, neighbor address, network-instance (VRF), session state (0/1 mapped to Down/Up with color), using the metric validated in #44 3. **Ports/interfaces table** — device, interface, oper-status (Down/Up colored), current in/out rate — joined via Grafana transformations (merge), not a PromQL join 4. **Throughput graphs — aggregated per site** (campus/core/dc), not per-link (61 links is too many to plot individually) — sum of interface rate() grouped by site ## Weathermap node status rework (new requirement) Current `node.statusQuery` from #48 is `"BGP {{device}}"`, one legend per device, sourced from the raw per-neighbor BGP session-state series. **Bug**: if a device has more than one BGP neighbor, multiple series collide on the same legend string and only one arbitrarily wins in Grafana's display-name map. Not previously caught because it wasn't visually obvious. New requirement — node is green only if: - **Device is up**: proposed as `min by (device) (interfaces_interface_state_oper_status)` (at least one interface active) — confirm this is the right proxy, device-level reachability has no direct metric - **AND all BGP sessions OK, if applicable**: `min by (device) (bgp_session_state)` — must reflect the worst session, not just any one. Devices with zero BGP sessions (e.g. access switches) must NOT be penalized by missing data — needs an explicit `or` fallback so "no BGP configured" reads as neutral/OK, not down. Combine both into a single per-device status series feeding `node.statusQuery`, replacing the current one-neighbor-wins query. Validate live via `/api/ds/query` (same method as #48) against at least one device with multiple BGP neighbors and one device with none, to confirm both edge cases resolve correctly. ## Template variables - `site` — campus/core/dc, derived from device naming convention or IPFabric site grouping - `device` — filtered by selected site - Both apply globally across all panels (table row filtering + throughput graph scope) ## Notes - Site is not an existing Prometheus label today — confirm whether it needs to be added via gnmic/Prometheus relabeling (derivable from hostname prefix: `campus-*`/`core*`/`dc-*`) or computed at query time - Reuse the `NetLab` datasource (uid `cfrlusac89se8a`) already validated in #48 - Dashboard-as-code in Gitea, same pattern as the weathermap panel JSON

Damien added the telemetry label 2026-07-10 09:15:41 +00:00

Damien commented

2026-07-10 09:46:35 +00:00

Implemented in b03464d (branch feat/telemetry). Dashboard regenerated via scripts/generate_weathermap.py (extended, not hand-edited) → configs/grafana/weathermap-dashboard.json, re-provisioned to Grafana (evpn-vxlan-fabric-weathermap, now dashboard version 4).

Part 1 — node status fix

Final PromQL (per device, refId: A, legendFormat: "STATUS {{device}}"):

min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"})
*
(
  min by (device) (network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{network_instance_name="default", device=~"<host_regex>"})
  or
  (min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"}) * 0 + 1)
)

node.statusQuery on all 28 nodes now points at "STATUS {{device}}".

Edge-case validation via /api/ds/query against the live in-topology Prometheus:

Multi-neighbor device (dc-spine1, 20 BGP neighbors): confirmed 20 raw series previously collided under the single legend "BGP dc-spine1" (the bug). New combined query returns exactly one series STATUS dc-spine1 = 1, correctly reflecting min() over all 20 sessions (worst wins, not first-match).
Zero-BGP device (dc-access1, access switch, 0 BGP sessions): raw BGP query returns empty for this device; new combined query still returns STATUS dc-access1 = 1 via the or fallback — not penalized as down.
All 28 devices resolved to a value (spot-checked full set, all healthy = 1 in current lab state, as expected).

Part 2 — site label

Resolved via Prometheus metric_relabel_configs (not label_replace() per query) — configs/prometheus/prometheus.yml:

- source_labels: [device]
  target_label: site
  regex: (campus|core|dc)-?.*
  replacement: $1
  action: replace

Chosen over per-query label_replace() since site is needed by every new panel (BGP table filter, ports table filter, throughput aggregation) — one relabel rule beats repeating the regex 5+ times. Applied by restarting the Prometheus container (no --web.enable-lifecycle, so /-/reload returned 403). Validated: count by (site) (interfaces_interface_state_oper_status) → campus=66, core=12, dc=120.

Surprise: right after restart, /api/v1/query briefly returned both the old (no-site) and new (site-tagged) series for the same device/interface, both with near-current timestamps — looked like a live duplicate-write bug. It was actually the pre-restart series sitting in Prometheus's 5-minute staleness/lookback window (frozen at its last real sample) alongside the new series. Resolved itself once >5 min had passed since restart. Worth remembering if this comes up again after any future relabel change.

Template variables

site: query variable, label_values(interfaces_interface_state_oper_status, site), multi + include-all
device: query variable chained on site, label_values(interfaces_interface_state_oper_status{site=~"$site"}, device), multi + include-all

New panels

BGP Sessions (table, instant): network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{device=~"$device"}, columns device/neighbor_address/network_instance_name(VRF)/state, 0/1 → Down(red)/Up(green) value mapping. Validated live: 180 series resolve with all needed labels present.
Ports/Interfaces (table): 3 instant queries (oper-status, in-rate, out-rate on interfaces_interface_state_counters_{in,out}_octets) combined with Grafana's merge transformation (matches on shared device/interface/site fields) — deliberately not a PromQL join, per the VXLAN join failure already logged in #44/#48. Validated all 3 queries independently resolve (198 series each) with matching label sets.
Throughput per Site (time series): sum by (site) (rate(interfaces_interface_state_counters_{in,out}_octets{site=~"$site"}[5m]) * 8), signed graph (out negated) rather than two separate panels — clearer at a glance which direction dominates per site. Validated live: campus/core/dc all return non-zero bps.

Naming surprises

None beyond the staleness-window one above — metric/label names matched #44/#45 exactly as documented, no new aliasing needed.

Outstanding

Manual visual check in the Grafana UI (node colors, table rendering, variable filtering) still needs a human pass — this Grafana instance has no image-renderer plugin installed and I have no browser tool available, so I could not screenshot it myself. Dashboard: $GRAFANA_URL/d/evpn-vxlan-fabric-weathermap.

Implemented in `b03464d` (branch `feat/telemetry`). Dashboard regenerated via `scripts/generate_weathermap.py` (extended, not hand-edited) → `configs/grafana/weathermap-dashboard.json`, re-provisioned to Grafana (`evpn-vxlan-fabric-weathermap`, now dashboard version 4). ## Part 1 — node status fix Final PromQL (per device, `refId: A`, `legendFormat: "STATUS {{device}}"`): ``` min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"}) * ( min by (device) (network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{network_instance_name="default", device=~"<host_regex>"}) or (min by (device) (interfaces_interface_state_oper_status{device=~"<host_regex>"}) * 0 + 1) ) ``` `node.statusQuery` on all 28 nodes now points at `"STATUS {{device}}"`. Edge-case validation via `/api/ds/query` against the live in-topology Prometheus: - **Multi-neighbor device** (`dc-spine1`, 20 BGP neighbors): confirmed 20 raw series previously collided under the single legend `"BGP dc-spine1"` (the bug). New combined query returns exactly one series `STATUS dc-spine1 = 1`, correctly reflecting `min()` over all 20 sessions (worst wins, not first-match). - **Zero-BGP device** (`dc-access1`, access switch, 0 BGP sessions): raw BGP query returns empty for this device; new combined query still returns `STATUS dc-access1 = 1` via the `or` fallback — not penalized as down. - All 28 devices resolved to a value (spot-checked full set, all healthy = `1` in current lab state, as expected). ## Part 2 — site label Resolved via Prometheus `metric_relabel_configs` (not `label_replace()` per query) — `configs/prometheus/prometheus.yml`: ```yaml - source_labels: [device] target_label: site regex: (campus|core|dc)-?.* replacement: $1 action: replace ``` Chosen over per-query `label_replace()` since `site` is needed by every new panel (BGP table filter, ports table filter, throughput aggregation) — one relabel rule beats repeating the regex 5+ times. Applied by restarting the Prometheus container (no `--web.enable-lifecycle`, so `/-/reload` returned 403). Validated: `count by (site) (interfaces_interface_state_oper_status)` → `campus=66, core=12, dc=120`. **Surprise**: right after restart, `/api/v1/query` briefly returned *both* the old (no-`site`) and new (`site`-tagged) series for the same device/interface, both with near-current timestamps — looked like a live duplicate-write bug. It was actually the pre-restart series sitting in Prometheus's 5-minute staleness/lookback window (frozen at its last real sample) alongside the new series. Resolved itself once >5 min had passed since restart. Worth remembering if this comes up again after any future relabel change. ## Template variables - `site`: query variable, `label_values(interfaces_interface_state_oper_status, site)`, multi + include-all - `device`: query variable chained on `site`, `label_values(interfaces_interface_state_oper_status{site=~"$site"}, device)`, multi + include-all ## New panels - **BGP Sessions** (table, instant): `network_instances_network_instance_protocols_protocol_bgp_neighbors_neighbor_state_session_state{device=~"$device"}`, columns device/neighbor_address/network_instance_name(VRF)/state, 0/1 → Down(red)/Up(green) value mapping. Validated live: 180 series resolve with all needed labels present. - **Ports/Interfaces** (table): 3 instant queries (oper-status, in-rate, out-rate on `interfaces_interface_state_counters_{in,out}_octets`) combined with Grafana's **merge** transformation (matches on shared `device`/`interface`/`site` fields) — deliberately not a PromQL join, per the VXLAN join failure already logged in #44/#48. Validated all 3 queries independently resolve (198 series each) with matching label sets. - **Throughput per Site** (time series): `sum by (site) (rate(interfaces_interface_state_counters_{in,out}_octets{site=~"$site"}[5m]) * 8)`, signed graph (out negated) rather than two separate panels — clearer at a glance which direction dominates per site. Validated live: campus/core/dc all return non-zero bps. ## Naming surprises None beyond the staleness-window one above — metric/label names matched #44/#45 exactly as documented, no new aliasing needed. ## Outstanding Manual visual check in the Grafana UI (node colors, table rendering, variable filtering) still needs a human pass — this Grafana instance has no image-renderer plugin installed and I have no browser tool available, so I could not screenshot it myself. Dashboard: `$GRAFANA_URL/d/evpn-vxlan-fabric-weathermap`.

Damien referenced this issue

2026-07-10 09:48:44 +00:00

VNI tooltip PromQL join breaks on transient duplicate series after Prometheus restart #50

Damien referenced this issue

2026-07-10 09:55:12 +00:00

Throughput-per-site panel includes Management0 telemetry traffic, not just fabric traffic #51

Damien referenced this issue

2026-07-10 10:05:30 +00:00

Split weathermap generation from dashboard composition (manual + generated merge) #52

Damien referenced this issue

2026-07-10 10:16:12 +00:00

Split weathermap generation from dashboard composition (manual + generated merge) #52