Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator #59

Damien · 2026-07-20T16:58:24Z

Damien commented

2026-07-20 16:58:24 +00:00

Summary

gnmic gNMI collection integrated into the Containerlab topology, scraped by a containerized Prometheus instance
scripts/generate_weathermap.py: generates the Grafana weathermap panel from live IPFabric topology + Prometheus metrics, merges it into a manually-authored dashboard base (configs/grafana/dashboard-base.json), and provisions it
Layered default node layout by hostname role (spine/core/border-leaf/leaf/access), live-Grafana position preserved across regeneration, consistent per-role node sizing
Full operational dashboard around the weathermap: BGP sessions table, ports/interfaces table, per-site throughput graphs, site/device template variables
scripts/generate_traffic.sh: iperf3 traffic generator exercising the DC↔Campus gold-VRF stitched EVPN Type-5 path, with a live terminal bandwidth dashboard
Fixed during final review pass:
- weathermap links only queried one direction (A tx / Z rx describing the same flow) — now each side queries its own tx, surfacing both directions (#57)
- iperf3 traffic ran campus→DC instead of DC→campus (service should push to consumer) — added -R (#58)
- scripts/README.md doc drift from both fixes above

Issues closed

Closes #43, Closes #44, Closes #45, Closes #47, Closes #49, Closes #52, Closes #53, Closes #54, Closes #57, Closes #58

Known gaps (tracked, not blocking)

VXLAN MAC-per-VNI join broken on live data (internal-vs-front-panel VLAN mismatch) — dropped from weathermap tooltip, needs its own table panel (noted in #44)
OSPF metrics for core1/core2 deferred

Test plan

python3 -m py_compile scripts/generate_weathermap.py
Weathermap direction fix validated live against iperf3 traffic (see commit 08124c8)
iperf3 reverse-direction fix validated (see commit 4719a77)
Reviewer: visual check of the merged dashboard in Grafana after merge

## Summary - gnmic gNMI collection integrated into the Containerlab topology, scraped by a containerized Prometheus instance - `scripts/generate_weathermap.py`: generates the Grafana weathermap panel from live IPFabric topology + Prometheus metrics, merges it into a manually-authored dashboard base (`configs/grafana/dashboard-base.json`), and provisions it - Layered default node layout by hostname role (spine/core/border-leaf/leaf/access), live-Grafana position preserved across regeneration, consistent per-role node sizing - Full operational dashboard around the weathermap: BGP sessions table, ports/interfaces table, per-site throughput graphs, site/device template variables - `scripts/generate_traffic.sh`: iperf3 traffic generator exercising the DC↔Campus gold-VRF stitched EVPN Type-5 path, with a live terminal bandwidth dashboard - Fixed during final review pass: - weathermap links only queried one direction (A tx / Z rx describing the same flow) — now each side queries its own tx, surfacing both directions (#57) - iperf3 traffic ran campus→DC instead of DC→campus (service should push to consumer) — added `-R` (#58) - `scripts/README.md` doc drift from both fixes above ## Issues closed Closes #43, Closes #44, Closes #45, Closes #47, Closes #49, Closes #52, Closes #53, Closes #54, Closes #57, Closes #58 ## Known gaps (tracked, not blocking) - VXLAN MAC-per-VNI join broken on live data (internal-vs-front-panel VLAN mismatch) — dropped from weathermap tooltip, needs its own table panel (noted in #44) - OSPF metrics for core1/core2 deferred ## Test plan - [x] `python3 -m py_compile scripts/generate_weathermap.py` - [x] Weathermap direction fix validated live against iperf3 traffic (see commit 08124c8) - [x] iperf3 reverse-direction fix validated (see commit 4719a77) - [x] Reviewer: visual check of the merged dashboard in Grafana after merge

Damien added 30 commits 2026-07-20 16:58:24 +00:00

Add HomeLab v1 infrastructure diagram 3104d7ea63

Remove HomeLab v1 diagram 553c713d17

Remove old Core.svg diagram d3b59a02e9

Remove outdated core reworked diagram d12e057bc8

Integrate gnmic for gNMI telemetry collection 71cc8de7d5

Add gnmic container node subscribing to interface, BGP, system, and
VXLAN gNMI paths across all Arista cEOS nodes, exposed via a
Prometheus exporter. gNMI runs in plaintext (no SSL profile on
management api gnmi), so insecure:true is used instead of TLS.

Refs #43, Refs #44

Add CLAUDE.md with project conventions and guidelines b6a49bfeac

Deploy containerized Prometheus for gnmic telemetry 1f8a3ef6d6

Add prometheus node to the Containerlab topology, scraping the gnmic
exporter (172.16.0.70:9273) at 5s interval with relabeling to clean
device/interface/neighbor_address/vlan/mac_address/afi_safi labels.
Validated against BGP and system metrics confirmed in #44. Existing
external Prometheus instance stays untouched in parallel.

Refs #45

Generate weathermap-ng panel config from IPFabric topology f711210379

Adds scripts/generate_weathermap.py: fetches device inventory and
connectivity-matrix from IPFabric, cross-checks IPFabric's abbreviated
interface names against the live gnmic exporter, and emits a Grafana
dashboard-as-code JSON (configs/grafana/weathermap-dashboard.json) with
the weathermap-ng panel wired to explicit legendFormat-resolved queries
for BGP status, interface tx/rx, and VXLAN MAC-per-VNI.

Refs #48

Fix PromQL regex escaping for hyphenated hostnames 0aff25f509

re.escape() escapes '-' as '\-', which Python's regex engine accepts
but PromQL's RE2 engine rejects outright (every hostname in this lab
is hyphenated, e.g. dc-leaf1). All 4 panel targets were failing with
"unknown escape sequence U+002D" until now. Also swaps the dashboard
to the newly-created NetLab datasource, which actually has gnmic data
(the previously-wired external Prometheus datasource had none).

Refs #48

Fix missing/wrong anchor fields crashing the weathermap panel 36c93d9ecd

tamirsuliman-weathermap-panel (the plugin actually installed, not the
allamiro fork the schema research was done against) requires node.anchors
-- a numLinks/numFilledLinks tally per anchor position -- and expects
link.sides.*.anchor as the plugin's numeric TS enum value (Center=0,
Top=1, Bottom=2, Left=3, Right=4), not an anchor name string. Omitting
anchors entirely (per the "safe to omit, auto-defaulted" schema-research
note, which turns out to be fork-specific) crashed the panel on load with
"TypeError: undefined is not an object (evaluating 'i.anchors[0]')".

Enum values reverse-engineered from the installed plugin's module.js.

Refs #48

Move weathermap generator docs to scripts/README.md 3c680a2b17

Root README now just links to it. Also updates the default
GRAFANA_WEATHERMAP_PLUGIN_ID to the fork actually installed
(tamirsuliman-weathermap-panel), confirmed while validating #48.

Refs #48

Extend weathermap dashboard with BGP/ports tables, per-site throughput, fix node status query b03464d0df

Fix node.statusQuery colliding legends: multi-neighbor devices previously
sourced from raw per-neighbor BGP session-state series that collide under
one legend, so an arbitrary neighbor won. Replace with a combined
device-up * worst-BGP-session query per device, with an explicit fallback
so zero-BGP devices (access switches) read as neutral/OK.

Add site label via Prometheus relabel (derived from device naming
convention) instead of per-query label_replace(). Add site/device template
variables, BGP sessions table, ports/interfaces table (Grafana merge
transform, not PromQL join), and per-site throughput graphs.

Refs #49

Guard VNI tooltip join against transient duplicate series b726848da0

Wrap the group_left RHS in max by (device, vlan) so a Prometheus restart
or relabel change (old/new label-set series briefly coexisting in the
staleness window) can't trip PromQL's many-to-many matching error.

Closes #50

Exclude Management0 from per-site throughput panel dba06c1607

gNMI telemetry traffic on the OOB management port swamped fabric traffic
in the site-aggregate sums, producing a misleading ~10x in/out asymmetry
that read as a real traffic anomaly.

Closes #51

Split weathermap generation from dashboard composition 632d146368

generate_weathermap.py now emits only the weathermap panel (targets +
options.weathermap) and merges it into a manually-authored dashboard base
(configs/grafana/dashboard-base.json) at a reserved slot panel, keeping
the base's gridPos so layout stays manual. The BGP sessions table,
ports/interfaces table, throughput graphs, and template variables move
out of the script entirely into that base file.

Verified zero regression: merged output is identical to the prior
generator-only output except for JSON key ordering, and all panel
queries re-validated via /api/ds/query.

Refs #52

Add sample environment file with Grafana and IPFabric credentials e73b789f92

Layered default layout by hostname role + preserve live node positions fcd7ea2567

Default node position for spine/core/border-leaf/leaf/access is now a
Y-tier by role (parsed from hostname, same trust level as the site
parsing from #49) with X grouped by site within each tier, replacing
the flat site-grid heuristic.

Position specifically is now read from the live Grafana dashboard before
computing anything, so a node manually drag-and-dropped in the UI keeps
its position across regeneration -- only nodes with no live position yet
(new devices, or first-ever run) get the layered default. Devices removed
from the topology are simply absent from this run's output, so no
orphaned position/link reference is possible.

Refs #53

Fix inconsistent node rectangle height via compactVerticalLinks 9b53ab0683

Confirmed root cause against the installed plugin's real module.js (not
the #48 schema doc, which doesn't cover this): node height is driven by
max(anchors[Left].numLinks, anchors[Right].numLinks) whenever
compactVerticalLinks is false, which is why campus-leaf1 (3 links) and
campus-leaf2 (4 links) rendered at different heights despite the same
role. Width is unaffected by link count -- it's always recomputed from
the label text at render time, no override field exists.

Set compactVerticalLinks: true on every generated node, which makes
height a constant (fontSize + 2*padding.vertical) regardless of link
count, for every node (padding/fontSize are already uniform fabric-wide).

Refs #54

Add end-to-end how-to guide + regenerate dashboard from a clean reset 8fb83b9446

Full reset validation: deleted weathermap-dashboard.json and the live
Grafana dashboard (dashboard-base.json untouched, it's the manual part).
Regenerated from scratch and reprovisioned -- confirms the first-ever-run
path (0 live positions found, full layered layout applied, all 4 panels'
queries resolve).

Documented the whole flow in scripts/README.md as a start-to-finish
how-to: credential acquisition (UI steps for the fiddly API paths --
Grafana service account token, datasource UID, plugin id), dry run,
provision, verification (UI walkthrough + API/Explore alternative),
re-running after topology changes, and a troubleshooting table. Added
GRAFANA_DATASOURCE_UID and the optional env vars to envrc.sample.

Update Grafana weathermap dashboard node positions 998014d144

Reword scripts/README.md for readability, add pipeline diagram 045179e098

Same content, restructured for scanning: tables instead of long
paragraphs (credentials, troubleshooting, env vars), shorter sentences,
deep rationale trimmed to a phrase + issue reference instead of inline
essays. Cut from 258 to 135 lines.

Added assets/weathermap-pipeline.svg showing the IPFabric/Prometheus ->
generator -> merge (with the manual base) -> Grafana flow, referenced at
the top of the README.

Update scripts README table formatting 64a8287518

Fix read-back arrow to terminate on dashboard-base.json's edge, not its middle 1cbea38522

Add Grafana weathermap dashboard asset 0f3aa27566

Add iperf3 traffic generator for DC<->Campus gold VRF path 5b12894289

Host containers sit idle otherwise, leaving IPFabric ARP/MAC tables and
Grafana throughput graphs empty until someone manually generates traffic.

Refs #55

Merge branch 'feat/telemetry' into feat/iperf-traffic-generator 3c3b28c987

Merge pull request 'Add iperf3 traffic generator for DC<->Campus gold VRF path' (#56 ) from feat/iperf-traffic-generator into feat/telemetry 8349791630

Reviewed-on: #56

Fix weathermap links to show both traffic directions 08124c8a1c

Z side queried z_host's rx, which measures the same A->Z flow as A
side's tx (same traffic, counted at each end) -- the Z->A direction
was never queried by either side. Use z_host's own tx instead, so
each side surfaces its own (opposite) direction.

Confirmed against the iperf3 traffic generator: DC access-leaf links
read ~500Kbps (ACK-only) before the fix despite ~18-20Mbps of real
downstream traffic to dc-server2/4, because that flow ran leaf->access
(Z->A), the unqueried direction. Campus links looked correct only by
coincidence -- campus hosts are the iperf3 clients, so their upload
traffic happens to run access->leaf (A->Z), the direction that was
queried.

Refs #57

Reverse iperf3 traffic direction to DC->campus (server pushes to client) 4719a77f21

Without -R, iperf3's client sends and the server only receives, so
traffic ran campus-host -> dc-server -- backwards for what these roles
represent: dc-serverN is the DC-hosted service, campus-hostN is the
consumer, and a real service delivers data to the consumer rather than
just absorbing uploads from it.

Refs #58

Fix scripts/README.md doc drift after iperf3/weathermap direction fixes 5e5d6b45df

Traffic direction docs still described the pre-fix client->server
iperf3 flow and the pre-fix tx/rx weathermap query pair.

Refs #57, Refs #58

Damien merged commit 31014bdfdf into main

2026-07-20 16:59:10 +00:00

Damien referenced this issue from a commit

2026-07-20 16:59:10 +00:00

Merge pull request 'Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator' (#59) from feat/telemetry into main

Damien deleted branch feat/telemetry

2026-07-20 16:59:10 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Damien/arista-evpn-vxlan-clab#59