arista-evpn-vxlan-clab/scripts/README.md

scripts/generate_weathermap.py

Generates a weathermap-ng panel only from live IPFabric topology + gnmic/Prometheus metrics, merges it into a manually-authored dashboard base, and writes the merged result to configs/grafana/weathermap-dashboard.json (committed to Gitea as the build artifact, same pattern as the retired Flow Panel YAML). Optionally provisions it into Grafana via the HTTP API.

File structure (see #52)

File	What it is
configs/grafana/dashboard-base.json	Manually authored, committed source of truth for everything except the weathermap panel: BGP sessions table, ports/interfaces table, throughput-per-site graphs, site/device template variables, panel layout. Edit this directly for anything in those panels. Contains a reserved slot panel titled __WEATHERMAP_SLOT__ (id 1, gridPos fixed) that the script splices the generated weathermap panel into.
configs/grafana/weathermap-dashboard.json	Generated build artifact — base + freshly-generated weathermap panel, merged. This is what actually gets provisioned to Grafana. Don't hand-edit; re-run the script.

The script itself only ever knows about the weathermap panel (targets + options.weathermap) — it has no knowledge of the other three panels or the template variables. That split is deliberate: topology-driven content (nodes/links, generated from live IPFabric+Prometheus data) is separated from hand-tuned dashboard composition (table layout, transformations, thresholds), which doesn't need to regenerate on every topology change.

See issue #48 for the panel schema research this is built against, and its comment thread for the live-validation history (auth quirks, plugin id, a PromQL escaping bug, a panel-crash fix — worth reading before touching the anchor/regex-escaping logic).

How to generate and provision the dashboard, start to finish

This walks through the whole thing from nothing — no weathermap-dashboard.json, no dashboard in Grafana — to a fully working, live-updating dashboard. Useful for a first-ever setup, or after a full reset (deleting the generated JSON and the Grafana dashboard, keeping dashboard-base.json — that file is the hand-authored part of the project and is never deleted/regenerated).

1. Gather credentials

Four values are required; two more are worth setting so --provision works without extra flags. Each has a fiddly API path and a much simpler UI path — UI is recommended unless you're scripting this.

IPFabric API token (IPFABRIC_TOKEN)

• UI: IPFabric → user menu (top right) → Settings → API Tokens → Add token. Needs Inventory/Snapshots read access.
• No simpler API alternative — tokens can only be created through the UI or by an admin via the same UI.

Grafana service account token (GRAFANA_TOKEN)

• UI (recommended — the API path is multiple chained calls: create the service account, then create a token under it, threading the returned service account ID between them):
  1. Grafana → Administration → Users and access → Service accounts → Add service account.
  2. Give it a name (e.g. weathermap-generator), role Editor (needs to create/update dashboards).
  3. Open the new service account → Add service account token → copy the token immediately, it's shown once.

Prometheus datasource UID (GRAFANA_DATASOURCE_UID)

• UI: Grafana → Connections → Data sources → click the Prometheus datasource that can see the gnmic metrics (NetLab in this lab) → the UID is the last segment of the page URL (.../datasources/edit/<uid>).
• API alternative: curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/datasources" | jq '.[] | {name, uid}'
• This must be a datasource that can actually see the gnmic metrics — see the note in the Environment variables table below; picking the wrong one renders the panel with no data (happened once, see #48).

Weathermap plugin ID (GRAFANA_WEATHERMAP_PLUGIN_ID, optional — only if a different fork than tamirsuliman-weathermap-panel is installed)

• UI: Grafana → Administration → Plugins and data → Plugins, search "weathermap", open it — the id is in the page URL (.../plugins/<id>).
• API alternative: curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/plugins?panelId=..." is more involved than it's worth; the UI is faster here.

IPFabric/Grafana URLs — just the base URLs you already use in a browser (e.g. https://ipfabric.example.com, https://grafana.example.com).

Copy envrc.sample to .envrc (already gitignored) and fill in the values, or export them directly in your shell:

cp envrc.sample .envrc   # then edit .envrc
source .envrc

2. Dry run first (no Grafana changes yet)

python3 scripts/generate_weathermap.py

This only needs IPFABRIC_URL/IPFABRIC_TOKEN (and optionally GRAFANA_URL/GRAFANA_TOKEN, to preserve any live node positions — see #53). It writes configs/grafana/weathermap-dashboard.json and prints a summary:

Fetching devices from IPFabric (...)...
  28 devices
Fetching connectivity-matrix...
  61 fabric links after Management-plane filter + dedup (702 raw rows)
Fetching interface speeds...
Fetching live node positions from ... (preserve manual repositioning)...
  0 node(s) with an existing live position          <- 0 is expected/correct on a first-ever run
Cross-checking interface names against live exporter (...)...
Loading manual dashboard base (configs/grafana/dashboard-base.json)...
Wrote configs/grafana/weathermap-dashboard.json (28 nodes, 61 links, 4 panels total)

If any "interface-name / bandwidth mismatch(es)" are printed to stderr, read them — they mean a link's query won't resolve to data until the underlying gnmic/IPFabric naming mismatch is fixed (not this script's job to fix, see "Known gaps" below), but the link itself is still generated, not dropped.

Open the written JSON and sanity-check it if you like before pushing it live — it's plain, readable JSON, no need to trust it blindly.

3. Provision it

python3 scripts/generate_weathermap.py --provision

Requires GRAFANA_URL, GRAFANA_TOKEN, GRAFANA_DATASOURCE_UID in addition to the IPFabric variables. On success:

Provisioning dashboard 'evpn-vxlan-fabric-weathermap' to https://...
  success: /d/evpn-vxlan-fabric-weathermap/evpn-vxlan-fabric-weathermap

4. Verify

Via the UI (recommended, and the only way to actually see it render): open $GRAFANA_URL/d/evpn-vxlan-fabric-weathermap in a browser. Check:

• The weathermap panel shows all nodes, roughly grouped spine-top/access-bottom (see #53), all colored green (up) with no data gaps.
• The site/device dropdown variables at the top of the dashboard filter the BGP Sessions and Ports/Interfaces tables when changed.
• BGP Sessions and Ports/Interfaces tables are populated, states colored (green=Up, red=Down).
• Throughput per Site shows a signed graph (in above the axis, out below) with plausible non-zero values.

This step has no good API substitute — dashboard rendering, panel colors, and transformations (the ports table's Grafana-side merge, in particular) only actually execute in a browser. If you don't have one handy, at minimum confirm the queries resolve (below), then ask someone who does have UI access to eyeball it.

Via the API (queries only, not rendering) — useful for CI/scripting or a quick sanity check without opening a browser:

# Any panel's query, pulled straight from the generated JSON, e.g.:
curl -s -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H "Content-Type: application/json" \
  "$GRAFANA_URL/api/ds/query" -d '{
    "queries": [{"refId":"A","datasource":{"type":"prometheus","uid":"'"$GRAFANA_DATASOURCE_UID"'"},
                 "expr":"min by (device) (interfaces_interface_state_oper_status)","instant":true}],
    "from":"now-5m","to":"now"}' | python3 -m json.tool

A much friendlier equivalent that doesn't need hand-built JSON: Grafana's Explore view (left sidebar → Explore, pick the Prometheus datasource, paste a PromQL expression from the panel JSON, run it). This is the manual alternative for every /api/ds/query validation call used during development of this dashboard (#49–#54) — same result, no curl/jq required.

5. Re-running after a topology change

python3 scripts/generate_weathermap.py --provision

Same command — the script always regenerates the weathermap panel fresh from live IPFabric data (no diffing/incremental state), so a plain re-run already picks up any topology change. There's no separate "detect changes" step, and no watcher/CI hook triggering it automatically yet — re-running after a change is a manual/operational step for now.

Nodes you've manually repositioned in the Grafana UI keep that position across the re-run (#53); only genuinely new nodes get the layered default.

If you only need to change the BGP/ports/throughput panels, template variables, or panel layout, edit configs/grafana/dashboard-base.json directly and re-run the script (or --provision to push it live) — no topology fetch is skipped today (the script always does the full IPFabric round-trip regardless), but no topology data is needed for that kind of edit to take effect.

Troubleshooting

Symptom	Likely cause	Check
Refusing to provision: set GRAFANA_DATASOURCE_UID...	Env var unset or still the placeholder	echo $GRAFANA_DATASOURCE_UID
Panel loads but shows no data at all	Wrong datasource UID — points at a Prometheus instance that can't see the gnmic metrics	Grafana UI → Explore → run up against the datasource you set; confirm gnmic-labeled series come back
Base dashboard has no panel titled '__WEATHERMAP_SLOT__'	dashboard-base.json was edited and the slot panel's title got changed/removed	Check the panel with id: 1 in dashboard-base.json still has that exact title
A node's manual position keeps resetting to the default layout on every rerun	GRAFANA_URL/GRAFANA_TOKEN not set for that run, so the live-position fetch was skipped entirely	Check the printed line Fetching live node positions from ... appears in the run's output — if it's missing, those two vars weren't set
Fetching live dashboard for position preservation failed: HTTP 401/403	Bad/expired Grafana token	Regenerate the service account token (UI steps above)

Environment variables

Variable	Required	Default	Notes
--base (CLI flag, not env)	no	configs/grafana/dashboard-base.json	Manual dashboard base to merge the weathermap panel into.
IPFABRIC_URL	yes	—	e.g. https://ipfabric.example.com
IPFABRIC_TOKEN	yes	—	Inventory/Snapshots read access. Sent as X-API-Token, not Authorization: Bearer — IPFabric's REST API does not use bearer auth.
IPFABRIC_SNAPSHOT	no	$last
PROMETHEUS_URL	no	http://172.16.0.71:9090	The in-topology Prometheus (see #45), used at generation time to validate the IPFabric→gnmic interface alias and to discover live VTEP/VLAN pairs. This does not have to be the same instance Grafana queries at render time — see below.
GRAFANA_URL	no (see #53)	—	Required for --provision. Also used, if set, to fetch the currently-provisioned dashboard and preserve any manually drag-and-drop-repositioned node positions — see step 3 below. Without it, every node gets the layered default position, even ones a human previously repositioned in the UI.
GRAFANA_TOKEN	no (see #53)	—	Grafana Service Account token, sent as Authorization: Bearer. Same conditions as GRAFANA_URL above.
GRAFANA_DATASOURCE_UID	with --provision	—	UID of the Prometheus datasource in Grafana that will actually back the panel. This must be a datasource that can see the gnmic metrics — an unrelated/external Prometheus instance with no gnmic data will make the panel render with no values (this happened once, see #48).
GRAFANA_DASHBOARD_UID	no	evpn-vxlan-fabric-weathermap
GRAFANA_WEATHERMAP_PLUGIN_ID	no	tamirsuliman-weathermap-panel	The installed weathermap-ng plugin id. Override if a different fork is installed — plugin ids don't always match the upstream repo name (the schema research in #48 was done against the allamiro fork's source; what's actually installed here is a different fork with a stricter/different runtime schema — see the ANCHOR handling in the script).

What the script does

1. Fetches device inventory + connectivity-matrix from IPFabric.
2. Filters the connectivity-matrix to physical Ethernet-to-Ethernet links only: drops Management-plane neighbor entries and .100/.200 subinterface rows (802.1Q tags used for the gold VRF stitching on Core — they ride the same physical port as their parent interface and gnmic only exports physical interface counters), and dedupes the two directions IPFabric reports for each physical link into one.
3. Computes node positions (see #53):
   • Default layered layout, for nodes with no known position yet: role is parsed from the hostname naming convention (*-spine*→spine, core*→core, *-border-leaf*→border-leaf, *-leaf*→leaf, *-access*→access — same trust level as the site parsing used for the Prometheus relabel, not IPFabric-derived), giving Y-tiers top to bottom spine → core → border-leaf → leaf → access. X position is grouped/columned by site within each tier, so e.g. dc leafs and campus leafs land in the same row but different column bands. A hostname matching no role pattern is logged (not silently placed in the wrong tier) and put in a fallback tier below access.
   • Live-position preservation: position is edited by drag-and-drop directly in the Grafana UI, not in the git-committed base file — a deliberate exception to the "file is the only source of truth" rule from #52, because that's simply not where this field is edited. Before computing the default layout above, the script fetches the currently provisioned dashboard's weathermap panel (if GRAFANA_URL/GRAFANA_TOKEN are set) and reuses each existing node's live position as-is. Only genuinely new nodes (not in that live map — first-ever run, or a fresh device added to the topology) get the layered default. A device removed from the topology simply has no entry in this run's node/link output at all, so no orphaned position or dangling link reference is possible.
4. Builds the panel's targets: one PromQL query per metric family (BGP status, interface tx, interface rx, VXLAN MAC/VNI), each with an explicit legendFormat so the resolved display name is predictable.
5. Builds nodes[] and links[] referencing those resolved legend strings, including the plugin's anchors tally (per-node count of link attachments per side) and numeric anchor enum on each link side — omitting these crashes the panel on load in the installed plugin fork, despite the (fork-specific) schema research saying it's safe to skip.
6. Cross-checks every IPFabric interface name, aliased to gnmic's naming (Et→Ethernet, Po→Port-Channel, Lo→Loopback, Vl→Vlan, Ma→Management), against the live exporter. Any link whose aliased name has no matching series is logged, not silently dropped — the actual fix for a real mismatch belongs in gnmic interface aliasing (#43), not in this script.
7. Loads configs/grafana/dashboard-base.json, substitutes the real Prometheus datasource UID into its __DATASOURCE_UID__ placeholders, finds the panel titled __WEATHERMAP_SLOT__ and splices in the generated weathermap panel content (keeping the slot's gridPos/id, so the manual layout is never repositioned).
8. Writes the merged dashboard JSON to configs/grafana/weathermap-dashboard.json, and provisions it via POST /api/dashboards/db if --provision is passed.

Known gaps

• VXLAN MAC-per-VNI target: the vlan join key used to correlate VLAN→VNI mapping with FDB entries (query verbatim from #44) doesn't actually match on live data for any VTEP node — likely an Arista internal-VLAN-vs-front-panel-VLAN translation the OpenConfig paths don't reconcile. Only affects the decorative per-VTEP tooltip metric, not node/link status or traffic coloring. Tracked in #44, not fixed here.