From 045179e09861618cfe7d7cb06b97d6530a6abbbb Mon Sep 17 00:00:00 2001 From: Damien Arnodo Date: Fri, 10 Jul 2026 13:18:05 +0000 Subject: [PATCH] Reword scripts/README.md for readability, add pipeline diagram Same content, restructured for scanning: tables instead of long paragraphs (credentials, troubleshooting, env vars), shorter sentences, deep rationale trimmed to a phrase + issue reference instead of inline essays. Cut from 258 to 135 lines. Added assets/weathermap-pipeline.svg showing the IPFabric/Prometheus -> generator -> merge (with the manual base) -> Grafana flow, referenced at the top of the README. --- assets/weathermap-pipeline.svg | 72 +++++++ scripts/README.md | 331 +++++++++++---------------------- 2 files changed, 176 insertions(+), 227 deletions(-) create mode 100644 assets/weathermap-pipeline.svg diff --git a/assets/weathermap-pipeline.svg b/assets/weathermap-pipeline.svg new file mode 100644 index 0000000..407b6e4 --- /dev/null +++ b/assets/weathermap-pipeline.svg @@ -0,0 +1,72 @@ + + + + + + + + + + + + IPFabric + topology, inventory + + + Prometheus + gnmic metrics, live labels + + + + generate_weathermap.py + weathermap panel only + targets + options.weathermap + + + + + + + dashboard-base.json + hand-authored, committed + BGP/ports/throughput/vars + + + + merge + splices into __WEATHERMAP_SLOT__ + + + + + + + weathermap-dashboard.json + generated build artifact + + + + + + Grafana + --provision (POST /api/dashboards/db) + + + + + + + live node positions read back before each regeneration (see #53) + + diff --git a/scripts/README.md b/scripts/README.md index b07384b..283597c 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,139 +1,67 @@ # scripts/generate_weathermap.py -Generates a [weathermap-ng](https://github.com/allamiro/grafana-network-weathermap-ng) -**panel only** from live IPFabric topology + gnmic/Prometheus metrics, merges -it into a manually-authored dashboard base, and writes the merged result to -`configs/grafana/weathermap-dashboard.json` (committed to Gitea as the build -artifact, same pattern as the retired Flow Panel YAML). Optionally provisions -it into Grafana via the HTTP API. +Generates the **weathermap panel only** from live IPFabric + Prometheus +data, merges it into a hand-authored dashboard base, and provisions the +result to Grafana. -## File structure (see #52) +![pipeline diagram](../assets/weathermap-pipeline.svg) -| File | What it is | -|---|---| -| `configs/grafana/dashboard-base.json` | **Manually authored**, committed source of truth for everything except the weathermap panel: BGP sessions table, ports/interfaces table, throughput-per-site graphs, `site`/`device` template variables, panel layout. Edit this directly for anything in those panels. Contains a reserved slot panel titled `__WEATHERMAP_SLOT__` (id `1`, `gridPos` fixed) that the script splices the generated weathermap panel into. | -| `configs/grafana/weathermap-dashboard.json` | **Generated build artifact** — base + freshly-generated weathermap panel, merged. This is what actually gets provisioned to Grafana. Don't hand-edit; re-run the script. | +## Files -The script itself only ever knows about the weathermap panel (`targets` + -`options.weathermap`) — it has no knowledge of the other three panels or the -template variables. That split is deliberate: topology-driven content -(nodes/links, generated from live IPFabric+Prometheus data) is separated from -hand-tuned dashboard composition (table layout, transformations, thresholds), -which doesn't need to regenerate on every topology change. +| File | What it is | Edit it? | +|---|---|---| +| `configs/grafana/dashboard-base.json` | Hand-authored: BGP table, ports table, throughput graphs, `site`/`device` variables, layout. Has one reserved slot panel (`title: "__WEATHERMAP_SLOT__"`) where the weathermap gets spliced in. | Yes, directly | +| `configs/grafana/weathermap-dashboard.json` | Generated build artifact — base + fresh weathermap panel, merged. What actually gets provisioned. | No — regenerate instead | -See issue #48 for the panel schema research this is built against, and its -comment thread for the live-validation history (auth quirks, plugin id, a -PromQL escaping bug, a panel-crash fix — worth reading before touching the -anchor/regex-escaping logic). +Why split like this: topology data (nodes/links) changes with the lab and +should regenerate every time; the rest of the dashboard (table layout, +thresholds, transformations) is hand-tuned and shouldn't be touched by a +topology refresh. See #52. -## How to generate and provision the dashboard, start to finish - -This walks through the whole thing from nothing — no `weathermap-dashboard.json`, -no dashboard in Grafana — to a fully working, live-updating dashboard. Useful -for a first-ever setup, or after a full reset (deleting the generated JSON -and the Grafana dashboard, keeping `dashboard-base.json` — that file is the -hand-authored part of the project and is never deleted/regenerated). - -### 1. Gather credentials - -Four values are required; two more are worth setting so `--provision` works -without extra flags. Each has a fiddly API path and a much simpler UI path — -UI is recommended unless you're scripting this. - -**IPFabric API token** (`IPFABRIC_TOKEN`) -- UI: IPFabric → user menu (top right) → **Settings** → **API Tokens** → **Add token**. Needs Inventory/Snapshots read access. -- No simpler API alternative — tokens can only be created through the UI or by an admin via the same UI. - -**Grafana service account token** (`GRAFANA_TOKEN`) -- UI (recommended — the API path is multiple chained calls: create the service account, then create a token under it, threading the returned service account ID between them): - 1. Grafana → **Administration** → **Users and access** → **Service accounts** → **Add service account**. - 2. Give it a name (e.g. `weathermap-generator`), role **Editor** (needs to create/update dashboards). - 3. Open the new service account → **Add service account token** → copy the token immediately, it's shown once. - -**Prometheus datasource UID** (`GRAFANA_DATASOURCE_UID`) -- UI: Grafana → **Connections** → **Data sources** → click the Prometheus datasource that can see the gnmic metrics (`NetLab` in this lab) → the UID is the last segment of the page URL (`.../datasources/edit/`). -- API alternative: `curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/datasources" | jq '.[] | {name, uid}'` -- **This must be a datasource that can actually see the gnmic metrics** — see the note in the Environment variables table below; picking the wrong one renders the panel with no data (happened once, see #48). - -**Weathermap plugin ID** (`GRAFANA_WEATHERMAP_PLUGIN_ID`, optional — only if a different fork than `tamirsuliman-weathermap-panel` is installed) -- UI: Grafana → **Administration** → **Plugins and data** → **Plugins**, search "weathermap", open it — the id is in the page URL (`.../plugins/`). -- API alternative: `curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/plugins?panelId=..." ` is more involved than it's worth; the UI is faster here. - -**IPFabric/Grafana URLs** — just the base URLs you already use in a browser (e.g. `https://ipfabric.example.com`, `https://grafana.example.com`). - -Copy `envrc.sample` to `.envrc` (already gitignored) and fill in the values, -or `export` them directly in your shell: +## Quickstart ```bash -cp envrc.sample .envrc # then edit .envrc +cp envrc.sample .envrc # fill in the values (see "Credentials" below) source .envrc + +python3 scripts/generate_weathermap.py # dry run: writes the JSON only +python3 scripts/generate_weathermap.py --provision # also pushes it to Grafana ``` -### 2. Dry run first (no Grafana changes yet) +Re-run the same `--provision` command any time the lab topology changes — +the script always rebuilds the weathermap panel from live data, so there's +nothing to "detect", just re-run it. It's not automated yet (no watcher/CI +hook), so re-running is a manual step for now. + +## Credentials + +| Value | Env var | Get it from the UI | Get it via API | +|---|---|---|---| +| IPFabric token | `IPFABRIC_TOKEN` | IPFabric → user menu → Settings → API Tokens → Add token | not possible, UI-only | +| Grafana service account token | `GRAFANA_TOKEN` | Grafana → Administration → Service accounts → Add → give it **Editor** role → Add token | possible but multi-step (create account, then create token under it) — UI is simpler | +| Prometheus datasource UID | `GRAFANA_DATASOURCE_UID` | Grafana → Connections → Data sources → open the one with gnmic data → UID is the last part of the URL | `curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/datasources" \| jq '.[] \| {name,uid}'` | +| Weathermap plugin ID | `GRAFANA_WEATHERMAP_PLUGIN_ID` (optional, only if not the default fork) | Grafana → Administration → Plugins → search "weathermap" → id is in the URL | — | + +⚠️ The datasource must be the one that can actually see gnmic metrics — +picking an unrelated Prometheus instance renders the panel with no data +(see #48). + +## Verifying the result + +**In the browser** (the only way to actually see it render — colors, +tables, and the ports-table merge transformation all happen client-side): + +1. Open `$GRAFANA_URL/d/evpn-vxlan-fabric-weathermap`. +2. Weathermap: nodes grouped spine-top → access-bottom, all green. +3. `site`/`device` dropdowns filter the two tables below. +4. BGP Sessions / Ports tables: populated, green=Up / red=Down. +5. Throughput per Site: signed graph, non-zero values. + +**Without a browser**, use Grafana's **Explore** view (paste a PromQL +expression from the panel JSON and run it) — same result as the UI check +above minus the rendering, no `curl` needed. Or hit the API directly: ```bash -python3 scripts/generate_weathermap.py -``` - -This only needs `IPFABRIC_URL`/`IPFABRIC_TOKEN` (and optionally `GRAFANA_URL`/`GRAFANA_TOKEN`, -to preserve any live node positions — see #53). It writes -`configs/grafana/weathermap-dashboard.json` and prints a summary: - -``` -Fetching devices from IPFabric (...)... - 28 devices -Fetching connectivity-matrix... - 61 fabric links after Management-plane filter + dedup (702 raw rows) -Fetching interface speeds... -Fetching live node positions from ... (preserve manual repositioning)... - 0 node(s) with an existing live position <- 0 is expected/correct on a first-ever run -Cross-checking interface names against live exporter (...)... -Loading manual dashboard base (configs/grafana/dashboard-base.json)... -Wrote configs/grafana/weathermap-dashboard.json (28 nodes, 61 links, 4 panels total) -``` - -If any "interface-name / bandwidth mismatch(es)" are printed to stderr, read -them — they mean a link's query won't resolve to data until the underlying -gnmic/IPFabric naming mismatch is fixed (not this script's job to fix, see -"Known gaps" below), but the link itself is still generated, not dropped. - -Open the written JSON and sanity-check it if you like before pushing it live -— it's plain, readable JSON, no need to trust it blindly. - -### 3. Provision it - -```bash -python3 scripts/generate_weathermap.py --provision -``` - -Requires `GRAFANA_URL`, `GRAFANA_TOKEN`, `GRAFANA_DATASOURCE_UID` in addition -to the IPFabric variables. On success: - -``` -Provisioning dashboard 'evpn-vxlan-fabric-weathermap' to https://... - success: /d/evpn-vxlan-fabric-weathermap/evpn-vxlan-fabric-weathermap -``` - -### 4. Verify - -**Via the UI (recommended, and the only way to actually see it render)**: -open `$GRAFANA_URL/d/evpn-vxlan-fabric-weathermap` in a browser. Check: -- The weathermap panel shows all nodes, roughly grouped spine-top/access-bottom (see #53), all colored green (up) with no data gaps. -- The `site`/`device` dropdown variables at the top of the dashboard filter the BGP Sessions and Ports/Interfaces tables when changed. -- BGP Sessions and Ports/Interfaces tables are populated, states colored (green=Up, red=Down). -- Throughput per Site shows a signed graph (in above the axis, out below) with plausible non-zero values. - -This step has no good API substitute — dashboard rendering, panel colors, -and transformations (the ports table's Grafana-side merge, in particular) -only actually execute in a browser. If you don't have one handy, at minimum -confirm the queries resolve (below), then ask someone who does have UI -access to eyeball it. - -**Via the API (queries only, not rendering)** — useful for CI/scripting or a -quick sanity check without opening a browser: - -```bash -# Any panel's query, pulled straight from the generated JSON, e.g.: curl -s -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H "Content-Type: application/json" \ "$GRAFANA_URL/api/ds/query" -d '{ "queries": [{"refId":"A","datasource":{"type":"prometheus","uid":"'"$GRAFANA_DATASOURCE_UID"'"}, @@ -141,118 +69,67 @@ curl -s -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H "Content-Type: appl "from":"now-5m","to":"now"}' | python3 -m json.tool ``` -A much friendlier equivalent that doesn't need hand-built JSON: Grafana's -**Explore** view (left sidebar → **Explore**, pick the Prometheus datasource, -paste a PromQL expression from the panel JSON, run it). This is the manual -alternative for every `/api/ds/query` validation call used during -development of this dashboard (#49–#54) — same result, no `curl`/`jq` -required. +## Troubleshooting -### 5. Re-running after a topology change - -```bash -python3 scripts/generate_weathermap.py --provision -``` - -Same command — the script always regenerates the weathermap panel fresh from -live IPFabric data (no diffing/incremental state), so a plain re-run already -picks up any topology change. There's no separate "detect changes" step, and -no watcher/CI hook triggering it automatically yet — re-running after a -change is a manual/operational step for now. - -Nodes you've manually repositioned in the Grafana UI keep that position -across the re-run (#53); only genuinely new nodes get the layered default. - -If you only need to change the BGP/ports/throughput panels, template -variables, or panel layout, edit `configs/grafana/dashboard-base.json` -directly and re-run the script (or `--provision` to push it live) — no -topology fetch is skipped today (the script always does the full IPFabric -round-trip regardless), but no topology data is *needed* for that kind of -edit to take effect. - -### Troubleshooting - -| Symptom | Likely cause | Check | +| Symptom | Cause | Check | |---|---|---| -| `Refusing to provision: set GRAFANA_DATASOURCE_UID...` | Env var unset or still the placeholder | `echo $GRAFANA_DATASOURCE_UID` | -| Panel loads but shows no data at all | Wrong datasource UID — points at a Prometheus instance that can't see the gnmic metrics | Grafana UI → Explore → run `up` against the datasource you set; confirm gnmic-labeled series come back | -| `Base dashboard has no panel titled '__WEATHERMAP_SLOT__'` | `dashboard-base.json` was edited and the slot panel's `title` got changed/removed | Check the panel with `id: 1` in `dashboard-base.json` still has that exact title | -| A node's manual position keeps resetting to the default layout on every rerun | `GRAFANA_URL`/`GRAFANA_TOKEN` not set for that run, so the live-position fetch was skipped entirely | Check the printed line `Fetching live node positions from ...` appears in the run's output — if it's missing, those two vars weren't set | -| `Fetching live dashboard for position preservation failed: HTTP 401/403` | Bad/expired Grafana token | Regenerate the service account token (UI steps above) | +| `Refusing to provision: set GRAFANA_DATASOURCE_UID...` | Env var unset | `echo $GRAFANA_DATASOURCE_UID` | +| Panel renders with no data | Wrong datasource — can't see gnmic metrics | Explore → run `up` against it, confirm gnmic series come back | +| `Base dashboard has no panel titled '__WEATHERMAP_SLOT__'` | Someone edited `dashboard-base.json` and renamed/removed that panel | Check panel `id: 1` still has that exact title | +| Manually-moved node keeps resetting position | `GRAFANA_URL`/`GRAFANA_TOKEN` weren't set for that run | Look for `Fetching live node positions from ...` in the run output | +| `HTTP 401/403` fetching live positions | Expired/bad Grafana token | Re-issue the service account token | ## Environment variables -| Variable | Required | Default | Notes | -|---|---|---|---| -| `--base` (CLI flag, not env) | no | `configs/grafana/dashboard-base.json` | Manual dashboard base to merge the weathermap panel into. | -| `IPFABRIC_URL` | yes | — | e.g. `https://ipfabric.example.com` | -| `IPFABRIC_TOKEN` | yes | — | Inventory/Snapshots read access. Sent as `X-API-Token`, not `Authorization: Bearer` — IPFabric's REST API does not use bearer auth. | -| `IPFABRIC_SNAPSHOT` | no | `$last` | | -| `PROMETHEUS_URL` | no | `http://172.16.0.71:9090` | The **in-topology** Prometheus (see #45), used at generation time to validate the IPFabric→gnmic interface alias and to discover live VTEP/VLAN pairs. This does not have to be the same instance Grafana queries at render time — see below. | -| `GRAFANA_URL` | no (see #53) | — | Required for `--provision`. Also used, if set, to fetch the currently-provisioned dashboard and preserve any manually drag-and-drop-repositioned node positions — see step 3 below. Without it, every node gets the layered default position, even ones a human previously repositioned in the UI. | -| `GRAFANA_TOKEN` | no (see #53) | — | Grafana Service Account token, sent as `Authorization: Bearer`. Same conditions as `GRAFANA_URL` above. | -| `GRAFANA_DATASOURCE_UID` | with `--provision` | — | UID of the Prometheus datasource in Grafana that will actually back the panel. **This must be a datasource that can see the gnmic metrics** — an unrelated/external Prometheus instance with no gnmic data will make the panel render with no values (this happened once, see #48). | -| `GRAFANA_DASHBOARD_UID` | no | `evpn-vxlan-fabric-weathermap` | | -| `GRAFANA_WEATHERMAP_PLUGIN_ID` | no | `tamirsuliman-weathermap-panel` | The installed weathermap-ng plugin id. Override if a different fork is installed — plugin ids don't always match the upstream repo name (the schema research in #48 was done against the `allamiro` fork's source; what's actually installed here is a different fork with a stricter/different runtime schema — see the `ANCHOR` handling in the script). | +| Variable | Required | Default | +|---|---|---| +| `IPFABRIC_URL` | yes | — | +| `IPFABRIC_TOKEN` | yes | — | +| `IPFABRIC_SNAPSHOT` | no | `$last` | +| `PROMETHEUS_URL` | no | `http://172.16.0.71:9090` | +| `GRAFANA_URL` | no* | — | +| `GRAFANA_TOKEN` | no* | — | +| `GRAFANA_DATASOURCE_UID` | with `--provision` | — | +| `GRAFANA_DASHBOARD_UID` | no | `evpn-vxlan-fabric-weathermap` | +| `GRAFANA_WEATHERMAP_PLUGIN_ID` | no | `tamirsuliman-weathermap-panel` | +| `--base` (CLI flag) | no | `configs/grafana/dashboard-base.json` | -## What the script does +\* `GRAFANA_URL`/`GRAFANA_TOKEN` are required for `--provision`, and +optional otherwise — if set on a plain (non-`--provision`) run, they're +used to read back live node positions so manual repositioning survives +the next regeneration (see #53). Skip them and every node just gets the +default layout. -1. Fetches device inventory + connectivity-matrix from IPFabric. -2. Filters the connectivity-matrix to physical Ethernet-to-Ethernet links - only: drops Management-plane neighbor entries and `.100`/`.200` - subinterface rows (802.1Q tags used for the gold VRF stitching on Core — - they ride the same physical port as their parent interface and gnmic only - exports physical interface counters), and dedupes the two directions - IPFabric reports for each physical link into one. -3. Computes node positions (see #53): - - **Default layered layout**, for nodes with no known position yet: role - is parsed from the hostname naming convention (`*-spine*`→spine, - `core*`→core, `*-border-leaf*`→border-leaf, `*-leaf*`→leaf, - `*-access*`→access — same trust level as the `site` parsing used for - the Prometheus relabel, not IPFabric-derived), giving Y-tiers top to - bottom `spine → core → border-leaf → leaf → access`. X position is - grouped/columned by site within each tier, so e.g. dc leafs and campus - leafs land in the same row but different column bands. A hostname - matching no role pattern is logged (not silently placed in the wrong - tier) and put in a fallback tier below `access`. - - **Live-position preservation**: position is edited by drag-and-drop - directly in the Grafana UI, not in the git-committed base file — a - deliberate exception to the "file is the only source of truth" rule - from #52, because that's simply not where this field is edited. Before - computing the default layout above, the script fetches the currently - provisioned dashboard's weathermap panel (if `GRAFANA_URL`/`GRAFANA_TOKEN` - are set) and reuses each existing node's live position as-is. Only - genuinely new nodes (not in that live map — first-ever run, or a fresh - device added to the topology) get the layered default. A device removed - from the topology simply has no entry in this run's node/link output at - all, so no orphaned position or dangling link reference is possible. -4. Builds the panel's `targets`: one PromQL query per metric family (BGP - status, interface tx, interface rx, VXLAN MAC/VNI), each with an explicit - `legendFormat` so the resolved display name is predictable. -5. Builds `nodes[]` and `links[]` referencing those resolved legend strings, - including the plugin's `anchors` tally (per-node count of link - attachments per side) and numeric anchor enum on each link side — - omitting these crashes the panel on load in the installed plugin fork, - despite the (fork-specific) schema research saying it's safe to skip. -6. Cross-checks every IPFabric interface name, aliased to gnmic's naming - (`Et`→`Ethernet`, `Po`→`Port-Channel`, `Lo`→`Loopback`, `Vl`→`Vlan`, - `Ma`→`Management`), against the live exporter. Any link whose aliased - name has no matching series is logged, not silently dropped — the actual - fix for a real mismatch belongs in gnmic interface aliasing (#43), not in - this script. -7. Loads `configs/grafana/dashboard-base.json`, substitutes the real - Prometheus datasource UID into its `__DATASOURCE_UID__` placeholders, - finds the panel titled `__WEATHERMAP_SLOT__` and splices in the generated - weathermap panel content (keeping the slot's `gridPos`/`id`, so the manual - layout is never repositioned). -8. Writes the merged dashboard JSON to `configs/grafana/weathermap-dashboard.json`, - and provisions it via `POST /api/dashboards/db` if `--provision` is passed. +## How it works + +1. Fetch device inventory + connectivity-matrix from IPFabric. +2. Keep only physical Ethernet↔Ethernet links (drop management-plane and + `.100`/`.200` subinterface rows), dedupe the two directions IPFabric + reports per link. +3. Position each node: + - **New node** → layered default: Y-tier by role parsed from the + hostname (`spine → core → border-leaf → leaf → access`), X grouped + by site within the tier. A hostname that matches no role is logged + and dropped into a fallback tier, never silently misplaced (#53). + - **Known node** → whatever position is currently live in Grafana, + unchanged. This is why `GRAFANA_URL`/`GRAFANA_TOKEN` matter even on + a dry run (#53). +4. Build the panel's PromQL `targets`: node status, link tx/rx, VXLAN + MAC/VNI tooltip — each with an explicit `legendFormat`. +5. Build `nodes[]`/`links[]`, including the plugin's `anchors` tally + (link count per side) — required by the installed plugin fork even + though the schema doc says it's optional (#48). +6. Cross-check every interface name against the live gnmic exporter + (`Et`→`Ethernet`, `Po`→`Port-Channel`, etc.); mismatches are logged, + not dropped — the link stays, just without data until fixed upstream. +7. Load `dashboard-base.json`, substitute the real datasource UID, splice + the generated panel into `__WEATHERMAP_SLOT__` (keeping that slot's + `gridPos`/`id` — layout stays manual). +8. Write `weathermap-dashboard.json`; provision it if `--provision`. ## Known gaps -- **VXLAN MAC-per-VNI target**: the `vlan` join key used to correlate - VLAN→VNI mapping with FDB entries (query verbatim from #44) doesn't - actually match on live data for any VTEP node — likely an Arista - internal-VLAN-vs-front-panel-VLAN translation the OpenConfig paths don't - reconcile. Only affects the decorative per-VTEP tooltip metric, not - node/link status or traffic coloring. Tracked in #44, not fixed here. +- **VXLAN MAC-per-VNI tooltip**: the `vlan` join key doesn't match on live + data for any VTEP node — likely an Arista internal-vs-front-panel VLAN + translation gap. Only affects that one decorative tooltip metric, not + node/link status or traffic coloring. Tracked in #44.