diff --git a/README.md b/README.md index 84c0f48..73b782b 100644 --- a/README.md +++ b/README.md @@ -384,42 +384,9 @@ curl -s 'http://172.16.0.71:9090/api/v1/query?query=system_memory_state_used' | ### Weathermap panel generation (dashboard-as-code) -`scripts/generate_weathermap.py` generates a [weathermap-ng](https://github.com/allamiro/grafana-network-weathermap-ng) -panel from live IPFabric topology + gnmic/Prometheus metrics, and writes a full -Grafana dashboard-as-code JSON to `configs/grafana/weathermap-dashboard.json` -(committed to Gitea as the source of truth, same pattern as the retired Flow -Panel YAML). See issue #48 for the schema research this is built against. - -```bash -export IPFABRIC_URL=https:// -export IPFABRIC_TOKEN= -python3 scripts/generate_weathermap.py # writes the JSON only - -export GRAFANA_URL=https:// -export GRAFANA_TOKEN= -export GRAFANA_DATASOURCE_UID= -python3 scripts/generate_weathermap.py --provision # also provisions via the Grafana API -``` - -- Re-run after any topology change (`evpn-lab.clab.yml`) to regenerate and re-provision. -- Node/link topology comes from IPFabric (`/tables/inventory/devices`, - `/tables/interfaces/connectivity-matrix`); node positions are computed as a - simple site-grouped grid (dc/core/campus bands) since IPFabric has no - layout data. -- IPFabric reports abbreviated interface names (`Et11`) while gnmic exposes - full OpenConfig names (`Ethernet11`); the script aliases known prefixes - (`Et`→`Ethernet`, `Po`→`Port-Channel`, `Lo`→`Loopback`, `Vl`→`Vlan`, - `Ma`→`Management`) and cross-checks the result against the live exporter, - logging (not silently dropping) any link whose aliased name has no matching - series — the actual fix for a real mismatch belongs in gnmic interface - aliasing (#43), not in this script. -- `.100`/`.200` subinterface rows in the connectivity-matrix (802.1Q tags used - for the gold VRF stitching on Core) are excluded — they ride the same - physical port as their parent interface and gnmic only exports physical - interface counters. -- The Grafana panel plugin id (`GRAFANA_WEATHERMAP_PLUGIN_ID`, default - `allamiro-weathermap-panel`) is a placeholder — confirm it against the - actually installed plugin before provisioning. +`scripts/generate_weathermap.py` generates a weathermap-ng Grafana panel from +live IPFabric topology + gnmic/Prometheus metrics. See +[`scripts/README.md`](scripts/README.md) for usage and details. ## 📁 Repository Structure @@ -449,6 +416,7 @@ arista-evpn-vxlan-clab/ │ └── grafana/ │ └── weathermap-dashboard.json ├── scripts/ +│ ├── README.md │ └── generate_weathermap.py └── hosts/ ├── README.md diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..2d5e4ea --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,79 @@ +# scripts/generate_weathermap.py + +Generates a [weathermap-ng](https://github.com/allamiro/grafana-network-weathermap-ng) +panel from live IPFabric topology + gnmic/Prometheus metrics, and writes a full +Grafana dashboard-as-code JSON to `configs/grafana/weathermap-dashboard.json` +(committed to Gitea as the source of truth, same pattern as the retired Flow +Panel YAML). Optionally provisions it into Grafana via the HTTP API. + +See issue #48 for the panel schema research this is built against, and its +comment thread for the live-validation history (auth quirks, plugin id, a +PromQL escaping bug, a panel-crash fix — worth reading before touching the +anchor/regex-escaping logic). + +## Usage + +```bash +export IPFABRIC_URL=https:// +export IPFABRIC_TOKEN= +python3 scripts/generate_weathermap.py # writes the JSON only + +export GRAFANA_URL=https:// +export GRAFANA_TOKEN= +export GRAFANA_DATASOURCE_UID= +python3 scripts/generate_weathermap.py --provision # also provisions via the Grafana API +``` + +Re-run after any topology change (`evpn-lab.clab.yml`) to regenerate and +re-provision. + +## Environment variables + +| Variable | Required | Default | Notes | +|---|---|---|---| +| `IPFABRIC_URL` | yes | — | e.g. `https://ipfabric.example.com` | +| `IPFABRIC_TOKEN` | yes | — | Inventory/Snapshots read access. Sent as `X-API-Token`, not `Authorization: Bearer` — IPFabric's REST API does not use bearer auth. | +| `IPFABRIC_SNAPSHOT` | no | `$last` | | +| `PROMETHEUS_URL` | no | `http://172.16.0.71:9090` | The **in-topology** Prometheus (see #45), used at generation time to validate the IPFabric→gnmic interface alias and to discover live VTEP/VLAN pairs. This does not have to be the same instance Grafana queries at render time — see below. | +| `GRAFANA_URL` | with `--provision` | — | | +| `GRAFANA_TOKEN` | with `--provision` | — | Grafana Service Account token, sent as `Authorization: Bearer`. | +| `GRAFANA_DATASOURCE_UID` | with `--provision` | — | UID of the Prometheus datasource in Grafana that will actually back the panel. **This must be a datasource that can see the gnmic metrics** — an unrelated/external Prometheus instance with no gnmic data will make the panel render with no values (this happened once, see #48). | +| `GRAFANA_DASHBOARD_UID` | no | `evpn-vxlan-fabric-weathermap` | | +| `GRAFANA_WEATHERMAP_PLUGIN_ID` | no | `tamirsuliman-weathermap-panel` | The installed weathermap-ng plugin id. Override if a different fork is installed — plugin ids don't always match the upstream repo name (the schema research in #48 was done against the `allamiro` fork's source; what's actually installed here is a different fork with a stricter/different runtime schema — see the `ANCHOR` handling in the script). | + +## What the script does + +1. Fetches device inventory + connectivity-matrix from IPFabric. +2. Filters the connectivity-matrix to physical Ethernet-to-Ethernet links + only: drops Management-plane neighbor entries and `.100`/`.200` + subinterface rows (802.1Q tags used for the gold VRF stitching on Core — + they ride the same physical port as their parent interface and gnmic only + exports physical interface counters), and dedupes the two directions + IPFabric reports for each physical link into one. +3. Computes node positions as a simple site-grouped grid (dc/core/campus + bands) — IPFabric has no layout data. +4. Builds the panel's `targets`: one PromQL query per metric family (BGP + status, interface tx, interface rx, VXLAN MAC/VNI), each with an explicit + `legendFormat` so the resolved display name is predictable. +5. Builds `nodes[]` and `links[]` referencing those resolved legend strings, + including the plugin's `anchors` tally (per-node count of link + attachments per side) and numeric anchor enum on each link side — + omitting these crashes the panel on load in the installed plugin fork, + despite the (fork-specific) schema research saying it's safe to skip. +6. Cross-checks every IPFabric interface name, aliased to gnmic's naming + (`Et`→`Ethernet`, `Po`→`Port-Channel`, `Lo`→`Loopback`, `Vl`→`Vlan`, + `Ma`→`Management`), against the live exporter. Any link whose aliased + name has no matching series is logged, not silently dropped — the actual + fix for a real mismatch belongs in gnmic interface aliasing (#43), not in + this script. +7. Writes the dashboard JSON, and provisions it via `POST + /api/dashboards/db` if `--provision` is passed. + +## Known gaps + +- **VXLAN MAC-per-VNI target**: the `vlan` join key used to correlate + VLAN→VNI mapping with FDB entries (query verbatim from #44) doesn't + actually match on live data for any VTEP node — likely an Arista + internal-VLAN-vs-front-panel-VLAN translation the OpenConfig paths don't + reconcile. Only affects the decorative per-VTEP tooltip metric, not + node/link status or traffic coloring. Tracked in #44, not fixed here. diff --git a/scripts/generate_weathermap.py b/scripts/generate_weathermap.py index 8014c7c..5540da1 100644 --- a/scripts/generate_weathermap.py +++ b/scripts/generate_weathermap.py @@ -20,8 +20,8 @@ Environment: GRAFANA_TOKEN required with --provision GRAFANA_DATASOURCE_UID Prometheus datasource UID in Grafana, required with --provision GRAFANA_DASHBOARD_UID default "evpn-vxlan-fabric-weathermap" - GRAFANA_WEATHERMAP_PLUGIN_ID default "allamiro-weathermap-panel" -- confirm - against the actually installed plugin id before provisioning. + GRAFANA_WEATHERMAP_PLUGIN_ID default "tamirsuliman-weathermap-panel" -- override + if a different weathermap-ng fork/plugin id is installed. """ import argparse import json @@ -458,7 +458,7 @@ def main(): dashboard_uid = env("GRAFANA_DASHBOARD_UID", "evpn-vxlan-fabric-weathermap") datasource_uid = env("GRAFANA_DATASOURCE_UID", "PROMETHEUS_DATASOURCE_UID_PLACEHOLDER") - plugin_id = env("GRAFANA_WEATHERMAP_PLUGIN_ID", "allamiro-weathermap-panel") + plugin_id = env("GRAFANA_WEATHERMAP_PLUGIN_ID", "tamirsuliman-weathermap-panel") dashboard_payload = build_dashboard(weathermap, targets, dashboard_uid, datasource_uid, plugin_id) os.makedirs(os.path.dirname(args.output) or ".", exist_ok=True)