Merge pull request 'Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator' (#59 ) from feat/telemetry into main

Reviewed-on: #59
Fix scripts/README.md doc drift after iperf3/weathermap direction fixes
2026-07-20 16:59:09 +00:00 · 2026-07-20 16:57:28 +00:00 · 2026-07-20 16:37:16 +00:00 · 2026-07-20 16:31:33 +00:00 · 2026-07-20 16:15:19 +00:00 · 2026-07-20 16:15:09 +00:00
7 changed files with 2758 additions and 2438 deletions
--- a/assets/grafana-weathermap-dashboard.png
+++ b/assets/grafana-weathermap-dashboard.png
--- a/assets/weathermap-pipeline.svg
+++ b/assets/weathermap-pipeline.svg
@@ -0,0 +1,72 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 350" font-family="Helvetica, Arial, sans-serif">
+  <defs>
+    <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
+      <path d="M 0 0 L 10 5 L 0 10 z" fill="#555"/>
+    </marker>
+  </defs>
+
+  <style>
+    .box { stroke-width: 1.5; rx: 6; }
+    .input { fill: #e8f0fe; stroke: #4285f4; }
+    .manual { fill: #fef7e0; stroke: #f9ab00; }
+    .script { fill: #e6f4ea; stroke: #34a853; }
+    .output { fill: #f3e8fd; stroke: #a142f4; }
+    .grafana { fill: #fce8e6; stroke: #ea4335; }
+    .label { font-size: 13px; fill: #202124; }
+    .sublabel { font-size: 10px; fill: #5f6368; }
+    .arrow { stroke: #555; stroke-width: 1.5; fill: none; marker-end: url(#arrow); }
+    .caption { font-size: 11px; fill: #5f6368; font-style: italic; }
+  </style>
+
+  <!-- inputs -->
+  <rect class="box input" x="20" y="30" width="150" height="50"/>
+  <text class="label" x="95" y="50" text-anchor="middle">IPFabric</text>
+  <text class="sublabel" x="95" y="66" text-anchor="middle">topology, inventory</text>
+
+  <rect class="box input" x="20" y="100" width="150" height="50"/>
+  <text class="label" x="95" y="120" text-anchor="middle">Prometheus</text>
+  <text class="sublabel" x="95" y="136" text-anchor="middle">gnmic metrics, live labels</text>
+
+  <!-- generator script -->
+  <rect class="box script" x="220" y="55" width="200" height="70"/>
+  <text class="label" x="320" y="82" text-anchor="middle" font-size="11.5">generate_weathermap.py</text>
+  <text class="sublabel" x="320" y="98" text-anchor="middle">weathermap panel only</text>
+  <text class="sublabel" x="320" y="112" text-anchor="middle">targets + options.weathermap</text>
+
+  <path class="arrow" d="M170,55 L220,80"/>
+  <path class="arrow" d="M170,125 L220,100"/>
+
+  <!-- manual base -->
+  <rect class="box manual" x="220" y="220" width="200" height="70"/>
+  <text class="label" x="320" y="247" text-anchor="middle">dashboard-base.json</text>
+  <text class="sublabel" x="320" y="263" text-anchor="middle">hand-authored, committed</text>
+  <text class="sublabel" x="320" y="277" text-anchor="middle">BGP/ports/throughput/vars</text>
+
+  <!-- merge -->
+  <rect class="box script" x="470" y="130" width="160" height="60"/>
+  <text class="label" x="550" y="155" text-anchor="middle">merge</text>
+  <text class="sublabel" x="550" y="171" text-anchor="middle" font-size="8.5">splices into __WEATHERMAP_SLOT__</text>
+
+  <path class="arrow" d="M420,90 L470,145"/>
+  <path class="arrow" d="M420,255 L470,175"/>
+
+  <!-- output artifact -->
+  <rect class="box output" x="660" y="60" width="210" height="60"/>
+  <text class="label" x="765" y="85" text-anchor="middle" font-size="11.5">weathermap-dashboard.json</text>
+  <text class="sublabel" x="765" y="101" text-anchor="middle">generated build artifact</text>
+
+  <path class="arrow" d="M630,150 L765,120"/>
+
+  <!-- grafana -->
+  <rect class="box grafana" x="660" y="210" width="210" height="60"/>
+  <text class="label" x="765" y="235" text-anchor="middle">Grafana</text>
+  <text class="sublabel" x="765" y="251" text-anchor="middle" font-size="9">--provision (POST /api/dashboards/db)</text>
+
+  <path class="arrow" d="M630,175 L765,210"/>
+  <path class="arrow" d="M765,120 L765,210" stroke-dasharray="3,3"/>
+
+  <!-- read-back for position preservation -->
+  <path class="arrow" d="M765,270 C 620,330 490,300 420,275" stroke-dasharray="3,3"/>
+  <text class="caption" x="545" y="325" text-anchor="middle">live node positions read back before each regeneration (see #53)</text>
+
+</svg>
--- a/configs/grafana/weathermap-dashboard.json
+++ b/configs/grafana/weathermap-dashboard.json
--- a/envrc.sample
+++ b/envrc.sample
@@ -1,4 +1,11 @@
 export GRAFANA_URL=""
 export GRAFANA_TOKEN=""
+export GRAFANA_DATASOURCE_UID=""
 export IPFABRIC_URL=""
 export IPFABRIC_TOKEN=""
+
+# Optional -- see scripts/README.md for defaults and when to override
+# export IPFABRIC_SNAPSHOT="$last"
+# export PROMETHEUS_URL="http://172.16.0.71:9090"
+# export GRAFANA_DASHBOARD_UID="evpn-vxlan-fabric-weathermap"
+# export GRAFANA_WEATHERMAP_PLUGIN_ID="tamirsuliman-weathermap-panel"
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -1,111 +1,167 @@
 # scripts/generate_weathermap.py

-Generates a [weathermap-ng](https://github.com/allamiro/grafana-network-weathermap-ng)
-**panel only** from live IPFabric topology + gnmic/Prometheus metrics, merges
-it into a manually-authored dashboard base, and writes the merged result to
-`configs/grafana/weathermap-dashboard.json` (committed to Gitea as the build
-artifact, same pattern as the retired Flow Panel YAML). Optionally provisions
-it into Grafana via the HTTP API.
+Generates the **weathermap panel only** from live IPFabric + Prometheus
+data, merges it into a hand-authored dashboard base, and provisions the
+result to Grafana.

-## File structure (see #52)
+![pipeline diagram](../assets/weathermap-pipeline.svg)

-| File | What it is |
-|---|---|
-| `configs/grafana/dashboard-base.json` | **Manually authored**, committed source of truth for everything except the weathermap panel: BGP sessions table, ports/interfaces table, throughput-per-site graphs, `site`/`device` template variables, panel layout. Edit this directly for anything in those panels. Contains a reserved slot panel titled `__WEATHERMAP_SLOT__` (id `1`, `gridPos` fixed) that the script splices the generated weathermap panel into. |
-| `configs/grafana/weathermap-dashboard.json` | **Generated build artifact** — base + freshly-generated weathermap panel, merged. This is what actually gets provisioned to Grafana. Don't hand-edit; re-run the script. |
+## Files

-The script itself only ever knows about the weathermap panel (`targets` +
-`options.weathermap`) — it has no knowledge of the other three panels or the
-template variables. That split is deliberate: topology-driven content
-(nodes/links, generated from live IPFabric+Prometheus data) is separated from
-hand-tuned dashboard composition (table layout, transformations, thresholds),
-which doesn't need to regenerate on every topology change.
+| File                                        | What it is                                                                                                                                                                                      | Edit it?                |
+| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
+| `configs/grafana/dashboard-base.json`       | Hand-authored: BGP table, ports table, throughput graphs, `site`/`device` variables, layout. Has one reserved slot panel (`title: "__WEATHERMAP_SLOT__"`) where the weathermap gets spliced in. | Yes, directly           |
+| `configs/grafana/weathermap-dashboard.json` | Generated build artifact — base + fresh weathermap panel, merged. What actually gets provisioned.                                                                                               | No — regenerate instead |

-See issue #48 for the panel schema research this is built against, and its
-comment thread for the live-validation history (auth quirks, plugin id, a
-PromQL escaping bug, a panel-crash fix — worth reading before touching the
-anchor/regex-escaping logic).
+Why split like this: topology data (nodes/links) changes with the lab and
+should regenerate every time; the rest of the dashboard (table layout,
+thresholds, transformations) is hand-tuned and shouldn't be touched by a
+topology refresh. See #52.

-## Usage
+## Quickstart

 ```bash
-export IPFABRIC_URL=https://<ipfabric-instance>
-export IPFABRIC_TOKEN=<token>
-python3 scripts/generate_weathermap.py                 # writes the merged JSON only
+cp envrc.sample .envrc   # fill in the values (see "Credentials" below)
+source .envrc

-export GRAFANA_URL=https://<external-grafana-instance>
-export GRAFANA_TOKEN=<token>
-export GRAFANA_DATASOURCE_UID=<prometheus-datasource-uid-in-grafana>
-python3 scripts/generate_weathermap.py --provision      # also provisions via the Grafana API
+python3 scripts/generate_weathermap.py              # dry run: writes the JSON only
+python3 scripts/generate_weathermap.py --provision   # also pushes it to Grafana
 ```

-Re-run after any topology change (`evpn-lab.clab.yml`) to regenerate the
-weathermap panel and re-merge/re-provision. The script always regenerates the
-weathermap panel fresh from live IPFabric data (no diffing/incremental
-state), so a plain re-run already picks up any topology change — there's no
-separate "detect changes" step. The only manual/operational step is
-*triggering* that re-run after a change; there's no watcher or CI hook doing
-it automatically yet.
+Re-run the same `--provision` command any time the lab topology changes —
+the script always rebuilds the weathermap panel from live data, so there's
+nothing to "detect", just re-run it. It's not automated yet (no watcher/CI
+hook), so re-running is a manual step for now.

-If you need to change the BGP/ports/throughput panels, template variables, or
-layout, edit `configs/grafana/dashboard-base.json` directly and re-run the
-script (or run it with `--provision` to push the edit live) — no topology
-data is needed for that path, the weathermap panel just gets regenerated
-alongside it.
+## Credentials
+
+| Value                         | Env var                                                                 | Get it from the UI                                                                                    | Get it via API                                                                                               |
+| ----------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
+| IPFabric token                | `IPFABRIC_TOKEN`                                                        | IPFabric → user menu → Settings → API Tokens → Add token                                              | not possible, UI-only                                                                                        |
+| Grafana service account token | `GRAFANA_TOKEN`                                                         | Grafana → Administration → Service accounts → Add → give it **Editor** role → Add token               | possible but multi-step (create account, then create token under it) — UI is simpler                         |
+| Prometheus datasource UID     | `GRAFANA_DATASOURCE_UID`                                                | Grafana → Connections → Data sources → open the one with gnmic data → UID is the last part of the URL | `curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" "$GRAFANA_URL/api/datasources" \| jq '.[] \| {name,uid}'` |
+| Weathermap plugin ID          | `GRAFANA_WEATHERMAP_PLUGIN_ID` (optional, only if not the default fork) | Grafana → Administration → Plugins → search "weathermap" → id is in the URL                           | —                                                                                                            |
+
+⚠️ The datasource must be the one that can actually see gnmic metrics —
+picking an unrelated Prometheus instance renders the panel with no data
+(see #48).
+
+## Verifying the result
+
+**In the browser** (the only way to actually see it render — colors,
+tables, and the ports-table merge transformation all happen client-side):
+
+1. Open `$GRAFANA_URL/d/evpn-vxlan-fabric-weathermap`.
+2. Weathermap: nodes grouped spine-top → access-bottom, all green.
+3. `site`/`device` dropdowns filter the two tables below.
+4. BGP Sessions / Ports tables: populated, green=Up / red=Down.
+5. Throughput per Site: signed graph, non-zero values.
+
+**Without a browser**, use Grafana's **Explore** view (paste a PromQL
+expression from the panel JSON and run it) — same result as the UI check
+above minus the rendering, no `curl` needed. Or hit the API directly:
+
+```bash
+curl -s -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H "Content-Type: application/json" \
+  "$GRAFANA_URL/api/ds/query" -d '{
+    "queries": [{"refId":"A","datasource":{"type":"prometheus","uid":"'"$GRAFANA_DATASOURCE_UID"'"},
+                 "expr":"min by (device) (interfaces_interface_state_oper_status)","instant":true}],
+    "from":"now-5m","to":"now"}' | python3 -m json.tool
+```
+
+## Troubleshooting
+
+| Symptom                                                    | Cause                                                               | Check                                                              |
+| ---------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------ |
+| `Refusing to provision: set GRAFANA_DATASOURCE_UID...`     | Env var unset                                                       | `echo $GRAFANA_DATASOURCE_UID`                                     |
+| Panel renders with no data                                 | Wrong datasource — can't see gnmic metrics                          | Explore → run `up` against it, confirm gnmic series come back      |
+| `Base dashboard has no panel titled '__WEATHERMAP_SLOT__'` | Someone edited `dashboard-base.json` and renamed/removed that panel | Check panel `id: 1` still has that exact title                     |
+| Manually-moved node keeps resetting position               | `GRAFANA_URL`/`GRAFANA_TOKEN` weren't set for that run              | Look for `Fetching live node positions from ...` in the run output |
+| `HTTP 401/403` fetching live positions                     | Expired/bad Grafana token                                           | Re-issue the service account token                                 |

 ## Environment variables

-| Variable | Required | Default | Notes |
-|---|---|---|---|
-| `--base` (CLI flag, not env) | no | `configs/grafana/dashboard-base.json` | Manual dashboard base to merge the weathermap panel into. |
-| `IPFABRIC_URL` | yes | — | e.g. `https://ipfabric.example.com` |
-| `IPFABRIC_TOKEN` | yes | — | Inventory/Snapshots read access. Sent as `X-API-Token`, not `Authorization: Bearer` — IPFabric's REST API does not use bearer auth. |
-| `IPFABRIC_SNAPSHOT` | no | `$last` | |
-| `PROMETHEUS_URL` | no | `http://172.16.0.71:9090` | The **in-topology** Prometheus (see #45), used at generation time to validate the IPFabric→gnmic interface alias and to discover live VTEP/VLAN pairs. This does not have to be the same instance Grafana queries at render time — see below. |
-| `GRAFANA_URL` | with `--provision` | — | |
-| `GRAFANA_TOKEN` | with `--provision` | — | Grafana Service Account token, sent as `Authorization: Bearer`. |
-| `GRAFANA_DATASOURCE_UID` | with `--provision` | — | UID of the Prometheus datasource in Grafana that will actually back the panel. **This must be a datasource that can see the gnmic metrics** — an unrelated/external Prometheus instance with no gnmic data will make the panel render with no values (this happened once, see #48). |
-| `GRAFANA_DASHBOARD_UID` | no | `evpn-vxlan-fabric-weathermap` | |
-| `GRAFANA_WEATHERMAP_PLUGIN_ID` | no | `tamirsuliman-weathermap-panel` | The installed weathermap-ng plugin id. Override if a different fork is installed — plugin ids don't always match the upstream repo name (the schema research in #48 was done against the `allamiro` fork's source; what's actually installed here is a different fork with a stricter/different runtime schema — see the `ANCHOR` handling in the script). |
+| Variable                       | Required           | Default                               |
+| ------------------------------ | ------------------ | ------------------------------------- |
+| `IPFABRIC_URL`                 | yes                | —                                     |
+| `IPFABRIC_TOKEN`               | yes                | —                                     |
+| `IPFABRIC_SNAPSHOT`            | no                 | `$last`                               |
+| `PROMETHEUS_URL`               | no                 | `http://172.16.0.71:9090`             |
+| `GRAFANA_URL`                  | no*                | —                                     |
+| `GRAFANA_TOKEN`                | no*                | —                                     |
+| `GRAFANA_DATASOURCE_UID`       | with `--provision` | —                                     |
+| `GRAFANA_DASHBOARD_UID`        | no                 | `evpn-vxlan-fabric-weathermap`        |
+| `GRAFANA_WEATHERMAP_PLUGIN_ID` | no                 | `tamirsuliman-weathermap-panel`       |
+| `--base` (CLI flag)            | no                 | `configs/grafana/dashboard-base.json` |

-## What the script does
+\* `GRAFANA_URL`/`GRAFANA_TOKEN` are required for `--provision`, and
+optional otherwise — if set on a plain (non-`--provision`) run, they're
+used to read back live node positions so manual repositioning survives
+the next regeneration (see #53). Skip them and every node just gets the
+default layout.

-1. Fetches device inventory + connectivity-matrix from IPFabric.
-2. Filters the connectivity-matrix to physical Ethernet-to-Ethernet links
-   only: drops Management-plane neighbor entries and `.100`/`.200`
-   subinterface rows (802.1Q tags used for the gold VRF stitching on Core —
-   they ride the same physical port as their parent interface and gnmic only
-   exports physical interface counters), and dedupes the two directions
-   IPFabric reports for each physical link into one.
-3. Computes node positions as a simple site-grouped grid (dc/core/campus
-   bands) — IPFabric has no layout data.
-4. Builds the panel's `targets`: one PromQL query per metric family (BGP
-   status, interface tx, interface rx, VXLAN MAC/VNI), each with an explicit
-   `legendFormat` so the resolved display name is predictable.
-5. Builds `nodes[]` and `links[]` referencing those resolved legend strings,
-   including the plugin's `anchors` tally (per-node count of link
-   attachments per side) and numeric anchor enum on each link side —
-   omitting these crashes the panel on load in the installed plugin fork,
-   despite the (fork-specific) schema research saying it's safe to skip.
-6. Cross-checks every IPFabric interface name, aliased to gnmic's naming
-   (`Et`→`Ethernet`, `Po`→`Port-Channel`, `Lo`→`Loopback`, `Vl`→`Vlan`,
-   `Ma`→`Management`), against the live exporter. Any link whose aliased
-   name has no matching series is logged, not silently dropped — the actual
-   fix for a real mismatch belongs in gnmic interface aliasing (#43), not in
-   this script.
-7. Loads `configs/grafana/dashboard-base.json`, substitutes the real
-   Prometheus datasource UID into its `__DATASOURCE_UID__` placeholders,
-   finds the panel titled `__WEATHERMAP_SLOT__` and splices in the generated
-   weathermap panel content (keeping the slot's `gridPos`/`id`, so the manual
-   layout is never repositioned).
-8. Writes the merged dashboard JSON to `configs/grafana/weathermap-dashboard.json`,
-   and provisions it via `POST /api/dashboards/db` if `--provision` is passed.
+## How it works
+
+1. Fetch device inventory + connectivity-matrix from IPFabric.
+2. Keep only physical Ethernet↔Ethernet links (drop management-plane and
+   `.100`/`.200` subinterface rows), dedupe the two directions IPFabric
+   reports per link.
+3. Position each node:
+    - **New node** → layered default: Y-tier by role parsed from the
+      hostname (`spine → core → border-leaf → leaf → access`), X grouped
+      by site within the tier. A hostname that matches no role is logged
+      and dropped into a fallback tier, never silently misplaced (#53).
+    - **Known node** → whatever position is currently live in Grafana,
+      unchanged. This is why `GRAFANA_URL`/`GRAFANA_TOKEN` matter even on
+      a dry run (#53).
+4. Build the panel's PromQL `targets`: node status, per-side link tx (each
+   side its own egress counter, so both directions are represented — #57),
+   VXLAN MAC/VNI tooltip — each with an explicit `legendFormat`.
+5. Build `nodes[]`/`links[]`, including the plugin's `anchors` tally
+   (link count per side) — required by the installed plugin fork even
+   though the schema doc says it's optional (#48).
+6. Cross-check every interface name against the live gnmic exporter
+   (`Et`→`Ethernet`, `Po`→`Port-Channel`, etc.); mismatches are logged,
+   not dropped — the link stays, just without data until fixed upstream.
+7. Load `dashboard-base.json`, substitute the real datasource UID, splice
+   the generated panel into `__WEATHERMAP_SLOT__` (keeping that slot's
+   `gridPos`/`id` — layout stays manual).
+8. Write `weathermap-dashboard.json`; provision it if `--provision`.

 ## Known gaps

- **VXLAN MAC-per-VNI target**: the `vlan` join key used to correlate
-  VLAN→VNI mapping with FDB entries (query verbatim from #44) doesn't
-  actually match on live data for any VTEP node — likely an Arista
-  internal-VLAN-vs-front-panel-VLAN translation the OpenConfig paths don't
-  reconcile. Only affects the decorative per-VTEP tooltip metric, not
-  node/link status or traffic coloring. Tracked in #44, not fixed here.
+- **VXLAN MAC-per-VNI tooltip**: the `vlan` join key doesn't match on live
+  data for any VTEP node, likely an Arista internal-vs-front-panel VLAN
+  translation gap. Only affects that one decorative tooltip metric, not
+  node/link status or traffic coloring. Tracked in #44.
+
+# scripts/generate_traffic.sh
+
+Generates real DC↔Campus traffic over VRF `gold` using `iperf3` (bundled
+in the `network-multitool` image every host container runs), with a live
+bandwidth dashboard. Without this, host containers sit idle and IPFabric
+ARP/MAC tables, Grafana throughput graphs, and the weathermap panel stay
+empty until someone manually generates traffic (see #55).
+
+## Quickstart
+
+```bash
+./scripts/generate_traffic.sh <duration_seconds>
+```
+
+## How it works
+
+- Starts `iperf3 -s` on the DC gold-VRF servers: `dc-server2`
+  (10.34.34.102), `dc-server4` (10.78.78.104).
+- Runs `iperf3 -c -R` from the paired campus gold-VRF hosts, reversing the
+  stream so the DC server pushes to the campus consumer: `dc-server2` →
+  `campus-host1`, `dc-server4` → `campus-host2` — exercising the full
+  DC→Core→Campus stitched EVPN Type-5 path end to end.
+- Redraws a terminal dashboard every second for the run duration: server
+  list, and live Mbits/sec per client session parsed from `iperf3 -i 1`
+  output.
+- On exit (duration end or Ctrl-C), kills the client processes and stops
+  the `iperf3 -s` processes on the DC servers — no leftover state.
+
+`dc-server1`/`dc-server3` (VLAN 40, VRF default, no gateway) are out of
+scope — this script only exercises the routed gold VRF path.
--- a/scripts/generate_traffic.sh
+++ b/scripts/generate_traffic.sh
@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Generates DC<->Campus traffic over VRF gold using iperf3 (bundled in the
+# network-multitool image every host container runs), with a live
+# server/client bandwidth dashboard. Refs #55.
+set -euo pipefail
+
+DURATION="${1:?Usage: $0 <duration_seconds>}"
+if ! [[ "$DURATION" =~ ^[0-9]+$ ]] || [[ "$DURATION" -lt 1 ]]; then
+    echo "Duration must be a positive integer (seconds)" >&2
+    exit 1
+fi
+
+LAB_PREFIX="clab-arista-evpn-fabric"
+PORT=5301
+
+# server_name:server_ip:client_name  — gold VRF pairs, stitched EVPN
+# Type-5 path DC -> Core -> Campus (see README Host Addressing table)
+PAIRS=(
+    "dc-server2:10.34.34.102:campus-host1"
+    "dc-server4:10.78.78.104:campus-host2"
+)
+
+WORKDIR="$(mktemp -d)"
+CLIENT_PIDS=()
+
+cleanup() {
+    for pid in "${CLIENT_PIDS[@]:-}"; do
+        kill "$pid" >/dev/null 2>&1 || true
+    done
+    for pair in "${PAIRS[@]}"; do
+        IFS=':' read -r server _ _ <<<"$pair"
+        docker exec "${LAB_PREFIX}-${server}" pkill -f "iperf3 -s -p ${PORT}" >/dev/null 2>&1 || true
+    done
+    rm -rf "$WORKDIR"
+}
+trap cleanup EXIT INT TERM
+
+echo "Starting iperf3 servers..."
+for pair in "${PAIRS[@]}"; do
+    IFS=':' read -r server server_ip _ <<<"$pair"
+    docker exec -d "${LAB_PREFIX}-${server}" iperf3 -s -p "$PORT"
+done
+sleep 1
+
+echo "Starting iperf3 clients for ${DURATION}s..."
+for pair in "${PAIRS[@]}"; do
+    IFS=':' read -r server server_ip client <<<"$pair"
+    logfile="${WORKDIR}/${client}.log"
+    # -R: DC hosts the service, campus is the consumer -- data should flow
+    # server -> client (download), not client -> server, to match how a
+    # real DC-hosted service/campus-consumer pair behaves.
+    docker exec "${LAB_PREFIX}-${client}" iperf3 -c "$server_ip" -p "$PORT" -R \
+        -t "$DURATION" -i 1 --forceflush -f m >"$logfile" 2>&1 &
+    CLIENT_PIDS+=("$!")
+done
+
+for ((elapsed = 0; elapsed <= DURATION; elapsed++)); do
+    clear
+    echo "EVPN/VXLAN lab traffic generator — ${elapsed}/${DURATION}s"
+    echo
+    echo "Servers (iperf3 -s):"
+    for pair in "${PAIRS[@]}"; do
+        IFS=':' read -r server server_ip _ <<<"$pair"
+        echo "  ${server} (${server_ip}:${PORT})"
+    done
+    echo
+    echo "Clients (live bandwidth):"
+    for pair in "${PAIRS[@]}"; do
+        IFS=':' read -r server server_ip client <<<"$pair"
+        logfile="${WORKDIR}/${client}.log"
+        last_line="$(grep -E 'Mbits/sec' "$logfile" 2>/dev/null | tail -1 || true)"
+        bw="$(sed -E 's/.*[[:space:]]([0-9.]+ Mbits\/sec).*/\1/' <<<"$last_line")"
+        [[ -z "$last_line" ]] && bw="waiting..."
+        printf "  %-13s -> %-13s : %s\n" "$server" "$client" "$bw"
+    done
+    sleep 1
+done
+
+wait "${CLIENT_PIDS[@]}" 2>/dev/null || true
+
+echo
+echo "Done. Summary:"
+for pair in "${PAIRS[@]}"; do
+    IFS=':' read -r server _ client <<<"$pair"
+    logfile="${WORKDIR}/${client}.log"
+    summary="$(grep -E 'receiver' "$logfile" 2>/dev/null || true)"
+    echo "  ${server} -> ${client}: ${summary:-no data}"
+done
--- a/scripts/generate_weathermap.py
+++ b/scripts/generate_weathermap.py
@@ -4,7 +4,7 @@ metrics, merge it into a manually-authored dashboard base, and optionally
 provision the result into Grafana.

 Scope (see #52): this script knows only about the weathermap panel --
-targets (node status / link tx / link rx / VXLAN tooltip queries) and
+targets (node status / link tx per side / VXLAN tooltip queries) and
 options.weathermap (nodes/links/scale/settings). It has no knowledge of the
 BGP sessions table, ports/interfaces table, or throughput panels -- those
 live in the manually-authored `configs/grafana/dashboard-base.json` and are
@@ -61,7 +61,25 @@ SITE_ORDER = ["dc", "core", "campus"]
 GRID_COLUMNS = 6
 GRID_SPACING_X = 180
 GRID_SPACING_Y = 150
-SITE_BAND_Y = {"dc": 0, "core": 450, "campus": 750}
+
+# Layered default layout (see #53): device role is parsed from the hostname
+# naming convention, same trust level as the `site` parsing already in place
+# for the Prometheus relabel (#49) -- not IPFabric-derived, not configurable.
+# Checked in this order so e.g. "campus-border-leaf1" matches border-leaf
+# before the more general leaf pattern.
+ROLE_PATTERNS = [
+    ("spine", re.compile(r"-spine")),
+    ("core", re.compile(r"^core")),
+    ("border-leaf", re.compile(r"-border-leaf")),
+    ("leaf", re.compile(r"-leaf")),
+    ("access", re.compile(r"-access")),
+]
+ROLE_TIER_ORDER = ["spine", "core", "border-leaf", "leaf", "access"]
+TIER_SPACING_Y = 300
+# Per-site X band start, wide enough that the largest role/site group (8 dc
+# leafs) doesn't spill into the next site's band at GRID_COLUMNS=6.
+SITE_X_OFFSET = {"dc": 100, "core": 1300, "campus": 1600}
+UNMATCHED_ROLE_TIER_Y = len(ROLE_TIER_ORDER) * TIER_SPACING_Y + 300

 # tamirsuliman-weathermap-panel's numeric anchor enum (Center=0, Top=1,
 # Bottom=2, Left=3, Right=4) -- reverse-engineered from module.js, since
@@ -181,27 +199,49 @@ def promql_escape(s):
 # Topology processing
 # --------------------------------------------------------------------------

-def build_layout(devices):
-    by_site = {}
+def parse_role(hostname):
+    """Device role from the hostname naming convention -- see ROLE_PATTERNS.
+    Returns None if nothing matches (logged by the caller, not silently
+    defaulted into the wrong tier)."""
+    for role, pattern in ROLE_PATTERNS:
+        if pattern.search(hostname):
+            return role
+    return None
+
+
+def build_layout(devices, mismatches):
+    """Default layered layout for nodes with no live (Grafana-drag-and-drop)
+    position yet -- see #53. Y-tier by role (spine top, access bottom), X
+    grouped/columned by site within each tier so same-role devices from
+    different sites don't overlap."""
+    groups = {}  # (role, site) -> [hostname, ...]
+    unmatched = []
    for dev in devices:
-        by_site.setdefault(dev["siteName"], []).append(dev["hostname"])
+        host, site = dev["hostname"], dev["siteName"]
+        role = parse_role(host)
+        if role is None:
+            unmatched.append(host)
+            continue
+        groups.setdefault((role, site), []).append(host)

    positions = {}
-    for site in SITE_ORDER:
-        band_y = SITE_BAND_Y.get(site, 0)
-        for idx, host in enumerate(sorted(by_site.get(site, []))):
-            col, row = idx % GRID_COLUMNS, idx // GRID_COLUMNS
-            positions[host] = [100 + col * GRID_SPACING_X, band_y + row * GRID_SPACING_Y]
+    for role_idx, role in enumerate(ROLE_TIER_ORDER):
+        tier_y = role_idx * TIER_SPACING_Y
+        for site in SITE_ORDER:
+            x_offset = SITE_X_OFFSET.get(site, max(SITE_X_OFFSET.values()) + 300)
+            for idx, host in enumerate(sorted(groups.get((role, site), []))):
+                col, row = idx % GRID_COLUMNS, idx // GRID_COLUMNS
+                positions[host] = [x_offset + col * GRID_SPACING_X, tier_y + row * GRID_SPACING_Y]

-    # Any site not in SITE_ORDER (shouldn't happen given the naming convention,
-    # but don't silently drop nodes if it does) gets stacked below everything else.
-    extra_band_y = max(SITE_BAND_Y.values()) + 300
-    for site, hosts in by_site.items():
-        if site in SITE_ORDER:
-            continue
-        for idx, host in enumerate(sorted(hosts)):
+    if unmatched:
+        mismatches.append(
+            f"{len(unmatched)} hostname(s) matched no role pattern (spine/core/border-leaf/leaf/access), "
+            f"placed in a fallback tier instead of guessing: {', '.join(sorted(unmatched))}"
+        )
+        for idx, host in enumerate(sorted(unmatched)):
            col, row = idx % GRID_COLUMNS, idx // GRID_COLUMNS
-            positions[host] = [100 + col * GRID_SPACING_X, extra_band_y + row * GRID_SPACING_Y]
+            positions[host] = [100 + col * GRID_SPACING_X, UNMATCHED_ROLE_TIER_Y + row * GRID_SPACING_Y]
+
    return positions


@@ -274,7 +314,21 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
            "position": positions[host],
            "isConnection": False,
            "useConstantSpacing": False,
-            "compactVerticalLinks": False,
+            # True (not the plugin default) so rendered node HEIGHT is
+            # constant, decoupled from per-node link count -- see #54. Ground
+            # truth confirmed against the installed plugin's real module.js
+            # (not the #48 schema doc, which doesn't cover this): height is
+            # `fontSize + 2*padding.vertical` when compactVerticalLinks is
+            # true, unconditionally; when false, it's the larger of that or
+            # a term proportional to max(anchors[Left].numLinks,
+            # anchors[Right].numLinks) -- exactly why campus-leaf1 (3 links)
+            # and campus-leaf2 (4 links) rendered at different heights.
+            # Width is unaffected either way: it's always recomputed from
+            # the label text at render time (no override field exists), and
+            # useConstantSpacing only pulls in Top/Bottom anchor link count,
+            # which this generator never uses (links always attach
+            # Right/Left -- see anchor_counts above).
+            "compactVerticalLinks": True,
            "padding": {"horizontal": 12, "vertical": 6},
            "colors": dict(NODE_COLORS),
            "nodeIcon": None,
@@ -309,6 +363,12 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
        link_defs.append({
            "id": f"{link['a_host']}-{a_gnmic_int}--{link['z_host']}-{z_gnmic_int}",
            "nodes": [{"id": link["a_host"]}, {"id": link["z_host"]}],
+            # Each side's query is that side's own tx (egress) counter, not the
+            # far end's rx -- see #57. a_host tx and z_host rx both describe the
+            # *same* A->Z flow measured from opposite ends (the a_host->z_host
+            # direction, counted twice), leaving the Z->A direction never
+            # queried by either side. Using each node's own tx gives two
+            # independent, opposite-direction measurements instead.
            "sides": {
                "A": {
                    "bandwidth": a_bw,
@@ -317,7 +377,7 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
                },
                "Z": {
                    "bandwidth": z_bw,
-                    "query": f"{link['z_host']} {z_gnmic_int} rx",
+                    "query": f"{link['z_host']} {z_gnmic_int} tx",
                    "labelOffset": 55, "anchor": ANCHOR["Left"], "dashboardLink": "",
                },
            },
@@ -356,11 +416,6 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
            "expr": f'rate(interfaces_interface_state_counters_out_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8',
            "legendFormat": "{{device}} {{interface}} tx",
        },
-        {
-            "refId": "C",
-            "expr": f'rate(interfaces_interface_state_counters_in_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8',
-            "legendFormat": "{{device}} {{interface}} rx",
-        },
    ]
    if vtep_hosts:
        vtep_regex = "|".join(promql_escape(h) for h in vtep_hosts)
@@ -413,6 +468,7 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
 # --------------------------------------------------------------------------

 WEATHERMAP_SLOT_TITLE = "__WEATHERMAP_SLOT__"
+WEATHERMAP_PANEL_TITLE = "Fabric Weathermap"  # what the slot is renamed to on merge


 def build_weathermap_panel(weathermap, targets, datasource_uid, plugin_id):
@@ -461,7 +517,7 @@ def merge_weathermap_into_base(base_dashboard, weathermap_panel, dashboard_uid):
            found = True
            merged = dict(panel)
            merged.update(weathermap_panel)
-            merged["title"] = "Fabric Weathermap"
+            merged["title"] = WEATHERMAP_PANEL_TITLE
            panels.append(merged)
        else:
            panels.append(panel)
@@ -471,6 +527,39 @@ def merge_weathermap_into_base(base_dashboard, weathermap_panel, dashboard_uid):
    return dashboard


+def fetch_live_node_positions(grafana_url, api_token, dashboard_uid):
+    """Node positions as currently provisioned in Grafana, keyed by node id
+    -- see #53. Position is edited live via drag-and-drop in the Grafana UI,
+    not in the git-committed base file, so it's the one weathermap field that
+    must be sourced from live Grafana state rather than regenerated or read
+    from the manual base -- a manual repositioning must survive every rerun.
+
+    Returns {} if the dashboard doesn't exist yet (first-ever run) or has no
+    weathermap panel yet -- everything falls back to the default layered
+    layout in that case. Any other HTTP error is treated as a real
+    misconfiguration (e.g. a bad token) and raised, rather than silently
+    treated as "no dashboard" -- that would risk quietly discarding every
+    manual position on a run that should have failed loudly instead."""
+    req = urllib.request.Request(
+        f"{grafana_url.rstrip('/')}/api/dashboards/uid/{dashboard_uid}",
+        headers={"Authorization": f"Bearer {api_token}"},
+        method="GET",
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            payload = json.load(resp)
+    except urllib.error.HTTPError as e:
+        if e.code == 404:
+            return {}
+        sys.exit(f"Fetching live dashboard for position preservation failed: HTTP {e.code} {e.read().decode()}")
+
+    for panel in payload.get("dashboard", {}).get("panels", []):
+        if panel.get("title") == WEATHERMAP_PANEL_TITLE:
+            nodes = panel.get("options", {}).get("weathermap", {}).get("nodes", [])
+            return {n["id"]: n["position"] for n in nodes if "position" in n}
+    return {}
+
+
 def provision_to_grafana(grafana_url, api_token, dashboard_payload):
    req = urllib.request.Request(
        f"{grafana_url.rstrip('/')}/api/dashboards/db",
@@ -501,6 +590,11 @@ def main():
    ipfabric_token = env("IPFABRIC_TOKEN", required=True)
    snapshot = env("IPFABRIC_SNAPSHOT", "$last")
    prometheus_url = env("PROMETHEUS_URL", "http://172.16.0.71:9090")
+    dashboard_uid = env("GRAFANA_DASHBOARD_UID", "evpn-vxlan-fabric-weathermap")
+    datasource_uid = env("GRAFANA_DATASOURCE_UID", "PROMETHEUS_DATASOURCE_UID_PLACEHOLDER")
+    plugin_id = env("GRAFANA_WEATHERMAP_PLUGIN_ID", "tamirsuliman-weathermap-panel")
+    grafana_url = env("GRAFANA_URL")
+    grafana_token = env("GRAFANA_TOKEN")

    print(f"Fetching devices from IPFabric ({ipfabric_url}, snapshot={snapshot})...")
    devices = fetch_devices(ipfabric_url, ipfabric_token, snapshot)
@@ -514,21 +608,33 @@ def main():
    print("Fetching interface speeds...")
    interface_speeds = fetch_interface_speeds(ipfabric_url, ipfabric_token, snapshot)

-    positions = build_layout(devices)
-
    mismatches = []
+    default_positions = build_layout(devices, mismatches)
+
+    # Position specifically is edited live in the Grafana UI (drag-and-drop),
+    # not in the git-committed base file -- see #53. Reuse whatever's live
+    # for nodes that already exist there; only brand-new nodes get the
+    # layered default. Iterating over default_positions (this run's device
+    # set) rather than the live map means a removed device's stale live
+    # position is simply never looked up again, no orphaned entry survives.
+    if grafana_url and grafana_token:
+        print(f"Fetching live node positions from {grafana_url} (preserve manual repositioning)...")
+        live_positions = fetch_live_node_positions(grafana_url, grafana_token, dashboard_uid)
+        print(f"  {len(live_positions)} node(s) with an existing live position")
+    else:
+        print("GRAFANA_URL/GRAFANA_TOKEN not set -- skipping live position fetch, using layered default for all nodes", file=sys.stderr)
+        live_positions = {}
+    positions = {host: live_positions.get(host, default) for host, default in default_positions.items()}
+
    print(f"Cross-checking interface names against live exporter ({prometheus_url})...")
    weathermap, targets = build_weathermap(devices, links, interface_speeds, positions, prometheus_url, mismatches)

    if mismatches:
-        print(f"\n{len(mismatches)} interface-name / bandwidth mismatch(es) found (link kept, not dropped):", file=sys.stderr)
+        print(f"\n{len(mismatches)} mismatch(es) found (link/position kept, not dropped):", file=sys.stderr)
        for m in mismatches:
            print(f"  - {m}", file=sys.stderr)
        print(file=sys.stderr)

-    dashboard_uid = env("GRAFANA_DASHBOARD_UID", "evpn-vxlan-fabric-weathermap")
-    datasource_uid = env("GRAFANA_DATASOURCE_UID", "PROMETHEUS_DATASOURCE_UID_PLACEHOLDER")
-    plugin_id = env("GRAFANA_WEATHERMAP_PLUGIN_ID", "tamirsuliman-weathermap-panel")
    weathermap_panel = build_weathermap_panel(weathermap, targets, datasource_uid, plugin_id)

    print(f"Loading manual dashboard base ({args.base})...")
@@ -545,8 +651,8 @@ def main():
          f"{len(dashboard['panels'])} panels total)")

    if args.provision:
-        grafana_url = env("GRAFANA_URL", required=True)
-        grafana_token = env("GRAFANA_TOKEN", required=True)
+        if not (grafana_url and grafana_token):
+            sys.exit("Refusing to provision: GRAFANA_URL and GRAFANA_TOKEN must both be set")
        if datasource_uid == "PROMETHEUS_DATASOURCE_UID_PLACEHOLDER":
            sys.exit("Refusing to provision: set GRAFANA_DATASOURCE_UID to the real Prometheus datasource UID first")
        print(f"Provisioning dashboard '{dashboard_uid}' to {grafana_url}...")
Author	SHA1	Message	Date
Damien Arnodo	31014bdfdf	Merge pull request 'Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator' (#59 ) from feat/telemetry into main Reviewed-on: #59	2026-07-20 16:59:09 +00:00
Damien Arnodo	5e5d6b45df	Fix scripts/README.md doc drift after iperf3/weathermap direction fixes Traffic direction docs still described the pre-fix client->server iperf3 flow and the pre-fix tx/rx weathermap query pair. Refs #57, Refs #58	2026-07-20 16:57:28 +00:00
Damien Arnodo	4719a77f21	Reverse iperf3 traffic direction to DC->campus (server pushes to client) Without -R, iperf3's client sends and the server only receives, so traffic ran campus-host -> dc-server -- backwards for what these roles represent: dc-serverN is the DC-hosted service, campus-hostN is the consumer, and a real service delivers data to the consumer rather than just absorbing uploads from it. Refs #58	2026-07-20 16:37:16 +00:00
Damien Arnodo	08124c8a1c	Fix weathermap links to show both traffic directions Z side queried z_host's rx, which measures the same A->Z flow as A side's tx (same traffic, counted at each end) -- the Z->A direction was never queried by either side. Use z_host's own tx instead, so each side surfaces its own (opposite) direction. Confirmed against the iperf3 traffic generator: DC access-leaf links read ~500Kbps (ACK-only) before the fix despite ~18-20Mbps of real downstream traffic to dc-server2/4, because that flow ran leaf->access (Z->A), the unqueried direction. Campus links looked correct only by coincidence -- campus hosts are the iperf3 clients, so their upload traffic happens to run access->leaf (A->Z), the direction that was queried. Refs #57	2026-07-20 16:31:33 +00:00
Damien Arnodo	8349791630	Merge pull request 'Add iperf3 traffic generator for DC<->Campus gold VRF path' (#56 ) from feat/iperf-traffic-generator into feat/telemetry Reviewed-on: #56	2026-07-20 16:15:19 +00:00
Damien Arnodo	3c3b28c987	Merge branch 'feat/telemetry' into feat/iperf-traffic-generator	2026-07-20 16:15:09 +00:00
Damien Arnodo	5b12894289	Add iperf3 traffic generator for DC<->Campus gold VRF path Host containers sit idle otherwise, leaving IPFabric ARP/MAC tables and Grafana throughput graphs empty until someone manually generates traffic. Refs #55	2026-07-20 16:09:39 +00:00
Damien	0f3aa27566	Add Grafana weathermap dashboard asset	2026-07-10 15:35:12 +02:00
Damien Arnodo	1cbea38522	Fix read-back arrow to terminate on dashboard-base.json's edge, not its middle	2026-07-10 13:22:56 +00:00
Damien Arnodo	64a8287518	Update scripts README table formatting	2026-07-10 13:19:43 +00:00
Damien Arnodo	045179e098	Reword scripts/README.md for readability, add pipeline diagram Same content, restructured for scanning: tables instead of long paragraphs (credentials, troubleshooting, env vars), shorter sentences, deep rationale trimmed to a phrase + issue reference instead of inline essays. Cut from 258 to 135 lines. Added assets/weathermap-pipeline.svg showing the IPFabric/Prometheus -> generator -> merge (with the manual base) -> Grafana flow, referenced at the top of the README.	2026-07-10 13:18:05 +00:00
Damien Arnodo	998014d144	Update Grafana weathermap dashboard node positions	2026-07-10 13:08:43 +00:00
Damien Arnodo	8fb83b9446	Add end-to-end how-to guide + regenerate dashboard from a clean reset Full reset validation: deleted weathermap-dashboard.json and the live Grafana dashboard (dashboard-base.json untouched, it's the manual part). Regenerated from scratch and reprovisioned -- confirms the first-ever-run path (0 live positions found, full layered layout applied, all 4 panels' queries resolve). Documented the whole flow in scripts/README.md as a start-to-finish how-to: credential acquisition (UI steps for the fiddly API paths -- Grafana service account token, datasource UID, plugin id), dry run, provision, verification (UI walkthrough + API/Explore alternative), re-running after topology changes, and a troubleshooting table. Added GRAFANA_DATASOURCE_UID and the optional env vars to envrc.sample.	2026-07-10 12:58:15 +00:00
Damien Arnodo	9b53ab0683	Fix inconsistent node rectangle height via compactVerticalLinks Confirmed root cause against the installed plugin's real module.js (not the #48 schema doc, which doesn't cover this): node height is driven by max(anchors[Left].numLinks, anchors[Right].numLinks) whenever compactVerticalLinks is false, which is why campus-leaf1 (3 links) and campus-leaf2 (4 links) rendered at different heights despite the same role. Width is unaffected by link count -- it's always recomputed from the label text at render time, no override field exists. Set compactVerticalLinks: true on every generated node, which makes height a constant (fontSize + 2*padding.vertical) regardless of link count, for every node (padding/fontSize are already uniform fabric-wide). Refs #54	2026-07-10 12:50:12 +00:00
Damien Arnodo	fcd7ea2567	Layered default layout by hostname role + preserve live node positions Default node position for spine/core/border-leaf/leaf/access is now a Y-tier by role (parsed from hostname, same trust level as the site parsing from #49) with X grouped by site within each tier, replacing the flat site-grid heuristic. Position specifically is now read from the live Grafana dashboard before computing anything, so a node manually drag-and-dropped in the UI keeps its position across regeneration -- only nodes with no live position yet (new devices, or first-ever run) get the layered default. Devices removed from the topology are simply absent from this run's output, so no orphaned position/link reference is possible. Refs #53	2026-07-10 10:40:42 +00:00