Merge pull request 'Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator' (#59) from feat/telemetry into main

Reviewed-on: #59

assets/weathermap-pipeline.svg

@@ -66,7 +66,7 @@
   <path class="arrow" d="M765,120 L765,210" stroke-dasharray="3,3"/>
 
   <!-- read-back for position preservation -->
-  <path class="arrow" d="M765,270 C 560,325 400,325 320,262" stroke-dasharray="3,3"/>
+  <path class="arrow" d="M765,270 C 620,330 490,300 420,275" stroke-dasharray="3,3"/>
   <text class="caption" x="545" y="325" text-anchor="middle">live node positions read back before each regeneration (see #53)</text>
 
 </svg>

scripts/README.md

@@ -114,8 +114,9 @@ default layout.
     - **Known node** → whatever position is currently live in Grafana,
       unchanged. This is why `GRAFANA_URL`/`GRAFANA_TOKEN` matter even on
       a dry run (#53).
-4. Build the panel's PromQL `targets`: node status, link tx/rx, VXLAN
-   MAC/VNI tooltip — each with an explicit `legendFormat`.
+4. Build the panel's PromQL `targets`: node status, per-side link tx (each
+   side its own egress counter, so both directions are represented — #57),
+   VXLAN MAC/VNI tooltip — each with an explicit `legendFormat`.
 5. Build `nodes[]`/`links[]`, including the plugin's `anchors` tally
    (link count per side) — required by the installed plugin fork even
    though the schema doc says it's optional (#48).
@@ -130,6 +131,37 @@ default layout.
 ## Known gaps
 
 - **VXLAN MAC-per-VNI tooltip**: the `vlan` join key doesn't match on live
-  data for any VTEP node — likely an Arista internal-vs-front-panel VLAN
+  data for any VTEP node, likely an Arista internal-vs-front-panel VLAN
   translation gap. Only affects that one decorative tooltip metric, not
   node/link status or traffic coloring. Tracked in #44.
+
+# scripts/generate_traffic.sh
+
+Generates real DC↔Campus traffic over VRF `gold` using `iperf3` (bundled
+in the `network-multitool` image every host container runs), with a live
+bandwidth dashboard. Without this, host containers sit idle and IPFabric
+ARP/MAC tables, Grafana throughput graphs, and the weathermap panel stay
+empty until someone manually generates traffic (see #55).
+
+## Quickstart
+
+```bash
+./scripts/generate_traffic.sh <duration_seconds>
+```
+
+## How it works
+
+- Starts `iperf3 -s` on the DC gold-VRF servers: `dc-server2`
+  (10.34.34.102), `dc-server4` (10.78.78.104).
+- Runs `iperf3 -c -R` from the paired campus gold-VRF hosts, reversing the
+  stream so the DC server pushes to the campus consumer: `dc-server2` →
+  `campus-host1`, `dc-server4` → `campus-host2` — exercising the full
+  DC→Core→Campus stitched EVPN Type-5 path end to end.
+- Redraws a terminal dashboard every second for the run duration: server
+  list, and live Mbits/sec per client session parsed from `iperf3 -i 1`
+  output.
+- On exit (duration end or Ctrl-C), kills the client processes and stops
+  the `iperf3 -s` processes on the DC servers — no leftover state.
+
+`dc-server1`/`dc-server3` (VLAN 40, VRF default, no gateway) are out of
+scope — this script only exercises the routed gold VRF path.

scripts/generate_traffic.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Generates DC<->Campus traffic over VRF gold using iperf3 (bundled in the
+# network-multitool image every host container runs), with a live
+# server/client bandwidth dashboard. Refs #55.
+set -euo pipefail
+
+DURATION="${1:?Usage: $0 <duration_seconds>}"
+if ! [[ "$DURATION" =~ ^[0-9]+$ ]] || [[ "$DURATION" -lt 1 ]]; then
+    echo "Duration must be a positive integer (seconds)" >&2
+    exit 1
+fi
+
+LAB_PREFIX="clab-arista-evpn-fabric"
+PORT=5301
+
+# server_name:server_ip:client_name  — gold VRF pairs, stitched EVPN
+# Type-5 path DC -> Core -> Campus (see README Host Addressing table)
+PAIRS=(
+    "dc-server2:10.34.34.102:campus-host1"
+    "dc-server4:10.78.78.104:campus-host2"
+)
+
+WORKDIR="$(mktemp -d)"
+CLIENT_PIDS=()
+
+cleanup() {
+    for pid in "${CLIENT_PIDS[@]:-}"; do
+        kill "$pid" >/dev/null 2>&1 || true
+    done
+    for pair in "${PAIRS[@]}"; do
+        IFS=':' read -r server _ _ <<<"$pair"
+        docker exec "${LAB_PREFIX}-${server}" pkill -f "iperf3 -s -p ${PORT}" >/dev/null 2>&1 || true
+    done
+    rm -rf "$WORKDIR"
+}
+trap cleanup EXIT INT TERM
+
+echo "Starting iperf3 servers..."
+for pair in "${PAIRS[@]}"; do
+    IFS=':' read -r server server_ip _ <<<"$pair"
+    docker exec -d "${LAB_PREFIX}-${server}" iperf3 -s -p "$PORT"
+done
+sleep 1
+
+echo "Starting iperf3 clients for ${DURATION}s..."
+for pair in "${PAIRS[@]}"; do
+    IFS=':' read -r server server_ip client <<<"$pair"
+    logfile="${WORKDIR}/${client}.log"
+    # -R: DC hosts the service, campus is the consumer -- data should flow
+    # server -> client (download), not client -> server, to match how a
+    # real DC-hosted service/campus-consumer pair behaves.
+    docker exec "${LAB_PREFIX}-${client}" iperf3 -c "$server_ip" -p "$PORT" -R \
+        -t "$DURATION" -i 1 --forceflush -f m >"$logfile" 2>&1 &
+    CLIENT_PIDS+=("$!")
+done
+
+for ((elapsed = 0; elapsed <= DURATION; elapsed++)); do
+    clear
+    echo "EVPN/VXLAN lab traffic generator — ${elapsed}/${DURATION}s"
+    echo
+    echo "Servers (iperf3 -s):"
+    for pair in "${PAIRS[@]}"; do
+        IFS=':' read -r server server_ip _ <<<"$pair"
+        echo "  ${server} (${server_ip}:${PORT})"
+    done
+    echo
+    echo "Clients (live bandwidth):"
+    for pair in "${PAIRS[@]}"; do
+        IFS=':' read -r server server_ip client <<<"$pair"
+        logfile="${WORKDIR}/${client}.log"
+        last_line="$(grep -E 'Mbits/sec' "$logfile" 2>/dev/null | tail -1 || true)"
+        bw="$(sed -E 's/.*[[:space:]]([0-9.]+ Mbits\/sec).*/\1/' <<<"$last_line")"
+        [[ -z "$last_line" ]] && bw="waiting..."
+        printf "  %-13s -> %-13s : %s\n" "$server" "$client" "$bw"
+    done
+    sleep 1
+done
+
+wait "${CLIENT_PIDS[@]}" 2>/dev/null || true
+
+echo
+echo "Done. Summary:"
+for pair in "${PAIRS[@]}"; do
+    IFS=':' read -r server _ client <<<"$pair"
+    logfile="${WORKDIR}/${client}.log"
+    summary="$(grep -E 'receiver' "$logfile" 2>/dev/null || true)"
+    echo "  ${server} -> ${client}: ${summary:-no data}"
+done

scripts/generate_weathermap.py

@@ -4,7 +4,7 @@ metrics, merge it into a manually-authored dashboard base, and optionally
 provision the result into Grafana.
 
 Scope (see #52): this script knows only about the weathermap panel --
-targets (node status / link tx / link rx / VXLAN tooltip queries) and
+targets (node status / link tx per side / VXLAN tooltip queries) and
 options.weathermap (nodes/links/scale/settings). It has no knowledge of the
 BGP sessions table, ports/interfaces table, or throughput panels -- those
 live in the manually-authored `configs/grafana/dashboard-base.json` and are
@@ -363,6 +363,12 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
         link_defs.append({
             "id": f"{link['a_host']}-{a_gnmic_int}--{link['z_host']}-{z_gnmic_int}",
             "nodes": [{"id": link["a_host"]}, {"id": link["z_host"]}],
+            # Each side's query is that side's own tx (egress) counter, not the
+            # far end's rx -- see #57. a_host tx and z_host rx both describe the
+            # *same* A->Z flow measured from opposite ends (the a_host->z_host
+            # direction, counted twice), leaving the Z->A direction never
+            # queried by either side. Using each node's own tx gives two
+            # independent, opposite-direction measurements instead.
             "sides": {
                 "A": {
                     "bandwidth": a_bw,
@@ -371,7 +377,7 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
                 },
                 "Z": {
                     "bandwidth": z_bw,
-                    "query": f"{link['z_host']} {z_gnmic_int} rx",
+                    "query": f"{link['z_host']} {z_gnmic_int} tx",
                     "labelOffset": 55, "anchor": ANCHOR["Left"], "dashboardLink": "",
                 },
             },
@@ -410,11 +416,6 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
             "expr": f'rate(interfaces_interface_state_counters_out_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8',
             "legendFormat": "{{device}} {{interface}} tx",
         },
-        {
-            "refId": "C",
-            "expr": f'rate(interfaces_interface_state_counters_in_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8',
-            "legendFormat": "{{device}} {{interface}} rx",
-        },
     ]
     if vtep_hosts:
         vtep_regex = "|".join(promql_escape(h) for h in vtep_hosts)