Compare commits

...

7 Commits

Author SHA1 Message Date
31014bdfdf Merge pull request 'Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator' (#59) from feat/telemetry into main
Reviewed-on: #59
2026-07-20 16:59:09 +00:00
5e5d6b45df Fix scripts/README.md doc drift after iperf3/weathermap direction fixes
Traffic direction docs still described the pre-fix client->server
iperf3 flow and the pre-fix tx/rx weathermap query pair.

Refs #57, Refs #58
2026-07-20 16:57:28 +00:00
4719a77f21 Reverse iperf3 traffic direction to DC->campus (server pushes to client)
Without -R, iperf3's client sends and the server only receives, so
traffic ran campus-host -> dc-server -- backwards for what these roles
represent: dc-serverN is the DC-hosted service, campus-hostN is the
consumer, and a real service delivers data to the consumer rather than
just absorbing uploads from it.

Refs #58
2026-07-20 16:37:16 +00:00
08124c8a1c Fix weathermap links to show both traffic directions
Z side queried z_host's rx, which measures the same A->Z flow as A
side's tx (same traffic, counted at each end) -- the Z->A direction
was never queried by either side. Use z_host's own tx instead, so
each side surfaces its own (opposite) direction.

Confirmed against the iperf3 traffic generator: DC access-leaf links
read ~500Kbps (ACK-only) before the fix despite ~18-20Mbps of real
downstream traffic to dc-server2/4, because that flow ran leaf->access
(Z->A), the unqueried direction. Campus links looked correct only by
coincidence -- campus hosts are the iperf3 clients, so their upload
traffic happens to run access->leaf (A->Z), the direction that was
queried.

Refs #57
2026-07-20 16:31:33 +00:00
8349791630 Merge pull request 'Add iperf3 traffic generator for DC<->Campus gold VRF path' (#56) from feat/iperf-traffic-generator into feat/telemetry
Reviewed-on: #56
2026-07-20 16:15:19 +00:00
3c3b28c987 Merge branch 'feat/telemetry' into feat/iperf-traffic-generator 2026-07-20 16:15:09 +00:00
Damien
0f3aa27566 Add Grafana weathermap dashboard asset 2026-07-10 15:35:12 +02:00
5 changed files with 2154 additions and 2157 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 639 KiB

File diff suppressed because it is too large Load Diff

View File

@@ -114,8 +114,9 @@ default layout.
- **Known node** → whatever position is currently live in Grafana, - **Known node** → whatever position is currently live in Grafana,
unchanged. This is why `GRAFANA_URL`/`GRAFANA_TOKEN` matter even on unchanged. This is why `GRAFANA_URL`/`GRAFANA_TOKEN` matter even on
a dry run (#53). a dry run (#53).
4. Build the panel's PromQL `targets`: node status, link tx/rx, VXLAN 4. Build the panel's PromQL `targets`: node status, per-side link tx (each
MAC/VNI tooltip — each with an explicit `legendFormat`. side its own egress counter, so both directions are represented — #57),
VXLAN MAC/VNI tooltip — each with an explicit `legendFormat`.
5. Build `nodes[]`/`links[]`, including the plugin's `anchors` tally 5. Build `nodes[]`/`links[]`, including the plugin's `anchors` tally
(link count per side) — required by the installed plugin fork even (link count per side) — required by the installed plugin fork even
though the schema doc says it's optional (#48). though the schema doc says it's optional (#48).
@@ -130,7 +131,7 @@ default layout.
## Known gaps ## Known gaps
- **VXLAN MAC-per-VNI tooltip**: the `vlan` join key doesn't match on live - **VXLAN MAC-per-VNI tooltip**: the `vlan` join key doesn't match on live
data for any VTEP node likely an Arista internal-vs-front-panel VLAN data for any VTEP node, likely an Arista internal-vs-front-panel VLAN
translation gap. Only affects that one decorative tooltip metric, not translation gap. Only affects that one decorative tooltip metric, not
node/link status or traffic coloring. Tracked in #44. node/link status or traffic coloring. Tracked in #44.
@@ -152,8 +153,9 @@ empty until someone manually generates traffic (see #55).
- Starts `iperf3 -s` on the DC gold-VRF servers: `dc-server2` - Starts `iperf3 -s` on the DC gold-VRF servers: `dc-server2`
(10.34.34.102), `dc-server4` (10.78.78.104). (10.34.34.102), `dc-server4` (10.78.78.104).
- Runs `iperf3 -c` from the paired campus gold-VRF hosts: `campus-host1` - Runs `iperf3 -c -R` from the paired campus gold-VRF hosts, reversing the
→ dc-server2, `campus-host2` → dc-server4 — exercising the full stream so the DC server pushes to the campus consumer: `dc-server2`
`campus-host1`, `dc-server4``campus-host2` — exercising the full
DC→Core→Campus stitched EVPN Type-5 path end to end. DC→Core→Campus stitched EVPN Type-5 path end to end.
- Redraws a terminal dashboard every second for the run duration: server - Redraws a terminal dashboard every second for the run duration: server
list, and live Mbits/sec per client session parsed from `iperf3 -i 1` list, and live Mbits/sec per client session parsed from `iperf3 -i 1`

View File

@@ -46,7 +46,10 @@ echo "Starting iperf3 clients for ${DURATION}s..."
for pair in "${PAIRS[@]}"; do for pair in "${PAIRS[@]}"; do
IFS=':' read -r server server_ip client <<<"$pair" IFS=':' read -r server server_ip client <<<"$pair"
logfile="${WORKDIR}/${client}.log" logfile="${WORKDIR}/${client}.log"
docker exec "${LAB_PREFIX}-${client}" iperf3 -c "$server_ip" -p "$PORT" \ # -R: DC hosts the service, campus is the consumer -- data should flow
# server -> client (download), not client -> server, to match how a
# real DC-hosted service/campus-consumer pair behaves.
docker exec "${LAB_PREFIX}-${client}" iperf3 -c "$server_ip" -p "$PORT" -R \
-t "$DURATION" -i 1 --forceflush -f m >"$logfile" 2>&1 & -t "$DURATION" -i 1 --forceflush -f m >"$logfile" 2>&1 &
CLIENT_PIDS+=("$!") CLIENT_PIDS+=("$!")
done done
@@ -68,7 +71,7 @@ for ((elapsed = 0; elapsed <= DURATION; elapsed++)); do
last_line="$(grep -E 'Mbits/sec' "$logfile" 2>/dev/null | tail -1 || true)" last_line="$(grep -E 'Mbits/sec' "$logfile" 2>/dev/null | tail -1 || true)"
bw="$(sed -E 's/.*[[:space:]]([0-9.]+ Mbits\/sec).*/\1/' <<<"$last_line")" bw="$(sed -E 's/.*[[:space:]]([0-9.]+ Mbits\/sec).*/\1/' <<<"$last_line")"
[[ -z "$last_line" ]] && bw="waiting..." [[ -z "$last_line" ]] && bw="waiting..."
printf " %-13s -> %-13s : %s\n" "$client" "$server" "$bw" printf " %-13s -> %-13s : %s\n" "$server" "$client" "$bw"
done done
sleep 1 sleep 1
done done
@@ -81,5 +84,5 @@ for pair in "${PAIRS[@]}"; do
IFS=':' read -r server _ client <<<"$pair" IFS=':' read -r server _ client <<<"$pair"
logfile="${WORKDIR}/${client}.log" logfile="${WORKDIR}/${client}.log"
summary="$(grep -E 'receiver' "$logfile" 2>/dev/null || true)" summary="$(grep -E 'receiver' "$logfile" 2>/dev/null || true)"
echo " ${client} -> ${server}: ${summary:-no data}" echo " ${server} -> ${client}: ${summary:-no data}"
done done

View File

@@ -4,7 +4,7 @@ metrics, merge it into a manually-authored dashboard base, and optionally
provision the result into Grafana. provision the result into Grafana.
Scope (see #52): this script knows only about the weathermap panel -- Scope (see #52): this script knows only about the weathermap panel --
targets (node status / link tx / link rx / VXLAN tooltip queries) and targets (node status / link tx per side / VXLAN tooltip queries) and
options.weathermap (nodes/links/scale/settings). It has no knowledge of the options.weathermap (nodes/links/scale/settings). It has no knowledge of the
BGP sessions table, ports/interfaces table, or throughput panels -- those BGP sessions table, ports/interfaces table, or throughput panels -- those
live in the manually-authored `configs/grafana/dashboard-base.json` and are live in the manually-authored `configs/grafana/dashboard-base.json` and are
@@ -363,6 +363,12 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
link_defs.append({ link_defs.append({
"id": f"{link['a_host']}-{a_gnmic_int}--{link['z_host']}-{z_gnmic_int}", "id": f"{link['a_host']}-{a_gnmic_int}--{link['z_host']}-{z_gnmic_int}",
"nodes": [{"id": link["a_host"]}, {"id": link["z_host"]}], "nodes": [{"id": link["a_host"]}, {"id": link["z_host"]}],
# Each side's query is that side's own tx (egress) counter, not the
# far end's rx -- see #57. a_host tx and z_host rx both describe the
# *same* A->Z flow measured from opposite ends (the a_host->z_host
# direction, counted twice), leaving the Z->A direction never
# queried by either side. Using each node's own tx gives two
# independent, opposite-direction measurements instead.
"sides": { "sides": {
"A": { "A": {
"bandwidth": a_bw, "bandwidth": a_bw,
@@ -371,7 +377,7 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
}, },
"Z": { "Z": {
"bandwidth": z_bw, "bandwidth": z_bw,
"query": f"{link['z_host']} {z_gnmic_int} rx", "query": f"{link['z_host']} {z_gnmic_int} tx",
"labelOffset": 55, "anchor": ANCHOR["Left"], "dashboardLink": "", "labelOffset": 55, "anchor": ANCHOR["Left"], "dashboardLink": "",
}, },
}, },
@@ -410,11 +416,6 @@ def build_weathermap(devices, links, interface_speeds, positions, prometheus_url
"expr": f'rate(interfaces_interface_state_counters_out_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8', "expr": f'rate(interfaces_interface_state_counters_out_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8',
"legendFormat": "{{device}} {{interface}} tx", "legendFormat": "{{device}} {{interface}} tx",
}, },
{
"refId": "C",
"expr": f'rate(interfaces_interface_state_counters_in_octets{{device=~"{host_regex}", interface=~"{interface_regex}"}}[5m]) * 8',
"legendFormat": "{{device}} {{interface}} rx",
},
] ]
if vtep_hosts: if vtep_hosts:
vtep_regex = "|".join(promql_escape(h) for h in vtep_hosts) vtep_regex = "|".join(promql_escape(h) for h in vtep_hosts)