Weathermap links only measure one direction (A tx / Z rx), reverse flow never queried #57

New Issue

Damien · 2026-07-20T16:29:09Z

Damien commented

2026-07-20 16:29:09 +00:00

Problem

scripts/generate_weathermap.py::build_weathermap builds each link's two
sides as:

"A": {"query": f"{a_host} {a_int} tx", ...}   # a_host's own egress
"Z": {"query": f"{z_host} {z_int} rx", ...}   # z_host's own ingress

Both queries describe the same physical flow (A → Z), just measured from
each end of the wire (a_host's tx counter and z_host's rx counter for the
same direction). The reverse direction (Z → A, i.e. z_host's own tx / a_host's
own rx) is never queried on either side.

Impact

Confirmed live against the iperf3 traffic generator (dc-server2/4 as
iperf3 -s, campus-host1/2 as clients):

Campus access→leaf links appeared to carry traffic in the weathermap
because campus hosts are the iperf3 clients (upload direction = A→Z,
the one direction that's queried). campus-access1 Et2 tx and
campus-leaf2 Et1 rx both read ~20.2 Mbps — same flow, counted twice at
each endpoint, not two opposing flows.
DC access→leaf links appeared idle (~500 Kbps, i.e. TCP ACKs only)
because DC hosts are the iperf3 servers (download direction = Z→A,
never queried). The real payload (dc-leaf3 Et1 tx / dc-access2 Et1
rx) measured ~20.1 Mbps at the same time.

So today the panel can only ever show traffic in whichever direction
happens to align with a link's a_host→z_host ordering — never true
bidirectional utilization, and silently wrong (reads near-zero) whenever
the dominant flow runs the other way.

Fix

Wire up the missing reverse-direction query per side, e.g.:

side A: a_host a_int tx (A→Z, existing) and/or a_host a_int rx (Z→A)
side Z: z_host z_int rx (A→Z, existing) and/or z_host z_int tx (Z→A)

so each link surfaces both directions instead of one direction measured
twice.

## Problem `scripts/generate_weathermap.py::build_weathermap` builds each link's two sides as: ```python "A": {"query": f"{a_host} {a_int} tx", ...} # a_host's own egress "Z": {"query": f"{z_host} {z_int} rx", ...} # z_host's own ingress ``` Both queries describe the *same* physical flow (A → Z), just measured from each end of the wire (a_host's tx counter and z_host's rx counter for the same direction). The reverse direction (Z → A, i.e. z_host's own tx / a_host's own rx) is never queried on either side. ## Impact Confirmed live against the iperf3 traffic generator (dc-server2/4 as `iperf3 -s`, campus-host1/2 as clients): - Campus `access→leaf` links appeared to carry traffic in the weathermap because campus hosts are the iperf3 *clients* (upload direction = A→Z, the one direction that's queried). `campus-access1` Et2 tx and `campus-leaf2` Et1 rx both read ~20.2 Mbps — same flow, counted twice at each endpoint, not two opposing flows. - DC `access→leaf` links appeared idle (~500 Kbps, i.e. TCP ACKs only) because DC hosts are the iperf3 *servers* (download direction = Z→A, never queried). The real payload (`dc-leaf3` Et1 tx / `dc-access2` Et1 rx) measured ~20.1 Mbps at the same time. So today the panel can only ever show traffic in whichever direction happens to align with a link's `a_host`→`z_host` ordering — never true bidirectional utilization, and silently wrong (reads near-zero) whenever the dominant flow runs the other way. ## Fix Wire up the missing reverse-direction query per side, e.g.: - side A: `a_host a_int tx` (A→Z, existing) **and/or** `a_host a_int rx` (Z→A) - side Z: `z_host z_int rx` (A→Z, existing) **and/or** `z_host z_int tx` (Z→A) so each link surfaces both directions instead of one direction measured twice.

Damien added reference feat/telemetry

2026-07-20 16:31:48 +00:00

Damien added the telemetry label 2026-07-20 16:32:08 +00:00

Damien referenced this issue from a commit

2026-07-20 16:38:11 +00:00

Fix weathermap links to show both traffic directions

Damien referenced this issue from a commit

2026-07-20 16:58:07 +00:00

Fix scripts/README.md doc drift after iperf3/weathermap direction fixes

Damien referenced a pull request that will close this issue

2026-07-20 16:58:24 +00:00

Telemetry: gnmic + Prometheus + Grafana weathermap dashboard, iperf3 traffic generator #59