VNI tooltip PromQL join breaks on transient duplicate series after Prometheus restart #50

New Issue

Damien · 2026-07-10T09:48:43Z

Damien commented

2026-07-10 09:48:43 +00:00

Weathermap panel's VNI-per-VTEP tooltip query (target D, from #44/#48) errors with:

found duplicate series for the match group {device="campus-leaf1", vlan="50"} on the
right-hand side of the operation ... many-to-many matching not allowed: matching
labels must be unique on one side

Root cause: the RHS of the group_left join,
interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni, is not
aggregated before joining. Any time Prometheus restarts or a relabel rule
changes (most recently the site relabel added in #49), the pre-restart
series (old label set, frozen) and post-restart series (new label set)
briefly coexist within Prometheus's 5-minute staleness/lookback window —
both matching the same (device, vlan) join key — which breaks the
uniqueness requirement on the RHS. Confirmed transient: series count back to
1-per-(device,vlan) 5+ minutes after the restart, but the panel errors for
every user who loads it in that window.

Fix: wrap the RHS in max by (device, vlan) (...) (or similar) so the join
key is always unique regardless of transient duplicate label sets — this
protects against every future Prometheus restart/relabel change, not just
this one.

Only affects the decorative per-VTEP MAC-count tooltip, not node/link status
or traffic coloring.

Weathermap panel's VNI-per-VTEP tooltip query (target D, from #44/#48) errors with: ``` found duplicate series for the match group {device="campus-leaf1", vlan="50"} on the right-hand side of the operation ... many-to-many matching not allowed: matching labels must be unique on one side ``` Root cause: the RHS of the `group_left` join, `interfaces_interface_arista_vxlan_vlan_to_vnis_vlan_to_vni_state_vni`, is not aggregated before joining. Any time Prometheus restarts or a relabel rule changes (most recently the `site` relabel added in #49), the pre-restart series (old label set, frozen) and post-restart series (new label set) briefly coexist within Prometheus's 5-minute staleness/lookback window — both matching the same `(device, vlan)` join key — which breaks the uniqueness requirement on the RHS. Confirmed transient: series count back to 1-per-(device,vlan) 5+ minutes after the restart, but the panel errors for every user who loads it in that window. Fix: wrap the RHS in `max by (device, vlan) (...)` (or similar) so the join key is always unique regardless of transient duplicate label sets — this protects against every future Prometheus restart/relabel change, not just this one. Only affects the decorative per-VTEP MAC-count tooltip, not node/link status or traffic coloring.

Damien added the telemetry label 2026-07-10 09:49:21 +00:00

Damien closed this issue

2026-07-10 09:51:54 +00:00

Damien referenced this issue

2026-07-10 10:16:12 +00:00

Split weathermap generation from dashboard composition (manual + generated merge) #52

Damien referenced this issue from a commit

2026-07-10 10:20:33 +00:00

Guard VNI tooltip join against transient duplicate series

Sign in to join this conversation.