Split weathermap generation from dashboard composition (manual + generated merge) #52
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #49. Two architectural changes to the dashboard generation pipeline, no change to the underlying metrics/queries already validated in #44/#45/#48/#49.
1. Split script scope: weathermap-only
scripts/generate_weathermap.pycurrently outputs a full dashboard JSON (weathermap + whatever else was in it at generation time). Scope narrows to: the script only ever produces the weathermap panel (targets +options.weathermap, node/link generation from live IPFabric topology, as already implemented). It must not know about or touch the BGP table, ports table, or throughput panels.Site label: no change — the existing hostname-regex Prometheus relabel (
campus-*/core*/dc-*→site) stays as-is. Site naming is a standardized convention, not something the script should adapt around; if a device ever breaks the convention, that's a human fix to the network/Prometheus config, not a case for the script to handle dynamically.2. Manual + generated dashboard composition
The rest of the dashboard (BGP sessions table, ports/interfaces table, throughput-per-site graphs, template variables, panel layout — all built in #49) becomes a manually-authored JSON file, living in the same directory as the weathermap panel output. For this POC, "manual" means agent-authored, committed directly — not derived from IPFabric or Prometheus discovery.
New merge step (extend
generate_weathermap.pyor a new small script, whichever is cleaner) that:idor title) and injects the generated content, preserving the base file'sgridPos(layout stays manual)--provisionstill pushes it to Grafana via the API, as todayWhy file-based merge, not live-fetch-from-Grafana
Considered fetching the current dashboard from Grafana's API and patching just the weathermap panel in place (avoids drift if someone hand-edits in the UI). Decided against it for this POC: the manual JSON file is the deliberate source of truth here, edited directly, not the live Grafana state. Simpler, and fits the "GitOps" pattern already used for the weathermap panel itself.
Idempotency / adapting to topology changes
No diffing logic needed — the script already fully regenerates the weathermap panel from live IPFabric data on every run (declarative, not incremental), so re-running it after a topology change already produces the correct up-to-date panel. The only operational gap is triggering: for now, documented as "rerun manually after a topology change"; scheduling/webhook automation is a stretch goal, not in scope here.
Tasks
generate_weathermap.pyto emit weathermap-panel-only JSONImplemented in
632d146(branchfeat/telemetry).File structure
scripts/generate_weathermap.pybuild_weathermap_panel()returns just{type, datasource, targets, options.weathermap}— nogridPos/id/titleof its own. No reference to BGP/ports/throughput panels or template variables anywhere in the script.configs/grafana/dashboard-base.jsontemplating(site/devicevars, ported verbatim from #49), and 4 panels: the reserved slot (id: 1,title: "__WEATHERMAP_SLOT__", fixedgridPos), BGP Sessions table, Ports/Interfaces table, Throughput per Site — all ported verbatim from #49's validated queries/transformations, including the #50 join-uniqueness guard and #51 Management0 exclusion (those live in the weathermap script and base file respectively, so both fixes carried through the split intact). Datasource UID is a__DATASOURCE_UID__placeholder, substituted at merge time.configs/grafana/weathermap-dashboard.jsonMerge step
Extended
generate_weathermap.pyitself (not a separate script — simplest option, avoids inter-script imports):main()now loads the base JSON, substitutes__DATASOURCE_UID__recursively, finds the panel titled__WEATHERMAP_SLOT__, anddict.update()s the generated panel content into it while preserving that panel'sgridPos/id. Renamed to"Fabric Weathermap"on merge.--provisionunchanged — still POSTs the merged payload to/api/dashboards/db.Site label mechanism: untouched, exactly as specified (hostname-regex Prometheus relabel from #49 stays as the standard).
Regression check
Diffed the merged output against the pre-refactor generator-only JSON (both generated back-to-back against the same live IPFabric/Prometheus state, no topology change in between): 1 line different, and that line is the same content (
"type": "tamirsuliman-weathermap-panel") just at a different position in the JSON — purely a key-ordering artifact ofdict.update()vs the old inline dict literal, not a content change.Re-validated live via
/api/ds/querypost-merge:max by (device, vlan)guard intactNo topology change was convenient to simulate in this pass (lab topology unchanged since #49/#50/#51) — re-running after a real topology change to confirm the panel updates in place is still an open manual check, but the mechanism is unchanged from #49 (full regeneration every run, no diffing), so no new risk introduced by the split.
Operational note (adapts to topology changes)
No new automation added — as scoped, this is a stretch goal, not in scope. Documented in
scripts/README.md: re-runninggenerate_weathermap.pyalways regenerates the weathermap panel fresh from live IPFabric+Prometheus and re-merges it into the (possibly separately-edited) base, so a topology change requires a manual re-run +--provision, same operational model as before. Editing the BGP/ports/throughput panels or variables now only requires editingdashboard-base.jsonand re-running — no IPFabric/topology fetch needed for that path (though the script currently always does that fetch regardless; a--base-onlyfast path that skips it wasn't requested and would be scope creep for this issue).