From b31632109acbfc281b6d82ef4dd75200433b196a Mon Sep 17 00:00:00 2001 From: Damien Arnodo Date: Thu, 26 Feb 2026 14:05:07 +0000 Subject: [PATCH] docs: update README to remove Infrahub schema/data references (#42) Remove all references to local Infrahub schemas, data files, and .infrahub.yml config. The orchestrator now treats Infrahub as an external Source of Truth queried via infrahub-sdk. Schema and data are managed in the arista-evpn-vxlan-clab repository. --- README.md | 223 ++++++++++++++++-------------------------------------- 1 file changed, 66 insertions(+), 157 deletions(-) diff --git a/README.md b/README.md index 2015aa5..edd6a2c 100644 --- a/README.md +++ b/README.md @@ -24,14 +24,13 @@ Think `terraform plan` and `terraform apply`, but for your network fabric — po We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source of Truth for several reasons: -| Feature | NetBox | InfraHub | -| ------------------- | --------------------- | ------------------------------------ | -| **Schema** | Fixed DCIM/IPAM model | Fully customizable YAML schema | -| **Git Integration** | External sync needed | Native - branches = data branches | -| **Versioning** | Changelog only | True Git-like versioning with merges | -| **Test/Redeploy** | Dump/restore | `git clone` = complete environment | -| **Transforms** | Limited | Built-in Jinja2 + Python transforms | -| **GraphQL** | Yes | Yes (auto-generated from schema) | +| Feature | NetBox | InfraHub | +| ------------------- | ----------------------- | -------------------------------------------- | +| **Schema** | Fixed DCIM/IPAM model | Fully customizable YAML schema | +| **Git Integration** | External sync needed | Native - branches = data branches | +| **Versioning** | Changelog only | True Git-like versioning with merges | +| **Transforms** | Limited | Built-in Jinja2 + Python transforms | +| **GraphQL** | Yes | Yes (auto-generated from schema) | **Key benefits for this project:** @@ -40,96 +39,40 @@ We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source o 3. **Transforms** - Generate device configs directly from InfraHub 4. **Branches** - Test fabric changes in isolated branches before merge -## 📦 Repository as InfraHub Backend - -This repository serves as the **single source of truth** for both code and infrastructure data: - -``` -fabric-orchestrator/ -├── .infrahub.yml # InfraHub repository config -│ -├── schemas/ # InfraHub schema definitions -│ └── fabric.yml # Custom EVPN-VXLAN fabric schema -│ -├── data/ # Infrastructure objects (YAML) -│ ├── topology/ -│ │ ├── sites.yml -│ │ └── devices.yml # Spines, Leafs, VTEP pairs -│ ├── network/ -│ │ ├── vlans.yml # VLANs + L2VNI mappings -│ │ ├── vrfs.yml # VRFs + L3VNI mappings -│ │ └── interfaces.yml # Interface configs -│ └── routing/ -│ ├── bgp_sessions.yml # Underlay + EVPN overlay -│ └── evpn.yml # Route targets, RDs -│ -├── transforms/ # Jinja2 templates for config generation -│ └── arista/ -│ ├── base.j2 -│ ├── interfaces.j2 -│ ├── bgp.j2 -│ └── evpn.j2 -│ -└── src/ # Python orchestration code -``` - -### Workflow - -```bash -# 1. Edit data files (e.g., add a VLAN) -vim data/network/vlans.yml - -# 2. Commit & push -git commit -am "Add VLAN 100 for production" -git push - -# 3. InfraHub syncs automatically from Git -# → Data available via GraphQL - -# 4. Prefect flow detects change → reconciles fabric -``` - -### Benefits - -- **Reproductibility**: `git clone` → `docker compose up` → complete environment -- **Code Review**: Infrastructure changes go through PR review -- **History**: Full audit trail via Git -- **Testing**: Create a branch, test changes, merge when validated - ## 🎛 Why Prefect? -| Feature | Benefit | -| -------------------------------- | ---------------------------------------------------------------------- | -| **Python-native workflows** | Use `@flow` and `@task` decorators — no YAML, just Python | -| **Free secrets management** | Native `Secret` blocks for credentials (free in OSS) | -| **Built-in UI** | Dashboard, logs, metrics, execution history via `prefect server start` | -| **No containerization required** | Run flows directly with `.serve()` — no Docker needed | -| **Event-driven triggers** | Schedule, webhooks (via FastAPI), flow triggers out of the box | -| **Task dependencies** | Automatic dependency ordering via task result passing or `wait_for` | -| **Retry & error handling** | Built-in retry policies with `@task(retries=3)` | -| **Human-in-the-loop** | Native `pause_flow_run()` for approval workflows | +| Feature | Benefit | +| -------------------------------- | ------------------------------------------------------------------------- | +| **Python-native workflows** | Use `@flow` and `@task` decorators — no YAML, just Python | +| **Free secrets management** | Native `Secret` blocks for credentials (free in OSS) | +| **Built-in UI** | Dashboard, logs, metrics, execution history via `prefect server start` | +| **No containerization required** | Run flows directly with `.serve()` — no Docker needed | +| **Event-driven triggers** | Schedule, webhooks (via FastAPI), flow triggers out of the box | +| **Task dependencies** | Automatic dependency ordering via task result passing or `wait_for` | +| **Retry & error handling** | Built-in retry policies with `@task(retries=3)` | +| **Human-in-the-loop** | Native `pause_flow_run()` for approval workflows | ## 🎯 Target Fabric -This project is designed for the Arista EVPN-VXLAN ContainerLab topology: +This project is designed for Arista EVPN-VXLAN fabrics: -- **2 Spines** (BGP Route Reflectors, AS 65000) -- **8 Leafs** (4 MLAG VTEP pairs, AS 65001-65004) +- **2 Spines** (BGP Route Reflectors) +- **8 Leafs** (4 MLAG VTEP pairs) - **cEOS 4.35.0F** with gNMI enabled - **EVPN Type-2** (L2 VXLAN) and **Type-5** (L3 VXLAN) support -Reference: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) +Reference lab topology: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) ## 📋 Project Phases Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/fabric-orchestrator/issues) or filter by phase: -| Phase | Description | Status | -| ----------- | -------------------------------------------------------------------- | ------------- | -| **Phase 1** | YANG Path Discovery - Map EOS 4.35.0F YANG models, validate gNMI | ✅ Complete | -| **Phase 2** | InfraHub Setup & Core Reconciler - Schema, diff engine, YANG mappers | 🔄 In Progress | -| **Phase 3** | Full Fabric Coverage - BGP, MLAG, VRFs mappers | 📋 Planned | -| **Phase 4** | Prefect Integration - Flows, webhooks, drift detection | 📋 Planned | +| Phase | Description | Status | +| ----------- | ---------------------------------------------------------------------------- | -------------- | +| **Phase 1** | YANG Path Discovery - Map EOS 4.35.0F YANG models, validate gNMI | ✅ Complete | +| **Phase 2** | InfraHub Client & Core Reconciler - SDK client, diff engine, YANG mappers | 🔄 In Progress | +| **Phase 3** | Full Fabric Coverage - BGP, MLAG, VRFs mappers | 📋 Planned | +| **Phase 4** | Prefect Integration - Flows, webhooks, drift detection | 📋 Planned | ## 📁 Project Structure @@ -137,52 +80,27 @@ Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/ fabric-orchestrator/ ├── README.md ├── pyproject.toml -├── .infrahub.yml # InfraHub config (points to schemas/) │ -├── schemas/ # InfraHub schema definitions -│ └── fabric.yml # Custom EVPN-VXLAN fabric schema -│ -├── data/ # Infrastructure data (YAML) -│ ├── topology/ -│ │ ├── sites.yml -│ │ └── devices.yml -│ ├── network/ -│ │ ├── vlans.yml -│ │ ├── vrfs.yml -│ │ └── interfaces.yml -│ └── routing/ -│ ├── bgp_sessions.yml -│ └── evpn.yml -│ -├── transforms/ # Jinja2 config templates -│ └── arista/ -│ └── *.j2 -│ -├── src/ # Python package +├── src/ # Python package │ ├── __init__.py -│ ├── cli.py # CLI for YANG discovery -│ │ -│ ├── flows/ # Prefect flows -│ │ ├── __init__.py -│ │ ├── reconcile.py # @flow fabric_reconcile -│ │ ├── drift.py # @flow handle_drift -│ │ └── remediation.py # @flow drift_remediation +│ ├── cli.py # CLI for YANG discovery │ │ │ ├── gnmi/ │ │ ├── __init__.py -│ │ ├── client.py # gNMI client wrapper (pygnmi) +│ │ ├── client.py # gNMI client wrapper (pygnmi) │ │ └── README.md │ │ -│ ├── infrahub/ # InfraHub integration +│ ├── infrahub/ # InfraHub integration │ │ ├── __init__.py -│ │ ├── client.py # InfraHub SDK wrapper -│ │ └── queries.py # GraphQL queries +│ │ ├── client.py # InfraHub SDK wrapper +│ │ ├── models.py # Pydantic intent models +│ │ └── exceptions.py # Client exceptions │ │ │ └── yang/ │ ├── __init__.py -│ ├── mapper.py # InfraHub intent → YANG paths -│ ├── paths.py # YANG path definitions -│ └── mappers/ # Resource-specific mappers +│ ├── mapper.py # InfraHub intent → YANG paths +│ ├── paths.py # YANG path definitions +│ └── mappers/ # Resource-specific mappers │ ├── vlan.py │ ├── interface.py │ ├── bgp.py @@ -197,21 +115,20 @@ fabric-orchestrator/ ## 🛠️ Technology Stack -| Component | Technology | Purpose | -| --------------- | -------------------------- | ------------------------------------ | -| Source of Truth | **InfraHub** | Intent definition via custom schema | -| Data Storage | **This Git repo** | Schema + data versioned together | -| Orchestrator | **Prefect** | Python-native workflow orchestration | -| Transport | gNMI | Configuration and telemetry | -| Data Models | YANG (OpenConfig + Arista) | Structured configuration | -| Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions | -| CLI | Click + Rich | YANG discovery tools | -| Validation | Pydantic v2 | Intent data validation | -| Lab | ContainerLab + cEOS | Development environment | +| Component | Technology | Purpose | +| --------------- | ------------------------------- | ------------------------------------------ | +| Source of Truth | **InfraHub** | Intent definition via custom schema | +| Orchestrator | **Prefect** | Python-native workflow orchestration | +| Transport | gNMI | Configuration and telemetry | +| Data Models | YANG (OpenConfig + Arista) | Structured configuration | +| Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions | +| CLI | Click + Rich | YANG discovery tools | +| Validation | Pydantic v2 | Intent data validation | +| Lab | ContainerLab + cEOS | Development environment | ## 🔗 Related Projects -- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Target fabric topology +- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Lab topology, InfraHub schemas & data - [InfraHub](https://github.com/opsmill/infrahub) - Source of Truth platform - [InfraHub Schema Library](https://github.com/opsmill/schema-library) - Reference schemas - [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions @@ -222,7 +139,6 @@ fabric-orchestrator/ - [InfraHub Documentation](https://docs.infrahub.app) - [InfraHub Schema Guide](https://docs.infrahub.app/guides/create-schema) - [InfraHub Python SDK](https://github.com/opsmill/infrahub-sdk-python) -- [InfraHub .infrahub.yml Reference](https://docs.infrahub.app/reference/dotinfrahub) ### Prefect - [Prefect Documentation](https://docs.prefect.io) @@ -243,31 +159,22 @@ fabric-orchestrator/ - Python 3.12+ - `uv` package manager -- Docker (for InfraHub) -- Access to ContainerLab with cEOS images +- Access to an InfraHub instance with the EVPN-VXLAN fabric schema loaded +- Access to ContainerLab with cEOS images (for lab testing) ### Quick Start ```bash -# Clone the repository (includes schema + data) +# Clone the repository git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git cd fabric-orchestrator # Install Python dependencies uv sync -# Start InfraHub (loads schema & data from this repo) -docker compose up -d - -# Configure Prefect secrets -python -c " -from prefect.blocks.system import Secret -from prefect.variables import Variable - -Secret(value='your-gnmi-password').save('gnmi-password', overwrite=True) -Variable.set('infrahub_url', 'http://localhost:8000') -Variable.set('gnmi_username', 'admin') -" +# Set InfraHub connection (point to your InfraHub instance) +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="your-token" # Verify gNMI connectivity uv run fabric-orch discover capabilities --target leaf1:6030 @@ -285,13 +192,15 @@ from prefect.variables import Variable @task(retries=2, retry_delay_seconds=10) -def get_fabric_intent(device: str | None = None) -> dict: +async def get_fabric_intent(device: str | None = None) -> dict: """Retrieve fabric intent from InfraHub.""" - from infrahub_sdk import InfrahubClient - - client = InfrahubClient(address=Variable.get("infrahub_url")) - # Query fabric intent via GraphQL - return client.query(...) + from src.infrahub.client import FabricInfrahubClient + + async with FabricInfrahubClient( + url=Variable.get("infrahub_url"), + api_token=Variable.get("infrahub_token"), + ) as client: + return await client.get_device(device) @task @@ -307,17 +216,17 @@ def fabric_reconcile(device: str | None = None, dry_run: bool = True) -> dict: intent = get_fabric_intent(device) current = get_current_state(device) changes = compute_diff(intent, current) - + if not changes: print("✅ Fabric is in sync") return {"in_sync": True} - + if not dry_run: apply_changes(changes) - + return {"changes": changes, "applied": not dry_run} ``` --- -**Status**: 🚧 Active Development - Phase 2 (InfraHub Setup & Core Reconciler) +**Status**: 🚧 Active Development - Phase 2 (InfraHub Client & Core Reconciler)