Damien/fabric-orchestrator
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

docs: update README to remove Infrahub schema/data references (#42)

Browse Source 
Remove all references to local Infrahub schemas, data files, and
.infrahub.yml config. The orchestrator now treats Infrahub as an
external Source of Truth queried via infrahub-sdk. Schema and data
are managed in the arista-evpn-vxlan-clab repository.
This commit is contained in:
Damien Arnodo
2026-02-26 14:05:07 +00:00
parent f663cc623c
commit b31632109a
1 changed files with 66 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files

223
README.md
View File

@@ -24,14 +24,13 @@ Think `terraform plan` and `terraform apply`, but for your network fabric — po
We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source of Truth for several reasons: We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source of Truth for several reasons:
| Feature | NetBox | InfraHub | | Feature | NetBox | InfraHub |
| ------------------- | --------------------- | ------------------------------------ | | ------------------- | ----------------------- | -------------------------------------------- |
| **Schema** | Fixed DCIM/IPAM model | Fully customizable YAML schema | | **Schema** | Fixed DCIM/IPAM model | Fully customizable YAML schema |
| **Git Integration** | External sync needed | Native - branches = data branches | | **Git Integration** | External sync needed | Native - branches = data branches |
| **Versioning** | Changelog only | True Git-like versioning with merges | | **Versioning** | Changelog only | True Git-like versioning with merges |
| **Test/Redeploy** | Dump/restore | `git clone` = complete environment | | **Transforms** | Limited | Built-in Jinja2 + Python transforms |
| **Transforms** | Limited | Built-in Jinja2 + Python transforms | | **GraphQL** | Yes | Yes (auto-generated from schema) |
| **GraphQL** | Yes | Yes (auto-generated from schema) |
**Key benefits for this project:** **Key benefits for this project:**
@@ -40,96 +39,40 @@ We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source o
3. **Transforms** - Generate device configs directly from InfraHub 3. **Transforms** - Generate device configs directly from InfraHub
4. **Branches** - Test fabric changes in isolated branches before merge 4. **Branches** - Test fabric changes in isolated branches before merge
## 📦 Repository as InfraHub Backend
This repository serves as the **single source of truth** for both code and infrastructure data:
```
fabric-orchestrator/
├── .infrahub.yml # InfraHub repository config
│
├── schemas/ # InfraHub schema definitions
│ └── fabric.yml # Custom EVPN-VXLAN fabric schema
│
├── data/ # Infrastructure objects (YAML)
│ ├── topology/
│ │ ├── sites.yml
│ │ └── devices.yml # Spines, Leafs, VTEP pairs
│ ├── network/
│ │ ├── vlans.yml # VLANs + L2VNI mappings
│ │ ├── vrfs.yml # VRFs + L3VNI mappings
│ │ └── interfaces.yml # Interface configs
│ └── routing/
│ ├── bgp_sessions.yml # Underlay + EVPN overlay
│ └── evpn.yml # Route targets, RDs
│
├── transforms/ # Jinja2 templates for config generation
│ └── arista/
│ ├── base.j2
│ ├── interfaces.j2
│ ├── bgp.j2
│ └── evpn.j2
│
└── src/ # Python orchestration code
```
### Workflow
```bash
# 1. Edit data files (e.g., add a VLAN)
vim data/network/vlans.yml
# 2. Commit & push
git commit -am "Add VLAN 100 for production"
git push
# 3. InfraHub syncs automatically from Git
# → Data available via GraphQL
# 4. Prefect flow detects change → reconciles fabric
```
### Benefits
- **Reproductibility**: `git clone` → `docker compose up` → complete environment
- **Code Review**: Infrastructure changes go through PR review
- **History**: Full audit trail via Git
- **Testing**: Create a branch, test changes, merge when validated
## 🎛 Why Prefect? ## 🎛 Why Prefect?
| Feature | Benefit | | Feature | Benefit |
| -------------------------------- | ---------------------------------------------------------------------- | | -------------------------------- | ------------------------------------------------------------------------- |
| **Python-native workflows** | Use `@flow` and `@task` decorators — no YAML, just Python | | **Python-native workflows** | Use `@flow` and `@task` decorators — no YAML, just Python |
| **Free secrets management** | Native `Secret` blocks for credentials (free in OSS) | | **Free secrets management** | Native `Secret` blocks for credentials (free in OSS) |
| **Built-in UI** | Dashboard, logs, metrics, execution history via `prefect server start` | | **Built-in UI** | Dashboard, logs, metrics, execution history via `prefect server start` |
| **No containerization required** | Run flows directly with `.serve()` — no Docker needed | | **No containerization required** | Run flows directly with `.serve()` — no Docker needed |
| **Event-driven triggers** | Schedule, webhooks (via FastAPI), flow triggers out of the box | | **Event-driven triggers** | Schedule, webhooks (via FastAPI), flow triggers out of the box |
| **Task dependencies** | Automatic dependency ordering via task result passing or `wait_for` | | **Task dependencies** | Automatic dependency ordering via task result passing or `wait_for` |
| **Retry & error handling** | Built-in retry policies with `@task(retries=3)` | | **Retry & error handling** | Built-in retry policies with `@task(retries=3)` |
| **Human-in-the-loop** | Native `pause_flow_run()` for approval workflows | | **Human-in-the-loop** | Native `pause_flow_run()` for approval workflows |
## 🎯 Target Fabric ## 🎯 Target Fabric
This project is designed for the Arista EVPN-VXLAN ContainerLab topology: This project is designed for Arista EVPN-VXLAN fabrics:
- **2 Spines** (BGP Route Reflectors, AS 65000) - **2 Spines** (BGP Route Reflectors)
- **8 Leafs** (4 MLAG VTEP pairs, AS 65001-65004) - **8 Leafs** (4 MLAG VTEP pairs)
- **cEOS 4.35.0F** with gNMI enabled - **cEOS 4.35.0F** with gNMI enabled
- **EVPN Type-2** (L2 VXLAN) and **Type-5** (L3 VXLAN) support - **EVPN Type-2** (L2 VXLAN) and **Type-5** (L3 VXLAN) support
Reference: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) Reference lab topology: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab)
## 📋 Project Phases ## 📋 Project Phases
Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/fabric-orchestrator/issues) or filter by phase: Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/fabric-orchestrator/issues) or filter by phase:
| Phase | Description | Status | | Phase | Description | Status |
| ----------- | -------------------------------------------------------------------- | ------------- | | ----------- | ---------------------------------------------------------------------------- | -------------- |
| **Phase 1** | YANG Path Discovery - Map EOS 4.35.0F YANG models, validate gNMI | ✅ Complete | | **Phase 1** | YANG Path Discovery - Map EOS 4.35.0F YANG models, validate gNMI | ✅ Complete |
| **Phase 2** | InfraHub Setup & Core Reconciler - Schema, diff engine, YANG mappers | 🔄 In Progress | | **Phase 2** | InfraHub Client & Core Reconciler - SDK client, diff engine, YANG mappers | 🔄 In Progress |
| **Phase 3** | Full Fabric Coverage - BGP, MLAG, VRFs mappers | 📋 Planned | | **Phase 3** | Full Fabric Coverage - BGP, MLAG, VRFs mappers | 📋 Planned |
| **Phase 4** | Prefect Integration - Flows, webhooks, drift detection | 📋 Planned | | **Phase 4** | Prefect Integration - Flows, webhooks, drift detection | 📋 Planned |
## 📁 Project Structure ## 📁 Project Structure
@@ -137,52 +80,27 @@ Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/
fabric-orchestrator/ fabric-orchestrator/
├── README.md ├── README.md
├── pyproject.toml ├── pyproject.toml
├── .infrahub.yml # InfraHub config (points to schemas/)
│ │
├── schemas/ # InfraHub schema definitions ├── src/ # Python package
│ └── fabric.yml # Custom EVPN-VXLAN fabric schema
│
├── data/ # Infrastructure data (YAML)
│ ├── topology/
│ │ ├── sites.yml
│ │ └── devices.yml
│ ├── network/
│ │ ├── vlans.yml
│ │ ├── vrfs.yml
│ │ └── interfaces.yml
│ └── routing/
│ ├── bgp_sessions.yml
│ └── evpn.yml
│
├── transforms/ # Jinja2 config templates
│ └── arista/
│ └── *.j2
│
├── src/ # Python package
│ ├── __init__.py │ ├── __init__.py
│ ├── cli.py # CLI for YANG discovery │ ├── cli.py # CLI for YANG discovery
│ │
│ ├── flows/ # Prefect flows
│ │ ├── __init__.py
│ │ ├── reconcile.py # @flow fabric_reconcile
│ │ ├── drift.py # @flow handle_drift
│ │ └── remediation.py # @flow drift_remediation
│ │ │ │
│ ├── gnmi/ │ ├── gnmi/
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── client.py # gNMI client wrapper (pygnmi) │ │ ├── client.py # gNMI client wrapper (pygnmi)
│ │ └── README.md │ │ └── README.md
│ │ │ │
│ ├── infrahub/ # InfraHub integration │ ├── infrahub/ # InfraHub integration
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── client.py # InfraHub SDK wrapper │ │ ├── client.py # InfraHub SDK wrapper
│ │ └── queries.py # GraphQL queries │ │ ├── models.py # Pydantic intent models
│ │ └── exceptions.py # Client exceptions
│ │ │ │
│ └── yang/ │ └── yang/
│ ├── __init__.py │ ├── __init__.py
│ ├── mapper.py # InfraHub intent → YANG paths │ ├── mapper.py # InfraHub intent → YANG paths
│ ├── paths.py # YANG path definitions │ ├── paths.py # YANG path definitions
│ └── mappers/ # Resource-specific mappers │ └── mappers/ # Resource-specific mappers
│ ├── vlan.py │ ├── vlan.py
│ ├── interface.py │ ├── interface.py
│ ├── bgp.py │ ├── bgp.py
@@ -197,21 +115,20 @@ fabric-orchestrator/
## 🛠️ Technology Stack ## 🛠️ Technology Stack
| Component | Technology | Purpose | | Component | Technology | Purpose |
| --------------- | -------------------------- | ------------------------------------ | | --------------- | ------------------------------- | ------------------------------------------ |
| Source of Truth | **InfraHub** | Intent definition via custom schema | | Source of Truth | **InfraHub** | Intent definition via custom schema |
| Data Storage | **This Git repo** | Schema + data versioned together | | Orchestrator | **Prefect** | Python-native workflow orchestration |
| Orchestrator | **Prefect** | Python-native workflow orchestration | | Transport | gNMI | Configuration and telemetry |
| Transport | gNMI | Configuration and telemetry | | Data Models | YANG (OpenConfig + Arista) | Structured configuration |
| Data Models | YANG (OpenConfig + Arista) | Structured configuration | | Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions |
| Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions | | CLI | Click + Rich | YANG discovery tools |
| CLI | Click + Rich | YANG discovery tools | | Validation | Pydantic v2 | Intent data validation |
| Validation | Pydantic v2 | Intent data validation | | Lab | ContainerLab + cEOS | Development environment |
| Lab | ContainerLab + cEOS | Development environment |
## 🔗 Related Projects ## 🔗 Related Projects
- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Target fabric topology - [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Lab topology, InfraHub schemas & data
- [InfraHub](https://github.com/opsmill/infrahub) - Source of Truth platform - [InfraHub](https://github.com/opsmill/infrahub) - Source of Truth platform
- [InfraHub Schema Library](https://github.com/opsmill/schema-library) - Reference schemas - [InfraHub Schema Library](https://github.com/opsmill/schema-library) - Reference schemas
- [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions - [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions
@@ -222,7 +139,6 @@ fabric-orchestrator/
- [InfraHub Documentation](https://docs.infrahub.app) - [InfraHub Documentation](https://docs.infrahub.app)
- [InfraHub Schema Guide](https://docs.infrahub.app/guides/create-schema) - [InfraHub Schema Guide](https://docs.infrahub.app/guides/create-schema)
- [InfraHub Python SDK](https://github.com/opsmill/infrahub-sdk-python) - [InfraHub Python SDK](https://github.com/opsmill/infrahub-sdk-python)
- [InfraHub .infrahub.yml Reference](https://docs.infrahub.app/reference/dotinfrahub)
### Prefect ### Prefect
- [Prefect Documentation](https://docs.prefect.io) - [Prefect Documentation](https://docs.prefect.io)
@@ -243,31 +159,22 @@ fabric-orchestrator/
- Python 3.12+ - Python 3.12+
- `uv` package manager - `uv` package manager
- Docker (for InfraHub) - Access to an InfraHub instance with the EVPN-VXLAN fabric schema loaded
- Access to ContainerLab with cEOS images - Access to ContainerLab with cEOS images (for lab testing)
### Quick Start ### Quick Start
```bash ```bash
# Clone the repository (includes schema + data) # Clone the repository
git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git
cd fabric-orchestrator cd fabric-orchestrator
# Install Python dependencies # Install Python dependencies
uv sync uv sync
# Start InfraHub (loads schema & data from this repo) # Set InfraHub connection (point to your InfraHub instance)
docker compose up -d export INFRAHUB_ADDRESS="http://localhost:8000"
export INFRAHUB_API_TOKEN="your-token"
# Configure Prefect secrets
python -c "
from prefect.blocks.system import Secret
from prefect.variables import Variable
Secret(value='your-gnmi-password').save('gnmi-password', overwrite=True)
Variable.set('infrahub_url', 'http://localhost:8000')
Variable.set('gnmi_username', 'admin')
"
# Verify gNMI connectivity # Verify gNMI connectivity
uv run fabric-orch discover capabilities --target leaf1:6030 uv run fabric-orch discover capabilities --target leaf1:6030
@@ -285,13 +192,15 @@ from prefect.variables import Variable
@task(retries=2, retry_delay_seconds=10) @task(retries=2, retry_delay_seconds=10)
def get_fabric_intent(device: str | None = None) -> dict: async def get_fabric_intent(device: str | None = None) -> dict:
"""Retrieve fabric intent from InfraHub.""" """Retrieve fabric intent from InfraHub."""
from infrahub_sdk import InfrahubClient from src.infrahub.client import FabricInfrahubClient
client = InfrahubClient(address=Variable.get("infrahub_url")) async with FabricInfrahubClient(
# Query fabric intent via GraphQL url=Variable.get("infrahub_url"),
return client.query(...) api_token=Variable.get("infrahub_token"),
) as client:
return await client.get_device(device)
@task @task
@@ -307,17 +216,17 @@ def fabric_reconcile(device: str | None = None, dry_run: bool = True) -> dict:
intent = get_fabric_intent(device) intent = get_fabric_intent(device)
current = get_current_state(device) current = get_current_state(device)
changes = compute_diff(intent, current) changes = compute_diff(intent, current)
if not changes: if not changes:
print("✅ Fabric is in sync") print("✅ Fabric is in sync")
return {"in_sync": True} return {"in_sync": True}
if not dry_run: if not dry_run:
apply_changes(changes) apply_changes(changes)
return {"changes": changes, "applied": not dry_run} return {"changes": changes, "applied": not dry_run}
``` ```
--- ---
**Status**: 🚧 Active Development - Phase 2 (InfraHub Setup & Core Reconciler) **Status**: 🚧 Active Development - Phase 2 (InfraHub Client & Core Reconciler)