Damien/fabric-orchestrator
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

feat(infrahub): add Infrahub client for fabric intent (#42) #54

Merged
Damien merged 5 commits from feature/42-infrahub-client into main 2026-02-27 12:08:36 +00:00
Conversation 0 Commits 5 Files Changed 10 +66 -157
1 changed files with 66 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit b31632109a - Show all commits

223
README.md
View File

@@ -24,14 +24,13 @@ Think `terraform plan` and `terraform apply`, but for your network fabric — po
We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source of Truth for several reasons:
| Feature | NetBox | InfraHub |
| ------------------- | --------------------- | ------------------------------------ |
| **Schema** | Fixed DCIM/IPAM model | Fully customizable YAML schema |
| **Git Integration** | External sync needed | Native - branches = data branches |
| **Versioning** | Changelog only | True Git-like versioning with merges |
| **Test/Redeploy** | Dump/restore | `git clone` = complete environment |
| **Transforms** | Limited | Built-in Jinja2 + Python transforms |
| **GraphQL** | Yes | Yes (auto-generated from schema) |
| Feature | NetBox | InfraHub |
| ------------------- | ----------------------- | -------------------------------------------- |
| **Schema** | Fixed DCIM/IPAM model | Fully customizable YAML schema |
| **Git Integration** | External sync needed | Native - branches = data branches |
| **Versioning** | Changelog only | True Git-like versioning with merges |
| **Transforms** | Limited | Built-in Jinja2 + Python transforms |
| **GraphQL** | Yes | Yes (auto-generated from schema) |
**Key benefits for this project:**
@@ -40,96 +39,40 @@ We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source o
3. **Transforms** - Generate device configs directly from InfraHub
4. **Branches** - Test fabric changes in isolated branches before merge
## 📦 Repository as InfraHub Backend
This repository serves as the **single source of truth** for both code and infrastructure data:
```
fabric-orchestrator/
├── .infrahub.yml # InfraHub repository config
│
├── schemas/ # InfraHub schema definitions
│ └── fabric.yml # Custom EVPN-VXLAN fabric schema
│
├── data/ # Infrastructure objects (YAML)
│ ├── topology/
│ │ ├── sites.yml
│ │ └── devices.yml # Spines, Leafs, VTEP pairs
│ ├── network/
│ │ ├── vlans.yml # VLANs + L2VNI mappings
│ │ ├── vrfs.yml # VRFs + L3VNI mappings
│ │ └── interfaces.yml # Interface configs
│ └── routing/
│ ├── bgp_sessions.yml # Underlay + EVPN overlay
│ └── evpn.yml # Route targets, RDs
│
├── transforms/ # Jinja2 templates for config generation
│ └── arista/
│ ├── base.j2
│ ├── interfaces.j2
│ ├── bgp.j2
│ └── evpn.j2
│
└── src/ # Python orchestration code
```
### Workflow
```bash
# 1. Edit data files (e.g., add a VLAN)
vim data/network/vlans.yml
# 2. Commit & push
git commit -am "Add VLAN 100 for production"
git push
# 3. InfraHub syncs automatically from Git
# → Data available via GraphQL
# 4. Prefect flow detects change → reconciles fabric
```
### Benefits
- **Reproductibility**: `git clone` → `docker compose up` → complete environment
- **Code Review**: Infrastructure changes go through PR review
- **History**: Full audit trail via Git
- **Testing**: Create a branch, test changes, merge when validated
## 🎛 Why Prefect?
| Feature | Benefit |
| -------------------------------- | ---------------------------------------------------------------------- |
| **Python-native workflows** | Use `@flow` and `@task` decorators — no YAML, just Python |
| **Free secrets management** | Native `Secret` blocks for credentials (free in OSS) |
| **Built-in UI** | Dashboard, logs, metrics, execution history via `prefect server start` |
| **No containerization required** | Run flows directly with `.serve()` — no Docker needed |
| **Event-driven triggers** | Schedule, webhooks (via FastAPI), flow triggers out of the box |
| **Task dependencies** | Automatic dependency ordering via task result passing or `wait_for` |
| **Retry & error handling** | Built-in retry policies with `@task(retries=3)` |
| **Human-in-the-loop** | Native `pause_flow_run()` for approval workflows |
| Feature | Benefit |
| -------------------------------- | ------------------------------------------------------------------------- |
| **Python-native workflows** | Use `@flow` and `@task` decorators — no YAML, just Python |
| **Free secrets management** | Native `Secret` blocks for credentials (free in OSS) |
| **Built-in UI** | Dashboard, logs, metrics, execution history via `prefect server start` |
| **No containerization required** | Run flows directly with `.serve()` — no Docker needed |
| **Event-driven triggers** | Schedule, webhooks (via FastAPI), flow triggers out of the box |
| **Task dependencies** | Automatic dependency ordering via task result passing or `wait_for` |
| **Retry & error handling** | Built-in retry policies with `@task(retries=3)` |
| **Human-in-the-loop** | Native `pause_flow_run()` for approval workflows |
## 🎯 Target Fabric
This project is designed for the Arista EVPN-VXLAN ContainerLab topology:
This project is designed for Arista EVPN-VXLAN fabrics:
- **2 Spines** (BGP Route Reflectors, AS 65000)
- **8 Leafs** (4 MLAG VTEP pairs, AS 65001-65004)
- **2 Spines** (BGP Route Reflectors)
- **8 Leafs** (4 MLAG VTEP pairs)
- **cEOS 4.35.0F** with gNMI enabled
- **EVPN Type-2** (L2 VXLAN) and **Type-5** (L3 VXLAN) support
Reference: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab)
Reference lab topology: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab)
## 📋 Project Phases
Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/fabric-orchestrator/issues) or filter by phase:
| Phase | Description | Status |
| ----------- | -------------------------------------------------------------------- | ------------- |
| **Phase 1** | YANG Path Discovery - Map EOS 4.35.0F YANG models, validate gNMI | ✅ Complete |
| **Phase 2** | InfraHub Setup & Core Reconciler - Schema, diff engine, YANG mappers | 🔄 In Progress |
| **Phase 3** | Full Fabric Coverage - BGP, MLAG, VRFs mappers | 📋 Planned |
| **Phase 4** | Prefect Integration - Flows, webhooks, drift detection | 📋 Planned |
| Phase | Description | Status |
| ----------- | ---------------------------------------------------------------------------- | -------------- |
| **Phase 1** | YANG Path Discovery - Map EOS 4.35.0F YANG models, validate gNMI | ✅ Complete |
| **Phase 2** | InfraHub Client & Core Reconciler - SDK client, diff engine, YANG mappers | 🔄 In Progress |
| **Phase 3** | Full Fabric Coverage - BGP, MLAG, VRFs mappers | 📋 Planned |
| **Phase 4** | Prefect Integration - Flows, webhooks, drift detection | 📋 Planned |
## 📁 Project Structure
@@ -137,52 +80,27 @@ Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/
fabric-orchestrator/
├── README.md
├── pyproject.toml
├── .infrahub.yml # InfraHub config (points to schemas/)
│
├── schemas/ # InfraHub schema definitions
│ └── fabric.yml # Custom EVPN-VXLAN fabric schema
│
├── data/ # Infrastructure data (YAML)
│ ├── topology/
│ │ ├── sites.yml
│ │ └── devices.yml
│ ├── network/
│ │ ├── vlans.yml
│ │ ├── vrfs.yml
│ │ └── interfaces.yml
│ └── routing/
│ ├── bgp_sessions.yml
│ └── evpn.yml
│
├── transforms/ # Jinja2 config templates
│ └── arista/
│ └── *.j2
│
├── src/ # Python package
├── src/ # Python package
│ ├── __init__.py
│ ├── cli.py # CLI for YANG discovery
│ │
│ ├── flows/ # Prefect flows
│ │ ├── __init__.py
│ │ ├── reconcile.py # @flow fabric_reconcile
│ │ ├── drift.py # @flow handle_drift
│ │ └── remediation.py # @flow drift_remediation
│ ├── cli.py # CLI for YANG discovery
│ │
│ ├── gnmi/
│ │ ├── __init__.py
│ │ ├── client.py # gNMI client wrapper (pygnmi)
│ │ ├── client.py # gNMI client wrapper (pygnmi)
│ │ └── README.md
│ │
│ ├── infrahub/ # InfraHub integration
│ ├── infrahub/ # InfraHub integration
│ │ ├── __init__.py
│ │ ├── client.py # InfraHub SDK wrapper
│ │ └── queries.py # GraphQL queries
│ │ ├── client.py # InfraHub SDK wrapper
│ │ ├── models.py # Pydantic intent models
│ │ └── exceptions.py # Client exceptions
│ │
│ └── yang/
│ ├── __init__.py
│ ├── mapper.py # InfraHub intent → YANG paths
│ ├── paths.py # YANG path definitions
│ └── mappers/ # Resource-specific mappers
│ ├── mapper.py # InfraHub intent → YANG paths
│ ├── paths.py # YANG path definitions
│ └── mappers/ # Resource-specific mappers
│ ├── vlan.py
│ ├── interface.py
│ ├── bgp.py
@@ -197,21 +115,20 @@ fabric-orchestrator/
## 🛠️ Technology Stack
| Component | Technology | Purpose |
| --------------- | -------------------------- | ------------------------------------ |
| Source of Truth | **InfraHub** | Intent definition via custom schema |
| Data Storage | **This Git repo** | Schema + data versioned together |
| Orchestrator | **Prefect** | Python-native workflow orchestration |
| Transport | gNMI | Configuration and telemetry |
| Data Models | YANG (OpenConfig + Arista) | Structured configuration |
| Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions |
| CLI | Click + Rich | YANG discovery tools |
| Validation | Pydantic v2 | Intent data validation |
| Lab | ContainerLab + cEOS | Development environment |
| Component | Technology | Purpose |
| --------------- | ------------------------------- | ------------------------------------------ |
| Source of Truth | **InfraHub** | Intent definition via custom schema |
| Orchestrator | **Prefect** | Python-native workflow orchestration |
| Transport | gNMI | Configuration and telemetry |
| Data Models | YANG (OpenConfig + Arista) | Structured configuration |
| Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions |
| CLI | Click + Rich | YANG discovery tools |
| Validation | Pydantic v2 | Intent data validation |
| Lab | ContainerLab + cEOS | Development environment |
## 🔗 Related Projects
- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Target fabric topology
- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Lab topology, InfraHub schemas & data
- [InfraHub](https://github.com/opsmill/infrahub) - Source of Truth platform
- [InfraHub Schema Library](https://github.com/opsmill/schema-library) - Reference schemas
- [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions
@@ -222,7 +139,6 @@ fabric-orchestrator/
- [InfraHub Documentation](https://docs.infrahub.app)
- [InfraHub Schema Guide](https://docs.infrahub.app/guides/create-schema)
- [InfraHub Python SDK](https://github.com/opsmill/infrahub-sdk-python)
- [InfraHub .infrahub.yml Reference](https://docs.infrahub.app/reference/dotinfrahub)
### Prefect
- [Prefect Documentation](https://docs.prefect.io)
@@ -243,31 +159,22 @@ fabric-orchestrator/
- Python 3.12+
- `uv` package manager
- Docker (for InfraHub)
- Access to ContainerLab with cEOS images
- Access to an InfraHub instance with the EVPN-VXLAN fabric schema loaded
- Access to ContainerLab with cEOS images (for lab testing)
### Quick Start
```bash
# Clone the repository (includes schema + data)
# Clone the repository
git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git
cd fabric-orchestrator
# Install Python dependencies
uv sync
# Start InfraHub (loads schema & data from this repo)
docker compose up -d
# Configure Prefect secrets
python -c "
from prefect.blocks.system import Secret
from prefect.variables import Variable
Secret(value='your-gnmi-password').save('gnmi-password', overwrite=True)
Variable.set('infrahub_url', 'http://localhost:8000')
Variable.set('gnmi_username', 'admin')
"
# Set InfraHub connection (point to your InfraHub instance)
export INFRAHUB_ADDRESS="http://localhost:8000"
export INFRAHUB_API_TOKEN="your-token"
# Verify gNMI connectivity
uv run fabric-orch discover capabilities --target leaf1:6030
@@ -285,13 +192,15 @@ from prefect.variables import Variable
@task(retries=2, retry_delay_seconds=10)
def get_fabric_intent(device: str | None = None) -> dict:
async def get_fabric_intent(device: str | None = None) -> dict:
"""Retrieve fabric intent from InfraHub."""
from infrahub_sdk import InfrahubClient
client = InfrahubClient(address=Variable.get("infrahub_url"))
# Query fabric intent via GraphQL
return client.query(...)
from src.infrahub.client import FabricInfrahubClient
async with FabricInfrahubClient(
url=Variable.get("infrahub_url"),
api_token=Variable.get("infrahub_token"),
) as client:
return await client.get_device(device)
@task
@@ -307,17 +216,17 @@ def fabric_reconcile(device: str | None = None, dry_run: bool = True) -> dict:
intent = get_fabric_intent(device)
current = get_current_state(device)
changes = compute_diff(intent, current)
if not changes:
print("✅ Fabric is in sync")
return {"in_sync": True}
if not dry_run:
apply_changes(changes)
return {"changes": changes, "applied": not dry_run}
```
---
**Status**: 🚧 Active Development - Phase 2 (InfraHub Setup & Core Reconciler)
**Status**: 🚧 Active Development - Phase 2 (InfraHub Client & Core Reconciler)