Damien/fabric-orchestrator
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

docs: Add Git-as-backend architecture for InfraHub

Browse Source 
- Repository now serves as InfraHub backend (schema + data in Git)
- Add data/ directory structure for infrastructure objects
- Add transforms/ for Jinja2 config templates
- Update architecture diagram to show Git-centric workflow
- Add "Repository as InfraHub Backend" section explaining benefits
- Simplify project structure to reflect new approach
This commit is contained in:
Damien Arnodo
2026-02-05 08:48:34 +00:00
parent 77ca22bd0a
commit 383aa6b35f
1 changed files with 134 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files

221
README.md
View File

@@ -21,15 +21,27 @@ Think `terraform plan` and `terraform apply`, but for your network fabric — po
``` ```
┌──────────────────────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────────────────────┐
│ INTENT LAYER │ │ INTENT LAYER │
│ ┌─────────────────────────┐ ┌──────────────────────────────────────────┐ │ │ │
│ │ InfraHub │ │ Git Repository │ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ (Source of Truth) │◄──►│ - Schema definitions (YAML) │ │ │ │ This Git Repository │ │
│ │ │ │ - Transforms (Jinja2/Python) │ │ │ │ │ │
│ │ • Custom fabric schema │ │ - Version-controlled intent │ │ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ │ │
│ │ • GraphQL API │ └──────────────────────────────────────────┘ │ │ │ │ schemas/ │ │ data/ │ │ transforms/ │ │ │
│ │ • Branch-based changes │ │ │ │ │ └─fabric.yml │ │ ├─devices.yml │ │ └─arista_config.j2 │ │ │
│ └────────────┬────────────┘ │ │ │ │ (InfraHub │ │ ├─vlans.yml │ │ (Jinja2 templates) │ │ │
└───────────────┼──────────────────────────────────────────────────────────────┘ │ │ │ schema) │ │ ├─bgp.yml │ │ │ │ │
│ │ │ │ │ └─... │ │ │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ InfraHub │ │
│ │ (Runtime + GraphQL API) │ │
│ │ Syncs schema & data from Git, exposes via GraphQL │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘
│
│ GraphQL / SDK │ GraphQL / SDK
▼ ▼
┌──────────────────────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────────────────────┐
@@ -50,11 +62,7 @@ Think `terraform plan` and `terraform apply`, but for your network fabric — po
│ │ │ (InfraHub→YANG) │ │ (Want vs Have) │ │ (pygnmi wrapper) ││ │ │ │ │ (InfraHub→YANG) │ │ (Want vs Have) │ │ (pygnmi wrapper) ││ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────────────┘│ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────────────┘│ │
│ └────────────────────────────────────────────────────────────────────────┘ │ │ └────────────────────────────────────────────────────────────────────────┘ │
│ │ └──────────────────────────────────────┬───────────────────────────────────────┘
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Prefect Server (UI) │ Prefect .serve() │ Webhook Receiver │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────┬───────────────────────────────────────────────────┘
│ gNMI Get/Set/Subscribe │ gNMI Get/Set/Subscribe
▼ ▼
┌──────────────────────────────────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────────────────────────────────────┐
@@ -86,9 +94,63 @@ We chose [InfraHub](https://github.com/opsmill/infrahub) over NetBox as Source o
3. **Transforms** - Generate device configs directly from InfraHub 3. **Transforms** - Generate device configs directly from InfraHub
4. **Branches** - Test fabric changes in isolated branches before merge 4. **Branches** - Test fabric changes in isolated branches before merge
## 🎛 Why Prefect? ## 📦 Repository as InfraHub Backend
We chose [Prefect](https://prefect.io) as the orchestration engine for several reasons: This repository serves as the **single source of truth** for both code and infrastructure data:
```
fabric-orchestrator/
├── .infrahub.yml # InfraHub repository config
│
├── schemas/ # InfraHub schema definitions
│ └── fabric.yml # Custom EVPN-VXLAN fabric schema
│
├── data/ # Infrastructure objects (YAML)
│ ├── topology/
│ │ ├── sites.yml
│ │ └── devices.yml # Spines, Leafs, VTEP pairs
│ ├── network/
│ │ ├── vlans.yml # VLANs + L2VNI mappings
│ │ ├── vrfs.yml # VRFs + L3VNI mappings
│ │ └── interfaces.yml # Interface configs
│ └── routing/
│ ├── bgp_sessions.yml # Underlay + EVPN overlay
│ └── evpn.yml # Route targets, RDs
│
├── transforms/ # Jinja2 templates for config generation
│ └── arista/
│ ├── base.j2
│ ├── interfaces.j2
│ ├── bgp.j2
│ └── evpn.j2
│
└── src/ # Python orchestration code
```
### Workflow
```bash
# 1. Edit data files (e.g., add a VLAN)
vim data/network/vlans.yml
# 2. Commit & push
git commit -am "Add VLAN 100 for production"
git push
# 3. InfraHub syncs automatically from Git
# → Data available via GraphQL
# 4. Prefect flow detects change → reconciles fabric
```
### Benefits
- **Reproductibility**: `git clone` → `docker compose up` → complete environment
- **Code Review**: Infrastructure changes go through PR review
- **History**: Full audit trail via Git
- **Testing**: Create a branch, test changes, merge when validated
## 🎛 Why Prefect?
| Feature | Benefit | | Feature | Benefit |
|---------|---------| |---------|---------|
@@ -129,49 +191,62 @@ Progress is tracked via issues. See [all issues](https://gitea.arnodo.fr/Damien/
fabric-orchestrator/ fabric-orchestrator/
├── README.md ├── README.md
├── pyproject.toml ├── pyproject.toml
├── .infrahub.yml # InfraHub config (points to schemas/)
│
├── schemas/ # InfraHub schema definitions
│ └── fabric.yml # Custom EVPN-VXLAN fabric schema
│
├── data/ # Infrastructure data (YAML)
│ ├── topology/
│ │ ├── sites.yml
│ │ └── devices.yml
│ ├── network/
│ │ ├── vlans.yml
│ │ ├── vrfs.yml
│ │ └── interfaces.yml
│ └── routing/
│ ├── bgp_sessions.yml
│ └── evpn.yml
│
├── transforms/ # Jinja2 config templates
│ └── arista/
│ └── *.j2
│ │
├── src/ # Python package ├── src/ # Python package
│ ├── __init__.py │ ├── __init__.py
│ ├── cli.py # CLI for YANG discovery (discover commands) │ ├── cli.py # CLI for YANG discovery
│ │ │ │
│ ├── flows/ # Prefect flows │ ├── flows/ # Prefect flows
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── reconcile.py # @flow fabric_reconcile (plan/apply) │ │ ├── reconcile.py # @flow fabric_reconcile
│ │ ├── drift.py # @flow handle_drift │ │ ├── drift.py # @flow handle_drift
│ │ └── remediation.py # @flow drift_remediation │ │ └── remediation.py # @flow drift_remediation
│ │ │ │
│ ├── api/ # FastAPI webhook receiver
│ │ ├── __init__.py
│ │ └── webhooks.py # InfraHub webhook endpoint
│ │
│ ├── services/ # Long-running services
│ │ ├── __init__.py
│ │ └── drift_monitor.py # gNMI Subscribe drift detection
│ │
│ ├── gnmi/ │ ├── gnmi/
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── client.py # gNMI client wrapper (pygnmi) │ │ ├── client.py # gNMI client wrapper (pygnmi)
│ │ └── README.md │ │ └── README.md
│ │ │ │
│ ├── infrahub/ # InfraHub integration (TODO) │ ├── infrahub/ # InfraHub integration
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── client.py # InfraHub SDK client │ │ ├── client.py # InfraHub SDK wrapper
│ │ └── models.py # Pydantic models for intent validation │ │ └── queries.py # GraphQL queries
│ │ │ │
│ └── yang/ │ └── yang/
│ ├── __init__.py │ ├── __init__.py
│ ├── mapper.py # InfraHub intent → YANG paths │ ├── mapper.py # InfraHub intent → YANG paths
│ ├── paths.py # YANG path definitions │ ├── paths.py # YANG path definitions
│ └── dependencies.py # Dependency ordering graph │ └── mappers/ # Resource-specific mappers
│ │ ├── vlan.py
├── schemas/ # InfraHub schema definitions (TODO) │ ├── interface.py
│ └── fabric.yml # Custom fabric schema │ ├── bgp.py
│ └── vxlan.py
│ │
├── tests/ ├── tests/
│ │
└── docs/ └── docs/
├── cli-user-guide.md # CLI documentation ├── cli-user-guide.md
└── yang-paths.md # Documented YANG paths └── yang-paths.md
``` ```
## 🛠️ Technology Stack ## 🛠️ Technology Stack
@@ -179,8 +254,8 @@ fabric-orchestrator/
| Component | Technology | Purpose | | Component | Technology | Purpose |
|-----------|------------|---------| |-----------|------------|---------|
| Source of Truth | **InfraHub** | Intent definition via custom schema | | Source of Truth | **InfraHub** | Intent definition via custom schema |
| Data Storage | **This Git repo** | Schema + data versioned together |
| Orchestrator | **Prefect** | Python-native workflow orchestration | | Orchestrator | **Prefect** | Python-native workflow orchestration |
| Webhooks | FastAPI | Receive InfraHub webhooks |
| Transport | gNMI | Configuration and telemetry | | Transport | gNMI | Configuration and telemetry |
| Data Models | YANG (OpenConfig + Arista) | Structured configuration | | Data Models | YANG (OpenConfig + Arista) | Structured configuration |
| Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions | | Python Library | pygnmi + infrahub-sdk | gNMI/InfraHub interactions |
@@ -194,7 +269,6 @@ fabric-orchestrator/
- [InfraHub](https://github.com/opsmill/infrahub) - Source of Truth platform - [InfraHub](https://github.com/opsmill/infrahub) - Source of Truth platform
- [InfraHub Schema Library](https://github.com/opsmill/schema-library) - Reference schemas - [InfraHub Schema Library](https://github.com/opsmill/schema-library) - Reference schemas
- [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions - [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions
- [Prefect Documentation](https://docs.prefect.io) - Orchestration platform docs
## 📚 References ## 📚 References
@@ -202,13 +276,12 @@ fabric-orchestrator/
- [InfraHub Documentation](https://docs.infrahub.app) - [InfraHub Documentation](https://docs.infrahub.app)
- [InfraHub Schema Guide](https://docs.infrahub.app/guides/create-schema) - [InfraHub Schema Guide](https://docs.infrahub.app/guides/create-schema)
- [InfraHub Python SDK](https://github.com/opsmill/infrahub-sdk-python) - [InfraHub Python SDK](https://github.com/opsmill/infrahub-sdk-python)
- [InfraHub .infrahub.yml Reference](https://docs.infrahub.app/reference/dotinfrahub)
### Prefect ### Prefect
- [Prefect Documentation](https://docs.prefect.io) - [Prefect Documentation](https://docs.prefect.io)
- [Prefect Flows](https://docs.prefect.io/latest/develop/write-flows/) - [Prefect Flows](https://docs.prefect.io/latest/develop/write-flows/)
- [Prefect Tasks](https://docs.prefect.io/latest/develop/write-tasks/) - [Prefect Tasks](https://docs.prefect.io/latest/develop/write-tasks/)
- [Prefect Deployments](https://docs.prefect.io/latest/deploy/run-flows-in-local-processes/)
- [Prefect Secrets](https://docs.prefect.io/latest/develop/blocks/#secret)
### YANG / gNMI ### YANG / gNMI
- [Arista gNMI Documentation](https://aristanetworks.github.io/openmgmt/configuration/gnmi/) - [Arista gNMI Documentation](https://aristanetworks.github.io/openmgmt/configuration/gnmi/)
@@ -217,7 +290,6 @@ fabric-orchestrator/
### EVPN-VXLAN ### EVPN-VXLAN
- [Arista BGP EVPN Configuration Example](https://overlaid.net/2019/01/27/arista-bgp-evpn-configuration-example/) - [Arista BGP EVPN Configuration Example](https://overlaid.net/2019/01/27/arista-bgp-evpn-configuration-example/)
- [Arista EVPN Deployment Guide](https://www.arista.com/en/solutions/evpn-vxlan)
## 🚀 Getting Started ## 🚀 Getting Started
@@ -225,21 +297,21 @@ fabric-orchestrator/
- Python 3.12+ - Python 3.12+
- `uv` package manager - `uv` package manager
- Access to ContainerLab with cEOS images
- Docker (for InfraHub) - Docker (for InfraHub)
- Access to ContainerLab with cEOS images
### Quick Start ### Quick Start
```bash ```bash
# Clone the repository # Clone the repository (includes schema + data)
git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git
cd fabric-orchestrator cd fabric-orchestrator
# Install Python dependencies # Install Python dependencies
uv sync uv sync
# Start InfraHub (see InfraHub docs for full setup) # Start InfraHub (loads schema & data from this repo)
# docker compose -f infrahub-docker-compose.yml up -d docker compose up -d
# Configure Prefect secrets # Configure Prefect secrets
python -c " python -c "
@@ -247,28 +319,22 @@ from prefect.blocks.system import Secret
from prefect.variables import Variable from prefect.variables import Variable
Secret(value='your-gnmi-password').save('gnmi-password', overwrite=True) Secret(value='your-gnmi-password').save('gnmi-password', overwrite=True)
Secret(value='your-infrahub-token').save('infrahub-token', overwrite=True)
Variable.set('infrahub_url', 'http://localhost:8000') Variable.set('infrahub_url', 'http://localhost:8000')
Variable.set('gnmi_username', 'admin') Variable.set('gnmi_username', 'admin')
" "
# Start Prefect server (optional, for UI) # Verify gNMI connectivity
prefect server start
# Verify gNMI connectivity to your fabric
uv run fabric-orch discover capabilities --target leaf1:6030 uv run fabric-orch discover capabilities --target leaf1:6030
# Explore YANG paths # Run reconciliation
uv run fabric-orch discover get --target leaf1:6030 \ uv run fabric-orch plan
--path "/interfaces/interface[name=Ethernet1]/state" uv run fabric-orch apply
``` ```
## Prefect Flow Example ## Prefect Flow Example
```python ```python
from prefect import flow, task from prefect import flow, task
from prefect.blocks.system import Secret
from prefect.variables import Variable from prefect.variables import Variable
@@ -277,13 +343,9 @@ def get_fabric_intent(device: str | None = None) -> dict:
"""Retrieve fabric intent from InfraHub.""" """Retrieve fabric intent from InfraHub."""
from infrahub_sdk import InfrahubClient from infrahub_sdk import InfrahubClient
infrahub_url = Variable.get("infrahub_url") client = InfrahubClient(address=Variable.get("infrahub_url"))
infrahub_token = Secret.load("infrahub-token").get()
client = InfrahubClient(address=infrahub_url, api_token=infrahub_token)
# Query fabric intent via GraphQL # Query fabric intent via GraphQL
# ... return client.query(...)
return intent
@task @task
@@ -293,46 +355,23 @@ def compute_diff(intent: dict, current: dict) -> list[dict]:
return diff_engine(want=intent, have=current) return diff_engine(want=intent, have=current)
@task(retries=1)
def apply_changes(changes: list[dict], dry_run: bool = True) -> dict:
"""Apply changes via gNMI Set."""
if dry_run:
return {"applied": False, "changes": changes}
# Apply via gNMI...
return {"applied": True, "changes": changes}
@flow(log_prints=True, name="fabric-reconcile") @flow(log_prints=True, name="fabric-reconcile")
def fabric_reconcile( def fabric_reconcile(device: str | None = None, dry_run: bool = True) -> dict:
device: str | None = None,
auto_apply: bool = False,
dry_run: bool = True
) -> dict:
"""Reconcile fabric state with InfraHub intent.""" """Reconcile fabric state with InfraHub intent."""
print(f"🔄 Starting fabric reconciliation")
intent = get_fabric_intent(device) intent = get_fabric_intent(device)
current = get_current_state(devices) current = get_current_state(device)
changes = compute_diff(intent, current) changes = compute_diff(intent, current)
if not changes: if not changes:
print("✅ No changes detected - fabric is in sync") print("✅ Fabric is in sync")
return {"changes": [], "in_sync": True} return {"in_sync": True}
should_apply = auto_apply and not dry_run if not dry_run:
result = apply_changes(changes, dry_run=not should_apply) apply_changes(changes)
return {"changes": changes, "applied": should_apply} return {"changes": changes, "applied": not dry_run}
if __name__ == "__main__":
fabric_reconcile.serve(
name="fabric-reconcile-scheduled",
cron="0 */6 * * *",
tags=["network", "fabric"]
)
``` ```
--- ---
**Status**: 🚧 Active Development - Migrating to InfraHub as Source of Truth **Status**: 🚧 Active Development - Phase 2 (InfraHub Setup & Core Reconciler)