diff --git a/README.md b/README.md index 3999fb5..6a3b869 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,217 @@ -# fabric-orchestrator +# Fabric Orchestrator -Declarative Network Fabric Orchestrator - Terraform-like infrastructure management for Arista EVPN-VXLAN using gNMI, YANG, and NetBox as Source of Truth \ No newline at end of file +**Declarative Network Infrastructure Management for Arista EVPN-VXLAN Fabrics** + +A Terraform-like orchestration system that uses NetBox as Source of Truth and gNMI/YANG for atomic configuration management of Arista data center fabrics. + +## 🎯 Project Vision + +Transform network infrastructure management from imperative scripting to true declarative infrastructure-as-code, where: + +- **Intent** is defined in NetBox (ConfigContexts, Custom Fields) +- **State** is continuously monitored via gNMI Subscribe +- **Changes** are computed as diffs and applied atomically via gNMI Set +- **Drift** is detected and optionally auto-remediated + +Think `terraform plan` and `terraform apply`, but for your network fabric. + +## πŸ—οΈ Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ INTENT LAYER β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ NetBox β”‚ β”‚ ConfigContexts β”‚ β”‚ Custom Fields / Tags β”‚ β”‚ +β”‚ β”‚ (SoT) │◄──►│ (Structured │◄──►│ (VLAN, VNI, VRF, BGP AS) β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Intent Data) β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ Webhook / Polling + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ ORCHESTRATION LAYER β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”β”‚ +β”‚ β”‚ State Reconciliation Engine β”‚β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚β”‚ +β”‚ β”‚ β”‚ Intent Parser β”‚ β”‚ Diff Engine β”‚ β”‚ Transaction Planner β”‚ β”‚β”‚ +β”‚ β”‚ β”‚ (NetBoxβ†’YANG) │──►│ (Want vs Have)│──►│ (Ordered gNMI SetReqs) β”‚ β”‚β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”β”‚ +β”‚ β”‚ Event Bus (Redis / NATS) β”‚β”‚ +β”‚ β”‚ β€’ config_drift_detected β€’ intent_changed β€’ apply_complete β”‚β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ gNMI Subscribe (Telemetry) β”‚ gNMI Set (Config) + β–Ό β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ DEVICE LAYER β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ spine1 β”‚ β”‚ spine2 β”‚ β”‚ leaf1 β”‚ β”‚ leaf2 β”‚ ... β”‚ +β”‚ β”‚ gNMI:6030 β”‚ β”‚ gNMI:6030 β”‚ β”‚ gNMI:6030 β”‚ β”‚ gNMI:6030 β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## πŸ”§ Target Fabric + +This project is designed for the Arista EVPN-VXLAN ContainerLab topology: + +- **2 Spines** (BGP Route Reflectors, AS 65000) +- **8 Leafs** (4 MLAG VTEP pairs, AS 65001-65004) +- **cEOS 4.35.0F** with gNMI enabled +- **EVPN Type-2** (L2 VXLAN) and **Type-5** (L3 VXLAN) support + +Reference: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) + +## πŸ“‹ Project Phases + +### Phase 1: YANG Path Discovery (Weeks 1-2) +> **Goal**: Map Arista EOS 4.35.0F YANG models and validate gNMI connectivity + +- [ ] Enable and test gNMI on cEOS devices +- [ ] Explore available YANG models (OpenConfig vs Native Arista) +- [ ] Document working paths for: VLANs, Interfaces, BGP, VXLAN, MLAG +- [ ] Create discovery tooling for path exploration +- [ ] Build YANG path reference documentation + +### Phase 2: Minimal Reconciler (Weeks 3-4) +> **Goal**: Prove the reconciliation pattern with VLANs and VNI mappings + +- [ ] Define NetBox ConfigContext schema for intent +- [ ] Implement NetBox β†’ YANG path mapper (VLANs only) +- [ ] Build basic diff engine (current vs desired state) +- [ ] Implement gNMI Get for state retrieval +- [ ] Implement gNMI Set for atomic changes +- [ ] Create CLI with `plan` and `apply` commands + +### Phase 3: Full Fabric Coverage (Weeks 5-8) +> **Goal**: Extend to complete EVPN-VXLAN fabric configuration + +- [ ] Add BGP underlay configuration +- [ ] Add BGP EVPN overlay configuration +- [ ] Add MLAG configuration (with coordinated peer updates) +- [ ] Add VRF and L3 VXLAN support +- [ ] Add interface configuration +- [ ] Implement dependency ordering for changes +- [ ] Add rollback capability + +### Phase 4: Event-Driven Automation (Weeks 9-10) +> **Goal**: Real-time drift detection and NetBox webhook integration + +- [ ] Implement gNMI Subscribe for config change telemetry +- [ ] Build drift detection engine +- [ ] Add NetBox webhook receiver (FastAPI) +- [ ] Implement event bus (Redis/NATS) +- [ ] Add auto-remediation option +- [ ] Create operational dashboard/logging + +## πŸ“ Project Structure + +``` +fabric-orchestrator/ +β”œβ”€β”€ README.md +β”œβ”€β”€ pyproject.toml +β”œβ”€β”€ docker-compose.yml # Redis, API server +β”œβ”€β”€ src/ +β”‚ β”œβ”€β”€ __init__.py +β”‚ β”œβ”€β”€ cli.py # CLI interface (plan, apply, drift) +β”‚ β”œβ”€β”€ api.py # FastAPI server for webhooks +β”‚ β”œβ”€β”€ reconciler/ +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ engine.py # Core reconciliation logic +β”‚ β”‚ β”œβ”€β”€ diff.py # State comparison +β”‚ β”‚ └── planner.py # Change ordering/dependencies +β”‚ β”œβ”€β”€ yang/ +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ mapper.py # NetBox intent β†’ YANG paths +β”‚ β”‚ β”œβ”€β”€ paths.py # YANG path definitions +β”‚ β”‚ └── validators.py # Schema validation +β”‚ β”œβ”€β”€ gnmi/ +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ client.py # gNMI client wrapper +β”‚ β”‚ └── transactions.py # Atomic operations +β”‚ β”œβ”€β”€ netbox/ +β”‚ β”‚ β”œβ”€β”€ __init__.py +β”‚ β”‚ β”œβ”€β”€ client.py # NetBox API client +β”‚ β”‚ └── models.py # Intent data models +β”‚ └── events/ +β”‚ β”œβ”€β”€ __init__.py +β”‚ β”œβ”€β”€ handlers.py # Event handlers +β”‚ └── bus.py # Event bus (Redis) +β”œβ”€β”€ tests/ +β”‚ β”œβ”€β”€ test_reconciler.py +β”‚ β”œβ”€β”€ test_yang_mapper.py +β”‚ └── fixtures/ +β”‚ β”œβ”€β”€ netbox_intent.json +β”‚ └── device_state.json +└── docs/ + β”œβ”€β”€ architecture.md + β”œβ”€β”€ yang-paths.md # Documented YANG paths + └── netbox-schema.md # ConfigContext schema +``` + +## πŸ› οΈ Technology Stack + +| Component | Technology | Purpose | +|-----------|------------|---------| +| Source of Truth | NetBox | Intent definition via ConfigContexts | +| Transport | gNMI | Configuration and telemetry | +| Data Models | YANG (OpenConfig + Arista) | Structured configuration | +| Orchestrator | Python (asyncio) | Reconciliation engine | +| CLI | Click + Rich | User interface | +| API | FastAPI | Webhook receiver | +| Event Bus | Redis | Async event handling | +| Lab | ContainerLab + cEOS | Development environment | + +## πŸ”— Related Projects + +- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Target fabric topology +- [projet-vxlan-automation](https://gitea.arnodo.fr/Damien/projet-vxlan-automation) - Previous NetBox RenderConfig work +- [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions + +## πŸ“š References + +### YANG / gNMI +- [Arista gNMI Documentation](https://aristanetworks.github.io/openmgmt/configuration/gnmi/) +- [OpenConfig Models](https://github.com/openconfig/public) +- [pygnmi Library](https://github.com/akarneliuk/pygnmi) + +### EVPN-VXLAN +- [Arista BGP EVPN Configuration Example](https://overlaid.net/2019/01/27/arista-bgp-evpn-configuration-example/) +- [Arista EVPN Deployment Guide](https://www.arista.com/en/solutions/evpn-vxlan) + +### Event-Driven Automation +- [NetBox Webhooks](https://docs.netbox.dev/en/stable/integrations/webhooks/) +- [Event-Driven Ansible](https://www.ansible.com/blog/introducing-event-driven-ansible) + +## πŸš€ Getting Started + +*Coming in Phase 1* + +```bash +# Clone the repository +git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git +cd fabric-orchestrator + +# Install dependencies +pip install -e . + +# Verify gNMI connectivity to your fabric +fabric-orch discover --target leaf1:6030 + +# Generate execution plan +fabric-orch plan + +# Apply changes +fabric-orch apply +``` + +## πŸ“„ License + +MIT License - See [LICENSE](LICENSE) for details. + +--- + +**Status**: 🚧 Active Development - Phase 1