docs: Add comprehensive README with architecture and project phases
This commit is contained in:
218
README.md
218
README.md
@@ -1,3 +1,217 @@
|
|||||||
# fabric-orchestrator
|
# Fabric Orchestrator
|
||||||
|
|
||||||
Declarative Network Fabric Orchestrator - Terraform-like infrastructure management for Arista EVPN-VXLAN using gNMI, YANG, and NetBox as Source of Truth
|
**Declarative Network Infrastructure Management for Arista EVPN-VXLAN Fabrics**
|
||||||
|
|
||||||
|
A Terraform-like orchestration system that uses NetBox as Source of Truth and gNMI/YANG for atomic configuration management of Arista data center fabrics.
|
||||||
|
|
||||||
|
## 🎯 Project Vision
|
||||||
|
|
||||||
|
Transform network infrastructure management from imperative scripting to true declarative infrastructure-as-code, where:
|
||||||
|
|
||||||
|
- **Intent** is defined in NetBox (ConfigContexts, Custom Fields)
|
||||||
|
- **State** is continuously monitored via gNMI Subscribe
|
||||||
|
- **Changes** are computed as diffs and applied atomically via gNMI Set
|
||||||
|
- **Drift** is detected and optionally auto-remediated
|
||||||
|
|
||||||
|
Think `terraform plan` and `terraform apply`, but for your network fabric.
|
||||||
|
|
||||||
|
## 🏗️ Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ INTENT LAYER │
|
||||||
|
│ ┌─────────────┐ ┌──────────────────┐ ┌─────────────────────────────┐ │
|
||||||
|
│ │ NetBox │ │ ConfigContexts │ │ Custom Fields / Tags │ │
|
||||||
|
│ │ (SoT) │◄──►│ (Structured │◄──►│ (VLAN, VNI, VRF, BGP AS) │ │
|
||||||
|
│ │ │ │ Intent Data) │ │ │ │
|
||||||
|
│ └──────┬──────┘ └──────────────────┘ └─────────────────────────────┘ │
|
||||||
|
└─────────┼───────────────────────────────────────────────────────────────────┘
|
||||||
|
│ Webhook / Polling
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ ORCHESTRATION LAYER │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────────────────────┐│
|
||||||
|
│ │ State Reconciliation Engine ││
|
||||||
|
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────────────────┐ ││
|
||||||
|
│ │ │ Intent Parser │ │ Diff Engine │ │ Transaction Planner │ ││
|
||||||
|
│ │ │ (NetBox→YANG) │──►│ (Want vs Have)│──►│ (Ordered gNMI SetReqs) │ ││
|
||||||
|
│ │ └───────────────┘ └───────────────┘ └───────────────────────────┘ ││
|
||||||
|
│ └─────────────────────────────────────────────────────────────────────────┘│
|
||||||
|
│ │ │
|
||||||
|
│ ┌─────────────────────────────────┼───────────────────────────────────────┐│
|
||||||
|
│ │ Event Bus (Redis / NATS) ││
|
||||||
|
│ │ • config_drift_detected • intent_changed • apply_complete ││
|
||||||
|
│ └─────────────────────────────────────────────────────────────────────────┘│
|
||||||
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
│ gNMI Subscribe (Telemetry) │ gNMI Set (Config)
|
||||||
|
▼ ▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ DEVICE LAYER │
|
||||||
|
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
|
||||||
|
│ │ spine1 │ │ spine2 │ │ leaf1 │ │ leaf2 │ ... │
|
||||||
|
│ │ gNMI:6030 │ │ gNMI:6030 │ │ gNMI:6030 │ │ gNMI:6030 │ │
|
||||||
|
│ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔧 Target Fabric
|
||||||
|
|
||||||
|
This project is designed for the Arista EVPN-VXLAN ContainerLab topology:
|
||||||
|
|
||||||
|
- **2 Spines** (BGP Route Reflectors, AS 65000)
|
||||||
|
- **8 Leafs** (4 MLAG VTEP pairs, AS 65001-65004)
|
||||||
|
- **cEOS 4.35.0F** with gNMI enabled
|
||||||
|
- **EVPN Type-2** (L2 VXLAN) and **Type-5** (L3 VXLAN) support
|
||||||
|
|
||||||
|
Reference: [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab)
|
||||||
|
|
||||||
|
## 📋 Project Phases
|
||||||
|
|
||||||
|
### Phase 1: YANG Path Discovery (Weeks 1-2)
|
||||||
|
> **Goal**: Map Arista EOS 4.35.0F YANG models and validate gNMI connectivity
|
||||||
|
|
||||||
|
- [ ] Enable and test gNMI on cEOS devices
|
||||||
|
- [ ] Explore available YANG models (OpenConfig vs Native Arista)
|
||||||
|
- [ ] Document working paths for: VLANs, Interfaces, BGP, VXLAN, MLAG
|
||||||
|
- [ ] Create discovery tooling for path exploration
|
||||||
|
- [ ] Build YANG path reference documentation
|
||||||
|
|
||||||
|
### Phase 2: Minimal Reconciler (Weeks 3-4)
|
||||||
|
> **Goal**: Prove the reconciliation pattern with VLANs and VNI mappings
|
||||||
|
|
||||||
|
- [ ] Define NetBox ConfigContext schema for intent
|
||||||
|
- [ ] Implement NetBox → YANG path mapper (VLANs only)
|
||||||
|
- [ ] Build basic diff engine (current vs desired state)
|
||||||
|
- [ ] Implement gNMI Get for state retrieval
|
||||||
|
- [ ] Implement gNMI Set for atomic changes
|
||||||
|
- [ ] Create CLI with `plan` and `apply` commands
|
||||||
|
|
||||||
|
### Phase 3: Full Fabric Coverage (Weeks 5-8)
|
||||||
|
> **Goal**: Extend to complete EVPN-VXLAN fabric configuration
|
||||||
|
|
||||||
|
- [ ] Add BGP underlay configuration
|
||||||
|
- [ ] Add BGP EVPN overlay configuration
|
||||||
|
- [ ] Add MLAG configuration (with coordinated peer updates)
|
||||||
|
- [ ] Add VRF and L3 VXLAN support
|
||||||
|
- [ ] Add interface configuration
|
||||||
|
- [ ] Implement dependency ordering for changes
|
||||||
|
- [ ] Add rollback capability
|
||||||
|
|
||||||
|
### Phase 4: Event-Driven Automation (Weeks 9-10)
|
||||||
|
> **Goal**: Real-time drift detection and NetBox webhook integration
|
||||||
|
|
||||||
|
- [ ] Implement gNMI Subscribe for config change telemetry
|
||||||
|
- [ ] Build drift detection engine
|
||||||
|
- [ ] Add NetBox webhook receiver (FastAPI)
|
||||||
|
- [ ] Implement event bus (Redis/NATS)
|
||||||
|
- [ ] Add auto-remediation option
|
||||||
|
- [ ] Create operational dashboard/logging
|
||||||
|
|
||||||
|
## 📁 Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
fabric-orchestrator/
|
||||||
|
├── README.md
|
||||||
|
├── pyproject.toml
|
||||||
|
├── docker-compose.yml # Redis, API server
|
||||||
|
├── src/
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── cli.py # CLI interface (plan, apply, drift)
|
||||||
|
│ ├── api.py # FastAPI server for webhooks
|
||||||
|
│ ├── reconciler/
|
||||||
|
│ │ ├── __init__.py
|
||||||
|
│ │ ├── engine.py # Core reconciliation logic
|
||||||
|
│ │ ├── diff.py # State comparison
|
||||||
|
│ │ └── planner.py # Change ordering/dependencies
|
||||||
|
│ ├── yang/
|
||||||
|
│ │ ├── __init__.py
|
||||||
|
│ │ ├── mapper.py # NetBox intent → YANG paths
|
||||||
|
│ │ ├── paths.py # YANG path definitions
|
||||||
|
│ │ └── validators.py # Schema validation
|
||||||
|
│ ├── gnmi/
|
||||||
|
│ │ ├── __init__.py
|
||||||
|
│ │ ├── client.py # gNMI client wrapper
|
||||||
|
│ │ └── transactions.py # Atomic operations
|
||||||
|
│ ├── netbox/
|
||||||
|
│ │ ├── __init__.py
|
||||||
|
│ │ ├── client.py # NetBox API client
|
||||||
|
│ │ └── models.py # Intent data models
|
||||||
|
│ └── events/
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── handlers.py # Event handlers
|
||||||
|
│ └── bus.py # Event bus (Redis)
|
||||||
|
├── tests/
|
||||||
|
│ ├── test_reconciler.py
|
||||||
|
│ ├── test_yang_mapper.py
|
||||||
|
│ └── fixtures/
|
||||||
|
│ ├── netbox_intent.json
|
||||||
|
│ └── device_state.json
|
||||||
|
└── docs/
|
||||||
|
├── architecture.md
|
||||||
|
├── yang-paths.md # Documented YANG paths
|
||||||
|
└── netbox-schema.md # ConfigContext schema
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🛠️ Technology Stack
|
||||||
|
|
||||||
|
| Component | Technology | Purpose |
|
||||||
|
|-----------|------------|---------|
|
||||||
|
| Source of Truth | NetBox | Intent definition via ConfigContexts |
|
||||||
|
| Transport | gNMI | Configuration and telemetry |
|
||||||
|
| Data Models | YANG (OpenConfig + Arista) | Structured configuration |
|
||||||
|
| Orchestrator | Python (asyncio) | Reconciliation engine |
|
||||||
|
| CLI | Click + Rich | User interface |
|
||||||
|
| API | FastAPI | Webhook receiver |
|
||||||
|
| Event Bus | Redis | Async event handling |
|
||||||
|
| Lab | ContainerLab + cEOS | Development environment |
|
||||||
|
|
||||||
|
## 🔗 Related Projects
|
||||||
|
|
||||||
|
- [arista-evpn-vxlan-clab](https://gitea.arnodo.fr/Damien/arista-evpn-vxlan-clab) - Target fabric topology
|
||||||
|
- [projet-vxlan-automation](https://gitea.arnodo.fr/Damien/projet-vxlan-automation) - Previous NetBox RenderConfig work
|
||||||
|
- [Arista YANG Models](https://github.com/aristanetworks/yang/tree/master/EOS-4.35.0F) - EOS 4.35.0F YANG definitions
|
||||||
|
|
||||||
|
## 📚 References
|
||||||
|
|
||||||
|
### YANG / gNMI
|
||||||
|
- [Arista gNMI Documentation](https://aristanetworks.github.io/openmgmt/configuration/gnmi/)
|
||||||
|
- [OpenConfig Models](https://github.com/openconfig/public)
|
||||||
|
- [pygnmi Library](https://github.com/akarneliuk/pygnmi)
|
||||||
|
|
||||||
|
### EVPN-VXLAN
|
||||||
|
- [Arista BGP EVPN Configuration Example](https://overlaid.net/2019/01/27/arista-bgp-evpn-configuration-example/)
|
||||||
|
- [Arista EVPN Deployment Guide](https://www.arista.com/en/solutions/evpn-vxlan)
|
||||||
|
|
||||||
|
### Event-Driven Automation
|
||||||
|
- [NetBox Webhooks](https://docs.netbox.dev/en/stable/integrations/webhooks/)
|
||||||
|
- [Event-Driven Ansible](https://www.ansible.com/blog/introducing-event-driven-ansible)
|
||||||
|
|
||||||
|
## 🚀 Getting Started
|
||||||
|
|
||||||
|
*Coming in Phase 1*
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone the repository
|
||||||
|
git clone https://gitea.arnodo.fr/Damien/fabric-orchestrator.git
|
||||||
|
cd fabric-orchestrator
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
pip install -e .
|
||||||
|
|
||||||
|
# Verify gNMI connectivity to your fabric
|
||||||
|
fabric-orch discover --target leaf1:6030
|
||||||
|
|
||||||
|
# Generate execution plan
|
||||||
|
fabric-orch plan
|
||||||
|
|
||||||
|
# Apply changes
|
||||||
|
fabric-orch apply
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📄 License
|
||||||
|
|
||||||
|
MIT License - See [LICENSE](LICENSE) for details.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Status**: 🚧 Active Development - Phase 1
|
||||||
|
|||||||
Reference in New Issue
Block a user