Damien/fabric-orchestrator
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: migrate orchestration layer to Kestra #27

Merged
Damien merged 1 commits from feat/kestra-orchestration into main 2026-01-10 14:35:43 +00:00
Conversation 0 Commits 1 Files Changed 1 +183 -76
Damien commented 2026-01-10 14:34:24 +00:00
Owner
Copy Link

Summary

This PR migrates the orchestration layer from a custom Python asyncio/Redis architecture to Kestra, a modern workflow orchestration platform.

Why Kestra?

Aspect Before (Custom) After (Kestra)
Event handling Redis pub/sub + custom handlers Native Flow Triggers + Webhooks
Webhooks FastAPI server to build Built-in Webhook Triggers
UI/Dashboard Custom to build Included (logs, metrics, executions)
Workflow definition Python code Declarative YAML
Scheduling Custom cron implementation Native cron triggers
Approval flows Not planned Native Pause/Resume tasks

Changes

README Updated

  • New architecture diagram with Kestra workflow layer
  • "Why Kestra?" section explaining the choice
  • Updated project structure for Kestra workflows
  • Updated technology stack table
  • Added Kestra workflow example

Issues Updated

Closed (replaced by Kestra native features):

  • #17 - FastAPI webhooks → Kestra Webhook Triggers
  • #18 - Redis event bus → Kestra Flow events
  • #20 - Dashboard/logging → Kestra UI

Adapted for Kestra:

  • #10 - CLI plan/apply → fabric-reconcile.yml workflow
  • #15 - Dependency ordering → Kestra DAG tasks
  • #16 - Drift detection → External service + Kestra webhook
  • #19 - Auto-remediation → drift-remediation.yml workflow

New issues created:

  • #24 - Setup Kestra infrastructure with Docker Compose
  • #25 - Package Python code as container image for Kestra
  • #26 - Create NetBox webhook trigger workflow

New Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         NetBox                                   │
│                    (Source of Truth)                             │
└──────────────────────────┬──────────────────────────────────────┘
                           │ Webhook / API
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                         Kestra                                   │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Workflows (YAML)                                           ││
│  │  - fabric-reconcile.yml    (plan/apply)                     ││
│  │  - netbox-webhook.yml      (intent changes)                 ││
│  │  - drift-detected.yml      (drift events)                   ││
│  │  - drift-remediation.yml   (auto-fix)                       ││
│  └─────────────────────────────────────────────────────────────┘│
│  ┌─────────────────────────────────────────────────────────────┐│
│  │  Python Scripts (containerized)                             ││
│  │  - get_fabric_intent.py                                     ││
│  │  - compute_diff.py                                          ││
│  │  - apply_changes.py                                         ││
│  └─────────────────────────────────────────────────────────────┘│
└──────────────────────────┬──────────────────────────────────────┘
                           │ gNMI Set/Get
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Arista Fabric                                 │
│              (Spines + Leafs via gNMI)                          │
└─────────────────────────────────────────────────────────────────┘

What's Preserved

  • ✅ src/netbox/ - NetBox client and Pydantic models
  • ✅ src/gnmi/ - gNMI client wrapper
  • ✅ src/yang/ - YANG path mappings
  • ✅ src/cli.py - Discovery commands (fabric-orch discover)
  • ✅ All documentation in docs/

Next Steps

After merging, the following issues should be tackled in order:

  1. #24 - Setup Kestra infrastructure (Docker Compose)
  2. #25 - Package Python code as container image
  3. #10 - Create fabric-reconcile.yml workflow
  4. #26 - Create NetBox webhook trigger workflow

Checklist

  • README updated with new architecture
  • Obsolete issues closed (#17, #18, #20)
  • Existing issues adapted for Kestra (#10, #15, #16, #19)
  • New infrastructure issues created (#24, #25, #26)
  • No breaking changes to existing code
## Summary This PR migrates the orchestration layer from a custom Python asyncio/Redis architecture to **Kestra**, a modern workflow orchestration platform. ## Why Kestra? | Aspect | Before (Custom) | After (Kestra) | |--------|-----------------|----------------| | **Event handling** | Redis pub/sub + custom handlers | Native Flow Triggers + Webhooks | | **Webhooks** | FastAPI server to build | Built-in Webhook Triggers | | **UI/Dashboard** | Custom to build | Included (logs, metrics, executions) | | **Workflow definition** | Python code | Declarative YAML | | **Scheduling** | Custom cron implementation | Native cron triggers | | **Approval flows** | Not planned | Native Pause/Resume tasks | ## Changes ### README Updated - New architecture diagram with Kestra workflow layer - "Why Kestra?" section explaining the choice - Updated project structure for Kestra workflows - Updated technology stack table - Added Kestra workflow example ### Issues Updated **Closed (replaced by Kestra native features):** - #17 - FastAPI webhooks → Kestra Webhook Triggers - #18 - Redis event bus → Kestra Flow events - #20 - Dashboard/logging → Kestra UI **Adapted for Kestra:** - #10 - CLI plan/apply → `fabric-reconcile.yml` workflow - #15 - Dependency ordering → Kestra DAG tasks - #16 - Drift detection → External service + Kestra webhook - #19 - Auto-remediation → `drift-remediation.yml` workflow **New issues created:** - #24 - Setup Kestra infrastructure with Docker Compose - #25 - Package Python code as container image for Kestra - #26 - Create NetBox webhook trigger workflow ## New Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ NetBox │ │ (Source of Truth) │ └──────────────────────────┬──────────────────────────────────────┘ │ Webhook / API ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Kestra │ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ Workflows (YAML) ││ │ │ - fabric-reconcile.yml (plan/apply) ││ │ │ - netbox-webhook.yml (intent changes) ││ │ │ - drift-detected.yml (drift events) ││ │ │ - drift-remediation.yml (auto-fix) ││ │ └─────────────────────────────────────────────────────────────┘│ │ ┌─────────────────────────────────────────────────────────────┐│ │ │ Python Scripts (containerized) ││ │ │ - get_fabric_intent.py ││ │ │ - compute_diff.py ││ │ │ - apply_changes.py ││ │ └─────────────────────────────────────────────────────────────┘│ └──────────────────────────┬──────────────────────────────────────┘ │ gNMI Set/Get ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Arista Fabric │ │ (Spines + Leafs via gNMI) │ └─────────────────────────────────────────────────────────────────┘ ``` ## What's Preserved - ✅ `src/netbox/` - NetBox client and Pydantic models - ✅ `src/gnmi/` - gNMI client wrapper - ✅ `src/yang/` - YANG path mappings - ✅ `src/cli.py` - Discovery commands (`fabric-orch discover`) - ✅ All documentation in `docs/` ## Next Steps After merging, the following issues should be tackled in order: 1. **#24** - Setup Kestra infrastructure (Docker Compose) 2. **#25** - Package Python code as container image 3. **#10** - Create `fabric-reconcile.yml` workflow 4. **#26** - Create NetBox webhook trigger workflow ## Checklist - [x] README updated with new architecture - [x] Obsolete issues closed (#17, #18, #20) - [x] Existing issues adapted for Kestra (#10, #15, #16, #19) - [x] New infrastructure issues created (#24, #25, #26) - [x] No breaking changes to existing code
Damien added 1 commit 2026-01-10 14:34:24 +00:00
docs: update README with Kestra orchestration architecture 7be72392b6 
- Replace custom Python asyncio/Redis architecture with Kestra
- Update architecture diagram to show Kestra workflow layer
- Add "Why Kestra?" section explaining the choice
- Update project structure for Kestra workflows
- Update technology stack table
- Add Kestra workflow example
- Update references with Kestra documentation
Damien merged commit 0d174cf6b8 into main 2026-01-10 14:35:43 +00:00
Damien deleted branch feat/kestra-orchestration 2026-01-10 14:35:47 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
documentation
Documentation tasks
 phase-1-yang-discovery
Phase 1: YANG Path Discovery and Validation
 phase-2-minimal-reconciler
Phase 2: Minimal Reconciler (VLANs, VNIs)
 phase-3-full-fabric
Phase 3: Full Fabric Coverage (BGP, MLAG, VRFs)
 phase-4-event-driven
Phase 4: Event-Driven Automation and CLI
 research
Research and investigation tasks
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Damien
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Damien/fabric-orchestrator#27
Delete Branch "feat/kestra-orchestration"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?