[Phase 4] Setup Prefect Infrastructure #38

New Issue

Damien · 2026-01-30T13:26:40Z

Damien commented

2026-01-30 13:26:40 +00:00

Description

Set up Prefect infrastructure for running orchestration flows. Prefect can run without containers via .serve(), but we'll set up a Prefect server for the UI and scheduling.

Context

Prefect offers multiple deployment options:

Local execution - Run flows directly with .serve() (no infrastructure needed)
Prefect server - Self-hosted UI, logging, scheduling via prefect server start
Prefect Cloud - Managed service (optional, free tier available)

For the homelab, the local Prefect server is ideal.

Tasks

Add Prefect to project dependencies
Create Prefect secrets and variables configuration script
Document local server setup (prefect server start)
Create Docker Compose for Prefect server (optional)
Test flow execution with .serve()
Configure InfraHub connection variables

Prefect Dependencies

# pyproject.toml
dependencies = [
    "prefect>=3.0.0",
    "infrahub-sdk>=1.0.0",
]

Secrets Configuration

# scripts/setup_prefect.py
from prefect.blocks.system import Secret
from prefect.variables import Variable

# Secrets (sensitive)
Secret(value="your-infrahub-api-token").save("infrahub-token", overwrite=True)
Secret(value="your-gnmi-password").save("gnmi-password", overwrite=True)

# Variables (non-sensitive)
Variable.set("infrahub_url", "http://localhost:8000")
Variable.set("gnmi_username", "admin")
Variable.set("gnmi_port", "6030")

# Fabric devices (from InfraHub, but can be cached here)
Variable.set("fabric_spines", '["spine1", "spine2"]')
Variable.set("fabric_leafs", '["leaf1", "leaf2", "leaf3", "leaf4", "leaf5", "leaf6", "leaf7", "leaf8"]')

Docker Compose (Optional)

# docker-compose.yml
services:
  # InfraHub (Source of Truth)
  infrahub:
    image: opsmill/infrahub:stable
    ports:
      - "8000:8000"
    volumes:
      - ./schemas:/opt/infrahub/schemas:ro
      - ./data:/opt/infrahub/data:ro
    environment:
      INFRAHUB_DB_TYPE: sqlite

  # Prefect Server (Orchestration UI)
  prefect-server:
    image: prefecthq/prefect:3-python3.12
    command: prefect server start --host 0.0.0.0
    ports:
      - "4200:4200"
    volumes:
      - prefect-data:/root/.prefect
    environment:
      PREFECT_API_URL: http://localhost:4200/api

  # Optional: Run flows in a worker
  prefect-worker:
    image: prefecthq/prefect:3-python3.12
    command: prefect worker start --pool default-agent-pool
    depends_on:
      - prefect-server
      - infrahub
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
    volumes:
      - ./src:/app/src

volumes:
  prefect-data:

Quick Start (No Docker)

# Install dependencies
uv sync

# Start InfraHub (loads schema + data from this repo)
# See InfraHub docs for setup

# Start Prefect server (terminal 1)
prefect server start

# Configure secrets (terminal 2)
python scripts/setup_prefect.py

# Run a flow with scheduling
python -m src.flows.reconcile

Output Files

scripts/setup_prefect.py - Secrets/variables setup
docker-compose.yml - Full stack (InfraHub + Prefect)
Updated pyproject.toml with Prefect + InfraHub SDK dependencies

Acceptance Criteria

Prefect server runs locally via prefect server start
Secrets and variables configured for InfraHub + gNMI
Flows can be executed and appear in UI
Docker Compose runs full stack (InfraHub + Prefect)
Documentation for setup added to README

Enables: #29 (Plan/Apply CLI), #36 (Drift Detection)
Reference: Prefect Self-Hosting
Reference: InfraHub Docker Setup

## Description Set up Prefect infrastructure for running orchestration flows. Prefect can run without containers via `.serve()`, but we'll set up a Prefect server for the UI and scheduling. ## Context Prefect offers multiple deployment options: 1. **Local execution** - Run flows directly with `.serve()` (no infrastructure needed) 2. **Prefect server** - Self-hosted UI, logging, scheduling via `prefect server start` 3. **Prefect Cloud** - Managed service (optional, free tier available) For the homelab, the local Prefect server is ideal. ## Tasks - [ ] Add Prefect to project dependencies - [ ] Create Prefect secrets and variables configuration script - [ ] Document local server setup (`prefect server start`) - [ ] Create Docker Compose for Prefect server (optional) - [ ] Test flow execution with `.serve()` - [ ] Configure InfraHub connection variables ## Prefect Dependencies ```toml # pyproject.toml dependencies = [ "prefect>=3.0.0", "infrahub-sdk>=1.0.0", ] ``` ## Secrets Configuration ```python # scripts/setup_prefect.py from prefect.blocks.system import Secret from prefect.variables import Variable # Secrets (sensitive) Secret(value="your-infrahub-api-token").save("infrahub-token", overwrite=True) Secret(value="your-gnmi-password").save("gnmi-password", overwrite=True) # Variables (non-sensitive) Variable.set("infrahub_url", "http://localhost:8000") Variable.set("gnmi_username", "admin") Variable.set("gnmi_port", "6030") # Fabric devices (from InfraHub, but can be cached here) Variable.set("fabric_spines", '["spine1", "spine2"]') Variable.set("fabric_leafs", '["leaf1", "leaf2", "leaf3", "leaf4", "leaf5", "leaf6", "leaf7", "leaf8"]') ``` ## Docker Compose (Optional) ```yaml # docker-compose.yml services: # InfraHub (Source of Truth) infrahub: image: opsmill/infrahub:stable ports: - "8000:8000" volumes: - ./schemas:/opt/infrahub/schemas:ro - ./data:/opt/infrahub/data:ro environment: INFRAHUB_DB_TYPE: sqlite # Prefect Server (Orchestration UI) prefect-server: image: prefecthq/prefect:3-python3.12 command: prefect server start --host 0.0.0.0 ports: - "4200:4200" volumes: - prefect-data:/root/.prefect environment: PREFECT_API_URL: http://localhost:4200/api # Optional: Run flows in a worker prefect-worker: image: prefecthq/prefect:3-python3.12 command: prefect worker start --pool default-agent-pool depends_on: - prefect-server - infrahub environment: PREFECT_API_URL: http://prefect-server:4200/api volumes: - ./src:/app/src volumes: prefect-data: ``` ## Quick Start (No Docker) ```bash # Install dependencies uv sync # Start InfraHub (loads schema + data from this repo) # See InfraHub docs for setup # Start Prefect server (terminal 1) prefect server start # Configure secrets (terminal 2) python scripts/setup_prefect.py # Run a flow with scheduling python -m src.flows.reconcile ``` ## Output Files - `scripts/setup_prefect.py` - Secrets/variables setup - `docker-compose.yml` - Full stack (InfraHub + Prefect) - Updated `pyproject.toml` with Prefect + InfraHub SDK dependencies ## Acceptance Criteria - Prefect server runs locally via `prefect server start` - Secrets and variables configured for InfraHub + gNMI - Flows can be executed and appear in UI - Docker Compose runs full stack (InfraHub + Prefect) - Documentation for setup added to README ## Related - Enables: #29 (Plan/Apply CLI), #36 (Drift Detection) - Reference: [Prefect Self-Hosting](https://docs.prefect.io/latest/manage/self-host/) - Reference: [InfraHub Docker Setup](https://docs.infrahub.app/getting-started/)

Damien added the phase-4-event-driven label 2026-01-30 13:26:52 +00:00

Damien referenced this issue

2026-02-05 08:51:56 +00:00

[Phase 2] Implement Plan/Apply CLI Commands #29

Damien referenced this issue

2026-02-05 08:52:38 +00:00

[Phase 4] Implement Drift Detection and Auto-Remediation #36

Damien added this to the Fabric Orchestrator project 2026-02-05 09:05:56 +00:00

Sign in to join this conversation.