[Phase 2] Setup Kestra infrastructure with Docker Compose #24

New Issue

Damien · 2026-01-10T13:16:42Z

Damien commented

2026-01-10 13:16:42 +00:00

Description

Mettre en place l'infrastructure Kestra pour l'orchestration des workflows fabric-orchestrator.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Docker Compose Stack                          │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐  │
│  │   Kestra    │  │  PostgreSQL │  │  fabric-orchestrator    │  │
│  │   :8080     │  │   :5432     │  │  (Python scripts)       │  │
│  └─────────────┘  └─────────────┘  └─────────────────────────┘  │
│         │                │                                       │
│         └────────────────┴───────────────────────────────────────┤
│                         Docker Network                           │
└─────────────────────────────────────────────────────────────────┘

Docker Compose

# kestra/docker-compose.yml
version: "3.8"

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-kestra}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kestra"]
      interval: 10s
      timeout: 5s
      retries: 5

  kestra:
    image: kestra/kestra:latest
    pull_policy: always
    command: server standalone
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: ${POSTGRES_PASSWORD:-kestra}
        kestra:
          server:
            basicAuth:
              enabled: true
              username: admin
              password: ${KESTRA_PASSWORD:-admin}
          repository:
            type: postgres
          storage:
            type: local
            local:
              basePath: /app/storage
          queue:
            type: postgres
    ports:
      - "8080:8080"
    volumes:
      - kestra_data:/app/storage
      - ./flows:/app/flows:ro
      - ./namespace-files:/app/namespace-files:ro
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  postgres_data:
  kestra_data:

Structure des dossiers

kestra/
├── docker-compose.yml
├── .env.example
├── flows/
│   ├── fabric-reconcile.yml
│   ├── drift-detected.yml
│   ├── drift-remediation.yml
│   └── netbox-webhook.yml
└── namespace-files/
    ├── scripts/
    │   ├── get_fabric_intent.py
    │   ├── get_current_state.py
    │   ├── compute_diff.py
    │   └── apply_changes.py
    └── policies/
        └── remediation.yml

Tasks

Créer kestra/docker-compose.yml
Créer .env.example avec les variables requises
Configurer le namespace network.fabric
Ajouter les secrets Kestra (NETBOX_URL, NETBOX_TOKEN, SLACK_WEBHOOK)
Tester le démarrage de la stack
Documenter l'installation dans le README
Ajouter au .gitignore les fichiers sensibles

Variables d'environnement

# .env.example
POSTGRES_PASSWORD=change_me
KESTRA_PASSWORD=change_me

# Secrets à configurer dans Kestra UI
# NETBOX_URL=https://netbox.example.com
# NETBOX_TOKEN=your_token
# SLACK_WEBHOOK=https://hooks.slack.com/...
# GNMI_USERNAME=admin
# GNMI_PASSWORD=admin

Output

kestra/docker-compose.yml
kestra/.env.example
Documentation dans README

## Description Mettre en place l'infrastructure Kestra pour l'orchestration des workflows fabric-orchestrator. ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Docker Compose Stack │ ├─────────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │ │ Kestra │ │ PostgreSQL │ │ fabric-orchestrator │ │ │ │ :8080 │ │ :5432 │ │ (Python scripts) │ │ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │ │ │ │ │ └────────────────┴───────────────────────────────────────┤ │ Docker Network │ └─────────────────────────────────────────────────────────────────┘ ``` ## Docker Compose ```yaml # kestra/docker-compose.yml version: "3.8" services: postgres: image: postgres:16-alpine environment: POSTGRES_DB: kestra POSTGRES_USER: kestra POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-kestra} volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U kestra"] interval: 10s timeout: 5s retries: 5 kestra: image: kestra/kestra:latest pull_policy: always command: server standalone environment: KESTRA_CONFIGURATION: | datasources: postgres: url: jdbc:postgresql://postgres:5432/kestra driverClassName: org.postgresql.Driver username: kestra password: ${POSTGRES_PASSWORD:-kestra} kestra: server: basicAuth: enabled: true username: admin password: ${KESTRA_PASSWORD:-admin} repository: type: postgres storage: type: local local: basePath: /app/storage queue: type: postgres ports: - "8080:8080" volumes: - kestra_data:/app/storage - ./flows:/app/flows:ro - ./namespace-files:/app/namespace-files:ro depends_on: postgres: condition: service_healthy healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 volumes: postgres_data: kestra_data: ``` ## Structure des dossiers ``` kestra/ ├── docker-compose.yml ├── .env.example ├── flows/ │ ├── fabric-reconcile.yml │ ├── drift-detected.yml │ ├── drift-remediation.yml │ └── netbox-webhook.yml └── namespace-files/ ├── scripts/ │ ├── get_fabric_intent.py │ ├── get_current_state.py │ ├── compute_diff.py │ └── apply_changes.py └── policies/ └── remediation.yml ``` ## Tasks - [ ] Créer `kestra/docker-compose.yml` - [ ] Créer `.env.example` avec les variables requises - [ ] Configurer le namespace `network.fabric` - [ ] Ajouter les secrets Kestra (NETBOX_URL, NETBOX_TOKEN, SLACK_WEBHOOK) - [ ] Tester le démarrage de la stack - [ ] Documenter l'installation dans le README - [ ] Ajouter au `.gitignore` les fichiers sensibles ## Variables d'environnement ```bash # .env.example POSTGRES_PASSWORD=change_me KESTRA_PASSWORD=change_me # Secrets à configurer dans Kestra UI # NETBOX_URL=https://netbox.example.com # NETBOX_TOKEN=your_token # SLACK_WEBHOOK=https://hooks.slack.com/... # GNMI_USERNAME=admin # GNMI_PASSWORD=admin ``` ## Output - `kestra/docker-compose.yml` - `kestra/.env.example` - Documentation dans README

Damien referenced this issue

2026-01-10 13:17:01 +00:00

[Phase 2] Package Python code as container image for Kestra #25

Damien referenced this issue

2026-01-10 13:17:37 +00:00

[Phase 4] Create NetBox webhook trigger workflow #26

Damien added the phase-2-minimal-reconciler label 2026-01-10 13:21:53 +00:00

Damien referenced this issue

2026-01-10 14:34:24 +00:00

feat: migrate orchestration layer to Kestra #27

Sign in to join this conversation.