Prefect Deployment

GitOps deployment for Prefect workflow orchestration with Tailscale HTTPS access.

Prerequisites

PostgreSQL Database

Connect to your PostgreSQL server and create the database:

CREATE DATABASE prefect;
CREATE USER prefect WITH PASSWORD 'your-secure-password';
GRANT ALL PRIVILEGES ON DATABASE prefect TO prefect;

-- PostgreSQL 15+ requires:
\c prefect
GRANT ALL ON SCHEMA public TO prefect;

Host Configuration

Download configuration files to /opt/prefect (first time only):

sudo mkdir -p /opt/prefect/tailscale
sudo curl -o /opt/prefect/tailscale/serve-config.json https://gitea.arnodo.fr/Damien/prefect-deployment/raw/branch/main/serve-config.json

Tailscale Auth Key

Generate a reusable auth key from https://login.tailscale.com/admin/settings/keys

Deployment

1. Create a new stack in Portainer
2. Select "Repository" and point to this repository
3. Portainer will load stack.env automatically
4. Override sensitive values (CHANGE_ME) in the environment variables section:
   • TS_AUTHKEY - Tailscale auth key (reusable recommended)
   • DB_PASSWORD - PostgreSQL password
   • S3_ACCESS_KEY - Garage S3 access key
   • S3_SECRET_KEY - Garage S3 secret key
5. Deploy

Access

Once deployed: https://prefect.taila5ad8.ts.net

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                         Tailscale Network                               │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌──────────────┐     ┌─────────────┐     ┌────────────────┐           │
│  │  Tailscale   │◀───▶│   Prefect   │◀───▶│   PostgreSQL   │           │
│  │   Sidecar    │     │   Server    │     │   (external)   │           │
│  │    :443      │     │   :4200     │     │                │           │
│  └──────────────┘     └──────┬──────┘     └────────────────┘           │
│         ▲                    │                                          │
│         │              ┌─────▼─────┐                                    │
│         │              │   Redis   │                                    │
│         │              │(messaging)│                                    │
│         │              └───────────┘                                    │
│         │                    ▲                                          │
│         │              ┌─────┴─────┐                                    │
│         │              │  Prefect  │                                    │
│         │              │ Services  │                                    │
│         │              │(background)│                                   │
│         │              └───────────┘                                    │
│         │                                                               │
│         │              ┌─────────────────┐     ┌─────────────┐         │
│         └─────────────▶│ Prefect Worker  │◀───▶│   Garage    │         │
│                        │ (pg-backup-pool)│     │    (S3)     │         │
│                        └─────────────────┘     └─────────────┘         │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Services

Service	Description	Image
tailscale	HTTPS ingress via Tailscale	tailscale/tailscale
redis	Messaging broker	redis:7-alpine
prefect-server	API + UI	prefecthq/prefect:3-latest
prefect-services	Background services	prefecthq/prefect:3-latest
prefect-worker-pg-backup	Worker for PostgreSQL backups	gitea.arnodo.fr/damien/prefect-worker-pg-backup

Work Pools

The prefect-worker-pg-backup service automatically creates and listens to the pg-backup-pool work pool (type: process).

To deploy a flow to this pool:

from prefect import flow

@flow
def my_backup_flow():
    ...

my_backup_flow.deploy(
    name="my-backup",
    work_pool_name="pg-backup-pool"
)

Directory Structure

/opt/prefect/
└── tailscale/
    └── serve-config.json     # Tailscale HTTPS configuration