130 lines
3.4 KiB
Markdown
130 lines
3.4 KiB
Markdown
# Prefect Deployment
|
|
|
|
GitOps deployment for Prefect workflow orchestration with Tailscale HTTPS access via Komodo.
|
|
|
|
## Prerequisites
|
|
|
|
### PostgreSQL Database
|
|
|
|
Connect to your PostgreSQL server and create the database:
|
|
|
|
```sql
|
|
CREATE DATABASE prefect;
|
|
CREATE USER prefect WITH PASSWORD 'your-secure-password';
|
|
GRANT ALL PRIVILEGES ON DATABASE prefect TO prefect;
|
|
|
|
\c prefect
|
|
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
|
GRANT ALL ON SCHEMA public TO prefect;
|
|
```
|
|
|
|
### Tailscale Auth Key
|
|
|
|
Generate a reusable auth key from https://login.tailscale.com/admin/settings/keys
|
|
|
|
## Deployment with Komodo
|
|
|
|
### 1. Add Git Provider (if using private repo)
|
|
|
|
In Komodo UI: Settings → Git Providers → Add your Gitea instance credentials.
|
|
|
|
### 2. Create Stack
|
|
|
|
1. Navigate to **Stacks** → **New Stack**
|
|
2. Configure:
|
|
- **Name**: `prefect`
|
|
- **Server**: Select your target server
|
|
- **Source**: Git Repo
|
|
- **Git Provider**: `gitea.arnodo.fr` (or your provider)
|
|
- **Repo**: `Damien/prefect-deployment`
|
|
- **Branch**: `main`
|
|
- **File Paths**: `docker-compose.yml`
|
|
|
|
### 3. Configure Environment Variables
|
|
|
|
In the stack configuration, add the following environment variables:
|
|
|
|
| Variable | Description | Example |
|
|
|----------|-------------|---------|
|
|
| `TS_AUTHKEY` | Tailscale auth key (reusable) | `tskey-auth-xxx` |
|
|
| `DB_HOST` | PostgreSQL host | `postgresql.taila5ad8.ts.net` |
|
|
| `DB_PORT` | PostgreSQL port | `5432` |
|
|
| `DB_USER` | Database user | `prefect` |
|
|
| `DB_PASSWORD` | Database password | *secret* |
|
|
|
|
> **Tip**: Use Komodo's secret variables (marked with 🔒) for sensitive values.
|
|
|
|
### 4. Deploy
|
|
|
|
Click **Deploy** in Komodo. The stack will clone the repository and start all services.
|
|
|
|
## GitOps Workflow
|
|
|
|
### Auto-deploy on Git Push
|
|
|
|
1. In Komodo, go to your stack settings
|
|
2. Enable **Auto Deploy** on push
|
|
3. Copy the webhook URL
|
|
4. Add it to your Gitea repository: Settings → Webhooks
|
|
|
|
## Access
|
|
|
|
Once deployed: https://prefect.taila5ad8.ts.net
|
|
|
|
## Services
|
|
|
|
| Service | Description | Image |
|
|
|---------|-------------|-------|
|
|
| `tailscale` | HTTPS ingress via Tailscale | `tailscale/tailscale` |
|
|
| `redis` | Messaging broker | `redis:7-alpine` |
|
|
| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` |
|
|
| `prefect-services` | Background services | `prefecthq/prefect:3-latest` |
|
|
| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/damien/prefect-worker-pg-backup` |
|
|
|
|
## Work Pools
|
|
|
|
The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process).
|
|
|
|
## Secrets Management
|
|
|
|
Flow-specific secrets (S3 credentials, database passwords, API keys, etc.) should be managed via **Prefect Blocks**, not environment variables in the compose file.
|
|
|
|
### Creating a Block (example with S3/Garage)
|
|
|
|
```python
|
|
from prefect_aws import AwsCredentials
|
|
|
|
creds = AwsCredentials(
|
|
aws_access_key_id="xxx",
|
|
aws_secret_access_key="xxx",
|
|
aws_endpoint_url="https://s3.taila5ad8.ts.net"
|
|
)
|
|
creds.save("garage-credentials")
|
|
```
|
|
|
|
### Using in a flow
|
|
|
|
```python
|
|
from prefect import flow
|
|
from prefect_aws import AwsCredentials
|
|
|
|
@flow
|
|
def my_backup_flow():
|
|
creds = AwsCredentials.load("garage-credentials")
|
|
# use creds...
|
|
|
|
my_backup_flow.deploy(
|
|
name="my-backup",
|
|
work_pool_name="pg-backup-pool"
|
|
)
|
|
```
|
|
|
|
## Repository Structure
|
|
|
|
```
|
|
prefect-deployment/
|
|
├── docker-compose.yml # Stack definition (relative paths)
|
|
├── serve-config.json # Tailscale HTTPS serve config
|
|
└── README.md
|
|
```
|