diff --git a/README.md b/README.md index fb76b5c..4029466 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,8 @@ Generate a reusable auth key from https://login.tailscale.com/admin/settings/key 4. Override sensitive values (`CHANGE_ME`) in the environment variables section: - `TS_AUTHKEY` - Tailscale auth key (reusable recommended) - `DB_PASSWORD` - PostgreSQL password + - `S3_ACCESS_KEY` - Garage S3 access key + - `S3_SECRET_KEY` - Garage S3 secret key 5. Deploy ## Access @@ -48,28 +50,62 @@ Once deployed: https://prefect.taila5ad8.ts.net ## Architecture ``` -┌─────────────────────────────────────────────────────────────┐ -│ Tailscale Network │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ │ -│ │ Tailscale │────▶│ Prefect │────▶│ PostgreSQL │ │ -│ │ Sidecar │ │ Server │ │ (external) │ │ -│ │ :443 │ │ :4200 │ │ │ │ -│ └─────────────┘ └──────┬──────┘ └──────────────┘ │ -│ │ │ -│ ┌──────▼──────┐ │ -│ │ Redis │ │ -│ │ (messaging)│ │ -│ └─────────────┘ │ -│ │ -│ ┌─────────────┐ │ -│ │ Prefect │ │ -│ │ Services │ │ -│ │ (background)│ │ -│ └─────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────────┘ +┌─────────────────────────────────────────────────────────────────────────┐ +│ Tailscale Network │ +├─────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────────┐ ┌─────────────┐ ┌────────────────┐ │ +│ │ Tailscale │◀───▶│ Prefect │◀───▶│ PostgreSQL │ │ +│ │ Sidecar │ │ Server │ │ (external) │ │ +│ │ :443 │ │ :4200 │ │ │ │ +│ └──────────────┘ └──────┬──────┘ └────────────────┘ │ +│ ▲ │ │ +│ │ ┌─────▼─────┐ │ +│ │ │ Redis │ │ +│ │ │(messaging)│ │ +│ │ └───────────┘ │ +│ │ ▲ │ +│ │ ┌─────┴─────┐ │ +│ │ │ Prefect │ │ +│ │ │ Services │ │ +│ │ │(background)│ │ +│ │ └───────────┘ │ +│ │ │ +│ │ ┌─────────────────┐ ┌─────────────┐ │ +│ └─────────────▶│ Prefect Worker │◀───▶│ Garage │ │ +│ │ (pg-backup-pool)│ │ (S3) │ │ +│ └─────────────────┘ └─────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +## Services + +| Service | Description | Image | +|---------|-------------|-------| +| `tailscale` | HTTPS ingress via Tailscale | `tailscale/tailscale` | +| `redis` | Messaging broker | `redis:7-alpine` | +| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` | +| `prefect-services` | Background services | `prefecthq/prefect:3-latest` | +| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/damien/prefect-worker-pg-backup` | + +## Work Pools + +The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process). + +To deploy a flow to this pool: + +```python +from prefect import flow + +@flow +def my_backup_flow(): + ... + +my_backup_flow.deploy( + name="my-backup", + work_pool_name="pg-backup-pool" +) ``` ## Directory Structure @@ -79,13 +115,3 @@ Once deployed: https://prefect.taila5ad8.ts.net └── tailscale/ └── serve-config.json # Tailscale HTTPS configuration ``` - -## Post-Deployment: Create Work Pool - -After the server is running, create a work pool for your workflows: - -```bash -export PREFECT_API_URL=https://prefect.taila5ad8.ts.net/api -prefect work-pool create backup-pool --type process -``` -