Damien/prefect-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Prefect worker for PostgreSQL backups #1

Merged
Damien merged 4 commits from feature/add-pg-backup-worker into main 2026-01-31 12:09:59 +00:00
Conversation 0 Commits 4 Files Changed 3 +58 -32
1 changed files with 58 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 18d2c59f7c - Show all commits

90
README.md
View File

@@ -39,6 +39,8 @@ Generate a reusable auth key from https://login.tailscale.com/admin/settings/key
4. Override sensitive values (`CHANGE_ME`) in the environment variables section: 4. Override sensitive values (`CHANGE_ME`) in the environment variables section:
- `TS_AUTHKEY` - Tailscale auth key (reusable recommended) - `TS_AUTHKEY` - Tailscale auth key (reusable recommended)
- `DB_PASSWORD` - PostgreSQL password - `DB_PASSWORD` - PostgreSQL password
- `S3_ACCESS_KEY` - Garage S3 access key
- `S3_SECRET_KEY` - Garage S3 secret key
5. Deploy 5. Deploy
## Access ## Access
@@ -48,28 +50,62 @@ Once deployed: https://prefect.taila5ad8.ts.net
## Architecture ## Architecture
``` ```
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────┐
│ Tailscale Network │ │ Tailscale Network │
├─────────────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────────────────────────┤
│ │ │ │
│ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ │ │ ┌──────────────┐ ┌─────────────┐ ┌────────────────┐ │
│ │ Tailscale │────▶│ Prefect │────▶│ PostgreSQL │ │ │ │ Tailscale │◀───▶│ Prefect │◀───▶│ PostgreSQL │ │
│ │ Sidecar │ │ Server │ │ (external) │ │ │ │ Sidecar │ │ Server │ │ (external) │ │
│ │ :443 │ │ :4200 │ │ │ │ │ │ :443 │ │ :4200 │ │ │ │
│ └─────────────┘ └──────┬──────┘ └──────────────┘ │ │ └──────────────┘ └──────┬──────┘ └────────────────┘ │
│ │ │ │ ▲ │ │
│ ┌──────▼──────┐ │ │ │ ┌─────▼─────┐ │
│ │ Redis │ │ │ │ │ Redis │ │
│ │ (messaging)│ │ │ │ │(messaging)│ │
│ └─────────────┘ │ │ │ └───────────┘ │
│ │ │ │ ▲ │
│ ┌─────────────┐ │ │ │ ┌─────┴─────┐ │
│ │ Prefect │ │ │ │ │ Prefect │ │
│ │ Services │ │ │ │ │ Services │ │
│ │ (background)│ │ │ │ │(background)│ │
│ └─────────────┘ │ │ │ └───────────┘ │
│ │ │ │ │
└─────────────────────────────────────────────────────────────┘ │ │ ┌─────────────────┐ ┌─────────────┐ │
│ └─────────────▶│ Prefect Worker │◀───▶│ Garage │ │
│ │ (pg-backup-pool)│ │ (S3) │ │
│ └─────────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
## Services
| Service | Description | Image |
|---------|-------------|-------|
| `tailscale` | HTTPS ingress via Tailscale | `tailscale/tailscale` |
| `redis` | Messaging broker | `redis:7-alpine` |
| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` |
| `prefect-services` | Background services | `prefecthq/prefect:3-latest` |
| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/damien/prefect-worker-pg-backup` |
## Work Pools
The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process).
To deploy a flow to this pool:
```python
from prefect import flow
@flow
def my_backup_flow():
...
my_backup_flow.deploy(
name="my-backup",
work_pool_name="pg-backup-pool"
)
``` ```
## Directory Structure ## Directory Structure
@@ -79,13 +115,3 @@ Once deployed: https://prefect.taila5ad8.ts.net
└── tailscale/ └── tailscale/
└── serve-config.json # Tailscale HTTPS configuration └── serve-config.json # Tailscale HTTPS configuration
``` ```
## Post-Deployment: Create Work Pool
After the server is running, create a work pool for your workflows:
```bash
export PREFECT_API_URL=https://prefect.taila5ad8.ts.net/api
prefect work-pool create backup-pool --type process
```