Damien/prefect-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README with worker architecture

Browse Source
This commit is contained in:
Damien Arnodo
2026-01-31 11:39:49 +00:00
parent c8e67e7bbb
commit 18d2c59f7c
1 changed files with 58 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
README.md
View File

@@ -39,6 +39,8 @@ Generate a reusable auth key from https://login.tailscale.com/admin/settings/key
4. Override sensitive values (`CHANGE_ME`) in the environment variables section:
- `TS_AUTHKEY` - Tailscale auth key (reusable recommended)
- `DB_PASSWORD` - PostgreSQL password
- `S3_ACCESS_KEY` - Garage S3 access key
- `S3_SECRET_KEY` - Garage S3 secret key
5. Deploy
## Access
@@ -48,28 +50,62 @@ Once deployed: https://prefect.taila5ad8.ts.net
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
┌─────────────────────────────────────────────────────────────────────────┐
│ Tailscale Network │
├─────────────────────────────────────────────────────────────┤
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ │
│ │ Tailscale │────▶│ Prefect │────▶│ PostgreSQL │ │
│ ┌──────────────┐ ┌─────────────┐ ┌────────────────┐ │
│ │ Tailscale │◀───▶│ Prefect │◀───▶│ PostgreSQL │ │
│ │ Sidecar │ │ Server │ │ (external) │ │
│ │ :443 │ │ :4200 │ │ │ │
│ └─────────────┘ └──────┬──────┘ └──────────────┘ │
│ └──────────────┘ └──────┬──────┘ └────────────────┘ │
│ ▲ │ │
│ │ ┌─────▼─────┐ │
│ │ │ Redis │ │
│ │ │(messaging)│ │
│ │ └───────────┘ │
│ │ ▲ │
│ │ ┌─────┴─────┐ │
│ │ │ Prefect │ │
│ │ │ Services │ │
│ │ │(background)│ │
│ │ └───────────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ Redis │ │
│ │ (messaging)│ │
│ └─────────────┘ │
│ │ ┌─────────────────┐ ┌─────────────┐ │
│ └─────────────▶│ Prefect Worker │◀───▶│ Garage │ │
│ │ (pg-backup-pool)│ │ (S3) │ │
│ └─────────────────┘ └─────────────┘ │
│ │
│ ┌─────────────┐ │
│ │ Prefect │ │
│ │ Services │ │
│ │ (background)│ │
│ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
└─────────────────────────────────────────────────────────────────────────┘
```
## Services
| Service | Description | Image |
|---------|-------------|-------|
| `tailscale` | HTTPS ingress via Tailscale | `tailscale/tailscale` |
| `redis` | Messaging broker | `redis:7-alpine` |
| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` |
| `prefect-services` | Background services | `prefecthq/prefect:3-latest` |
| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/damien/prefect-worker-pg-backup` |
## Work Pools
The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process).
To deploy a flow to this pool:
```python
from prefect import flow
@flow
def my_backup_flow():
...
my_backup_flow.deploy(
name="my-backup",
work_pool_name="pg-backup-pool"
)
```
## Directory Structure
@@ -79,13 +115,3 @@ Once deployed: https://prefect.taila5ad8.ts.net
└── tailscale/
└── serve-config.json # Tailscale HTTPS configuration
```
## Post-Deployment: Create Work Pool
After the server is running, create a work pool for your workflows:
```bash
export PREFECT_API_URL=https://prefect.taila5ad8.ts.net/api
prefect work-pool create backup-pool --type process
```