Damien/prefect-deployment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update README with worker architecture

Browse Source
This commit is contained in:
Damien Arnodo
2026-01-31 11:39:49 +00:00
parent c8e67e7bbb
commit 18d2c59f7c
1 changed files with 58 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
README.md
View File

@@ -39,6 +39,8 @@ Generate a reusable auth key from https://login.tailscale.com/admin/settings/key
4. Override sensitive values (`CHANGE_ME`) in the environment variables section: 4. Override sensitive values (`CHANGE_ME`) in the environment variables section:
- `TS_AUTHKEY` - Tailscale auth key (reusable recommended) - `TS_AUTHKEY` - Tailscale auth key (reusable recommended)
- `DB_PASSWORD` - PostgreSQL password - `DB_PASSWORD` - PostgreSQL password
- `S3_ACCESS_KEY` - Garage S3 access key
- `S3_SECRET_KEY` - Garage S3 secret key
5. Deploy 5. Deploy
## Access ## Access
@@ -48,28 +50,62 @@ Once deployed: https://prefect.taila5ad8.ts.net
## Architecture ## Architecture
``` ```
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────────────────┐
│ Tailscale Network │ │ Tailscale Network │
├─────────────────────────────────────────────────────────────┤ ├─────────────────────────────────────────────────────────────────────────┤
│ │ │ │
│ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ │ │ ┌──────────────┐ ┌─────────────┐ ┌────────────────┐ │
│ │ Tailscale │────▶│ Prefect │────▶│ PostgreSQL │ │ │ │ Tailscale │◀───▶│ Prefect │◀───▶│ PostgreSQL │ │
│ │ Sidecar │ │ Server │ │ (external) │ │ │ │ Sidecar │ │ Server │ │ (external) │ │
│ │ :443 │ │ :4200 │ │ │ │ │ │ :443 │ │ :4200 │ │ │ │
│ └─────────────┘ └──────┬──────┘ └──────────────┘ │ │ └──────────────┘ └──────┬──────┘ └────────────────┘ │
│ ▲ │ │
│ │ ┌─────▼─────┐ │
│ │ │ Redis │ │
│ │ │(messaging)│ │
│ │ └───────────┘ │
│ │ ▲ │
│ │ ┌─────┴─────┐ │
│ │ │ Prefect │ │
│ │ │ Services │ │
│ │ │(background)│ │
│ │ └───────────┘ │
│ │ │ │ │ │
│ ┌──────▼──────┐ │ │ │ ┌─────────────────┐ ┌─────────────┐ │
│ │ Redis │ │ │ └─────────────▶│ Prefect Worker │◀───▶│ Garage │ │
│ │ (messaging)│ │ │ │ (pg-backup-pool)│ │ (S3) │ │
│ └─────────────┘ │ │ └─────────────────┘ └─────────────┘ │
│ │ │ │
│ ┌─────────────┐ │ └─────────────────────────────────────────────────────────────────────────┘
│ │ Prefect │ │ ```
│ │ Services │ │
│ │ (background)│ │ ## Services
│ └─────────────┘ │
│ │ | Service | Description | Image |
└─────────────────────────────────────────────────────────────┘ |---------|-------------|-------|
| `tailscale` | HTTPS ingress via Tailscale | `tailscale/tailscale` |
| `redis` | Messaging broker | `redis:7-alpine` |
| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` |
| `prefect-services` | Background services | `prefecthq/prefect:3-latest` |
| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/damien/prefect-worker-pg-backup` |
## Work Pools
The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process).
To deploy a flow to this pool:
```python
from prefect import flow
@flow
def my_backup_flow():
...
my_backup_flow.deploy(
name="my-backup",
work_pool_name="pg-backup-pool"
)
``` ```
## Directory Structure ## Directory Structure
@@ -79,13 +115,3 @@ Once deployed: https://prefect.taila5ad8.ts.net
└── tailscale/ └── tailscale/
└── serve-config.json # Tailscale HTTPS configuration └── serve-config.json # Tailscale HTTPS configuration
``` ```
## Post-Deployment: Create Work Pool
After the server is running, create a work pool for your workflows:
```bash
export PREFECT_API_URL=https://prefect.taila5ad8.ts.net/api
prefect work-pool create backup-pool --type process
```