Compare commits
7 Commits
0493b81108
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 580fba95cd | |||
| 10eb46d9d8 | |||
| a91e70fa81 | |||
| 6fa6ba5496 | |||
| 115a690d8f | |||
| 7bf75f85e1 | |||
| beb84cf11d |
29
README.md
29
README.md
@@ -16,6 +16,7 @@ GRANT ALL PRIVILEGES ON DATABASE prefect TO prefect;
|
|||||||
\c prefect
|
\c prefect
|
||||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||||
GRANT ALL ON SCHEMA public TO prefect;
|
GRANT ALL ON SCHEMA public TO prefect;
|
||||||
|
ALTER SCHEMA public OWNER TO prefect;
|
||||||
```
|
```
|
||||||
|
|
||||||
### Tailscale Auth Key
|
### Tailscale Auth Key
|
||||||
@@ -51,9 +52,6 @@ In the stack configuration, add the following environment variables:
|
|||||||
| `DB_PORT` | PostgreSQL port | `5432` |
|
| `DB_PORT` | PostgreSQL port | `5432` |
|
||||||
| `DB_USER` | Database user | `prefect` |
|
| `DB_USER` | Database user | `prefect` |
|
||||||
| `DB_PASSWORD` | Database password | *secret* |
|
| `DB_PASSWORD` | Database password | *secret* |
|
||||||
| `S3_ACCESS_KEY` | Garage S3 access key | *secret* |
|
|
||||||
| `S3_SECRET_KEY` | Garage S3 secret key | *secret* |
|
|
||||||
| `S3_ENDPOINT_URL` | Garage S3 endpoint | `https://s3.taila5ad8.ts.net` |
|
|
||||||
|
|
||||||
> **Tip**: Use Komodo's secret variables (marked with 🔒) for sensitive values.
|
> **Tip**: Use Komodo's secret variables (marked with 🔒) for sensitive values.
|
||||||
|
|
||||||
@@ -82,20 +80,39 @@ Once deployed: https://prefect.taila5ad8.ts.net
|
|||||||
| `redis` | Messaging broker | `redis:7-alpine` |
|
| `redis` | Messaging broker | `redis:7-alpine` |
|
||||||
| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` |
|
| `prefect-server` | API + UI | `prefecthq/prefect:3-latest` |
|
||||||
| `prefect-services` | Background services | `prefecthq/prefect:3-latest` |
|
| `prefect-services` | Background services | `prefecthq/prefect:3-latest` |
|
||||||
| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/damien/prefect-worker-pg-backup` |
|
| `prefect-worker-pg-backup` | Worker for PostgreSQL backups | `gitea.arnodo.fr/Damien/prefect-flows-pg-backup` |
|
||||||
|
|
||||||
## Work Pools
|
## Work Pools
|
||||||
|
|
||||||
The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process).
|
The `prefect-worker-pg-backup` service automatically creates and listens to the `pg-backup-pool` work pool (type: process).
|
||||||
|
|
||||||
To deploy a flow to this pool:
|
## Secrets Management
|
||||||
|
|
||||||
|
Flow-specific secrets (S3 credentials, database passwords, API keys, etc.) should be managed via **Prefect Blocks**, not environment variables in the compose file.
|
||||||
|
|
||||||
|
### Creating a Block (example with S3/Garage)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from prefect_aws import AwsCredentials
|
||||||
|
|
||||||
|
creds = AwsCredentials(
|
||||||
|
aws_access_key_id="xxx",
|
||||||
|
aws_secret_access_key="xxx",
|
||||||
|
aws_endpoint_url="https://s3.taila5ad8.ts.net"
|
||||||
|
)
|
||||||
|
creds.save("garage-credentials")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using in a flow
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from prefect import flow
|
from prefect import flow
|
||||||
|
from prefect_aws import AwsCredentials
|
||||||
|
|
||||||
@flow
|
@flow
|
||||||
def my_backup_flow():
|
def my_backup_flow():
|
||||||
...
|
creds = AwsCredentials.load("garage-credentials")
|
||||||
|
# use creds...
|
||||||
|
|
||||||
my_backup_flow.deploy(
|
my_backup_flow.deploy(
|
||||||
name="my-backup",
|
name="my-backup",
|
||||||
|
|||||||
@@ -78,12 +78,7 @@ services:
|
|||||||
depends_on:
|
depends_on:
|
||||||
- prefect-server
|
- prefect-server
|
||||||
environment:
|
environment:
|
||||||
# Prefect API connection (via Tailscale)
|
|
||||||
- PREFECT_API_URL=http://localhost:4200/api
|
- PREFECT_API_URL=http://localhost:4200/api
|
||||||
# S3 credentials for Garage
|
|
||||||
- AWS_ACCESS_KEY_ID=${S3_ACCESS_KEY}
|
|
||||||
- AWS_SECRET_ACCESS_KEY=${S3_SECRET_KEY}
|
|
||||||
- AWS_ENDPOINT_URL=${S3_ENDPOINT_URL}
|
|
||||||
command: prefect worker start --pool pg-backup-pool --type process
|
command: prefect worker start --pool pg-backup-pool --type process
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user