feat(workers): Concurrent task processing with resource-based backoff

Workers can now process multiple tasks concurrently (default: 3 max).
Self-regulate based on resource usage - back off at 85% memory or 90% CPU.

Backend changes:
- TaskWorker handles concurrent tasks using async Maps
- Resource monitoring (memory %, CPU %) with backoff logic
- Heartbeat reports active_task_count, max_concurrent_tasks, resource stats
- Decommission support via worker_commands table

Frontend changes:
- Workers Dashboard shows tasks per worker (N/M format)
- Resource badges with color-coded thresholds
- Pod visualization with clickable selection
- Decommission controls per worker

New env vars:
- MAX_CONCURRENT_TASKS (default: 3)
- MEMORY_BACKOFF_THRESHOLD (default: 0.85)
- CPU_BACKOFF_THRESHOLD (default: 0.90)
- BACKOFF_DURATION_MS (default: 10000)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Kelly
2025-12-11 11:47:24 -07:00
parent be251c6fb3
commit 8e2f07c941
10 changed files with 1243 additions and 416 deletions

View File

@@ -1,5 +1,5 @@
# RBAC configuration for scraper pod to control worker scaling
# Allows the scraper to read and scale the scraper-worker deployment
# Allows the scraper to read and scale the scraper-worker statefulset
apiVersion: v1
kind: ServiceAccount
metadata:
@@ -12,13 +12,13 @@ metadata:
name: worker-scaler
namespace: dispensary-scraper
rules:
# Allow reading deployment status
# Allow reading deployment and statefulset status
- apiGroups: ["apps"]
resources: ["deployments"]
resources: ["deployments", "statefulsets"]
verbs: ["get", "list"]
# Allow scaling deployments (read/write the scale subresource)
# Allow scaling deployments and statefulsets
- apiGroups: ["apps"]
resources: ["deployments/scale"]
resources: ["deployments/scale", "statefulsets/scale"]
verbs: ["get", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1