feat(workers): Concurrent task processing with resource-based backoff
Workers can now process multiple tasks concurrently (default: 3 max). Self-regulate based on resource usage - back off at 85% memory or 90% CPU. Backend changes: - TaskWorker handles concurrent tasks using async Maps - Resource monitoring (memory %, CPU %) with backoff logic - Heartbeat reports active_task_count, max_concurrent_tasks, resource stats - Decommission support via worker_commands table Frontend changes: - Workers Dashboard shows tasks per worker (N/M format) - Resource badges with color-coded thresholds - Pod visualization with clickable selection - Decommission controls per worker New env vars: - MAX_CONCURRENT_TASKS (default: 3) - MEMORY_BACKOFF_THRESHOLD (default: 0.85) - CPU_BACKOFF_THRESHOLD (default: 0.90) - BACKOFF_DURATION_MS (default: 10000) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,5 @@
|
||||
# RBAC configuration for scraper pod to control worker scaling
|
||||
# Allows the scraper to read and scale the scraper-worker deployment
|
||||
# Allows the scraper to read and scale the scraper-worker statefulset
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
@@ -12,13 +12,13 @@ metadata:
|
||||
name: worker-scaler
|
||||
namespace: dispensary-scraper
|
||||
rules:
|
||||
# Allow reading deployment status
|
||||
# Allow reading deployment and statefulset status
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments"]
|
||||
resources: ["deployments", "statefulsets"]
|
||||
verbs: ["get", "list"]
|
||||
# Allow scaling deployments (read/write the scale subresource)
|
||||
# Allow scaling deployments and statefulsets
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments/scale"]
|
||||
resources: ["deployments/scale", "statefulsets/scale"]
|
||||
verbs: ["get", "patch", "update"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
|
||||
Reference in New Issue
Block a user