diff --git a/CLAUDE.md b/CLAUDE.md index 8a94047a..57a290ea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,6 +17,35 @@ Never deploy unless user explicitly says: "CLAUDE — DEPLOYMENT IS NOW AUTHORIZ ### 5. DB POOL ONLY Never import `src/db/migrate.ts` at runtime. Use `src/db/pool.ts` for DB access. +### 6. K8S POD LIMITS — CRITICAL +**MAX 8 PODS** for `scraper-worker` deployment. NEVER EXCEED THIS. + +**Pods vs Workers:** +- **Pod** = Kubernetes container instance (MAX 8) +- **Worker** = Concurrent task runner INSIDE a pod (controlled by `MAX_CONCURRENT_TASKS` env var) +- Formula: `8 pods × MAX_CONCURRENT_TASKS = total concurrent workers` + +**To increase workers:** Change `MAX_CONCURRENT_TASKS` env var, NOT replicas. +```bash +# CORRECT - increase workers per pod +kubectl set env deployment/scraper-worker -n dispensary-scraper MAX_CONCURRENT_TASKS=5 + +# WRONG - never scale above 8 replicas +kubectl scale deployment/scraper-worker --replicas=20 # NEVER DO THIS +``` + +**If K8s API returns ServiceUnavailable:** STOP IMMEDIATELY. Do not retry. The cluster is overloaded. + +### 7. K8S REQUIRES EXPLICIT PERMISSION +**NEVER run kubectl commands without explicit user permission.** + +Before running ANY `kubectl` command (scale, rollout, set env, delete, apply, etc.): +1. Tell the user what you want to do +2. Wait for explicit approval +3. Only then execute the command + +This applies to ALL kubectl operations - even read-only ones like `kubectl get pods`. + --- ## Quick Reference diff --git a/backend/src/tasks/task-worker.ts b/backend/src/tasks/task-worker.ts index 1a8062e6..578b7535 100644 --- a/backend/src/tasks/task-worker.ts +++ b/backend/src/tasks/task-worker.ts @@ -98,7 +98,7 @@ const API_BASE_URL = process.env.API_BASE_URL || 'http://localhost:3010'; // Maximum number of tasks this worker will run concurrently // Tune based on workload: I/O-bound tasks benefit from higher concurrency -const MAX_CONCURRENT_TASKS = parseInt(process.env.MAX_CONCURRENT_TASKS || '3'); +const MAX_CONCURRENT_TASKS = parseInt(process.env.MAX_CONCURRENT_TASKS || '15'); // When heap memory usage exceeds this threshold (as decimal 0.0-1.0), stop claiming new tasks // Default 85% - gives headroom before OOM