1861e18396
Workers now follow the correct flow: 1. Check what pools have pending tasks 2. Claim a pool (e.g., Phoenix AZ) 3. Get Evomi proxy for that geo 4. Run preflight with geo proxy 5. Pull tasks from pool (up to 6 stores) 6. Execute tasks 7. Release pool when exhausted (6 stores visited) Task pools group dispensaries by metro area (100mi radius): - Phoenix AZ, Tucson AZ - Los Angeles CA, San Francisco CA, San Diego CA, Sacramento CA - Denver CO, Chicago IL, Boston MA, Detroit MI - Las Vegas NV, Reno NV, Newark NJ, New York NY - Oklahoma City OK, Tulsa OK, Portland OR, Seattle WA Benefits: - Workers know geo BEFORE getting proxy (no more "No geo assigned") - IP diversity within metro area (Phoenix worker can use Tempe IP) - Simpler worker logic - just match pool geo - Pre-organized tasks, not grouped at claim time New files: - migrations/113_task_pools.sql - schema, seed data, functions - src/services/task-pool.ts - TypeScript service Env vars: - USE_TASK_POOLS=true (new system) - USE_IDENTITY_POOL=false (disabled) 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
78 lines
2.5 KiB
YAML
78 lines
2.5 KiB
YAML
# Task Worker Deployment
|
||
#
|
||
# Simple Deployment that runs task-worker.js to process tasks from worker_tasks queue.
|
||
# Workers pull tasks using DB-level locking (FOR UPDATE SKIP LOCKED).
|
||
#
|
||
# The worker will wait up to 60 minutes for active proxies to be added before failing.
|
||
# This allows deployment to succeed even if proxies aren't configured yet.
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: scraper-worker
|
||
namespace: dispensary-scraper
|
||
spec:
|
||
# MAX 8 PODS - See CLAUDE.md rule #6
|
||
# Each pod runs up to MAX_CONCURRENT_TASKS browsers (~400MB each)
|
||
# Scale pods for throughput, not concurrent tasks per pod
|
||
replicas: 8
|
||
selector:
|
||
matchLabels:
|
||
app: scraper-worker
|
||
template:
|
||
metadata:
|
||
labels:
|
||
app: scraper-worker
|
||
spec:
|
||
imagePullSecrets:
|
||
- name: regcred
|
||
containers:
|
||
- name: worker
|
||
image: code.cannabrands.app/creationshop/dispensary-scraper:latest
|
||
command: ["node"]
|
||
args: ["dist/tasks/task-worker.js"]
|
||
envFrom:
|
||
- configMapRef:
|
||
name: scraper-config
|
||
- secretRef:
|
||
name: scraper-secrets
|
||
env:
|
||
- name: WORKER_MODE
|
||
value: "true"
|
||
- name: POD_NAME
|
||
valueFrom:
|
||
fieldRef:
|
||
fieldPath: metadata.name
|
||
- name: API_BASE_URL
|
||
value: "http://scraper"
|
||
- name: NODE_OPTIONS
|
||
value: "--max-old-space-size=1500"
|
||
# Browser memory limits - see docs/WORKER_TASK_ARCHITECTURE.md
|
||
# 3 browsers × ~400MB = ~1.3GB (safe for 2GB pod limit)
|
||
- name: MAX_CONCURRENT_TASKS
|
||
value: "3"
|
||
# Task Pool System (geo-based pools)
|
||
# Correct flow: check pools → claim pool → get proxy → preflight → pull tasks
|
||
- name: USE_TASK_POOLS
|
||
value: "true"
|
||
# Disable legacy identity pool
|
||
- name: USE_IDENTITY_POOL
|
||
value: "false"
|
||
resources:
|
||
requests:
|
||
memory: "1Gi"
|
||
cpu: "100m"
|
||
limits:
|
||
memory: "2Gi"
|
||
cpu: "500m"
|
||
livenessProbe:
|
||
exec:
|
||
command:
|
||
- /bin/sh
|
||
- -c
|
||
- "pgrep -f 'task-worker' > /dev/null"
|
||
initialDelaySeconds: 60
|
||
periodSeconds: 30
|
||
failureThreshold: 3
|
||
terminationGracePeriodSeconds: 60
|