56cc171287
## Worker System - Role-agnostic workers that can handle any task type - Pod-based architecture with StatefulSet (5-15 pods, 5 workers each) - Custom pod names (Aethelgard, Xylos, Kryll, etc.) - Worker registry with friendly names and resource monitoring - Hub-and-spoke visualization on JobQueue page ## Stealth & Anti-Detection (REQUIRED) - Proxies are MANDATORY - workers fail to start without active proxies - CrawlRotator initializes on worker startup - Loads proxies from `proxies` table - Auto-rotates proxy + fingerprint on 403 errors - 12 browser fingerprints (Chrome, Firefox, Safari, Edge) - Locale/timezone matching for geographic consistency ## Task System - Renamed product_resync → product_refresh - Task chaining: store_discovery → entry_point → product_discovery - Priority-based claiming with FOR UPDATE SKIP LOCKED - Heartbeat and stale task recovery ## UI Updates - JobQueue: Pod visualization, resource monitoring on hover - WorkersDashboard: Simplified worker list - Removed unused filters from task list ## Other - IP2Location service for visitor analytics - Findagram consumer features scaffolding - Documentation updates 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
58 lines
1.4 KiB
YAML
58 lines
1.4 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: dutchie-backend
|
|
namespace: default
|
|
labels:
|
|
app: dutchie-backend
|
|
spec:
|
|
replicas: 2
|
|
selector:
|
|
matchLabels:
|
|
app: dutchie-backend
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: dutchie-backend
|
|
spec:
|
|
containers:
|
|
- name: backend
|
|
image: YOUR_REGISTRY/dutchie-backend:latest # Replace with your Docker registry
|
|
ports:
|
|
- containerPort: 3010
|
|
name: http
|
|
envFrom:
|
|
- configMapRef:
|
|
name: dutchie-backend-config
|
|
- secretRef:
|
|
name: dutchie-backend-secret
|
|
env:
|
|
- name: IP2LOCATION_DB_PATH
|
|
value: /data/ip2location/IP2LOCATION-LITE-DB5.BIN
|
|
volumeMounts:
|
|
- name: ip2location-data
|
|
mountPath: /data/ip2location
|
|
resources:
|
|
requests:
|
|
memory: "256Mi"
|
|
cpu: "250m"
|
|
limits:
|
|
memory: "512Mi"
|
|
cpu: "500m"
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: 3010
|
|
initialDelaySeconds: 30
|
|
periodSeconds: 10
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: 3010
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 5
|
|
volumes:
|
|
- name: ip2location-data
|
|
persistentVolumeClaim:
|
|
claimName: ip2location-pvc
|