56cc171287
## Worker System - Role-agnostic workers that can handle any task type - Pod-based architecture with StatefulSet (5-15 pods, 5 workers each) - Custom pod names (Aethelgard, Xylos, Kryll, etc.) - Worker registry with friendly names and resource monitoring - Hub-and-spoke visualization on JobQueue page ## Stealth & Anti-Detection (REQUIRED) - Proxies are MANDATORY - workers fail to start without active proxies - CrawlRotator initializes on worker startup - Loads proxies from `proxies` table - Auto-rotates proxy + fingerprint on 403 errors - 12 browser fingerprints (Chrome, Firefox, Safari, Edge) - Locale/timezone matching for geographic consistency ## Task System - Renamed product_resync → product_refresh - Task chaining: store_discovery → entry_point → product_discovery - Priority-based claiming with FOR UPDATE SKIP LOCKED - Heartbeat and stale task recovery ## UI Updates - JobQueue: Pod visualization, resource monitoring on hover - WorkersDashboard: Simplified worker list - Removed unused filters from task list ## Other - IP2Location service for visitor analytics - Findagram consumer features scaffolding - Documentation updates 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
70 lines
1.8 KiB
YAML
70 lines
1.8 KiB
YAML
apiVersion: batch/v1
|
|
kind: CronJob
|
|
metadata:
|
|
name: ip2location-update
|
|
namespace: default
|
|
spec:
|
|
# Run on the 1st of every month at 3am UTC
|
|
schedule: "0 3 1 * *"
|
|
concurrencyPolicy: Forbid
|
|
successfulJobsHistoryLimit: 3
|
|
failedJobsHistoryLimit: 3
|
|
jobTemplate:
|
|
spec:
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: ip2location-updater
|
|
image: curlimages/curl:latest
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
set -e
|
|
echo "Downloading IP2Location LITE DB5..."
|
|
|
|
# Download to temp
|
|
cd /tmp
|
|
curl -L -o ip2location.zip "https://www.ip2location.com/download/?token=${IP2LOCATION_TOKEN}&file=DB5LITEBIN"
|
|
|
|
# Extract
|
|
unzip -o ip2location.zip
|
|
|
|
# Find and copy the BIN file
|
|
BIN_FILE=$(ls *.BIN 2>/dev/null | head -1)
|
|
if [ -z "$BIN_FILE" ]; then
|
|
echo "ERROR: No BIN file found"
|
|
exit 1
|
|
fi
|
|
|
|
# Copy to shared volume
|
|
cp "$BIN_FILE" /data/IP2LOCATION-LITE-DB5.BIN
|
|
|
|
echo "Done! Database updated: /data/IP2LOCATION-LITE-DB5.BIN"
|
|
env:
|
|
- name: IP2LOCATION_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: dutchie-backend-secret
|
|
key: IP2LOCATION_TOKEN
|
|
volumeMounts:
|
|
- name: ip2location-data
|
|
mountPath: /data
|
|
restartPolicy: OnFailure
|
|
volumes:
|
|
- name: ip2location-data
|
|
persistentVolumeClaim:
|
|
claimName: ip2location-pvc
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: ip2location-pvc
|
|
namespace: default
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 100Mi
|