Kelly f82eed4dc3 feat(workers): Add proxy reload, staggered tasks, and bulk proxy import ...

- Periodic proxy reload: Workers now reload proxies every 60s to pick up changes
- Staggered task scheduling: New API endpoints for creating tasks with delays
- Bulk proxy import: Script supports multiple URL formats including host:port:user:pass
- Proxy URL column: Migration 086 adds proxy_url for non-standard formats

Key changes:
- crawl-rotator.ts: Added reloadIfStale(), isStale(), setReloadInterval()
- task-worker.ts: Calls reloadIfStale() in main loop
- task-service.ts: Added createStaggeredTasks() and createAZStoreTasks()
- tasks.ts: Added POST /batch/staggered and /batch/az-stores endpoints
- import-proxies.ts: New script for bulk proxy import
- CLAUDE.md: Documented staggered task workflow

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2025-12-12 01:53:15 -07:00

..

archive

feat: Add stale process monitor, users route, landing page, archive old scripts

2025-12-05 04:07:31 -07:00

docs

chore: Clean up deprecated code and docs

2025-12-11 22:17:40 -07:00

k8s

feat(workers): Add preflight schema and StatefulSet

2025-12-11 23:45:04 -07:00

migrations

feat(workers): Add proxy reload, staggered tasks, and bulk proxy import

2025-12-12 01:53:15 -07:00

new-scrapers

fix(monitor): remove non-existent worker columns from job_run_logs query

2025-12-03 18:45:05 -07:00

node_modules

fix: Add curl to Docker, add active flag to worker_tasks

2025-12-10 23:12:09 -07:00

public

chore: Add WordPress plugin v1.6.0 download files

2025-12-10 17:23:25 -07:00

scripts

feat: Stealth worker system with mandatory proxy rotation

2025-12-10 00:44:59 -07:00

src

feat(workers): Add proxy reload, staggered tasks, and bulk proxy import

2025-12-12 01:53:15 -07:00

.dockerignore

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

.env

feat: AZ dispensary harmonization with Dutchie source of truth

2025-12-08 10:19:49 -07:00

.env.example

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

.gitignore

feat: Stealth worker system with mandatory proxy rotation

2025-12-10 00:44:59 -07:00

API_TOKENS_GUIDE.md

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

API_USAGE.md

feat: Add stale process monitor, users route, landing page, archive old scripts

2025-12-05 04:07:31 -07:00

docker-compose.local.yml

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

Dockerfile

fix: Add curl to Docker, add active flag to worker_tasks

2025-12-10 23:12:09 -07:00

Dockerfile.dev

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

openapi.yaml

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

package-lock.json

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

package.json

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

setup-local.sh

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

stop-local.sh

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

test-intercept.js

chore: Clean up deprecated code and docs

2025-12-11 22:17:40 -07:00

tsconfig.json

fix: Restore hydration and product_refresh for store updates

2025-12-11 23:03:39 -07:00