Kelly 4949b22457 feat(tasks): Refactor task workflow with payload/refresh separation ...

Major changes:
- Split crawl into payload_fetch (API → disk) and product_refresh (disk → DB)
- Add task chaining: store_discovery → product_discovery → payload_fetch → product_refresh
- Add payload storage utilities for gzipped JSON on filesystem
- Add /api/payloads endpoints for payload access and diffing
- Add DB-driven TaskScheduler with schedule persistence
- Track newDispensaryIds through discovery promotion for chaining
- Add stealth improvements: HTTP fingerprinting, proxy rotation enhancements
- Add Workers dashboard K8s scaling controls

New files:
- src/tasks/handlers/payload-fetch.ts - Fetches from API, saves to disk
- src/services/task-scheduler.ts - DB-driven schedule management
- src/utils/payload-storage.ts - Payload save/load utilities
- src/routes/payloads.ts - Payload API endpoints
- src/services/http-fingerprint.ts - Browser fingerprint generation
- docs/TASK_WORKFLOW_2024-12-10.md - Complete workflow documentation

Migrations:
- 078: Proxy consecutive 403 tracking
- 079: task_schedules table
- 080: raw_crawl_payloads table
- 081: payload column and last_fetch_at

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2025-12-10 22:15:35 -07:00

..

archive

feat: Add stale process monitor, users route, landing page, archive old scripts

2025-12-05 04:07:31 -07:00

docs

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

k8s

feat: Stealth worker system with mandatory proxy rotation

2025-12-10 00:44:59 -07:00

migrations

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

new-scrapers

fix(monitor): remove non-existent worker columns from job_run_logs query

2025-12-03 18:45:05 -07:00

node_modules

feat: Stealth worker system with mandatory proxy rotation

2025-12-10 00:44:59 -07:00

public

chore: Add WordPress plugin v1.6.0 download files

2025-12-10 17:23:25 -07:00

scripts

feat: Stealth worker system with mandatory proxy rotation

2025-12-10 00:44:59 -07:00

src

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

.dockerignore

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

.env

feat: AZ dispensary harmonization with Dutchie source of truth

2025-12-08 10:19:49 -07:00

.env.example

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

.gitignore

feat: Stealth worker system with mandatory proxy rotation

2025-12-10 00:44:59 -07:00

API_TOKENS_GUIDE.md

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

API_USAGE.md

feat: Add stale process monitor, users route, landing page, archive old scripts

2025-12-05 04:07:31 -07:00

docker-compose.local.yml

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

Dockerfile

fix(docker): Use npm install instead of npm ci for reliability

2025-12-10 11:28:29 -07:00

Dockerfile.dev

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

openapi.yaml

Initial commit - Dutchie dispensary scraper

2025-11-28 19:45:44 -07:00

package-lock.json

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

package.json

feat(tasks): Refactor task workflow with payload/refresh separation

2025-12-10 22:15:35 -07:00

setup-local.sh

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

stop-local.sh

feat: Add v2 architecture with multi-state support and orchestrator services

2025-12-07 11:30:57 -07:00

tsconfig.json

feat: AZ dispensary harmonization with Dutchie source of truth

2025-12-08 10:19:49 -07:00