Creationshop/cannaiq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1861e18396b19d0a1a970146e8040029083a53f0
cannaiq/backend/migrations
History
Kelly 1861e18396 feat(workers): Implement geo-based task pools 
Workers now follow the correct flow:
1. Check what pools have pending tasks
2. Claim a pool (e.g., Phoenix AZ)
3. Get Evomi proxy for that geo
4. Run preflight with geo proxy
5. Pull tasks from pool (up to 6 stores)
6. Execute tasks
7. Release pool when exhausted (6 stores visited)

Task pools group dispensaries by metro area (100mi radius):
- Phoenix AZ, Tucson AZ
- Los Angeles CA, San Francisco CA, San Diego CA, Sacramento CA
- Denver CO, Chicago IL, Boston MA, Detroit MI
- Las Vegas NV, Reno NV, Newark NJ, New York NY
- Oklahoma City OK, Tulsa OK, Portland OR, Seattle WA

Benefits:
- Workers know geo BEFORE getting proxy (no more "No geo assigned")
- IP diversity within metro area (Phoenix worker can use Tempe IP)
- Simpler worker logic - just match pool geo
- Pre-organized tasks, not grouped at claim time

New files:
- migrations/113_task_pools.sql - schema, seed data, functions
- src/services/task-pool.ts - TypeScript service

Env vars:
- USE_TASK_POOLS=true (new system)
- USE_IDENTITY_POOL=false (disabled)

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-14 01:41:52 -07:00
..
005_api_tokens.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
005_notifications.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
007_proxy_test_jobs.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
008_proxy_locations.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
009_image_sizes.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
010_store_logo.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
011_create_dispensaries_table.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
012_create_dispensary_changes_table.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
013_populate_dispensaries_from_azdhs.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
014_add_dispensary_fk_to_tables.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
015_crawler_schedule.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
016_extend_dispensaries_for_multi_provider.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
017_create_crawler_sandboxes_and_templates.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
018_multi_category_intelligence.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
019_link_stores_to_dispensaries.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
020_scheduler_orchestrator_fields.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
021_dispensary_crawl_schedule.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
022_scheduler_dispensary_link_fields.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
023_dispensary_crawl_status_add_link_fields.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
024_product_availability_tracking.sql
Fix category-crawler-jobs store lookup query
2025-12-01 00:07:00 -07:00
025_dispensary_crawl_fields.sql
Add crawl fields directly to dispensaries table
2025-11-30 23:52:48 -07:00
026_update_crawl_status_view.sql
Update admin panel to use unified dispensaries table
2025-12-01 00:13:41 -07:00
027_brand_history_tracking.sql
Add brand history tracking for new/dropped brands detection
2025-12-01 00:18:47 -07:00
028_snapshot_architecture.sql
Remove incorrect migration 029, add snapshot architecture, improve scraper
2025-12-01 08:52:54 -07:00
029_add_store_id_to_wp_api_permissions.sql
Add store selection to API permissions
2025-12-01 13:59:01 -07:00
030_add_dispensary_id_to_wp_api_permissions.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
031_consolidate_schema.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
031_product_normalized_fields.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
032_dutchie_graphql_schema.sql
Add brand history view to dutchie GraphQL migration
2025-12-02 11:34:01 -07:00
032_menu_type_and_local_images.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
033_add_platform_id_to_crawl_status.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
033_job_queue_claiming.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
034_crawl_status_fields.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
034_dispensary_failure_tracking.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
035_multi_domain_users.sql
feat: Add Findagram and FindADispo consumer frontends
2025-12-05 16:10:15 -07:00
036_findadispo_dispensary_fields.sql
feat: Add Findagram and FindADispo consumer frontends
2025-12-05 16:10:15 -07:00
037_dispensary_crawler_profiles.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
038_profile_status_field.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
039_crawl_orchestration_traces.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
040_dispensary_dba_name.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
041_cannaiq_canonical_schema.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
043_add_states_table.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
044_add_provider_detection_data.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
045_add_image_columns.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
046_crawler_reliability.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
046_raw_payloads_table.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
047_analytics_infrastructure.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
048_production_sync_monitoring.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
050_cannaiq_canonical_v2.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_cannaiq_canonical_safe_bootstrap.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_create_mv_state_metrics.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_worker_definitions.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
052_add_provider_data_columns.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_add_state_cannabis_flags.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_hydration_schema_alignment.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_seo_settings.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
053_analytics_indexes.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
053_dutchie_discovery_schema.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
054_worker_metadata.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
055_workforce_enhancements.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
056_fix_worker_and_run_logs.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
057_crawl_enabled_and_dutchie_verified.sql
feat: AZ dispensary harmonization with Dutchie source of truth
2025-12-08 10:19:49 -07:00
057_visibility_tracking_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
058_add_id_resolution_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
059_job_queue_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
060_consumer_verification_notifications.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
061_product_click_events.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
062_click_analytics_enhancements.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
063_seo_pages.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
064_ai_settings.sql
feat: Add AI settings to database (provider/model configurable via UI)
2025-12-07 23:18:52 -07:00
065_slug_verification_tracking.sql
feat: AZ dispensary harmonization with Dutchie source of truth
2025-12-08 10:19:49 -07:00
066_dutchie_field_alignment.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
067_promotion_log.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
068_crawler_status_alerts.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
069_six_stage_status.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
070_product_variants.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
071_harmonize_store_products.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
072_product_views.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
073_proxy_timezone.sql
feat(images): Add local image storage with on-demand resizing
2025-12-09 11:04:50 -07:00
074_worker_commands.sql
feat(workers): Concurrent task processing with resource-based backoff
2025-12-11 11:47:24 -07:00
074_worker_task_queue.sql
feat(tasks): Add unified task-based worker architecture
2025-12-09 16:27:03 -07:00
075_consecutive_misses.sql
fix: Public API column mappings and OOS detection
2025-12-09 20:44:53 -07:00
076_visitor_analytics.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
076_worker_registry.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
077_click_events_location.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
078_proxy_consecutive_403.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
079_task_schedules.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
080_raw_crawl_payloads.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
081_payload_fetch_columns.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
082_proxy_notification_trigger.sql
feat(worker): Listen for proxy_added notifications
2025-12-10 22:58:00 -07:00
083_discovery_runs.sql
feat(tasks): Add proxy_test task handler and discovery run tracking
2025-12-11 21:07:58 -07:00
084_dual_transport_preflight.sql
feat(workers): Add dual-transport preflight system
2025-12-11 22:47:52 -07:00
085_preflight_ip_fingerprint.sql
feat(workers): Add preflight schema and StatefulSet
2025-12-11 23:45:04 -07:00
085_trusted_origins.sql
feat(api): Add payload query API and trusted origins management
2025-12-11 23:28:05 -07:00
086_proxy_url_column.sql
feat(workers): Add proxy reload, staggered tasks, and bulk proxy import
2025-12-12 01:53:15 -07:00
088_discovery_payloads.sql
feat(scheduler): Immutable schedules and HTTP-only pipeline
2025-12-12 09:24:08 -07:00
089_immutable_schedules.sql
feat(discovery): Add self-healing and rename schedule
2025-12-13 14:14:21 -07:00
090_modification_tracking.sql
feat: Parallelized store discovery, modification tracking, and task deduplication
2025-12-12 22:15:04 -07:00
091_store_discovery_tracking.sql
feat: Auto-healing entry_point_discovery with browser-first transport
2025-12-12 22:55:21 -07:00
092_fix_trulieve_urls.sql
fix: Intelligence stores endpoint and UI consistency
2025-12-12 23:37:28 -07:00
092_store_intelligence_cache.sql
perf: Add store_intelligence_cache for fast /intelligence/stores
2025-12-13 00:13:41 -07:00
093_fix_mv_state_metrics.sql
fix: Correct mv_state_metrics to use brand_name_raw
2025-12-12 23:50:53 -07:00
094_import_evomi_proxies.sql
feat: Import 500 Evomi residential proxies
2025-12-13 00:05:21 -07:00
095_proxy_auto_retry.sql
feat: Auto-retry failed proxies after cooldown period
2025-12-13 00:08:44 -07:00
096_product_search_indexes.sql
fix(analytics): Fix market-summary store count and add search indexes
2025-12-13 00:35:17 -07:00
097_worker_tasks_worker_id_index.sql
perf: Optimize dashboard queries for faster load times
2025-12-13 01:09:02 -07:00
098_dispensary_stage_tracking.sql
fix: Add consecutive_failures column to migration
2025-12-13 01:20:59 -07:00
099_working_hours.sql
feat: Add working hours for natural traffic patterns
2025-12-13 03:24:42 -07:00
100_worker_timezone.sql
feat: Add working hours for natural traffic patterns
2025-12-13 03:24:42 -07:00
101_worker_preflight_timezone.sql
feat: Add working hours for natural traffic patterns
2025-12-13 03:24:42 -07:00
102_check_working_hours.sql
feat: Add working hours for natural traffic patterns
2025-12-13 03:24:42 -07:00
103_schedule_dispensary_id.sql
feat: Support per-dispensary schedules (not just per-state)
2025-12-13 12:03:08 -07:00
104_task_source_tracking.sql
feat: Remove Run Now, add source tracking, optimize dashboard
2025-12-13 13:23:35 -07:00
105_dashboard_performance_indexes.sql
feat: Remove Run Now, add source tracking, optimize dashboard
2025-12-13 13:23:35 -07:00
106_rename_store_discovery_schedule.sql
feat(discovery): Add self-healing and rename schedule
2025-12-13 14:14:21 -07:00
107_proxy_tracking.sql
feat: Platform isolation, Evomi geo-targeting, proxy management
2025-12-13 15:16:48 -07:00
108_worker_geo_sessions.sql
feat: Worker geo sessions for state-based task assignment
2025-12-13 16:00:09 -07:00
109_worker_identity_pool.sql
feat: Add identity pool for diverse IP/fingerprint rotation
2025-12-13 18:46:58 -07:00
110_trusted_origins.sql
feat: Add trusted origins management UI at /users
2025-12-13 19:54:26 -07:00
111_system_settings.sql
fix(identity): Use unique session IDs for proxy rotation + add task pool gate
2025-12-13 20:17:52 -07:00
112_worker_session_pool.sql
feat(workers): Session pool system - claim tasks first, then get IP
2025-12-13 22:54:45 -07:00
113_task_pools.sql
feat(workers): Implement geo-based task pools
2025-12-14 01:41:52 -07:00
add-stock-columns.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-scrape-jobs.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00