Creationshop/cannaiq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f82eed4dc3826d0ebb4ea3cd27056a92e2d72690
cannaiq/backend/migrations
History
Kelly f82eed4dc3 feat(workers): Add proxy reload, staggered tasks, and bulk proxy import 
- Periodic proxy reload: Workers now reload proxies every 60s to pick up changes
- Staggered task scheduling: New API endpoints for creating tasks with delays
- Bulk proxy import: Script supports multiple URL formats including host:port:user:pass
- Proxy URL column: Migration 086 adds proxy_url for non-standard formats

Key changes:
- crawl-rotator.ts: Added reloadIfStale(), isStale(), setReloadInterval()
- task-worker.ts: Calls reloadIfStale() in main loop
- task-service.ts: Added createStaggeredTasks() and createAZStoreTasks()
- tasks.ts: Added POST /batch/staggered and /batch/az-stores endpoints
- import-proxies.ts: New script for bulk proxy import
- CLAUDE.md: Documented staggered task workflow

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-12 01:53:15 -07:00
..
005_api_tokens.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
005_notifications.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
007_proxy_test_jobs.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
008_proxy_locations.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
009_image_sizes.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
010_store_logo.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
011_create_dispensaries_table.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
012_create_dispensary_changes_table.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
013_populate_dispensaries_from_azdhs.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
014_add_dispensary_fk_to_tables.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
015_crawler_schedule.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
016_extend_dispensaries_for_multi_provider.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
017_create_crawler_sandboxes_and_templates.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
018_multi_category_intelligence.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
019_link_stores_to_dispensaries.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
020_scheduler_orchestrator_fields.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
021_dispensary_crawl_schedule.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
022_scheduler_dispensary_link_fields.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
023_dispensary_crawl_status_add_link_fields.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
024_product_availability_tracking.sql
Fix category-crawler-jobs store lookup query
2025-12-01 00:07:00 -07:00
025_dispensary_crawl_fields.sql
Add crawl fields directly to dispensaries table
2025-11-30 23:52:48 -07:00
026_update_crawl_status_view.sql
Update admin panel to use unified dispensaries table
2025-12-01 00:13:41 -07:00
027_brand_history_tracking.sql
Add brand history tracking for new/dropped brands detection
2025-12-01 00:18:47 -07:00
028_snapshot_architecture.sql
Remove incorrect migration 029, add snapshot architecture, improve scraper
2025-12-01 08:52:54 -07:00
029_add_store_id_to_wp_api_permissions.sql
Add store selection to API permissions
2025-12-01 13:59:01 -07:00
030_add_dispensary_id_to_wp_api_permissions.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
031_consolidate_schema.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
031_product_normalized_fields.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
032_dutchie_graphql_schema.sql
Add brand history view to dutchie GraphQL migration
2025-12-02 11:34:01 -07:00
032_menu_type_and_local_images.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
033_add_platform_id_to_crawl_status.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
033_job_queue_claiming.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
034_crawl_status_fields.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
034_dispensary_failure_tracking.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
035_multi_domain_users.sql
feat: Add Findagram and FindADispo consumer frontends
2025-12-05 16:10:15 -07:00
036_findadispo_dispensary_fields.sql
feat: Add Findagram and FindADispo consumer frontends
2025-12-05 16:10:15 -07:00
037_dispensary_crawler_profiles.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
038_profile_status_field.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
039_crawl_orchestration_traces.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
040_dispensary_dba_name.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
041_cannaiq_canonical_schema.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
043_add_states_table.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
044_add_provider_detection_data.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
045_add_image_columns.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
046_crawler_reliability.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
046_raw_payloads_table.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
047_analytics_infrastructure.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
048_production_sync_monitoring.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
050_cannaiq_canonical_v2.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_cannaiq_canonical_safe_bootstrap.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_create_mv_state_metrics.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_worker_definitions.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
052_add_provider_data_columns.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_add_state_cannabis_flags.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_hydration_schema_alignment.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_seo_settings.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
053_analytics_indexes.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
053_dutchie_discovery_schema.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
054_worker_metadata.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
055_workforce_enhancements.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
056_fix_worker_and_run_logs.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
057_crawl_enabled_and_dutchie_verified.sql
feat: AZ dispensary harmonization with Dutchie source of truth
2025-12-08 10:19:49 -07:00
057_visibility_tracking_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
058_add_id_resolution_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
059_job_queue_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
060_consumer_verification_notifications.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
061_product_click_events.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
062_click_analytics_enhancements.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
063_seo_pages.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
064_ai_settings.sql
feat: Add AI settings to database (provider/model configurable via UI)
2025-12-07 23:18:52 -07:00
065_slug_verification_tracking.sql
feat: AZ dispensary harmonization with Dutchie source of truth
2025-12-08 10:19:49 -07:00
066_dutchie_field_alignment.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
067_promotion_log.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
068_crawler_status_alerts.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
069_six_stage_status.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
070_product_variants.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
071_harmonize_store_products.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
072_product_views.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
073_proxy_timezone.sql
feat(images): Add local image storage with on-demand resizing
2025-12-09 11:04:50 -07:00
074_worker_commands.sql
feat(workers): Concurrent task processing with resource-based backoff
2025-12-11 11:47:24 -07:00
074_worker_task_queue.sql
feat(tasks): Add unified task-based worker architecture
2025-12-09 16:27:03 -07:00
075_consecutive_misses.sql
fix: Public API column mappings and OOS detection
2025-12-09 20:44:53 -07:00
076_visitor_analytics.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
076_worker_registry.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
077_click_events_location.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
078_proxy_consecutive_403.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
079_task_schedules.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
080_raw_crawl_payloads.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
081_payload_fetch_columns.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
082_proxy_notification_trigger.sql
feat(worker): Listen for proxy_added notifications
2025-12-10 22:58:00 -07:00
083_discovery_runs.sql
feat(tasks): Add proxy_test task handler and discovery run tracking
2025-12-11 21:07:58 -07:00
084_dual_transport_preflight.sql
feat(workers): Add dual-transport preflight system
2025-12-11 22:47:52 -07:00
085_preflight_ip_fingerprint.sql
feat(workers): Add preflight schema and StatefulSet
2025-12-11 23:45:04 -07:00
086_proxy_url_column.sql
feat(workers): Add proxy reload, staggered tasks, and bulk proxy import
2025-12-12 01:53:15 -07:00
add-stock-columns.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-scrape-jobs.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00