Creationshop/cannaiq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4949b22457e5257fe92b6fc6416d4b62d7b72343
cannaiq/backend/migrations
History
Kelly 4949b22457 feat(tasks): Refactor task workflow with payload/refresh separation 
Major changes:
- Split crawl into payload_fetch (API → disk) and product_refresh (disk → DB)
- Add task chaining: store_discovery → product_discovery → payload_fetch → product_refresh
- Add payload storage utilities for gzipped JSON on filesystem
- Add /api/payloads endpoints for payload access and diffing
- Add DB-driven TaskScheduler with schedule persistence
- Track newDispensaryIds through discovery promotion for chaining
- Add stealth improvements: HTTP fingerprinting, proxy rotation enhancements
- Add Workers dashboard K8s scaling controls

New files:
- src/tasks/handlers/payload-fetch.ts - Fetches from API, saves to disk
- src/services/task-scheduler.ts - DB-driven schedule management
- src/utils/payload-storage.ts - Payload save/load utilities
- src/routes/payloads.ts - Payload API endpoints
- src/services/http-fingerprint.ts - Browser fingerprint generation
- docs/TASK_WORKFLOW_2024-12-10.md - Complete workflow documentation

Migrations:
- 078: Proxy consecutive 403 tracking
- 079: task_schedules table
- 080: raw_crawl_payloads table
- 081: payload column and last_fetch_at

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-10 22:15:35 -07:00
..
005_api_tokens.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
005_notifications.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
007_proxy_test_jobs.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
008_proxy_locations.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
009_image_sizes.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
010_store_logo.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
011_create_dispensaries_table.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
012_create_dispensary_changes_table.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
013_populate_dispensaries_from_azdhs.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
014_add_dispensary_fk_to_tables.sql
fix: Make all migrations idempotent
2025-12-07 23:48:35 -07:00
015_crawler_schedule.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
016_extend_dispensaries_for_multi_provider.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
017_create_crawler_sandboxes_and_templates.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
018_multi_category_intelligence.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
019_link_stores_to_dispensaries.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
020_scheduler_orchestrator_fields.sql
Add crawler scheduler, orchestrator, and multi-category intelligence
2025-11-30 09:29:15 -07:00
021_dispensary_crawl_schedule.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
022_scheduler_dispensary_link_fields.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
023_dispensary_crawl_status_add_link_fields.sql
Switch scheduler UI to dispensary-based API
2025-11-30 18:54:52 -07:00
024_product_availability_tracking.sql
Fix category-crawler-jobs store lookup query
2025-12-01 00:07:00 -07:00
025_dispensary_crawl_fields.sql
Add crawl fields directly to dispensaries table
2025-11-30 23:52:48 -07:00
026_update_crawl_status_view.sql
Update admin panel to use unified dispensaries table
2025-12-01 00:13:41 -07:00
027_brand_history_tracking.sql
Add brand history tracking for new/dropped brands detection
2025-12-01 00:18:47 -07:00
028_snapshot_architecture.sql
Remove incorrect migration 029, add snapshot architecture, improve scraper
2025-12-01 08:52:54 -07:00
029_add_store_id_to_wp_api_permissions.sql
Add store selection to API permissions
2025-12-01 13:59:01 -07:00
030_add_dispensary_id_to_wp_api_permissions.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
031_consolidate_schema.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
031_product_normalized_fields.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
032_dutchie_graphql_schema.sql
Add brand history view to dutchie GraphQL migration
2025-12-02 11:34:01 -07:00
032_menu_type_and_local_images.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
033_add_platform_id_to_crawl_status.sql
Add CLAUDE guidelines for consolidated pipeline
2025-12-02 13:28:23 -07:00
033_job_queue_claiming.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
034_crawl_status_fields.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
034_dispensary_failure_tracking.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
035_multi_domain_users.sql
feat: Add Findagram and FindADispo consumer frontends
2025-12-05 16:10:15 -07:00
036_findadispo_dispensary_fields.sql
feat: Add Findagram and FindADispo consumer frontends
2025-12-05 16:10:15 -07:00
037_dispensary_crawler_profiles.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
038_profile_status_field.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
039_crawl_orchestration_traces.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
040_dispensary_dba_name.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
041_cannaiq_canonical_schema.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
043_add_states_table.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
044_add_provider_detection_data.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
045_add_image_columns.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
046_crawler_reliability.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
046_raw_payloads_table.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
047_analytics_infrastructure.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
048_production_sync_monitoring.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
050_cannaiq_canonical_v2.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_cannaiq_canonical_safe_bootstrap.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_create_mv_state_metrics.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
051_worker_definitions.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
052_add_provider_data_columns.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_add_state_cannabis_flags.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_hydration_schema_alignment.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
052_seo_settings.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
053_analytics_indexes.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
053_dutchie_discovery_schema.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
054_worker_metadata.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
055_workforce_enhancements.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
056_fix_worker_and_run_logs.sql
feat: Add v2 architecture with multi-state support and orchestrator services
2025-12-07 11:30:57 -07:00
057_crawl_enabled_and_dutchie_verified.sql
feat: AZ dispensary harmonization with Dutchie source of truth
2025-12-08 10:19:49 -07:00
057_visibility_tracking_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
058_add_id_resolution_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
059_job_queue_columns.sql
feat(cannaiq): Add Workers Dashboard and visibility tracking
2025-12-07 11:04:12 -07:00
060_consumer_verification_notifications.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
061_product_click_events.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
062_click_analytics_enhancements.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
063_seo_pages.sql
feat: Responsive admin UI, SEO pages, and click analytics
2025-12-07 22:48:21 -07:00
064_ai_settings.sql
feat: Add AI settings to database (provider/model configurable via UI)
2025-12-07 23:18:52 -07:00
065_slug_verification_tracking.sql
feat: AZ dispensary harmonization with Dutchie source of truth
2025-12-08 10:19:49 -07:00
066_dutchie_field_alignment.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
067_promotion_log.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
068_crawler_status_alerts.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
069_six_stage_status.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
070_product_variants.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
071_harmonize_store_products.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
072_product_views.sql
feat: SEO template library, discovery pipeline, and orchestrator enhancements
2025-12-09 00:05:34 -07:00
073_proxy_timezone.sql
feat(images): Add local image storage with on-demand resizing
2025-12-09 11:04:50 -07:00
074_worker_task_queue.sql
feat(tasks): Add unified task-based worker architecture
2025-12-09 16:27:03 -07:00
075_consecutive_misses.sql
fix: Public API column mappings and OOS detection
2025-12-09 20:44:53 -07:00
076_visitor_analytics.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
076_worker_registry.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
077_click_events_location.sql
feat: Stealth worker system with mandatory proxy rotation
2025-12-10 00:44:59 -07:00
078_proxy_consecutive_403.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
079_task_schedules.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
080_raw_crawl_payloads.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
081_payload_fetch_columns.sql
feat(tasks): Refactor task workflow with payload/refresh separation
2025-12-10 22:15:35 -07:00
add-stock-columns.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-scrape-jobs.sql
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00