Commit Graph

213 Commits

Author SHA1 Message Date
Kelly
e345707db2 Add store selection to API permissions
- Add store_id and store_name columns to wp_dutchie_api_permissions
- Backend: Add /stores endpoint, require store_id when creating permissions
- Frontend: Add store selector dropdown to API Permissions form
- WordPress plugin v1.3.0: Remove store_id from shortcodes (store is tied to token)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 13:59:01 -07:00
Kelly
d614979569 Increase navigation timeout to 90s for parallel browser crawling
Dutchie pages can take 30-40s to load, and when running 3 parallel browsers
they compete for resources, causing some to hit the 60s timeout. Increased
to 90s to accommodate slow-loading React SPAs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 10:37:52 -07:00
Kelly
81606447d2 Skip proxies for Dutchie - datacenter IPs are blocked
Dutchie blocks all our datacenter proxy IPs, returning empty/different
content. Direct connection from pod IP works fine (100 products found).
Added PROXY_SKIP_DOMAINS list for sites that block datacenter IPs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 09:53:55 -07:00
Kelly
e518bb8169 Fix Dutchie scraper to wait for React content to load
- Change waitUntil from 'domcontentloaded' to 'networkidle2' for SPAs
- Add waitForSelector to wait for product elements before parsing
- WordPress plugin: update API endpoints to use hardcoded URL

The scraper was returning 0 products because it wasn't waiting for
React to render the product list. Now it properly waits for either
the product list items or an empty state indicator.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 09:33:32 -07:00
Kelly
199b6a8a23 Remove incorrect migration 029, add snapshot architecture, improve scraper
- Delete migration 029 that was incorrectly creating duplicate dispensaries
- Add migration 028 for snapshot architecture
- Improve downloader with proxy/UA rotation
- Update scraper monitor and tools pages
- Various scraper improvements

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 08:52:54 -07:00
Kelly
9de0d709b2 Update admin panel to use unified dispensaries table
- Add migration 026 to update dispensary_crawl_status view with new fields
- Update dashboard API to use dispensaries table (not stores)
- Show current inventory counts (products seen in last 7 days)
- Update ScraperSchedule UI to show provider_type correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 00:13:41 -07:00
Kelly
9d8972aa86 Fix category-crawler-jobs store lookup query
- Fix column name from s.dutchie_plus_url to s.dutchie_url
- Add availability tracking and product freshness APIs
- Add crawl script for sequential dispensary processing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-01 00:07:00 -07:00
Kelly
5306c3f4ca Switch scheduler UI to dispensary-based API
- Add migrations 021-023 for dispensary_crawl_schedule tables and views
- Add dispensary-orchestrator service and bootstrap-discovery script
- Update schedule routes with dispensary endpoints (/api/schedule/dispensaries)
- Switch frontend scheduler to use canonical dispensaries table (182 AZDHS entries)
- Add dispensary schedule API methods to frontend api.ts
- Remove "Unmapped" badge logic - all dispensaries now linked properly
- Add proper URL linking to dispensary detail pages (/dispensaries/:state/:city/:slug)
- Update Jobs table to show dispensary_name

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 18:54:52 -07:00
Kelly
0083f6a510 Add version display in admin sidebar footer
- Add /api/version endpoint that returns build info from env vars
- Add version footer to Layout.tsx showing build version, git SHA, and image tag
- Update Dockerfile to accept build args for version info (APP_BUILD_VERSION, APP_GIT_SHA, APP_BUILD_TIME, CONTAINER_IMAGE_TAG)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 10:01:10 -07:00
Kelly
afc71e4225 Fix migration idempotency for constraint creation
Make products_store_id_slug_unique constraint creation idempotent
by checking if it exists before attempting to add it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 09:55:51 -07:00
Kelly
3861a31a3b Add crawler scheduler, orchestrator, and multi-category intelligence
- Add scheduler UI with store schedules, job queue, and global settings
- Add store crawl orchestrator for intelligent crawl workflow
- Add multi-category intelligence detection (product, specials, brands, metadata)
- Add CrawlerLogger for structured JSON logging
- Add migrations for scheduler tables and dispensary linking
- Add dispensary → scheduler navigation link
- Support production/sandbox crawler modes per provider

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 09:29:15 -07:00
Kelly
8b4292fbb2 Add local product detail page with Dutchie comparison
- Add ProductDetail page for viewing products locally
- Add Dutchie and Details buttons to product cards in Products and StoreDetail pages
- Add Last Updated display showing data freshness
- Add parallel scrape scripts and routes
- Add K8s deployment configurations
- Add frontend Dockerfile with nginx

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 06:34:38 -07:00
Kelly
5757a8e9bd Initial commit - Dutchie dispensary scraper 2025-11-28 19:45:44 -07:00