Creationshop/cannaiq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
95fc8bb4cc5493bd8b9598ee7c60038e96dfd9c3
cannaiq/backend
History
Kelly 95fc8bb4cc Improve menu detection to extract platform ID from URL and crawl proprietary domains 
- Add extractFromMenuUrl() to discovery.ts that extracts either cName or platformId directly
  from Dutchie URLs (handles /api/v2/embedded-menu/<id>.js pattern)
- Add isObjectId() helper to identify MongoDB ObjectIds in URLs
- Update menu-detection.ts to skip GraphQL resolution when URL contains platformId directly
- For proprietary domains (curaleaf, sol), crawl website to find actual menu provider
  instead of blindly marking as not_crawlable
- If website crawl finds Dutchie embedded menu, set menu_type='dutchie' and resolve platform ID
- Tested successfully with consumeaz.com which discovers Dutchie embedded menu JS URL

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-03 21:41:45 -07:00
..
dist
fix(menu-detection): remove non-existent platform_dispensary_id_resolved_at column
2025-12-03 19:40:22 -07:00
k8s
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
migrations
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
new-scrapers
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
node_modules
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
public
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
src
Improve menu detection to extract platform ID from URL and crawl proprietary domains
2025-12-03 21:41:45 -07:00
.dockerignore
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
.env
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-curaleaf-az-stores.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-curaleaf-docker.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-discount-columns.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-geo-fields.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-price-history.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-proxy-location-columns.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-sol-flower-stores.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
add-test-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
API_TOKENS_GUIDE.md
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
API_USAGE.md
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
cancel-pending-job.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
capture-age-gate-cookies.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-48th-store.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-brand-jobs.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-brand-names.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-brands-table.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-db.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-deeply-rooted.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-discounts.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-enrichment-progress.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-for-prices.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-jobs.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-jobs.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-leaks.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-product-data.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-product-detail-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-product-prices.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-products-schema.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-products.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-proxies.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-proxy-stats.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-scraper-data.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-select-prices.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-special-product.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-store.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-stores.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-tables.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-treez-categories.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-treez-data.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
check-treez-pagination.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
complete-schema-migration.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
count-products.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-azdhs-table.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-brands-table.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-brands-tables.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
create-table.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
curaleaf-cookies.json
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-after-state-select.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-age-gate-detailed.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-age-gate-elements.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-azdhs-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-brands-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-curaleaf-buttons.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-dutchie-detection.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-dutchie-selectors.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-google-scraper.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-product-card.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-products-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-sale-price.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-scrape.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-sol-flower.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
debug-specials-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
delete-dispensary-products.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
diagnose-curaleaf-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
Dockerfile
Rename WordPress plugin from Dutchie Menus to Crawlsy Menus v1.3.0
2025-12-01 16:29:21 -07:00
Dockerfile.dev
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
enrich-azdhs-from-google-maps.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
enrich-dispensaries-from-google.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
enrich-prices.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
enrich-single-dispensary.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
explore-curaleaf-menu.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
find-dutchie-menu-curaleaf.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
find-price-location.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
find-products-curaleaf.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
fix-brand-names.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
fix-brands-table.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
fix-category-urls.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
fix-docker-db-urls.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
fix-stuck-job.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
get-best-dispensary.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
get-dispensary-ids.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
import-proxies.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
inspect-treez-structure.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
launch-25-scrapers.sh
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
list-dispensaries.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
list-stores.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
migrate-proxy-locations.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
openapi.yaml
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
package-lock.json
Add local product detail page with Dutchie comparison
2025-11-30 06:34:38 -07:00
package.json
Add AZ schema migration script and health check
2025-12-02 10:40:06 -07:00
parse-azdhs-copied-data.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
populate-jobs.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
populate-proxy-locations.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
preview-azdhs-import.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
reset-jobs.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
reset-one-job.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
run-location-migration.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
run-logo-migration.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
run-migration.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
run-scrape-jobs-migration.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
schema-dump.sql
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
scrape-48th-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-azdhs-api.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-azdhs-auto.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-azdhs-better.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-azdhs-manual-scroll.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-azdhs-map.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-brands-48th.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-bypass-age-gate.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-curaleaf-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-curaleaf-simple.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-curaleaf-stores.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-curaleaf-with-proxy.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-dutchie-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-google-maps.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-leafly-arizona.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-leafly-map-markers.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-leafly-map.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-mobile-chrome.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-parallel-brands.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-products.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-real-clicks.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-treez.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-two-gates.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-with-age-gate.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
scrape-with-jobs.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
SESSION_SUMMARY.md
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
setup-api-permissions-table.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
start-proxy-test.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
sync-by-address.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
sync-dispensaries-to-dutchie.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-azdhs-parser.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-brand-products.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-brand-scrape.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-brands-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-category-discovery.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-curaleaf-playwright-advanced.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-curaleaf-playwright-final.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-curaleaf-playwright.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-deeply-rooted.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-dutchie-simple.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-dutchie-template.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-filter-interaction.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-graphql-curl.sh
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
test-improved-age-gate.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-llm-scraper-deeply-rooted.ts
fix(monitor): remove non-existent worker columns from job_run_logs query
2025-12-03 18:45:05 -07:00
test-no-proxy.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-playwright-scraper-curaleaf.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-puppeteer-click.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-scrape-flower.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-sol-flower-age-gate.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-sol-flower-category-discovery.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-sol-flower-scraper.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-stealth-sol-flower.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-stock-status.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-template-integration.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
test-treez-page.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
tsconfig.json
Add local product detail page with Dutchie comparison
2025-11-30 06:34:38 -07:00
update-best-dispensary.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
update-curaleaf-dutchie-urls.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
verify-categories.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
verify-curaleaf-urls.js
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
verify-curaleaf.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00
verify-treez-total-count.ts
Initial commit - Dutchie dispensary scraper
2025-11-28 19:45:44 -07:00