a35976b9e9
- Move deprecated directories to src/_deprecated/: - hydration/ (old pipeline approach) - scraper-v2/ (old Puppeteer scraper) - canonical-hydration/ (merged into tasks) - Unused services: availability, crawler-logger, geolocation, etc - Unused utils: age-gate-playwright, HomepageValidator, stealthBrowser - Archive outdated docs to docs/_archive/: - ANALYTICS_RUNBOOK.md - ANALYTICS_V2_EXAMPLES.md - BRAND_INTELLIGENCE_API.md - CRAWL_PIPELINE.md - TASK_WORKFLOW_2024-12-10.md - WORKER_TASK_ARCHITECTURE.md - ORGANIC_SCRAPING_GUIDE.md - Add docs/CODEBASE_MAP.md as single source of truth - Add warning files to deprecated/archived directories - Slim down CLAUDE.md to essential rules only 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
219 lines
7.8 KiB
Markdown
219 lines
7.8 KiB
Markdown
# CannaiQ Backend Codebase Map
|
|
|
|
**Last Updated:** 2025-12-12
|
|
**Purpose:** Help Claude and developers understand which code is current vs deprecated
|
|
|
|
---
|
|
|
|
## Quick Reference: What to Use
|
|
|
|
### For Crawling/Scraping
|
|
| Task | Use This | NOT This |
|
|
|------|----------|----------|
|
|
| Fetch products | `src/tasks/handlers/payload-fetch.ts` | `src/hydration/*` |
|
|
| Process products | `src/tasks/handlers/product-refresh.ts` | `src/scraper-v2/*` |
|
|
| GraphQL client | `src/platforms/dutchie/client.ts` | `src/dutchie-az/services/graphql-client.ts` |
|
|
| Worker system | `src/tasks/task-worker.ts` | `src/dutchie-az/services/worker.ts` |
|
|
|
|
### For Database
|
|
| Task | Use This | NOT This |
|
|
|------|----------|----------|
|
|
| Get DB pool | `src/db/pool.ts` | `src/dutchie-az/db/connection.ts` |
|
|
| Run migrations | `src/db/migrate.ts` (CLI only) | Never import at runtime |
|
|
| Query products | `store_products` table | `products`, `dutchie_products` |
|
|
| Query stores | `dispensaries` table | `stores` table |
|
|
|
|
### For Discovery
|
|
| Task | Use This |
|
|
|------|----------|
|
|
| Discover stores | `src/discovery/*.ts` |
|
|
| Run discovery | `npx tsx src/scripts/run-discovery.ts` |
|
|
|
|
---
|
|
|
|
## Directory Status
|
|
|
|
### ACTIVE DIRECTORIES (Use These)
|
|
|
|
```
|
|
src/
|
|
├── auth/ # JWT/session auth, middleware
|
|
├── db/ # Database pool, migrations
|
|
├── discovery/ # Dutchie store discovery pipeline
|
|
├── middleware/ # Express middleware
|
|
├── multi-state/ # Multi-state query support
|
|
├── platforms/ # Platform-specific clients (Dutchie, Jane, etc)
|
|
│ └── dutchie/ # THE Dutchie client - use this one
|
|
├── routes/ # Express API routes
|
|
├── services/ # Core services (logger, scheduler, etc)
|
|
├── tasks/ # Task system (workers, handlers, scheduler)
|
|
│ └── handlers/ # Task handlers (payload_fetch, product_refresh, etc)
|
|
├── types/ # TypeScript types
|
|
└── utils/ # Utilities (storage, image processing)
|
|
```
|
|
|
|
### DEPRECATED DIRECTORIES (DO NOT USE)
|
|
|
|
```
|
|
src/
|
|
├── hydration/ # DEPRECATED - Old pipeline approach
|
|
├── scraper-v2/ # DEPRECATED - Old scraper engine
|
|
├── canonical-hydration/# DEPRECATED - Merged into tasks/handlers
|
|
├── dutchie-az/ # PARTIAL - Some parts deprecated, some active
|
|
│ ├── db/ # DEPRECATED - Use src/db/pool.ts
|
|
│ └── services/ # PARTIAL - worker.ts still runs, graphql-client.ts deprecated
|
|
├── portals/ # FUTURE - Not yet implemented
|
|
├── seo/ # PARTIAL - Settings work, templates WIP
|
|
└── system/ # DEPRECATED - Old orchestration system
|
|
```
|
|
|
|
### DEPRECATED FILES (DO NOT USE)
|
|
|
|
```
|
|
src/dutchie-az/db/connection.ts # Use src/db/pool.ts instead
|
|
src/dutchie-az/services/graphql-client.ts # Use src/platforms/dutchie/client.ts
|
|
src/hydration/*.ts # Entire directory deprecated
|
|
src/scraper-v2/*.ts # Entire directory deprecated
|
|
```
|
|
|
|
---
|
|
|
|
## Key Files Reference
|
|
|
|
### Entry Points
|
|
| File | Purpose | Status |
|
|
|------|---------|--------|
|
|
| `src/index.ts` | Main Express server | ACTIVE |
|
|
| `src/dutchie-az/services/worker.ts` | Worker process entry | ACTIVE |
|
|
| `src/tasks/task-worker.ts` | Task worker (new system) | ACTIVE |
|
|
|
|
### Dutchie Integration
|
|
| File | Purpose | Status |
|
|
|------|---------|--------|
|
|
| `src/platforms/dutchie/client.ts` | GraphQL client, hashes, curl | **PRIMARY** |
|
|
| `src/platforms/dutchie/queries.ts` | High-level query functions | ACTIVE |
|
|
| `src/platforms/dutchie/index.ts` | Re-exports | ACTIVE |
|
|
|
|
### Task Handlers
|
|
| File | Purpose | Status |
|
|
|------|---------|--------|
|
|
| `src/tasks/handlers/payload-fetch.ts` | Fetch products from Dutchie | **PRIMARY** |
|
|
| `src/tasks/handlers/product-refresh.ts` | Process payload into DB | **PRIMARY** |
|
|
| `src/tasks/handlers/menu-detection.ts` | Detect menu type | ACTIVE |
|
|
| `src/tasks/handlers/id-resolution.ts` | Resolve platform IDs | ACTIVE |
|
|
| `src/tasks/handlers/image-download.ts` | Download product images | ACTIVE |
|
|
|
|
### Database
|
|
| File | Purpose | Status |
|
|
|------|---------|--------|
|
|
| `src/db/pool.ts` | Canonical DB pool | **PRIMARY** |
|
|
| `src/db/migrate.ts` | Migration runner (CLI only) | CLI ONLY |
|
|
| `src/db/auto-migrate.ts` | Auto-run migrations on startup | ACTIVE |
|
|
|
|
### Configuration
|
|
| File | Purpose | Status |
|
|
|------|---------|--------|
|
|
| `.env` | Environment variables | ACTIVE |
|
|
| `package.json` | Dependencies | ACTIVE |
|
|
| `tsconfig.json` | TypeScript config | ACTIVE |
|
|
|
|
---
|
|
|
|
## GraphQL Hashes (CRITICAL)
|
|
|
|
The correct hashes are in `src/platforms/dutchie/client.ts`:
|
|
|
|
```typescript
|
|
export const GRAPHQL_HASHES = {
|
|
FilteredProducts: 'ee29c060826dc41c527e470e9ae502c9b2c169720faa0a9f5d25e1b9a530a4a0',
|
|
GetAddressBasedDispensaryData: '13461f73abf7268770dfd05fe7e10c523084b2bb916a929c08efe3d87531977b',
|
|
ConsumerDispensaries: '0a5bfa6ca1d64ae47bcccb7c8077c87147cbc4e6982c17ceec97a2a4948b311b',
|
|
GetAllCitiesByState: 'ae547a0466ace5a48f91e55bf6699eacd87e3a42841560f0c0eabed5a0a920e6',
|
|
};
|
|
```
|
|
|
|
**ALWAYS** use `Status: 'Active'` for FilteredProducts (not `null` or `'All'`).
|
|
|
|
---
|
|
|
|
## Scripts Reference
|
|
|
|
### Useful Scripts (in `src/scripts/`)
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| `run-discovery.ts` | Run Dutchie discovery |
|
|
| `crawl-single-store.ts` | Test crawl a single store |
|
|
| `test-dutchie-graphql.ts` | Test GraphQL queries |
|
|
|
|
### One-Off Scripts (probably don't need)
|
|
| Script | Purpose |
|
|
|--------|---------|
|
|
| `harmonize-az-dispensaries.ts` | One-time data cleanup |
|
|
| `bootstrap-stores-for-dispensaries.ts` | One-time migration |
|
|
| `backfill-*.ts` | Historical backfill scripts |
|
|
|
|
---
|
|
|
|
## API Routes
|
|
|
|
### Active Routes (in `src/routes/`)
|
|
| Route File | Mount Point | Purpose |
|
|
|------------|-------------|---------|
|
|
| `auth.ts` | `/api/auth` | Login/logout/session |
|
|
| `stores.ts` | `/api/stores` | Store CRUD |
|
|
| `dashboard.ts` | `/api/dashboard` | Dashboard stats |
|
|
| `workers.ts` | `/api/workers` | Worker monitoring |
|
|
| `pipeline.ts` | `/api/pipeline` | Crawl triggers |
|
|
| `discovery.ts` | `/api/discovery` | Discovery management |
|
|
| `analytics.ts` | `/api/analytics` | Analytics queries |
|
|
| `wordpress.ts` | `/api/v1/wordpress` | WordPress plugin API |
|
|
|
|
---
|
|
|
|
## Documentation Files
|
|
|
|
### Current Docs (in `backend/docs/`)
|
|
| Doc | Purpose | Currency |
|
|
|-----|---------|----------|
|
|
| `TASK_WORKFLOW_2024-12-10.md` | Task system architecture | CURRENT |
|
|
| `WORKER_TASK_ARCHITECTURE.md` | Worker/task design | CURRENT |
|
|
| `CRAWL_PIPELINE.md` | Crawl pipeline overview | CURRENT |
|
|
| `ORGANIC_SCRAPING_GUIDE.md` | Browser-based scraping | CURRENT |
|
|
| `CODEBASE_MAP.md` | This file | CURRENT |
|
|
| `ANALYTICS_V2_EXAMPLES.md` | Analytics API examples | CURRENT |
|
|
| `BRAND_INTELLIGENCE_API.md` | Brand API docs | CURRENT |
|
|
|
|
### Root Docs
|
|
| Doc | Purpose | Currency |
|
|
|-----|---------|----------|
|
|
| `CLAUDE.md` | Claude instructions | **PRIMARY** |
|
|
| `README.md` | Project overview | NEEDS UPDATE |
|
|
|
|
---
|
|
|
|
## Common Mistakes to Avoid
|
|
|
|
1. **Don't use `src/hydration/`** - It's an old approach that was superseded by the task system
|
|
|
|
2. **Don't use `src/dutchie-az/db/connection.ts`** - Use `src/db/pool.ts` instead
|
|
|
|
3. **Don't import `src/db/migrate.ts` at runtime** - It will crash. Only use for CLI migrations.
|
|
|
|
4. **Don't query `stores` table** - It's empty. Use `dispensaries`.
|
|
|
|
5. **Don't query `products` table** - It's empty. Use `store_products`.
|
|
|
|
6. **Don't use wrong GraphQL hash** - Always get hash from `GRAPHQL_HASHES` in client.ts
|
|
|
|
7. **Don't use `Status: null`** - It returns 0 products. Use `Status: 'Active'`.
|
|
|
|
---
|
|
|
|
## When in Doubt
|
|
|
|
1. Check if the file is imported in `src/index.ts` - if not, it may be deprecated
|
|
2. Check the last modified date - older files may be stale
|
|
3. Look for `DEPRECATED` comments in the code
|
|
4. Ask: "Is there a newer version of this in `src/tasks/` or `src/platforms/`?"
|
|
5. Read the relevant doc in `docs/` before modifying code
|