Add Getting Started guide for development and deployment
Includes: - Architecture overview with k8s diagram - Local development setup - Database information (local vs remote) - K8s deployment instructions - Running crawlers remotely - Proxy system documentation - Common tasks and troubleshooting 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Kelly
291
GETTING-STARTED.md
Normal file
291
GETTING-STARTED.md
Normal file
@@ -0,0 +1,291 @@
|
|||||||
|
# Dispensary Scraper - Getting Started Guide
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ K8s Cluster (Production) │
|
||||||
|
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
|
||||||
|
│ │ frontend │ │ scraper │ │ postgres │ │
|
||||||
|
│ │ (React Admin) │ │ (Node + API) │ │ (Database) │ │
|
||||||
|
│ │ port: 80 │ │ port: 3010 │ │ port: 5432 │ │
|
||||||
|
│ └────────────────┘ └────────────────┘ └────────────────┘ │
|
||||||
|
│ │ │ │
|
||||||
|
│ └─────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ Ingress: dispos.crawlsy.com │
|
||||||
|
│ / → frontend │
|
||||||
|
│ /api → scraper │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Components
|
||||||
|
|
||||||
|
| Component | Description | Image |
|
||||||
|
|-----------|-------------|-------|
|
||||||
|
| **frontend** | React admin panel for monitoring crawls | `code.cannabrands.app/creationshop/dispensary-scraper-frontend:latest` |
|
||||||
|
| **scraper** | Node.js backend with API + Puppeteer crawlers | `code.cannabrands.app/creationshop/dispensary-scraper:latest` |
|
||||||
|
| **postgres** | PostgreSQL database | `postgres:15` |
|
||||||
|
|
||||||
|
### Key Directories
|
||||||
|
|
||||||
|
```
|
||||||
|
dispensary-scraper/
|
||||||
|
├── backend/ # Node.js backend
|
||||||
|
│ ├── src/
|
||||||
|
│ │ ├── routes/ # API endpoints
|
||||||
|
│ │ ├── services/ # Business logic
|
||||||
|
│ │ ├── scraper-v2/ # Scraper engine (Scrapy-inspired)
|
||||||
|
│ │ └── scripts/ # CLI scripts
|
||||||
|
│ └── migrations/ # SQL migrations
|
||||||
|
├── frontend/ # React admin panel
|
||||||
|
├── k8s/ # Kubernetes manifests
|
||||||
|
├── wordpress-plugin/ # WP plugin for dispensary sites
|
||||||
|
└── docs/ # Architecture documentation
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Local Development
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- Node.js 20+
|
||||||
|
- Docker & Docker Compose
|
||||||
|
- kubectl configured for the k8s cluster
|
||||||
|
- SSH tunnel to production database (port 54320)
|
||||||
|
|
||||||
|
### 1. Start Local Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/kelly/git/dispensary-scraper
|
||||||
|
|
||||||
|
# Start local postgres and minio
|
||||||
|
docker start dutchie-postgres dutchie-minio
|
||||||
|
|
||||||
|
# Or use docker-compose for fresh setup
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Environment Variables
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Production database (via SSH tunnel)
|
||||||
|
export DATABASE_URL="postgresql://dutchie:dutchie_local_pass@localhost:54320/dutchie_menus"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Run Backend
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
npm install
|
||||||
|
npm run dev
|
||||||
|
# API available at http://localhost:3010
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Run Frontend
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd frontend
|
||||||
|
npm install
|
||||||
|
npm run dev
|
||||||
|
# UI available at http://localhost:5173
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Run Migrations
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
DATABASE_URL="postgresql://dutchie:dutchie_local_pass@localhost:54320/dutchie_menus" npx tsx src/db/migrate.ts
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Database Architecture
|
||||||
|
|
||||||
|
### Two Databases
|
||||||
|
|
||||||
|
| Database | Port | User | Purpose |
|
||||||
|
|----------|------|------|---------|
|
||||||
|
| `dutchie_menus` | 54320 | dutchie | **Production** - SSH tunnel to remote k8s postgres |
|
||||||
|
| `dispensary_scraper` | 5432 | scraper | **Remote only** - Accessed via k8s pods |
|
||||||
|
|
||||||
|
**Local development uses port 54320** (SSH tunnel to production).
|
||||||
|
|
||||||
|
### Key Tables
|
||||||
|
|
||||||
|
| Table | Purpose |
|
||||||
|
|-------|---------|
|
||||||
|
| `dispensaries` | Master list of dispensaries (from AZDHS) with crawl metadata |
|
||||||
|
| `stores` | Legacy table - being migrated to dispensaries |
|
||||||
|
| `products` | Scraped product data |
|
||||||
|
| `proxies` | Proxy pool for bot detection avoidance |
|
||||||
|
| `crawl_jobs` | Crawl job history and status |
|
||||||
|
|
||||||
|
### Dispensary Crawl Fields
|
||||||
|
|
||||||
|
The `dispensaries` table now contains crawl metadata:
|
||||||
|
- `provider_type` - Menu provider (dutchie, jane, etc.)
|
||||||
|
- `menu_provider` - Detected provider for crawling
|
||||||
|
- `crawler_mode` - 'production' or 'sandbox'
|
||||||
|
- `crawl_status` - Current status (idle, running, ok, error)
|
||||||
|
- `menu_url` - URL to crawl
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Kubernetes Deployment
|
||||||
|
|
||||||
|
### Namespace
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get all -n dispensary-scraper
|
||||||
|
```
|
||||||
|
|
||||||
|
### Image Registry
|
||||||
|
|
||||||
|
Images are stored at `code.cannabrands.app/creationshop/`:
|
||||||
|
- `dispensary-scraper:latest` - Backend
|
||||||
|
- `dispensary-scraper-frontend:latest` - Frontend
|
||||||
|
|
||||||
|
### Deploy New Code
|
||||||
|
|
||||||
|
The CI/CD pipeline builds and deploys automatically on push to master. To manually deploy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Restart deployment to pull latest image
|
||||||
|
kubectl rollout restart deployment/scraper -n dispensary-scraper
|
||||||
|
|
||||||
|
# Check status
|
||||||
|
kubectl rollout status deployment/scraper -n dispensary-scraper
|
||||||
|
|
||||||
|
# View logs
|
||||||
|
kubectl logs -f deployment/scraper -n dispensary-scraper
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run Migrations on Remote
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Copy migration file to pod
|
||||||
|
kubectl cp backend/migrations/029_link_dutchie_stores_to_dispensaries.sql \
|
||||||
|
dispensary-scraper/scraper-POD-NAME:/tmp/migration.sql
|
||||||
|
|
||||||
|
# Execute via psql in pod
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB -f /tmp/migration.sql'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Running Crawlers
|
||||||
|
|
||||||
|
### Remote (Production)
|
||||||
|
|
||||||
|
Crawlers should run on the k8s cluster, not locally.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run crawl script inside pod
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
node dist/scripts/queue-dispensaries.js --production
|
||||||
|
|
||||||
|
# Or for a specific dispensary
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
node -e "require('./dist/services/crawler-jobs').runDutchieMenuCrawlJob(112)"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Trigger via API
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Queue all Dutchie dispensaries
|
||||||
|
curl -X POST https://dispos.crawlsy.com/api/crawl/queue-all?type=dutchie
|
||||||
|
```
|
||||||
|
|
||||||
|
### Local Testing
|
||||||
|
|
||||||
|
For code development/testing only:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend
|
||||||
|
DATABASE_URL="postgresql://dutchie:dutchie_local_pass@localhost:54320/dutchie_menus" \
|
||||||
|
npx tsx src/scripts/queue-dispensaries.ts --production --limit=1
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proxy System
|
||||||
|
|
||||||
|
### How It Works
|
||||||
|
|
||||||
|
1. Proxies stored in `proxies` table with `active=true/false`
|
||||||
|
2. `getActiveProxy()` selects random active proxy
|
||||||
|
3. On bot detection, proxy gets 35-second in-memory cooldown
|
||||||
|
4. Permanent failures mark `active=false` in DB
|
||||||
|
|
||||||
|
### Adding New Proxies
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO proxies (ip, port, protocol, active)
|
||||||
|
VALUES ('123.45.67.89', 3128, 'http', true);
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Tasks
|
||||||
|
|
||||||
|
### Check Crawl Status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Via kubectl
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
node -e "require('pg').Pool({connectionString: process.env.DATABASE_URL}).query('SELECT id, name, crawl_status FROM dispensaries WHERE provider_type = \\'dutchie\\' LIMIT 10').then(r => console.log(r.rows))"
|
||||||
|
|
||||||
|
# Via admin UI
|
||||||
|
# https://dispos.crawlsy.com/schedule
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Recent Products
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
node -e "require('pg').Pool({connectionString: process.env.DATABASE_URL}).query('SELECT COUNT(*), MAX(updated_at) FROM products').then(r => console.log(r.rows))"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restart Stuck Crawl
|
||||||
|
|
||||||
|
```sql
|
||||||
|
UPDATE dispensaries SET crawl_status = 'idle' WHERE crawl_status = 'running';
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Pod Not Starting
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl describe pod -n dispensary-scraper -l app=scraper
|
||||||
|
kubectl logs -n dispensary-scraper -l app=scraper --previous
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Connection Issues
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test from pod
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
node -e "require('pg').Pool({connectionString: process.env.DATABASE_URL}).query('SELECT 1').then(() => console.log('OK'))"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Proxy Issues
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check active proxy count
|
||||||
|
kubectl exec -n dispensary-scraper deployment/scraper -- \
|
||||||
|
node -e "require('pg').Pool({connectionString: process.env.DATABASE_URL}).query('SELECT COUNT(*) FROM proxies WHERE active = true').then(r => console.log('Active proxies:', r.rows[0].count))"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- `docs/CRAWL_OPERATIONS.md` - Crawl scheduling and data philosophy
|
||||||
|
- `docs/PRODUCT_BRAND_INTELLIGENCE_ARCHITECTURE.md` - Data processing
|
||||||
|
- `DATABASE-GUIDE.md` - Database usage details
|
||||||
Reference in New Issue
Block a user