fix(security): Add auth middleware to unprotected API endpoints

Security audit identified 8 endpoint groups that were publicly accessible
without authentication. Added authMiddleware and requireRole where appropriate.

Protected endpoints:
- /api/payloads/* - authMiddleware (trusted origins or API token)
- /api/job-queue/* - authMiddleware + requireRole('admin')
- /api/workers/* - authMiddleware
- /api/worker-registry/* - authMiddleware (pods access via trusted IPs)
- /api/k8s/* - authMiddleware + requireRole('admin')
- /api/pipeline/* - authMiddleware + requireRole('admin')
- /api/tasks/* - authMiddleware + requireRole('admin')
- /api/admin/orchestrator/* - authMiddleware + requireRole('admin')

Also:
- Added API_SECURITY.md documentation
- Filter AI settings from /settings page (managed in /ai-settings)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

CLAUDE.md

@@ -205,6 +205,58 @@ These binaries mimic real browser TLS fingerprints to avoid detection.
 
 ---
 
+## Worker Architecture (Kubernetes)
+
+### Persistent Workers (StatefulSet)
+
+Workers run as a **StatefulSet** with 8 persistent pods. They maintain identity across restarts.
+
+**Pod Names**: `scraper-worker-0` through `scraper-worker-7`
+
+**Key Properties**:
+- `updateStrategy: OnDelete` - Pods only update when manually deleted (no automatic restarts)
+- `podManagementPolicy: Parallel` - All pods start simultaneously
+- Workers register with their pod name as identity
+
+**K8s Manifest**: `backend/k8s/scraper-worker-statefulset.yaml`
+
+### Worker Lifecycle
+
+1. **Startup**: Worker registers in `worker_registry` table with pod name
+2. **Preflight**: Runs dual-transport preflights (curl + http), reports IPs and fingerprint
+3. **Task Loop**: Polls for tasks, executes them, reports status
+4. **Shutdown**: Graceful 60-second termination period
+
+### NEVER Restart Workers Unnecessarily
+
+**Claude must NOT**:
+- Restart workers unless explicitly requested
+- Use `kubectl rollout restart` on workers
+- Use `kubectl set image` on workers (this triggers restart)
+
+**To update worker code** (only when user authorizes):
+1. Build and push new image with version tag
+2. Update StatefulSet image reference
+3. Manually delete pods one at a time when ready: `kubectl delete pod scraper-worker-0 -n dispensary-scraper`
+
+### Worker Registry API
+
+**Endpoint**: `GET /api/worker-registry/workers`
+
+**Response Fields**:
+| Field | Description |
+|-------|-------------|
+| `pod_name` | Kubernetes pod name |
+| `worker_id` | Internal worker UUID |
+| `status` | active, idle, offline |
+| `curl_ip` | IP from curl preflight |
+| `http_ip` | IP from Puppeteer preflight |
+| `preflight_status` | pending, passed, failed |
+| `preflight_at` | Timestamp of last preflight |
+| `fingerprint_data` | Browser fingerprint JSON |
+
+---
+
 ## Documentation
 
 | Doc | Purpose |

backend/docs/API_SECURITY.md

@@ -0,0 +1,175 @@
+# API Security Documentation
+
+This document describes the authentication and authorization configuration for all CannaiQ API endpoints.
+
+## Authentication Methods
+
+### 1. Trusted Origins (No Token Required)
+
+Requests from trusted sources are automatically authenticated with `internal` role:
+
+**Trusted IPs:**
+- `127.0.0.1` (localhost IPv4)
+- `::1` (localhost IPv6)
+- `::ffff:127.0.0.1` (IPv4-mapped IPv6)
+
+**Trusted Domains:**
+- `https://cannaiq.co`
+- `https://www.cannaiq.co`
+- `https://findadispo.com`
+- `https://www.findadispo.com`
+- `https://findagram.co`
+- `https://www.findagram.co`
+- `http://localhost:3010`
+- `http://localhost:8080`
+- `http://localhost:5173`
+
+**Trusted Patterns:**
+- `*.cannabrands.app`
+- `*.cannaiq.co`
+
+**Internal Header:**
+- `X-Internal-Request` header matching `INTERNAL_REQUEST_SECRET` env var
+
+### 2. Bearer Token Authentication
+
+External requests must include a valid token:
+
+```
+Authorization: Bearer <token>
+```
+
+**Token Types:**
+- **JWT Token**: User session tokens (7-day expiry)
+- **API Token**: Long-lived tokens for integrations (stored in `api_tokens` table)
+
+## Authorization Levels
+
+### Public (No Auth)
+Routes accessible without authentication:
+- `GET /health` - Health check
+- `GET /api/health/*` - Comprehensive health endpoints
+- `GET /outbound-ip` - Server's outbound IP
+- `GET /api/v1/deals` - Public deals endpoint
+
+### Authenticated (Trusted Origin or Token)
+Routes requiring authentication but no specific role:
+
+| Route | Description |
+|-------|-------------|
+| `/api/payloads/*` | Raw crawl payload access |
+| `/api/workers/*` | Worker monitoring |
+| `/api/worker-registry/*` | Worker registration and heartbeats |
+| `/api/stores/*` | Store CRUD |
+| `/api/products/*` | Product listing |
+| `/api/dispensaries/*` | Dispensary data |
+
+### Admin Only (Requires `admin` or `superadmin` role)
+Routes restricted to administrators:
+
+| Route | Description |
+|-------|-------------|
+| `/api/job-queue/*` | Job queue management |
+| `/api/k8s/*` | Kubernetes control (scaling) |
+| `/api/pipeline/*` | Pipeline stage transitions |
+| `/api/tasks/*` | Task queue management |
+| `/api/admin/orchestrator/*` | Orchestrator dashboard |
+| `/api/admin/trusted-origins/*` | Manage trusted origins |
+| `/api/admin/debug/*` | Debug endpoints |
+
+**Note:** The `internal` role (localhost/trusted origins) bypasses role checks, granting automatic admin access for local development and internal services.
+
+## Endpoint Security Matrix
+
+| Endpoint Group | Auth Required | Role Required | Notes |
+|----------------|---------------|---------------|-------|
+| `/api/payloads/*` | Yes | None | Query API for raw crawl data |
+| `/api/job-queue/*` | Yes | admin | Legacy job queue (deprecated) |
+| `/api/workers/*` | Yes | None | Worker status monitoring |
+| `/api/worker-registry/*` | Yes | None | Workers register via trusted IPs |
+| `/api/k8s/*` | Yes | admin | K8s scaling controls |
+| `/api/pipeline/*` | Yes | admin | Store pipeline transitions |
+| `/api/tasks/*` | Yes | admin | Task queue CRUD |
+| `/api/admin/orchestrator/*` | Yes | admin | Orchestrator metrics/alerts |
+| `/api/admin/trusted-origins/*` | Yes | admin | Auth bypass management |
+| `/api/v1/*` | Varies | Varies | Public API (per-endpoint) |
+| `/api/consumer/*` | Varies | Varies | Consumer features |
+
+## Implementation Details
+
+### Middleware Stack
+
+```typescript
+// Authentication middleware - validates token or trusted origin
+import { authMiddleware } from '../auth/middleware';
+
+// Role requirement middleware - checks user role
+import { requireRole } from '../auth/middleware';
+
+// Usage in route files:
+router.use(authMiddleware);                           // All routes need auth
+router.use(requireRole('admin', 'superadmin'));       // Admin-only routes
+```
+
+### Auth Middleware Flow
+
+```
+Request → Check Bearer Token
+         ├─ Valid JWT → Set user from token → Continue
+         ├─ Valid API Token → Set user as api_token role → Continue
+         └─ No Token → Check Trusted Origin
+                      ├─ Trusted → Set user as internal role → Continue
+                      └─ Not Trusted → 401 Unauthorized
+```
+
+### Role Check Flow
+
+```
+Request → authMiddleware → requireRole('admin')
+                          ├─ role === 'internal' → Continue (bypass)
+                          ├─ role in ['admin', 'superadmin'] → Continue
+                          └─ else → 403 Forbidden
+```
+
+## Worker Pod Authentication
+
+Worker pods (in Kubernetes) authenticate via:
+
+1. **Internal IP**: Pods communicate via cluster IPs, which are trusted
+2. **Internal Header**: Optional `X-Internal-Request` header for explicit trust
+
+Endpoints used by workers:
+- `POST /api/worker-registry/register` - Report for duty
+- `POST /api/worker-registry/heartbeat` - Stay alive
+- `POST /api/worker-registry/deregister` - Graceful shutdown
+- `POST /api/worker-registry/task-completed` - Report task completion
+
+## API Token Management
+
+API tokens are managed via:
+- `GET /api/api-tokens` - List tokens
+- `POST /api/api-tokens` - Create token
+- `DELETE /api/api-tokens/:id` - Revoke token
+
+Token properties:
+- `token`: The bearer token value
+- `name`: Human-readable identifier
+- `rate_limit`: Requests per minute
+- `expires_at`: Optional expiration
+- `active`: Enable/disable toggle
+- `allowed_endpoints`: Optional endpoint restrictions
+
+## Security Best Practices
+
+1. **Never expose tokens in URLs** - Use Authorization header
+2. **Use HTTPS in production** - All traffic encrypted
+3. **Rotate API tokens periodically** - Set expiration dates
+4. **Monitor rate limits** - Prevent abuse
+5. **Audit access logs** - Track API usage via `api_usage_logs` table
+
+## Related Files
+
+- `src/auth/middleware.ts` - Auth middleware implementation
+- `src/routes/api-tokens.ts` - Token management endpoints
+- `src/middleware/apiTokenTracker.ts` - Usage tracking
+- `src/middleware/trustedDomains.ts` - Domain trust markers

backend/src/routes/job-queue.ts

@@ -15,9 +15,14 @@
 
 import { Router, Request, Response } from 'express';
 import { pool } from '../db/pool';
+import { authMiddleware, requireRole } from '../auth/middleware';
 
 const router = Router();
 
+// All job-queue routes require authentication and admin role
+router.use(authMiddleware);
+router.use(requireRole('admin', 'superadmin'));
+
 // In-memory queue state (would be in Redis in production)
 let queuePaused = false;
 

backend/src/routes/k8s.ts

@@ -7,9 +7,14 @@
 
 import { Router, Request, Response } from 'express';
 import * as k8s from '@kubernetes/client-node';
+import { authMiddleware, requireRole } from '../auth/middleware';
 
 const router = Router();
 
+// K8s control routes require authentication and admin role
+router.use(authMiddleware);
+router.use(requireRole('admin', 'superadmin'));
+
 // K8s client setup - lazy initialization
 let appsApi: k8s.AppsV1Api | null = null;
 let k8sError: string | null = null;

backend/src/routes/orchestrator-admin.ts

@@ -11,9 +11,14 @@ import { getLatestTrace, getTracesForDispensary, getTraceById } from '../service
 import { getProviderDisplayName } from '../utils/provider-display';
 import * as fs from 'fs';
 import * as path from 'path';
+import { authMiddleware, requireRole } from '../auth/middleware';
 
 const router = Router();
 
+// Orchestrator admin routes require authentication and admin role
+router.use(authMiddleware);
+router.use(requireRole('admin', 'superadmin'));
+
 // ============================================================
 // ORCHESTRATOR METRICS
 // ============================================================

backend/src/routes/payloads.ts

@@ -21,9 +21,13 @@ import {
   listPayloadMetadata,
 } from '../utils/payload-storage';
 import { Pool } from 'pg';
+import { authMiddleware } from '../auth/middleware';
 
 const router = Router();
 
+// All payload routes require authentication (trusted origins or API token)
+router.use(authMiddleware);
+
 // Get pool instance for queries
 const getDbPool = (): Pool => getPool() as unknown as Pool;
 

backend/src/routes/pipeline.ts

@@ -18,9 +18,14 @@
 
 import { Router, Request, Response } from 'express';
 import { pool } from '../db/pool';
+import { authMiddleware, requireRole } from '../auth/middleware';
 
 const router = Router();
 
+// Pipeline routes require authentication and admin role
+router.use(authMiddleware);
+router.use(requireRole('admin', 'superadmin'));
+
 // Valid stages
 const STAGES = ['discovered', 'validated', 'promoted', 'sandbox', 'production', 'failing'] as const;
 type Stage = typeof STAGES[number];

backend/src/routes/tasks.ts

@@ -19,9 +19,14 @@ import {
   resumeTaskPool,
   getTaskPoolStatus,
 } from '../tasks/task-pool-state';
+import { authMiddleware, requireRole } from '../auth/middleware';
 
 const router = Router();
 
+// Task routes require authentication and admin role
+router.use(authMiddleware);
+router.use(requireRole('admin', 'superadmin'));
+
 /**
  * GET /api/tasks
  * List tasks with optional filters

backend/src/routes/worker-registry.ts

@@ -23,9 +23,14 @@
 import { Router, Request, Response } from 'express';
 import { pool } from '../db/pool';
 import os from 'os';
+import { authMiddleware } from '../auth/middleware';
 
 const router = Router();
 
+// Worker registry routes require authentication
+// Note: Internal workers (pods) can access via trusted IP (localhost, in-cluster)
+router.use(authMiddleware);
+
 // ============================================================
 // WORKER REGISTRATION
 // ============================================================

backend/src/routes/workers.ts

@@ -26,9 +26,13 @@
 import { Router, Request, Response } from 'express';
 import { pool } from '../db/pool';
 import * as k8s from '@kubernetes/client-node';
+import { authMiddleware } from '../auth/middleware';
 
 const router = Router();
 
+// All worker routes require authentication (trusted origins or API token)
+router.use(authMiddleware);
+
 // ============================================================
 // K8S SCALING CONFIGURATION (added 2024-12-10)
 // Per TASK_WORKFLOW_2024-12-10.md: Admin can scale workers from UI

cannaiq/src/pages/Settings.tsx

@@ -14,11 +14,27 @@ export function Settings() {
     loadSettings();
   }, []);
 
+  // AI-related settings are managed in /ai-settings, filter them out here
+  const AI_SETTING_KEYS = [
+    'ai_model',
+    'ai_provider',
+    'anthropic_api_key',
+    'openai_api_key',
+    'anthropic_model',
+    'openai_model',
+    'anthropic_enabled',
+    'openai_enabled',
+  ];
+
   const loadSettings = async () => {
     setLoading(true);
     try {
       const data = await api.getSettings();
-      setSettings(data.settings);
+      // Filter out AI settings - those are managed in /ai-settings
+      const filteredSettings = (data.settings || []).filter(
+        (s: any) => !AI_SETTING_KEYS.includes(s.key)
+      );
+      setSettings(filteredSettings);
     } catch (error) {
       console.error('Failed to load settings:', error);
     } finally {