- Add backend stale process monitoring API (/api/stale-processes) - Add users management route - Add frontend landing page and stale process monitor UI on /scraper-tools - Move old development scripts to backend/archive/ - Update frontend build with new features 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
56 lines
2.0 KiB
TypeScript
56 lines
2.0 KiB
TypeScript
import { scrapeCategoryPlaywright } from './src/services/scraper-playwright';
|
|
import { getRandomProxy } from './src/utils/proxyManager';
|
|
import { pool } from './src/db/migrate';
|
|
|
|
async function testTemplateIntegration() {
|
|
console.log('🧪 Testing Dutchie template integration\n');
|
|
console.log('This test verifies that:');
|
|
console.log(' 1. The scraper detects Dutchie URLs');
|
|
console.log(' 2. The template is automatically used');
|
|
console.log(' 3. Products are extracted correctly\n');
|
|
|
|
const categoryUrl = 'https://dutchie.com/dispensary/sol-flower-dispensary/products/flower';
|
|
const categoryName = 'Flower';
|
|
const state = 'Arizona';
|
|
|
|
console.log(`📂 Category: ${categoryName}`);
|
|
console.log(`🔗 URL: ${categoryUrl}`);
|
|
console.log(`📍 State: ${state}\n`);
|
|
|
|
try {
|
|
// Get proxy
|
|
const proxy = await getRandomProxy();
|
|
if (proxy) {
|
|
console.log(`🔍 Using proxy: ${proxy.server}\n`);
|
|
} else {
|
|
console.log('⚠️ No proxy available, proceeding without proxy\n');
|
|
}
|
|
|
|
console.log('🎭 Starting scrape with template integration...\n');
|
|
|
|
const products = await scrapeCategoryPlaywright(categoryUrl, categoryName, state, proxy || undefined);
|
|
|
|
console.log(`\n✅ Scrape completed!`);
|
|
console.log(`📦 Found ${products.length} products\n`);
|
|
|
|
if (products.length > 0) {
|
|
console.log('🎉 SUCCESS! Template integration working!\n');
|
|
console.log('First 10 products:');
|
|
products.slice(0, 10).forEach((product, i) => {
|
|
console.log(`\n${i + 1}. ${product.name}`);
|
|
if (product.brand) console.log(` Brand: ${product.brand}`);
|
|
if (product.price) console.log(` Price: $${product.price}`);
|
|
console.log(` URL: ${product.product_url}`);
|
|
});
|
|
} else {
|
|
console.log('⚠️ No products found - integration may need debugging');
|
|
}
|
|
} catch (error) {
|
|
console.error('❌ Error:', error);
|
|
} finally {
|
|
await pool.end();
|
|
}
|
|
}
|
|
|
|
testTemplateIntegration();
|