B&H Photo Video uses aggressive Cloudflare protection that blocks
headless browsers even with stealth plugins. Removing the site-specific
scraper for now. The Puppeteer fallback remains in place for other
sites with less aggressive protection.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The headless browser was being detected by Cloudflare and stuck on the
"Just a moment..." challenge page. Added puppeteer-extra with the stealth
plugin which patches browser fingerprinting to avoid bot detection. Also
added logic to wait for Cloudflare challenges to complete.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When HTTP requests are blocked with 403 (e.g., B&H Photo's Cloudflare
protection), the scraper now automatically retries using a headless
Chrome browser via Puppeteer. Also updated Dockerfile to include
Chromium dependencies for container deployment.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Parse JSON-LD structured data for price, name, image, availability
- Add fallback HTML selectors using data-selenium attributes
- Detect stock status from add-to-cart and notify buttons
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Parse Walmart's __NEXT_DATA__ JSON for accurate product data
- Extract price, name, image, and availability from embedded JSON
- Add fallback HTML selectors if JSON parsing fails
- Make stock status detection more conservative
- Avoid false "out of stock" from unrelated page text
- Only mark out of stock when explicitly indicated
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Features:
- Target price alerts: Set a specific price target and get notified when reached
- Historical low indicator: Badge showing when current price is at/near all-time low
- Bulk actions: Select multiple products to delete at once
- Dashboard summary: Shows total products, items at lowest price, at target, biggest drops
Backend changes:
- Add target_price column to products table
- Add target_price notification type with Telegram/Discord support
- Include min_price in product queries for historical low detection
- Update scheduler to check target price conditions
Frontend changes:
- Add target price input to ProductDetail notification settings
- Show target price badge on product cards
- Add "Lowest Price" and "Near Low" badges to product cards
- Add bulk selection mode with checkboxes
- Add dashboard summary cards at top of product list
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ability to create new users from admin panel
- Add role dropdown (User/Admin) for each user
- Replace toggle buttons with select dropdown for role management
- Admin users can access the Admin section in settings
- Regular users see only Profile and Notifications sections
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add sidebar navigation to settings page
- Add profile section for name management and password change
- Add admin section for user management and registration toggle
- Add profile API endpoints (GET/PUT /profile, PUT /profile/password)
- Add admin API endpoints (users CRUD, system settings)
- Add system_settings table for registration control
- Add name and is_admin columns to users table
- First registered user automatically becomes admin
- Check registration status on register/login page
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add multiple price selectors for robustness
- Combine dollar/cents from price-current element
- Add JSON-LD fallback for price extraction
- Add explicit stock status detection for Newegg
- Prevents false out-of-stock detection from generic detector
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add next_check_at column to track when each product should be checked
- New products get random initial delay (0 to refresh_interval) to spread them out
- Each check adds ±5 minute jitter so products naturally drift apart over time
- Randomize delay between requests (2-5 seconds instead of fixed 2s)
This prevents all products from being checked at the same time,
reducing the risk of being rate-limited or blocked by retailers.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add refresh button to product list items with spinning animation
- Add editable refresh interval dropdown on product detail page
- Add user profile dropdown with settings link in navbar
- Create Settings page for Telegram and Discord configuration
- Add per-product notification options (price drop threshold, back in stock)
- Integrate notifications into scheduler for automatic alerts
- Add notification service supporting Telegram Bot API and Discord webhooks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add stock_status column to products table (in_stock/out_of_stock/unknown)
- Detect out-of-stock status on Amazon by checking:
- #availability text for "currently unavailable"
- #outOfStock element presence
- Missing "Add to Cart" button
- Add generic stock status detection for other sites
- Allow adding out-of-stock products (they just won't have a price)
- Update background scheduler to track stock status changes
- Display stock status badge in product list and detail pages
- Dim out-of-stock products in the dashboard
- Show "Currently Unavailable" badge instead of price when out of stock
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add detection for coupon/savings containers and skip prices within them
- Check parent elements for coupon-related IDs, classes, and text
- Add minimum price threshold of $2 (coupons are typically $1-5)
- Add fallback to parse Amazon's whole/fraction price format directly
- Increase findMostLikelyPrice threshold from $0.99 to $5
This fixes the issue where $1 coupon savings were being scraped
instead of the actual $25.99 product price.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add sparkline component for 7-day price history visualization
- Convert product cards to horizontal list items
- Add search functionality to filter products by name/URL
- Backend returns sparkline data and 7-day price change with products
- Show price trend indicator (green for drops, red for increases)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix cheerio import to use named exports
- Add proper interfaces for JSON-LD data
- Fix type annotations for CheerioAPI
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
npm ci requires package-lock.json which wasn't generated.
Using npm install instead for builds.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
