When enabled, AI verifies every scraped price to ensure accuracy.
This catches issues like scraped "savings" amounts instead of actual prices.
- Add ai_verification_enabled column to users table
- Create verification prompt and functions for Anthropic, OpenAI, Ollama
- Integrate verification step into scraper after traditional scraping
- Add verification toggle to Settings page (separate from AI extraction)
- AI verification is independent of AI extraction fallback
Flow: Traditional scraping -> AI verification (if enabled) -> AI extraction fallback (if no price found)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add helper to detect savings/combo/bundle containers
- Prioritize JSON-LD data as primary price source (most reliable)
- Skip price elements inside savings containers
- Add minimum price threshold to filter out discount amounts
Fixes issue where $189.99 bundle savings was extracted instead of
actual $675.59 product price for items like AMD Ryzen 9 9950X3D.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ntfy_topic and ntfy_enabled columns to database
- Add sendNtfyNotification function with emoji tags
- Add /settings/notifications/test/ntfy endpoint
- Add ntfy section in Settings UI with topic input
- Show ntfy badge in ProductDetail notification status
ntfy.sh is a free, simple notification service - no account needed.
Users just pick a topic name and subscribe in the ntfy app.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Backend:
- Update /settings/notifications to return actual tokens/keys
- Update /settings/ai to return actual API keys
Frontend:
- Update NotificationSettings and AISettings types
- Populate form fields with actual saved values on load
- Eye toggle now reveals actual stored values
- Always show toggle button for consistent UX
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add database migration for ollama_base_url and ollama_model columns
- Update backend models and queries for Ollama settings
- Add extractWithOllama function using Ollama's /api/chat endpoint
- Add /api/settings/ai/test-ollama endpoint to test connection and list models
- Update frontend Settings page with Ollama configuration UI
- Support model selection from dropdown after testing connection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create stock_status_history table to track status changes over time
- Add stockStatusHistoryQueries with getByProductId, recordChange, getStats
- Update scheduler to record status changes
- Update product creation and manual refresh to record initial/changed status
- Add GET /products/:id/stock-history API endpoint
- Create StockTimeline component with:
- Visual timeline bar showing in-stock (green) vs out-of-stock (red)
- Availability percentage
- Outage count and duration stats
- Integrate timeline into ProductDetail page
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add CHF to currency detection patterns in priceParser
- Add getCurrencySymbol helper in notifications service
- Update all frontend price formatting to support CHF
- Swiss francs display as "CHF 29.99" format
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add telegram_enabled, discord_enabled, pushover_enabled columns to database
- Update notification service to check enabled status before sending
- Add toggle switches in Settings UI for each configured channel
- Update ProductDetail to only show badges for enabled channels
- Channels default to enabled so existing users keep notifications
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add pushover_user_key and pushover_app_token columns to users table
- Add sendPushoverNotification function to notifications service
- Add Pushover test endpoint
- Add Pushover settings UI in Settings page
- User provides both User Key and App Token (self-hosted friendly)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add name field to auth login/register responses
- Update User interface in AuthContext to include name
- Show name (or email as fallback) in navbar dropdown
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Extract JSON-LD scripts BEFORE removing script tags
- Add logging for prepared HTML, AI responses, and parsed data
- Include more detailed error messages in test endpoint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add AI extraction service supporting Anthropic (Claude) and OpenAI
- Add AI settings UI in Settings page with provider selection
- Add database migration for AI settings columns
- Integrate AI fallback into scraper when standard methods fail
- Add API endpoints for AI settings and test extraction
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Now extracts availability/stock status from JSON-LD structured data
(e.g., "https://schema.org/OutOfStock"). This fixes stock detection
for sites like Ubiquiti Store that provide availability in JSON-LD.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Some sites (like Ubiquiti Store) use the priceSpecification nested
format in their JSON-LD structured data instead of direct price
property. Now checks offer.priceSpecification.price as fallback.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
B&H Photo Video uses aggressive Cloudflare protection that blocks
headless browsers even with stealth plugins. Removing the site-specific
scraper for now. The Puppeteer fallback remains in place for other
sites with less aggressive protection.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The headless browser was being detected by Cloudflare and stuck on the
"Just a moment..." challenge page. Added puppeteer-extra with the stealth
plugin which patches browser fingerprinting to avoid bot detection. Also
added logic to wait for Cloudflare challenges to complete.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When HTTP requests are blocked with 403 (e.g., B&H Photo's Cloudflare
protection), the scraper now automatically retries using a headless
Chrome browser via Puppeteer. Also updated Dockerfile to include
Chromium dependencies for container deployment.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Parse JSON-LD structured data for price, name, image, availability
- Add fallback HTML selectors using data-selenium attributes
- Detect stock status from add-to-cart and notify buttons
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Parse Walmart's __NEXT_DATA__ JSON for accurate product data
- Extract price, name, image, and availability from embedded JSON
- Add fallback HTML selectors if JSON parsing fails
- Make stock status detection more conservative
- Avoid false "out of stock" from unrelated page text
- Only mark out of stock when explicitly indicated
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Features:
- Target price alerts: Set a specific price target and get notified when reached
- Historical low indicator: Badge showing when current price is at/near all-time low
- Bulk actions: Select multiple products to delete at once
- Dashboard summary: Shows total products, items at lowest price, at target, biggest drops
Backend changes:
- Add target_price column to products table
- Add target_price notification type with Telegram/Discord support
- Include min_price in product queries for historical low detection
- Update scheduler to check target price conditions
Frontend changes:
- Add target price input to ProductDetail notification settings
- Show target price badge on product cards
- Add "Lowest Price" and "Near Low" badges to product cards
- Add bulk selection mode with checkboxes
- Add dashboard summary cards at top of product list
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ability to create new users from admin panel
- Add role dropdown (User/Admin) for each user
- Replace toggle buttons with select dropdown for role management
- Admin users can access the Admin section in settings
- Regular users see only Profile and Notifications sections
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add sidebar navigation to settings page
- Add profile section for name management and password change
- Add admin section for user management and registration toggle
- Add profile API endpoints (GET/PUT /profile, PUT /profile/password)
- Add admin API endpoints (users CRUD, system settings)
- Add system_settings table for registration control
- Add name and is_admin columns to users table
- First registered user automatically becomes admin
- Check registration status on register/login page
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add multiple price selectors for robustness
- Combine dollar/cents from price-current element
- Add JSON-LD fallback for price extraction
- Add explicit stock status detection for Newegg
- Prevents false out-of-stock detection from generic detector
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add next_check_at column to track when each product should be checked
- New products get random initial delay (0 to refresh_interval) to spread them out
- Each check adds ±5 minute jitter so products naturally drift apart over time
- Randomize delay between requests (2-5 seconds instead of fixed 2s)
This prevents all products from being checked at the same time,
reducing the risk of being rate-limited or blocked by retailers.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add refresh button to product list items with spinning animation
- Add editable refresh interval dropdown on product detail page
- Add user profile dropdown with settings link in navbar
- Create Settings page for Telegram and Discord configuration
- Add per-product notification options (price drop threshold, back in stock)
- Integrate notifications into scheduler for automatic alerts
- Add notification service supporting Telegram Bot API and Discord webhooks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add stock_status column to products table (in_stock/out_of_stock/unknown)
- Detect out-of-stock status on Amazon by checking:
- #availability text for "currently unavailable"
- #outOfStock element presence
- Missing "Add to Cart" button
- Add generic stock status detection for other sites
- Allow adding out-of-stock products (they just won't have a price)
- Update background scheduler to track stock status changes
- Display stock status badge in product list and detail pages
- Dim out-of-stock products in the dashboard
- Show "Currently Unavailable" badge instead of price when out of stock
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add detection for coupon/savings containers and skip prices within them
- Check parent elements for coupon-related IDs, classes, and text
- Add minimum price threshold of $2 (coupons are typically $1-5)
- Add fallback to parse Amazon's whole/fraction price format directly
- Increase findMostLikelyPrice threshold from $0.99 to $5
This fixes the issue where $1 coupon savings were being scraped
instead of the actual $25.99 product price.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add sparkline component for 7-day price history visualization
- Convert product cards to horizontal list items
- Add search functionality to filter products by name/URL
- Backend returns sparkline data and 7-day price change with products
- Show price trend indicator (green for drops, red for increases)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix cheerio import to use named exports
- Add proper interfaces for JSON-LD data
- Fix type annotations for CheerioAPI
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
npm ci requires package-lock.json which wasn't generated.
Using npm install instead for builds.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
