Opening Problem Statement
Meet Emma, a market analyst at an e-commerce company. Every week, Emma needs to gather up-to-date product details like titles, prices, brands, and promotional offers from multiple competitors’ websites. Previously, Emma manually copied and pasted this information into spreadsheets — a tedious process taking hours and prone to errors. Missed price changes and inconsistent data led to lost opportunities and inaccurate reports.
This is a common but very specific pain point: scraping precise product data from multiple web pages with fluctuating site structures, all while ensuring data accuracy and minimal manual effort. Emma needs a reliable, automated way to extract this info efficiently and store it in a format ready for analysis.
What This Automation Does
This n8n workflow leverages a smart, vision-based AI scraping agent integrated with Google Sheets and ScrapingBee to automate product data extraction from any e-commerce site.
- Reads a list of product page URLs from a Google Sheets document.
- Uses ScrapingBee’s API to capture full-page screenshots for AI vision scraping.
- Employs Google Gemini-1.5-Pro multimodal model to extract detailed product data directly from screenshots, including titles, prices, brands, and promotions.
- Fallbacks to HTML-based scraping using ScrapingBee’s page HTML if the screenshot data is incomplete or unclear.
- Parses the extracted data into structured JSON format suitable for direct insertion into Google Sheets.
- Updates the ‘Results’ sheet with organized product information, ready for Emma’s analysis and reporting.
By automating this, Emma saves hours per week on repetitive tasks, reduces errors, and gains consistent, comprehensive competitor data faster.
Prerequisites ⚙️
- n8n account (cloud or self-hosted) 🔌
- Google Sheets account with service account credentials 🔑
- ScrapingBee API key (for webpage screenshots and HTML retrieval) 🔐
- Google PaLM API credentials for Google Gemini model 🔑
- Basic knowledge of how to use n8n and set credentials 📊
Step-by-Step Guide
Step 1: Set Up the Trigger Node
Start by opening your workflow in n8n. The trigger node here is the Manual Trigger named “When clicking ‘Test workflow’”. This means the workflow runs only when you manually trigger it. You can customize this later to an automatic trigger if desired.
Navigate: Click on the trigger node labeled “When clicking ‘Test workflow’”.
Outcome: You’re ready to test the workflow manually.
Common Mistake: Forgetting to trigger the workflow manually will result in no actions executing.
Step 2: Fetch URLs from Google Sheets
Next, the workflow fetches a list of URLs from a Google Sheet. Configure the Google Sheets – Get list of URLs node.
Navigate: Select the ‘Google Sheets – Get list of URLs’ node.
Enter: Provide the Google Sheet document ID and select the sheet with URLs (usually the first sheet named “List of URLs”). Ensure you use service account authentication.
Visual: You should see the URLs loading as JSON items.
Common Mistake: Incorrect document ID or missing access permission leads to a fetch failure.
Step 3: Prepare URL Field
This Set fields node extracts and assigns the URL from the fetched data to send downstream.
Navigate: Click on the ‘Set fields’ node.
Configure: Assign a string parameter ‘url’ with the value {{$json.url}}.
Outcome: Each item now contains a clean ‘url’ field for the next node.
Common Mistake: Missing field assignment leads to empty URLs passed forward.
Step 4: Capture Full-Page Screenshot with ScrapingBee
The crucial ScrapingBee – Get page screenshot node calls ScrapingBee API to get a full-page screenshot of each URL.
Navigate: Select the HTTP Request node named ‘ScrapingBee – Get page screenshot’.
Configure: Set method to GET, API URL to “https://app.scrapingbee.com/api/v1”, add query params ‘api_key’ with your API key, ‘url’ with {{$json.url}}, and ‘screenshot_full_page’ set to ‘true’. Add appropriate headers like User-Agent.
Outcome: The node receives full-page screenshots to use as visual input for AI.
Common Mistake: Forgetting “screenshot_full_page=true” results in partial images, reducing data accuracy.
Step 5: Vision-Based Scraping AI Agent with Google Gemini
This is the key AI agent node named Vision-based Scraping Agent. It uses screenshots to extract product data using Google Gemini (PaLM) chat model, with fallback on HTML scraping if needed.
Navigate: Click the LangChain Agent node ‘Vision-based Scraping Agent’.
Configuration: The system prompt instructs the model to extract product titles, prices, brands, and promotions from the screenshot. If the data extracted is missing or unclear, it triggers an HTML scraping fallback tool.
Outcome: Produces structured JSON with product info.
Common Mistake: Not providing PaLM credentials or entering wrong prompts may cause extraction failures.
Step 6: HTML Scraping Fallback Workflow
The HTML-Scraping Tool workflow is called if the AI-agent can’t retrieve sufficient data from the screenshot. This sub-workflow fetches raw page HTML and converts it to Markdown before sending it back to the AI agent for detailed parsing.
Navigate: Explore the sub-workflow linked in the ‘HTML-based Scraping Tool’ node.
Overview: It includes a trigger node, ScrapingBee HTML fetch node, and Markdown conversion node.
Common Mistake: Missing to map query parameters can result in empty HTML data sent back.
Step 7: Parse AI Output into Structured JSON
The Structured Output Parser node formats the AI extracted data cleanly in JSON according to a defined schema focusing on e-commerce product fields.
Navigate: Click on the ‘Structured Output Parser’ node.
Visual: JSON outputs like:[{ product_title, product_price, product_brand, promo, promo_percentage }]
Common Mistake: Not customizing the JSON schema to match your target data may cause parsing errors.
Step 8: Split Output Array into Individual Items
The Split Out node breaks the structured JSON array into individual data rows for insertion.
Navigate: Select the ‘Split Out’ node.
Outcome: Each product is handled as a separate item.
Common Mistake: Missing configuring the node’s field to ‘output’ results in no split.
Step 9: Append Parsed Data to Google Sheets Results
The final Google Sheets – Create Rows node appends the extracted data into the ‘Results’ sheet of your Google Sheet document.
Navigate: Open the ‘Google Sheets – Create Rows’ node.
Configure: Map parsed fields such as promo, category (from URL), product_title, brand, price, and promo percentage to the respective columns.
Outcome: Your structured data appears neatly in the sheet for Emma’s analysis.
Common Mistake: Unaligned result sheet columns cause data to appear incorrectly.
Customizations ✏️
- Change the Data Schema: In ‘Structured Output Parser’, modify the JSON schema to add or remove fields like product ratings or stock status.
- Use Different AI Models: Replace the ‘Google Gemini Chat Model’ with other LangChain-compatible LLMs if you want to experiment with different AI capabilities.
- Automate Trigger: Replace the Manual Trigger with a Time Trigger node to run the scraping daily or weekly.
- Custom URL Filters: Add a Filter node after fetching URLs from Google Sheets to process only specific domains or categories.
- Partial Screenshot Capture: Configure the ScrapingBee screenshot to capture only relevant page portions to save API usage and improve AI focus.
Troubleshooting 🔧
Problem: “No data returned from AI Agent”
Cause: Incorrect or incomplete API credentials for Google Gemini model.
Solution: Double-check your PaLM API key setup under credentials, and test the connection in n8n.
Problem: “ScrapingBee API returns error or no screenshot”
Cause: Missing API key or wrong query parameters like missing ‘screenshot_full_page=true’.
Solution: Verify API key validity, ensure parameters are correctly passed in the HTTP Request node.
Problem: “Google Sheets update fails”
Cause: Mismatched columns between structured JSON and Google Sheets ‘Results’ tab.
Solution: Align column headers exactly and refresh permissions on the Google Sheets node.
Pre-Production Checklist ✅
- Confirm API keys for ScrapingBee and Google PaLM are active.
- Test Google Sheets access and sheet names for “List of URLs” and “Results”.
- Run the workflow manually and observe console logs for errors.
- Review AI extraction output for completeness and accuracy.
- Backup your Google Sheets data before launch.
Deployment Guide
To deploy, activate the workflow and trigger manually or via your chosen trigger. Monitor execution logs in n8n to ensure smooth operation. Errors in AI extraction or HTTP requests should be addressed promptly with logs. For scaling, consider batching URLs or scheduling periodic runs.
Conclusion
You’ve just built a vision-based AI web scraper using n8n, Google Gemini, and ScrapingBee integrated with Google Sheets. This automation empowers Emma and others to transform tedious manual data gathering into a seamless, accurate, and repeatable process that saves valuable time and resources.
By leveraging screenshots for AI vision extraction and graceful fallback on HTML scraping, this workflow improves data fidelity and automates complex e-commerce scraping tasks easily.
Next steps? Experiment with custom data schemas, introduce email alerts for scraped results, or expand to social media or other visual content scraping using the same AI agent methodology.
Enjoy your automation journey! ⚙️