Step-by-Step: Scrape Any Website with n8n’s Tools

To scrape a website with n8n, you’ll build a visual workflow primarily using two core nodes: the HTTP Request node to fetch the website’s raw HTML code and the HTML Extract node to parse that code and pull out specific pieces of information. By targeting data using CSS selectors, you can extract anything from product prices and article titles to contact information. Once extracted, you can use other n8n nodes to sort, format, and send this structured data to applications like Google Sheets, a database, or even an email client, creating a fully automated data collection pipeline.

Ever looked at a website and thought, “I wish I could just grab all that data without spending hours copying and pasting?” Whether you’re tracking competitor prices, gathering sales leads, or collecting data for a research project, web scraping is your secret weapon. And let’s be honest, while you could write a complex Python script, who has the time for that? This is where the magic of n8n comes in. I’m here to show you how you can scrape a website with n8n, turning a tedious manual task into an automated, set-it-and-forget-it workflow.

Before You Scrape: The Golden Rules

Before we dive into the fun stuff, we need to have a quick chat about the ethics of web scraping. Think of it like being a guest in someone’s house. You wouldn’t just walk in and start rummaging through their drawers, right? The same principle applies here.

1. Check the Terms of Service (ToS): Most websites have a ToS page that outlines what you can and can’t do. Some explicitly forbid scraping. Always check this first, as it’s the legally binding document.
2. Respect robots.txt: This is a file found on most websites (e.g., www.example.com/robots.txt) that gives guidelines to bots. While not legally binding, it’s polite to follow its instructions. It’s the website owner’s way of saying, “Please don’t look in these areas.”
3. Be a Good Web Citizen: Don’t hammer a website with hundreds of requests per second. This can slow down or even crash their server. Use a Wait node in n8n to space out your requests. Be gentle!

Being responsible not only keeps you out of trouble but also ensures that valuable data sources remain accessible for everyone.

The Core Tools: Your n8n Scraping Toolkit

Building a scraper in n8n revolves around a few key nodes. Understanding these building blocks is the first step to mastering web scraping.

The HTTP Request Node: The First Knock on the Door

This is where it all begins. The HTTP Request node is your way of asking a website’s server for its content. You simply provide the URL, and the node fetches the entire HTML source code of that page, which is the raw material for our scraping operation.

The HTML Extract Node: The Treasure Map

Once you have the HTML, the HTML Extract node is how you find your treasure. It uses CSS Selectors to pinpoint the exact data you want. What’s a CSS Selector? It’s simply a pattern that identifies specific elements on a page. For example, a selector like h2.product-title would tell n8n to find all the <h2> headings that have the class product-title. It’s like having a treasure map where ‘X’ marks the spot for your data.

A Practical Example: Let’s Scrape an Online Bookstore

Talk is cheap. Let’s build a real workflow! We’ll use Books to Scrape, a website specifically designed for this purpose. Our goal: to extract the title and price of every book on the front page and save it to a spreadsheet.

Step 1: Fetching the Page

Start your workflow with an HTTP Request node.

• Request Method: Set to GET.
• URL: Enter http://books.toscrape.com.
• Response Format: Set to String.

Execute the node. You should see a giant block of HTML code in the output. That’s our page!

Step 2: Extracting Each Book’s Information

Now, add an HTML Extract node. We first need to isolate each book’s container.

1. Go to the website, right-click on a book, and select “Inspect.”
2. You’ll see that each book is contained within an <li> element inside a <article class="product_pod">. A good CSS selector to grab all books would be article.product_pod.
3. In the HTML Extract node:
   • Source Data: Leave as JSON.
   • JSON Property: Set to data (or whatever the output of the previous node is called).
   • Key: books
   • CSS Selector: article.product_pod
   • Return Value: HTML
   • Enable Return Array.

Execute this. You’ll now have an array of HTML snippets, one for each book.

Step 3: Getting the Title and Price

We’re not done yet. We need to process each of those snippets. First, add a Split Out node to turn our array of 20 books into 20 separate items. Then, add another HTML Extract node to pull the specifics from each book snippet.

• JSON Property: books (the field from our first HTML extract)
• Click Add Extraction Value to create two extractors:
  1. For the Title:
     • Key: title
     • CSS Selector: h3 > a (This targets the <a> tag inside the <h3>)
     • Return Value: Text
  2. For the Price:
     • Key: price
     • CSS Selector: p.price_color
     • Return Value: Text

Run the workflow. Voila! You now have a structured list of items, each with a title and a price. How cool is that?

Step 4: Storing Your Data

From here, the world is your oyster. You can connect a Google Sheets node to append this data into a new row automatically. Or use the Convert to File node to create a CSV, and then the Gmail node to email it to yourself. This is the power of n8n—it’s not just about getting the data; it’s about what you do with it.

Level Up: Tackling Advanced Scraping Challenges

Of course, not all websites are as straightforward as our example.

• Pagination: What about sites with multiple pages? You’ll need to create a loop. Scrape the data on the first page, then use another HTML Extract node to find the link to the “Next” page. You can then feed that URL back into your HTTP Request node to continue the process until there are no more “Next” buttons.

• Dynamic Content: Some websites load data with JavaScript after the page initially loads. A simple HTTP request won’t see this data. For these tougher cases, you might need to use the Execute Command node to run a headless browser script (like Puppeteer) or integrate with a third-party scraping API that can handle JavaScript rendering.

• Staying Under the Radar: If you’re making many requests, use the Wait node to add a delay of a few seconds between each call. For very large-scale scraping, you might explore using a proxy service to rotate your IP address, though for most personal or small business projects, just being slow and respectful is enough.

So, Why Scrape with n8n?

Let’s be real, a developer could code all this. But using n8n to scrape websites offers some incredible advantages, especially for those of us who value speed and simplicity.

Conclusion: Start Building Your Scraper Today

Web scraping can seem intimidating, but with n8n, it becomes a surprisingly accessible and powerful tool. You’ve seen how to go from a blank canvas to a functioning workflow that extracts and stores data from the web—all without writing a single line of complex code. So go ahead, find a website, and start building. You’ll be amazed at what you can automate.