Recursively Scraping A Blog With Scrapy

Scrapy is a web crawling and scraping framework
written in python. The framework is really simple to understand and easy
to get started with. If you know little bit of Python, you should be
able to build your own web scraper within few minutes. In this post I’m
going to describe how you can use Scrapy to build recursive blog
crawler. Building a recursive scraper using Scrapy is pretty simple, yet
it’s getting started guide doesn’t help people who are unfamiliar with
the framework to write a recursive scraper.

I’m going to use as the target
blog and techniques I am discussing will work on other sites as well
with simple changes to information extraction queries. If you are new to
Scrapy it’s required to read the Scrapy tutorial first.
I also assume that you have Scrapy installed in your machine.

Let’s create a Scrapy project first using following command:

scrapy startproject scrapy_sample

 Defining the scrapy item

Next step is to define the Item which is the container Scrapy spider
used to store the scraped data. I’m going to extract the blog post link,
post title and text content of the post. So my Item definition will look like

from scrapy.item import Item, Field

class ScrapySampleItem(Item):
    title = Field()
    link = Field()
    content = Field()

 Implementing the spider

Our spider will define initial URL to download content from, how to
follow pagination links and how to extract blog posts in a page and
creating items from the posts.

Your spider class must be a subclass of scrapy.spider.BaseSpider and
you need to define three main mandatory attributes.

Our spider implementation will look like following:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
from scrapy_sample.items import ScrapySampleItem

class ScrapyOrgSpider(BaseSpider):
    name = "scrapy"
    allowed_domains = [""]
    start_urls = [""]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)

        next_page =
        if not not next_page:
            yield Request(next_page[0], self.parse)

        posts ="//div[@class='post']")
        items = []
        for post in posts:
            item = ScrapySampleItem()
            item["title"] ="div[@class='bodytext']/h2/a/text()").extract()
            item["link"] ="div[@class='bodytext']/h2/a/@href").extract()
            item["content"] ="div[@class='bodytext']/p/text()").extract()
        for item in items:
            yield item

First we create HtmlXpathSelector giving the response object.
This will allow you to select elements in response HTML using XPath selectors. Then we extract the link to the next page of the blog using “//div[@class=‘pagination’]/a[@class=‘next_page’]/@href” XPath selector and selector you need to use in your code will depend on the web site you are going to crawl. Once we get the URL of the next page we check whether there are any URLs in the retirned list by selector, because last page will not have a next page link and Scrapy will throw a error when tried to go to empty URL while in the last page of the crawl. Main trick here is we are returning a python generator for the recursive call. You can learn more about reason behind this from this stackoverflow conversation. Last thing we are doing inside our parse method is extracting blog posts in the current page and creating list of Scrapy Items for blog posts.

 Running the scraper

Now you can execute your scraper by running following command while in
the root directory of your Scrapy project.

scrapy crawl scrapy

Scrapy allows you to save the scraped items into a JSON formatted file.
All you have to do is add -o filename.json -t json option to previous
crawl command. This will save the scraped items into a JSON file with
the given name.

You can find more information about Scrapy from here. I strongly recommend you to read the full documentation if you like to dig deeper into Scrapy.

Source code for the sample can be found here.

This post was moved from my old blog.


Now read this

Interesting Resources on Writing

Starting to participate in #The100DayProject by writing every day for 100 days got me into research more about writing. Writing is major part of the life as grad student. But I was far behind my writing and I wanted to improve by writing... Continue →