simple crawler with scrapy

Installation (python 3.4)
We need the Scrapy library (v1.3.3) along with PyMongo (v3.4.0) (latest version when this blog created) for storing the data in MongoDB. You need to install MongoDB as well(not covered).

$ pip install Scrapy==1.3.3
$ pip freeze > requirements.txt

$ pip install pymongo==3.4.0
$ pip freeze > requirements.txt

start project

$ scrapy startproject stack

Specify Data
Those familiar with Django will notice that Scrapy Items are declared similar to Django Models, except that Scrapy Items are much simpler as there is no concept of different field types.

In items.py file

#stack/items.py
from scrapy.item import Item, Field

class StackItem(Item):
    title = Field()
    url = Field()

Create the Spider
Create a file called stack_spider.py in the “spiders” directory.
Using Chrome -> inspect to copy XPath of the craped element.

# stack/spider/stack_spider.py file
from scrapy import Spider
from scrapy.selector import Selector

from stack.items import StackItem


class StackSpider(Spider):
    name = "stack"
    allowed_domains = ["stackoverflow.com"]
    start_urls = [
        "http://stackoverflow.com/questions?pagesize=50&sort=newest",
    ]

    def parse(self, response):
        questions = Selector(response).xpath('//div[@class="summary"]/h3')

        for question in questions:
            item = StackItem()
            item['title'] = question.xpath(
                'a[@class="question-hyperlink"]/text()').extract()[0]
            item['url'] = question.xpath(
                'a[@class="question-hyperlink"]/@href').extract()[0]
            yield item

Test

$ scrapy crawl stack -o items.json -t json