本篇内容需要大家对scrapy框架有了解，并完成了入门学习才能继续使用。

创建项目

scrape startproject tutorial

定义Item如下:

import scrapy

class TutorialItem(scrapy.Item):

# define the fields for your item here like:

# name = scrapy.Field()

title = scrapy.Field()

movieInfo = scrapy.Field()

star = scrapy.Field()

quote = scrapy.Field()

制作爬虫（Spider）

代码内容如下：

import scrapy

class Douban(scrapy.Spider):

name = "douban"

start_urls = ['http://movie.douban.com/top250']

def parse(self,response):

print response.body

运行一下看看

scrapy crawl douban

INFO: Closing spider (finished)表明爬虫已经成功运行并且自行关闭了。

创建主函数

创建文件main.py文件，内容如下：

from scrapy import cmdline

cmdline.execute("scrapy crawl douban".split())

DEBUG1：HTTP status code is not handled or not allowed

DEBUG: Ignoring response <403 http://movie.douban.com/top250>: HTTP status code is not handled or not allowed

Answer:被屏蔽了，在settings.py里加上USER_AGENT：

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5'

取数据

OK，获取网页后，接下去就是取数据了。

导入包：

import scrapy

from tutorial.items import TutorialItem

from scrapy.selector import Selector

from scrapy.http import Request

class Douban(scrapy.Spider):

name = "douban250"

start_urls = ['http://movie.douban.com/top250']

url = 'http://movie.douban.com/top250'

def parse(self,response):

item = TutorialItem()

selector = Selector(response)

#print selector

Movies = selector.xpath('//div[@class="info"]')

#print Movies

for eachMoive in Movies:

title = eachMoive.xpath('div[@class="hd"]/a/span/text()').extract()

fullTitle = ''

for each in title:

fullTitle += each

movieInfo = eachMoive.xpath('div[@class="bd"]/p/text()').extract()

star = eachMoive.xpath('div[@class="bd"]/div[@class="star"]/span[@class="rating_num"]/text()').extract()[0]

quote = eachMoive.xpath('div[@class="bd"]/p[@class="quote"]/span/text()').extract()

if quote:

quote = quote[0]

else:

quote = ''

#print fullTitle

#print movieInfo

#print star

#print quote

item['title'] = fullTitle

item['movieInfo'] = ';'.join(movieInfo)

item['star'] = star

item['quote'] = quote

yield item

nextLink = selector.xpath('//span[@class="next"]/link/@href').extract()

if nextLink:

nextLink = nextLink[0]

print nextLink

yield Request(self.url + nextLink, callback=self.parse)

存储数据一

-o 后面是导出文件名，-t 后面是导出类型。

scrapy crawl douban -o items.csv -t csv

然后用numbers程序打开即可看到250个电影排序了。

存储数据二

可以直接在settings.py文件中设置输出的位置和文件类型，如下：

FEED_URI = u'file:///E:/douban/douban.csv'

FEED_FORMAT = 'CSV'