一、抓取名人名言
名人名言的地址：http://quotes.toscrape.com/
1.查看网页代码，获取待抓取元素的css：
（1）打开页面，在谷歌浏览器中使用开发者工具查看网页源代码
网页源代码
（2）在本次抓取过程中主要采集sayings(名言)、author(作者)、tags(标签)，接下来查看获取对应的css：
①quotes_list对应的css
"div.quote"
②sayings(名言)对应的css
"span.text::text"
③author(作者)对应的css
"span small::text"
④tags(标签)对应的css
"div.tags a.tag::text"
⑤next_page(下一页)对应的css
"li.next a::attr(href)"
2.新建爬虫项目，开始抓取数据
（1）新建项目
scrapy startproject quotesSpider
(2)在/quotesSpider/spiders文件下新建quotes_spider.py文件，编写code。
直接编写quotes_spider.py文件：
from scrapy.spiders import CrawlSpider
from scrapy.selector import Selector
from quotesSpider.items import *
class QuotesSpider(CrawlSpider):
name = "quotes"
start_urls = ["http://quotes.toscrape.com/"]
def parse(self, response):
quotes_list = response.css("div.quote").extract()
for quotes in quotes_list:
quotes = Selector(text=quotes)
yield {
'text': quotes.css('span.text::text').extract_first(),
'author': quotes.css('span small::text').extract_first(),
'tags': quotes.css('div.tags a.tag::text').extract(),
}
next_page = response.css("li.next a::attr(href)").extract_first()
if next_page: #若存在下一页则继续
yield scrapy.Request(response.urljoin(url=next_page), callback=self.parse)
或者：
先编辑好items.py：
import scrapy
class QuotesSpiderItem(scrapy.Item):
sayings = scrapy.Field()
author = scrapy.Field()
tags = scrapy.Field()
pass
接着编写quotes_spider.py：
from scrapy.spiders import CrawlSpider
from quotesSpider.items import *
class QuotesSpider(CrawlSpider):
name = "quotes"
allowed_domains = ["quotes.toscrape.com"]
start_urls = ['http://quotes.toscrape.com/']
def parse(self, response):
quotes = response.css('div.quote')
for quote in quotes:
item = QuotesSpiderItem()
item['sayings'] = quote.css('span.text::text').extract_first()
item['author'] = quote.css('span small::text').extract_first()
item['tags'] = quote.css('div.tags a.tag::text').extract()
yield item
next_page = response.css("li.next a::attr(href)").extract_first()
if next_page: # 若存在下一页则继续
yield scrapy.Request(response.urljoin(url=next_page), callback=self.parse)
(3)在命令行中，进入quotesSpider文件夹,输入下面的内容，开始运行爬虫
scrapy crawl quotes -o data.json #将爬取到的内容以json 格式存放在data.json文件夹下
(4)爬取结果如下（展示部分）
[
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["change", "deep-thoughts", "thinking", "world"]},
{"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d", "author": "J.K. Rowling", "tags": ["abilities", "choices"]},
{"text": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "author": "Albert Einstein", "tags": ["inspirational", "life", "live", "miracle", "miracles"]},
{"text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d", "author": "Jane Austen", "tags": ["aliteracy", "books", "classic", "humor"]},
{"text": "\u201cImperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.\u201d", "author": "Marilyn Monroe", "tags": ["be-yourself", "inspirational"]},
{"text": "\u201cTry not to become a man of success. Rather become a man of value.\u201d", "author": "Albert Einstein", "tags": ["adulthood", "success", "value"]},
{"text": "\u201cIt is better to be hated for what you are than to be loved for what you are not.\u201d", "author": "Andr\u00e9 Gide", "tags": ["life", "love"]},
{"text": "\u201cI have not failed. I've just found 10,000 ways that won't work.\u201d", "author": "Thomas A. Edison", "tags": ["edison", "failure", "inspirational", "paraphrased"]},
{"text": "\u201cA woman is like a tea bag; you never know how strong it is until it's in hot water.\u201d", "author": "Eleanor Roosevelt", "tags": ["misattributed-eleanor-roosevelt"]},
{"text": "\u201cA day without sunshine is like, you know, night.\u201d", "author": "Steve Martin", "tags": ["humor", "obvious", "simile"]},
{"text": "\u201cThis life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.\u201d", "author": "Marilyn Monroe", "tags": ["friends", "heartbreak", "inspirational", "life", "love", "sisters"]},
{"text": "\u201cIt takes a great deal of bravery to stand up to our enemies, but just as much to stand up to our friends.\u201d", "author": "J.K. Rowling", "tags": ["courage", "friends"]},
{"text": "\u201cIf you can't explain it to a six year old, you don't understand it yourself.\u201d", "author": "Albert Einstein", "tags": ["simplicity", "understand"]},
{"text": "\u201cYou may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not perfect\u2014you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being human and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a part of her that she knows you can break\u2014her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when she makes you happy, let her know when she makes you mad, and miss her when she's not there.\u201d", "author": "Bob Marley", "tags": ["love"]},
{"text": "\u201cI like nonsense, it wakes up the brain cells. Fantasy is a necessary ingredient in living.\u201d", "author": "Dr. Seuss", "tags": ["fantasy"]},
{"text": "\u201cI may not have gone where I intended to go, but I think I have ended up where I needed to be.\u201d", "author": "Douglas Adams", "tags": ["life", "navigation"]}
]
由下图可以看到爬取回来一共有100条结果:
爬取结果条数展示
网友评论