Python爬虫实战笔记_1-2

一个小爬虫，爬一个本地的网页。

Screen Shot 2016-06-26 at 10.07.06 PM.png

需要提取的信息包括：
页面中显示的所有商品的标题，价格，点击量，评分以及商品的图片地址。

#!usr/bin/env python
#_*_ coding: utf-8 _*_
#
# filter out specifical info from a local webpage
#
from bs4 import BeautifulSoup

path = '1_2answer_of_homework/index.html'

with open(path, 'r') as webpage:
    content = webpage.read()
    #print content
    soup = BeautifulSoup(content, "lxml")

    images = soup.select('body > div > div > div.col-md-9 > div > div > div > img')
    titles = soup.select('body > div > div > div.col-md-9 > div > div > div > div.caption > h4 > a')
    prices = soup.select('body > div > div > div.col-md-9 > div > div > div > div.caption > h4.pull-right')
    reviews = soup.select('body > div > div > div.col-md-9 > div > div > div > div.ratings > p.pull-right')
    stars = soup.select('body > div > div > div.col-md-9 > div > div > div > div.ratings')
    #print stars
    for (img, title, price, review, star) in zip(images, titles, prices, reviews, stars):
        img_content = img.get('src')
        title_content = title.get_text()
        price_content = price.get_text()
        review_content = review.get_text()
        star_content = len(star.select('p > span.glyphicon-star'))
    #print stars

        data = {
            "title": title_content,
            "review": review_content,
            "image": img_content,
            "price": price_content,
            "rating": star_content
        }
        print(data)

运行结果

{'image': 'img/pic_0000_073a9256d9624c92a05dc680fc28865f.jpg', 'price': '$24.99', 'review': '65 reviews', 'rating': 5, 'title': 'EarPod'}
{'image': 'img/pic_0005_828148335519990171_c234285520ff.jpg', 'price': '$64.99', 'review': '12 reviews', 'rating': 4, 'title': 'New Pocket'}
{'image': 'img/pic_0006_949802399717918904_339a16e02268.jpg', 'price': '$74.99', 'review': '31 reviews', 'rating': 4, 'title': 'New sunglasses'}
{'image': 'img/pic_0008_975641865984412951_ade7a767cfc8.jpg', 'price': '$84.99', 'review': '6 reviews', 'rating': 3, 'title': 'Art Cup'}
{'image': 'img/pic_0001_160243060888837960_1c3bcd26f5fe.jpg', 'price': '$94.99', 'review': '18 reviews', 'rating': 4, 'title': 'iphone gamepad'}
{'image': 'img/pic_0002_556261037783915561_bf22b24b9e4e.jpg', 'price': '$214.5', 'review': '18 reviews', 'rating': 4, 'title': 'Best Bed'}
{'image': 'img/pic_0011_1032030741401174813_4e43d182fce7.jpg', 'price': '$500', 'review': '35 reviews', 'rating': 4, 'title': 'iWatch'}
{'image': 'img/pic_0010_1027323963916688311_09cc2d7648d9.jpg', 'price': '$15.5', 'review': '8 reviews', 'rating': 4, 'title': 'Park tickets'}

总结

• 爬之前要先了解网页的结构
• 熟悉soup.select()如何筛选目标元素
• 学习如何从筛选出的元素中提取目标信息，get_text(), get('class')方法的使用
• 将目标信息保存为一个dict, 方便整理
• 计算星星个数的方法: star_content = len(star.select('p > span.glyphicon-star'))，
  先筛选出目标标签再用len()函数得到个数