Python爬虫笔记（9）：多进程爬取瓜子二手车信息，简单的ma

本次利用pyquery和multiprocessing多进程爬取了瓜子二手车1万多条数据，存入MySQL数据库，并做简单的matplotlib绘图分析。
代码如下：

import requests
from pyquery import PyQuery as pq
from multiprocessing import Pool
import re
import pymysql

db = pymysql.connect('localhost', 'root', '8802667', 'guazi', 3306)
curser = db.cursor()
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36',
    'Cookie': 'uuid=c6ae8249-8eda-4649-e64f-f203aa1a6002; ganji_uuid=8209532471253319517082; lg=1; antipas=9d40v4V7G57458195967v4968; clueSourceCode=%2A%2300; sessionid=4440bacc-c05b-4fa4-9108-d0dfa1334a48; cainfo=%7B%22ca_s%22%3A%22sem_baiduss%22%2C%22ca_n%22%3A%22bdpc_sye%22%2C%22ca_i%22%3A%22-%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22%25E5%25A4%25A9%25E6%25B4%25A5%25E4%25BA%258C%25E6%2589%258B%25E8%25BD%25A6%22%2C%22ca_content%22%3A%22-%22%2C%22ca_campaign%22%3A%22-%22%2C%22ca_kw%22%3A%22%25e4%25ba%258c%25e6%2589%258b%25e8%25bd%25a6%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%2277188931804%22%2C%22scode%22%3A%2210103188612%22%2C%22ca_transid%22%3Anull%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22ca_b%22%3A%22-%22%2C%22ca_a%22%3A%22-%22%2C%22display_finance_flag%22%3A%22-%22%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%22c6ae8249-8eda-4649-e64f-f203aa1a6002%22%2C%22sessionid%22%3A%224440bacc-c05b-4fa4-9108-d0dfa1334a48%22%7D; close_finance_popup=2018-12-31; cityDomain=www; preTime=%7B%22last%22%3A1546220511%2C%22this%22%3A1545915036%2C%22pre%22%3A1545915036%7D'
}
base_url = 'https://www.guazi.com'


def get_url(url):
    r = requests.get(url, headers=headers)
    r.encoding = r.apparent_encoding
    doc = pq(r.text)
    items = doc('.carlist li').items()
    for item in items:
        href = item.find('a').attr('href')
        href = base_url + href
        get_detail(href)


def get_detail(href):
    r = requests.get(href, headers=headers)
    r.encoding = r.apparent_encoding
    doc = pq(r.text)
    title = doc('.titlebox').remove('span').text()
    brand = title.split(' ')[0]
    series = title.split(' ')[1]
    match = re.search('[A-Za-z0-9]', brand)
    if match:
        match = match.group()
        series = match + brand.split(match)[1]
        brand = brand.split(match)[0]
    types = title.split(' ')[-1]
    car_license = doc('.one div').text()
    mileage = doc('.two div').text().split('万')[0]
    site = doc('.three div').text()
    standard = doc('.four div').remove('em').text()
    gearbox = doc('.five div').text()
    cc = doc('.six div').text()
    price = doc('.pricestype').remove('.f14').text().split('¥')[-1]
    print(brand, series, types, car_license, mileage, site, standard, gearbox, cc, price)
    sql = """
    insert into ershouche(brand,series,types,car_license,mileage,site,standard,gearbox,cc,price)
    values(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
    """
    try:
        curser.execute(sql, (brand, series, types, car_license, mileage, site, standard, gearbox, cc, price))
        db.commit()
    except:
        db.rollback()


if __name__ == '__main__':
    urls = ['https://www.guazi.com/www/buy/o{}c-1/#bread'.format(str(i)) for i in range(1, 11642)]
    pool = Pool(processes=4)
    pool.map(get_url, urls)
    curser.close()
    db.close()

需要补充的是，爬取瓜子网，必须要加cookies，否则会抓不到数据。
下面是简单的绘图分析：

1 手动自动变速箱占比

Figure_1.png

不知道大家买车时有没有朋友总是强烈推荐自动挡车型：‘市区开车不累’，‘简单容易上手’，‘手动挡快淘汰了’等等。从图中也可以看出，自动档变速箱占据绝对的数量优势，但是手动变速箱并没有我想象中的少，看来手动变速箱还是有一定的受众人群的。

2 各国标占比

Figure_2.png

可以看出，国四车型保有量最高，也算意料之中，国五车型现在基本上还算是新车，国三面临淘汰，国四自然是二手车市场的主力军了。

3 二手车保有量前十品牌

Figure_3.png

大众神车果然不负众望，以压倒性优势夺得第一。前十之中自主品牌只有哈弗一支独苗。

4 各价位二手车保有量

Figure_4.png

10万以下的二手车，数量异常庞大。同时，这也是大多数买二手车的人考虑最多的价格区间。毕竟，花个几十万买辆二手车的人还是稀有的吧。