第一次爬虫实践

• 简介： 这次爬虫爬取的是工业和信息化部网站上公示的车辆信息，爬取js动态生成的链接。

首页，总的有123页，打开其他页的时候地址栏里的地址不变，也就是说新的页面是用js动态生成的。

1.PNG

它对应的每一个子页打开都是一样的，只要解决一个子页面，再获取所有子页的url就可以解决了。

image.png

• 在主页上右键->检查,选择NetWork，然后在左边页面中点第二页，看它发送接收的信息，可以看到在标4的地方，有一个paramJson{"pageNo"：2,...........},里面的pageNo就是页数 3.PNG
  

  鼠标往上滑，在Genernal下面Request URL就是当前页面的请求地址，仔细观察里面包含了paramJson pageNo,那也就是说页面的信息页包含在里面了。

  
  image.png
  回到paramJson{"pageNo"：2,...........}的地方 4.PNG
  点击view URL encoded，变成 5.PNG
  

  它就是general URL里面的一部分，也就是说general URL就是由下面的这些项目编码后生成的，对应一下它字符编码方式就是

 {   ==>   %7B
"   ==>   %22
英文字母  ==>  英文字母
,   ==>   %3A
2   ==>   2

想要获得全部页面就只需要一个循环，把pageNo:2换成其他的页码即可。

• 1.用request_html循环获取所有子页页面上的URL链接,在每一页上用r.html.links得到url，加上网页的头就可以得到所有车辆对应的url了。

from requests_html import HTMLSession
import csv
#import requests
from lxml import etree

session = HTMLSession()

url_list=[]#all url list

pageNo = 123# the number of all the pages
# url = ".....%7B%22pageNo%22%3A{}%2C%22pageSize......".format(num)
for num in range(1,pageNo):
    url = "https://www.miit.gov.cn/api-gateway/jpaas-publish-server/front/page/build/unit?webId=b3eba6883f9240e2b51025f690afbae8&pageId=02621da61f2548b2ab3ca69e348d72bc&parseType=bulidstatic&pageType=column&tagId=%E4%BF%A1%E6%81%AF%E5%88%97%E8%A1%A8&tplSetId=9a9a7b87a4444169bdef99ff1f84e1aa&unitUrl=%2Fapi-gateway%2Fjpaas-publish-server%2Ffront%2Fpage%2Fbuild%2Funit&paramJson=%7B%22pageNo%22%3A{}%2C%22pageSize%22%3A20%2C%22loadEnabled%22%3Atrue%2C%22search%22%3A%22%7B%5C%22title%5C%22%3A%5C%22%5C%22%2C%5C%22PICI%5C%22%3A%5C%22338%5C%22%2C%5C%22QYMC%5C%22%3A%5C%22%5C%22%2C%5C%22CPSB%5C%22%3A%5C%22%5C%22%2C%5C%22CPMC%5C%22%3A%5C%22%5C%22%2C%5C%22CPXH%5C%22%3A%5C%22%5C%22%7D%22%7D".format(num)
    r = session.get(url)
    all_links =  r.html.links
    for item in all_links:
        url_list.append("https://www.miit.gov.cn/"+item[2:-2])

[out]:all_links
{'\\"/datainfo/cpgg/xcpgs/art/2020/art_0665c69d3bc34a66ad7b30bfc98922cb.html\\"',
  '\\"/datainfo/cpgg/xcpgs/art/2020/art_f45286e89b4f425cbd1db8597aa43cad.html\\"',
 '\\"/datainfo/cpgg/xcpgs/art/2020/art_fcf6daa662134326a31294769adc94a1.html\\"'}

[out]:url_list[:3]
['https://www.miit.gov.cn//datainfo/cpgg/xcpgs/art/2020/art_882da80d6b1248548cb077c5f91d7237.html',
 'https://www.miit.gov.cn//datainfo/cpgg/xcpgs/art/2020/art_f38cd9d91e6a431bb8c07677c3f3db42.html',
 'https://www.miit.gov.cn//datainfo/cpgg/xcpgs/art/2020/art_f8aa6c4543564f7890c7506c7a55ac53.html']

• 2. 现在所有子页的所有车辆信息都在url_list 中了，下面就是获取每车的页面上的信息了，以外形尺寸为例，右键-检查，选择外形尺寸所在的位置，右键-Copy-Copy full Xpath,然后用etree获取内容

full Xpath:  /html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[1]/td

image.png
核心流程到这里也就完成了，下面的内容就是python的一些处理了，想深入了解lxml.etree的点我

• 3. 保存下来的表格

     
     image.png

完整代码

from requests_html import HTMLSession
import csv
#import requests
from lxml import etree

# get all url
session = HTMLSession()
url_list=[]#all url list
pageNo = 123# the number of all the pages

for num in range(1,pageNo):
    url = "https://www.miit.gov.cn/api-gateway/jpaas-publish-server/front/page/build/unit?webId=b3eba6883f9240e2b51025f690afbae8&pageId=02621da61f2548b2ab3ca69e348d72bc&parseType=bulidstatic&pageType=column&tagId=%E4%BF%A1%E6%81%AF%E5%88%97%E8%A1%A8&tplSetId=9a9a7b87a4444169bdef99ff1f84e1aa&unitUrl=%2Fapi-gateway%2Fjpaas-publish-server%2Ffront%2Fpage%2Fbuild%2Funit&paramJson=%7B%22pageNo%22%3A{}%2C%22pageSize%22%3A20%2C%22loadEnabled%22%3Atrue%2C%22search%22%3A%22%7B%5C%22title%5C%22%3A%5C%22%5C%22%2C%5C%22PICI%5C%22%3A%5C%22338%5C%22%2C%5C%22QYMC%5C%22%3A%5C%22%5C%22%2C%5C%22CPSB%5C%22%3A%5C%22%5C%22%2C%5C%22CPMC%5C%22%3A%5C%22%5C%22%2C%5C%22CPXH%5C%22%3A%5C%22%5C%22%7D%22%7D".format(num)
    r = session.get(url)
    all_links =  r.html.links
    for item in all_links:
        url_list.append("https://www.miit.gov.cn/"+item[2:-2])


names = ["产品商标","产品型号","产品名称","企业名称","外形尺寸(mm)","货箱栏板内尺寸(mm)",
         "排放依据标准","燃料种类","最高车速(km/h)","整备质量(kg)","准拖挂车总质量(kg)",
         "轴距(mm)","其他","发动机型号","发动机企业","排量(ml)","功率(kw)","油耗(L/100km)",
         "底盘型号","总质量","额定载质量","轴数","轮胎数","轮胎规格","板簧片数",
         "半挂鞍座最大允许承载质量","驾驶室准乘人数","接近角/离去角","防抱死制动系统",
         "车辆识别代号(VIN)","前悬/后悬","底盘生产企业"]
b=["最高车速","总质量","额定载质量","轴数","轮胎数","轮胎规格","板簧片数","半挂鞍座最大允许承载质量",
   "驾驶室准乘人数","接近角/离去角","防抱死制动系统","车辆识别代号(VIN)","前悬/后悬","底盘生产企业"]

#item xPath
item_list=[
    "/html/body/div[1]/div/table[1]/tbody/tr[1]/td[0]/text()",#产品商标
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[1]/tbody/tr[1]/td[2]/text()",#产品型号
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[1]/tbody/tr[1]/td[3]/text()",#产品名称"
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[1]/tbody/tr[2]/td[1]/text()",#企业名称"
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[1]/td/text()",#"外形尺寸(mm)"
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[2]/td/text()",#"货箱栏板内尺寸(mm)"
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[3]/td[1]/text()",#"排放依据标准
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[3]/td[2]/text()",#"燃料种类"
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[4]/td[1]/text()",#"最高车速(km/h)
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[6]/td[2]/text()",#"整备质量(kg)
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[7]/td[2]/text()",#"准拖挂车总质量(kg)
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[8]/td[1]/text()",#"轴距(mm)
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[16]/td/text()",#"其他
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[4]/tbody/tr[2]/td[1]/text()",#"发动机型号
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[4]/tbody/tr[2]/td[2]/text()",#"发动机企业
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[4]/tbody/tr[2]/td[3]/text()",#"排量(ml)
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[4]/tbody/tr[2]/td[4]/text()",#"功率(kw)
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[4]/tbody/tr[2]/td[5]/text()",#"油耗(L/100km)"
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[3]/tbody/tr[2]/td[3]/text()",#底盘型号"
    
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[4]/td[2]/text()",#总质量、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[5]/td[2]/text()",#额定载质量、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[7]/td[1]/text()",#轴数、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[10]/td[1]/text()",#轮胎数、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[8]/td[2]/text()",#轮胎规格、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[9]/td[1]/text()",#板簧片数、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[9]/td[2]/text()",#半挂鞍座最大允许承载质量、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[10]/td[2]/text()",#驾驶室准乘人数、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[12]/td[2]/text()",#接近角/离去角、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[14]/td[2]/text()",#防抱死制动系统、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[15]/td[1]/text()",#车辆识别代号（VIN）、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[2]/tbody/tr[15]/td[2]/text()",#前悬/后悬、
    "/html/body/div[1]/div[3]/div[2]/div/div[2]/table[3]/tbody/tr[2]/td[4]/text()"#底盘生产企业
    
]

result = []#all result
rawdata=[]#a raw data
url_num=1

for item in url_list:
#     if url_num>100:
#         break
    print("current url:",item)
    print ('times:{}/{}'.format(url_num,len(url_list)))
    response_new = session.get(item)
    html_new = etree.HTML(response_new.content.decode('utf-8'))
    for i in range(len(item_list)):
        tag_str = html_new.xpath(item_list[i])
        #print(tag_str)
        if len(tag_str)!=0:
            rawdata.append(tag_str[0])
        else:
            rawdata.append(" ")
    result.append(rawdata)
    url_num += 1
    rawdata=[]#a raw data

#save data
with open("./mydata.csv","a+") as csvfile: 
    writer = csv.writer(csvfile)

    #先写入columns_name
    writer.writerow(names)
    #写入多行用writerows
    writer.writerows(result)