美文网首页爬虫
解析网页速度比较(BeautifulSoup、PyQuery、l

解析网页速度比较(BeautifulSoup、PyQuery、l

作者: SeanCheney | 来源:发表于2019-01-30 16:06 被阅读75次

    用标题中的四种方式解析网页,比较其解析速度。复习PyQuery和PySpider,PySpider这个项目有点老了,现在还是使用被淘汰的PhantomJS。

    系统配置、Python版本对解析速度也有影响,下面是我的结果(lxml与xpath最快,bs最慢):

    ==== Python version: 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14) 
    [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] =====
    
    ==== Total trials: 10000 =====
    bs4 total time: 7.7
    pq total time: 0.9
    lxml (cssselect) total time: 0.9
    lxml (xpath) total time: 0.6
    regex total time: 1.0 (doesn't find all p)
    

    拷贝下面代码可以自测:

    import re
    import sys
    import time
    import requests
    from lxml.html import fromstring
    from pyquery import PyQuery as pq
    from bs4 import BeautifulSoup as bs
    
    def Timer():
        a = time.time()
        while True:
            c = time.time()
            yield time.time()-a
            a = c
    timer = Timer()
    url = "http://www.python.org/"
    html = requests.get(url).text
    num = 10000
    print ('\n==== Python version: %s =====' %sys.version)
    print ('\n==== Total trials: %s =====' %num)
    next(timer)
    soup = bs(html, 'lxml')
    for x in range(num):
        paragraphs = soup.findAll('p')
    t = next(timer)
    print ('bs4 total time: %.1f' %t)
    d = pq(html)
    for x in range(num):
        paragraphs = d('p')
    t = next(timer)
    print ('pq total time: %.1f' %t)
    tree = fromstring(html)
    for x in range(num):
        paragraphs = tree.cssselect('p')
    t = next(timer)
    print ('lxml (cssselect) total time: %.1f' %t)
    tree = fromstring(html)
    for x in range(num):
        paragraphs = tree.xpath('.//p')
    t = next(timer)
    print ('lxml (xpath) total time: %.1f' %t)
    for x in range(num):
        paragraphs = re.findall('<[p ]>.*?</p>', html)
    t = next(timer)
    print ('regex total time: %.1f (doesn\'t find all p)\n' %t)
    

    借PyQuery复习CSS选择器。

    PyQuery支持下载网页为文本,是通过urllib或Requests实现的:

    from pyquery import PyQuery as pq
    
    url = 'https://www.feixiaohao.com/currencies/bitcoin/'
    headers = {
                'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 '
                              '(KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
            }
    doc = pq(url=url, headers=headers, method='get')
    
    # 也支持发送表单数据
    # pq(your_url, {'q': 'foo'}, method='post', verify=True)
    
    # btc价格
    btc_price = doc('.mainPrice span.convert').text()
    print(btc_price)
    
    # btc logo
    btc_logo = doc('img.coinLogo').attr('src')
    print(btc_logo)
    

    也可以直接加载文档字符串或html文档:

    from pyquery import PyQuery as pq
    
    html = '''
    <div>
        <ul>
             <li class="item-0">first item</li>
             <li class="item-1"><a href="link2.html">second item</a></li>
             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
             <li class="item-1 active"><a href="link4.html">fourth item</a></li>
             <li class="item-0"><a href="link5.html">fifth item</a></li>
        </ul>
    </div>
    '''
    
    doc = pq(html)
    # doc = pq(filename='demo.html')
    
    # 使用eq可以按次序选择
    print(doc('li').eq(1))
    

    查找上下级元素可以通过find(),children(),parent(),parents(),siblings()。CSS选择器举例如下:

    Pyspider的选择器是PyQuery。下面的例子是使用PySpider抓取IMDB250信息,fetch_type设为了js,存入MongoDB。

    #!/usr/bin/env python
    # -*- encoding: utf-8 -*-
    # Created on 2019-01-30 16:22:03
    # Project: imdb
    
    from pyspider.libs.base_handler import *
    
    
    class Handler(BaseHandler):
        crawl_config = {
        }
    
        @every(minutes=5)
        def on_start(self):
            self.crawl('https://www.imdb.com/chart/top', callback=self.index_page)
    
        @config(age=10 * 24 * 60 * 60)
        def index_page(self, response):
            for each in response.doc('.titleColumn > a').items():
                self.crawl(each.attr.href, callback=self.detail_page, fetch_type='js')
    
        @config(priority=2)
        def detail_page(self, response):
            return {            
                "title": response.doc('h1').text(),
                "voters": response.doc('[itemprop="aggregateRating"] > a > span').text(),
                "score": response.doc('strong > span').text()
            }
    
        # 需要再init中定义mongoclient
        def on_result(self, result):
            self.mongo.insert_result(result)
            super(Handler, self).on_result(result)
    

    PySpider的文档
    http://docs.pyspider.org/en/latest/

    PySpider目前还不支持Python3.7,所以只好用Python3.6。

    在MacOSX上安装Pyspider时,反复弹出这个错误Failed building wheel for pycurlerror: command 'gcc' failed with exit status 1。是SSL导致的错误,解决方法如下:

    (env)$ pip uninstall pycurl
    (env)$ pip install --upgrade pip
    (env)$ export LDFLAGS=-L/usr/local/opt/openssl/lib
    (env)$ export CPPFLAGS=-I/usr/local/opt/openssl/include
    (env)$ export PYCURL_SSL_LIBRARY=openssl
    (env)$ pip install pycurl
    

    相关文章

      网友评论

        本文标题:解析网页速度比较(BeautifulSoup、PyQuery、l

        本文链接:https://www.haomeiwen.com/subject/jufcsqtx.html