Python 爬虫

一、背景

某个比赛要从网上抓取图像，于是做了简单爬虫入门。

二、基础

1、首先是python3的urllib，获取普通baidu网页

import urllib.request
import urllib.error

url = "http://www.baidu.com"
try:
    response = urllib.request.urlopen(url)
    html = response.read().decode()
    print(html)
except:
    print('Warning: Could not download from %s' % url)

2、python2的urllib2，获取普通baidu网页

import urllib2 #这里变成了urllib2

url = "http://www.baidu.com"
try:
    response = urllib2.urlopen(url) #这里变成了urllib2
    html = response.read().decode()
    print(html)
except:
    print('Warning: Could not download from %s' % url)

3、使用request库，不区分python2还是python3

import requests #这里变成了requests

url = "http://www.baidu.com"
try:
    response = requests.get(url=url)
    response.encoding = 'utf-8'
    html = response.text
    print(html)
except:
    print('Warning: Could not download from %s' % url)

三、使用代理

  要使用代理，首先要确定本地http和https代理设置，一般为“127.0.0.1：xxxx”。其中，xxxx为端口号。其次要确认headers里的‘user-agent’，可以在https://developers.whatismybrowser.com/useragents/explore/ 网站查找。
  获取图像，某网页图片如下图：

image.png

  下面给出了python2和python3的两种实现，其中python3区分了字节流BytesIO和字符流StringIO，而python2不区分，需要特别注意。

import urllib.request
import urllib.error
from PIL import Image
from io import BytesIO  #python3
# from StringIO import StringIO    #python2

url = "https://lh3.googleusercontent.com/-4YE2L3n5W1s/WEn3Ks0KnNI/AAAAAAAASSI/_HP83M-1cIQEYq7CQVj0lHmZ1P1kMilLwCOcB/s1600/"
try:
    proxies = {'https': 'https://127.0.0.1:12333','http' : 'http://127.0.0.1:12333'}
    headers = {
        #73.0.3683.86为chrome的版本号
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'
    }


    opener = urllib.request.build_opener(urllib.request.ProxyHandler(proxies))
    urllib.request.install_opener(opener)
    req = urllib.request.Request(url, headers=headers)
    image_data = response.read()
except:
    print('Warning: Could not download image from %s' % url)

try:
    #python3
    pil_image = Image.open(BytesIO(image_data))
   #phyton2
   #pil_image = Image.open(StringIO(image_data))
except:
    print('Warning: Failed to parse image from %s' % url)


  try:
    pil_image_rgb = pil_image.convert('RGB') #RGB的图像
  except:
    print('Warning: Failed to convert image %s to RGB' % key)

四、总结

  Python2和Python3库兼容性非常不好，使用时要特别注意。