美文网首页python爬虫学习
同时使用ip代理池和用户代理池

同时使用ip代理池和用户代理池

作者: 薛落花随泪绽放 | 来源:发表于2017-11-02 21:07 被阅读6次
#如何同时使用用户代理池和IP代理池
import urllib.request
import re
import random
uapools=[
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 BIDUBrowser/8.7 Safari/537.36",
]
def api():
        print("这一次调用了接口")
        thisall=urllib.request.urlopen("网址").read().decode("utf-8","ignore")
        ippools=[]
        for item in thisall:
            ippools.append(item.decode("utf-8","ignore"))
        return ippools
def ip(ippools,time,uapools):
    thisua=random.choice(uapools)
    print(thisua)
    headers=("User-Agent",thisua)
    thisip=ippools[time]
    print("当前用的IP是:"+ippools[time])
    proxy=urllib.request.ProxyHandler({"http":thisip})
    opener=urllib.request.build_opener(proxy.urllib.request.HTTPHandler)
    urllib.request.install_opener(opener)
x=0
for i in range(0,35): 
    try:
        if(x%10==0):
            time=x%10
            ippools=api()
            ip(ippools,time,uapools)
        url="http://www.baidu.com"
        data1=urllib.request.urlopen(url).read()
        data=data1.decode("utf-8","ignore")
        print(len(data))
        fh=open("F:/pycharmxuexi/rst/ip_baidu")
        fh.write(data1)
        fh.close()
    except Exception as err:
        print(err)
        x+=1

相关文章

网友评论

    本文标题:同时使用ip代理池和用户代理池

    本文链接:https://www.haomeiwen.com/subject/eoxipxtx.html