美文网首页
scrapy调用httpproxy

scrapy调用httpproxy

作者: Sirius之剣 | 来源:发表于2020-02-04 10:33 被阅读0次

    middleware.py中构建proxy中间件

    class ProxyMiddleware(object):
        def process_request(self, request, spider):
            request.meta['proxy'] =  crawler.settings.get('HTTPPROXY')
            pass
    

    settings.py中启用proxymiddleware 和设定proxy

    DOWNLOADER_MIDDLEWARES = {
        'mingyan.middlewares.ProxyMiddleware': 1,
    }
    HTTPPROXY="http://127.0.0.1:80"
    

    相关文章

      网友评论

          本文标题:scrapy调用httpproxy

          本文链接:https://www.haomeiwen.com/subject/kgzlxhtx.html