middleware.py中构建proxy中间件
class ProxyMiddleware(object):
def process_request(self, request, spider):
request.meta['proxy'] = crawler.settings.get('HTTPPROXY')
pass
settings.py中启用proxymiddleware 和设定proxy
DOWNLOADER_MIDDLEWARES = {
'mingyan.middlewares.ProxyMiddleware': 1,
}
HTTPPROXY="http://127.0.0.1:80"
网友评论