方法一：直接在spider中设置代理
该方法只对一个spider有效
import sprapy
from bs4 import BeautifulSoup as bs
class appledaily(scrapy.Spider):
name = "appledaily"
start_url = ['*****']
def start_requests(self):
yield scrapy.Request(url = self.start_url[0],callback = self.parse,meta={'proxy':"127.0.0.1:port"})
def parse(self,response):
html = bs(response.body)
print(html) #验证
方法二：整个爬虫项目设置代理
首先在middlewares中定义一个代理设置得类
class proxy(object):
def process_request(self,request,spider):
request.meta['proxy'] = "http://127.0.0.1:port"
然后再setting类中修改设置：
网友评论