利用Python进行简单的一些图片网站爬虫。
我们分为三部分来完成这个爬虫
获取页数的url - 解析页面的HTLM - 下载图片
PAGE的url是这样的:www.91doutu.com/category/qq表情包/page/1
我们可以用for循环来遍历出我们需要爬虫的页数。
BASE_PAGE_URL = 'http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85/page/'
for i in range(0,11):
print BASE_PAGE_URL + str(i)
这样就获取到了我们需要的page_url了。
接下来我们来完成第二步
解析页面的HTML源码 获取我们需要的部分。
#encoding
import requests
from bs4 import BeautifulSoup
response = requests.get('http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85')
content = response.content
soup = BeautifulSoup(content,'lxml')
img_list = soup.find_all('img',attrs={'class':'thumb'})
for img in img_list:
print img['data-src']
这样就获取到了我们需要的图片url了。
Img_List第三步-下载
只需要用到一个函数就轻轻松松搞定。
首先分割url 取list最后一个元素来当做我们的文件名,然后再下载到images目录下。
#encoding
import requests
from bs4 import BeautifulSoup
import os
import urllib
def download_image(url):
split_list = url.split('/')
filename = split_list.pop()
path = os.path.join('images',filename)
urllib.urlretrieve(url,filename=path)
response = requests.get('http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85')
content = response.content
soup = BeautifulSoup(content,'lxml')
img_list = soup.find_all('img',attrs={'class':'thumb'})
for img in img_list:
url = img['data-src']
download_image(url)
download_img
完整的Code:
#encoding
#_PlugName_ = Spider_Img
#__Author__ = Search__
# @Time : 2017/8/29
#__Refer___ = http://www.jianshu.com/u/d743d12d1d77
import requests
from bs4 import BeautifulSoup
import os
import urllib
BASE_PAGE_URL = 'http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85/page/'
PAGE_URL_LIST = []
for x in range(7,10):
url = BASE_PAGE_URL + str(x)
PAGE_URL_LIST.append(url)
def download_image(url):
split_list = url.split('/')
filename = split_list.pop()
path = os.path.join('images',filename)
urllib.urlretrieve(url,filename=path)
def get_page(page_url):
response = requests.get(page_url)
content = response.content
soup = BeautifulSoup(content,'lxml')
img_list = soup.find_all('img',attrs={'class':'thumb'})
for img in img_list:
url = img['data-src']
download_image(url)
def main():
for page_url in PAGE_URL_LIST:
get_page(page_url)
if __name__ == "__main__":
main()
网友评论