Python实战计划学习笔记（5）获取网页中的动态数据

什么是异步数据

通过js技术，不需要用户请求即可不断加载的数据

爬取方法

1. 通过浏览器network/XHR数据观察确定自动加载的页面请求URL
2. 按规律自动生成需要爬取的URL
3. 逐页爬取信息

教学代码

from bs4 import BeautifulSoup
import requests
import time

url = 'https://knewone.com/discover?page='
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}

def get_page(url,data=None):
    web_data = requests.get(url,headers = headers)
    soup = BeautifulSoup(web_data.text,'lxml')
    imgs = soup.select('article > header > a > img')
    titles = soup.select('article > section > h4 > a')
    links = soup.select('article > section > h4 > a')
    if data == None:
         for img,title,link in zip(imgs,titles,links):
            data ={
                'img':img.get('src'),
                'title':title.get('title'),
                'link':link.get('href')
            }
            print(data)
def get_more_pages(start,end):
    for one in range(start,end):
        get_page(url+str(one))
        time.sleep(2)
get_more_pages(1,5)

运行结果

作业

爬取58同城二手商品一页列表中的商品详细信息