随机请求头模块：pip install fake-useragent

用法简单记录

from fake_useragent import UserAgent
for i in range(5):
    print(UserAgent().random)

from fake_useragent import UserAgent
print(UserAgent().chrome)
print(UserAgent().ie)
print(UserAgent().firefox)
print(UserAgent().opera)
print(UserAgent().safari)

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent

session = requests.Session()
headers= {'User-Agent':str(UserAgent().random)}
print(headers)
url = "https://www.whatismybrowser.com/developers/what-http-headers-is-my-browser-sending"
req = session.get(url, headers=headers)

bsObj = BeautifulSoup(req.text, "html.parser")
print(bsObj.find("table", {'class':'table-striped'}).get_text)

请求头可以让网站改变布局格式。

例如，移动设备浏览网站时，通常会看到一个没有广告、Flash以及其他干扰的简化网站版本。这样的页面更容易采集。