从今天开始自学Python网络爬虫实战了，买到一本好书，和大家一起分享学习，也建议大家要多写多练。今天的收获感觉好多呢。越来越觉得Python有意思了。今天结合书上练习，自己实践了一把。书上的部分代码和实际代码有出入，根据书上的方法，经过一天的研究，最终把10页的新闻列表提取到了WORD文档里^_^
一、获取网度新闻headers
二、获取网页源代码
import requests
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'}
url = 'https://www.baidu.com/s?tn=news&rtt=1&bsst=1&cl=2&wd=考察'
res = requests.get(url, headers=headers).text
print(res)
三、编写正则表达式提取新闻信息
import re
res = '''
<div class="news-source">
<div class="c-img c-img1 c-img-circle news-source-icon_1tdlx c-gap-right-xsmall">
<span class="c-img-border c-img-circle"></span>
<img class="source-img_33bs5" src="https://timg01.bdimg.com/timg?pacompress=&imgtype=0&sec=1439619614&autorotate=1&di=834cbb7d72ef5d6290e356c3a9b82679&quality=90&size=b870_10000&src=http%3A%2F%2Fpic.rmb.bdstatic.com%2Fb5aef7a1e77791d0387d001e5fa2d184.png">
</div>
<span class="c-color-gray c-font-normal c-gap-right">网易新闻</span>
<span class="c-color-gray2 c-font-normal">2020年12月27日 18:37</span>
</div>
'''
p_info = '<div class="news-source">(.*?)</div>'
info = re.findall(p_info, res, re.S)
print(info)
四、编写正则表达式提取新闻链接
import re
res = '''
<h3 class="news-title_1YtI1">
<a href="https://finance.ifeng.com/c/82Z0Nx2QiJ6" target="_blank" class="news-title-font_1xS-F" data-click="{
'f0':'77A717EA',
'f1':'9F63F1E4',
'f2':'4CA6DE6E',
'f3':'54E5243F',
't':'1609115182',
}"><!--s-text--><em>阿里巴巴</em>某某某某某某,由...<!--/s-text--></a>
</h3>
'''
p_href = '<h3 class="news-title_1YtI1">.*?<a href="(.*?)"'
href = re.findall(p_href, res, re.S)
print(href) # ['https://finance.ifeng.com/c/82Z0Nx2QiJ6']
五、编写正则表达式提取新闻标题
import re
res = '''
<h3 class="news-title_1YtI1">
<a href="https://finance.ifeng.com/c/82Z0Nx2QiJ6" target="_blank" class="news-title-font_1xS-F" data-click="{
'f0':'77A717EA',
'f1':'9F63F1E4',
'f2':'4CA6DE6E',
'f3':'54E5243F',
't':'1609115182',
}"><!--s-text--><em>阿里巴巴</em>在港公告:董事会已授权增加本公司的股份回购计划总额,由...<!--/s-text--></a>
</h3>
'''
p_title = '<h3 class="news-title_1YtI1">.*?>(.*?)</a>'
title = re.findall(p_title, res, re.S)
print(title) # ['<!--s-text--><em>阿里巴巴</em>在港公告:董事会已授权增加本公司的股份回购计划总额,由...<!--/s-text-->']
六、数据清洗并打印输出
1.新闻标题清洗
import re
res = '''
<h3 class="news-title_1YtI1">
<a href="https://finance.ifeng.com/c/82Z0Nx2QiJ6" target="_blank" class="news-title-font_1xS-F" data-click="{
'f0':'77A717EA',
'f1':'9F63F1E4',
'f2':'4CA6DE6E',
'f3':'54E5243F',
't':'1609115182',
}"> <!--s-text--><em>阿里巴巴</em>在港公告:董事会已授权增加本公司的股份回购计划总额,由...<!--/s-text--></a>
</h3>
'''
p_title = '<h3 class="news-title_1YtI1">.*?>(.*?)</a>'
title = re.findall(p_title, res, re.S)
# strip()函数，清理空格和换行符
# 该方法只能删除开头或是结尾的字符，不能删除中间部分的字符。
for i in range(len(title)): # len(title) title的长度
title[i] = title[i].strip()
print(title[i])
2.新闻来源和日期清洗
import re
res = '''
<div class="news-source">
<div class="c-img c-img1 c-img-circle news-source-icon_1tdlx c-gap-right-xsmall">
<span class="c-img-border c-img-circle"></span>
<img class="source-img_33bs5" src="https://timg01.bdimg.com/timg?pacompress=&imgtype=0&sec=1439619614&autorotate=1&di=834cbb7d72ef5d6290e356c3a9b82679&quality=90&size=b870_10000&src=http%3A%2F%2Fpic.rmb.bdstatic.com%2Fb5aef7a1e77791d0387d001e5fa2d184.png">
</div>
<span class="c-color-gray c-font-normal c-gap-right">网易新闻</span>
<span class="c-color-gray2 c-font-normal">2020年12月27日 18:37</span>
</div>
'''
p_source = '<span class="c-color-gray c-font-normal c-gap-right">(.*?)</span>'
source = re.findall(p_source, res, re.S)
for i in range(len(source)):
source[i] = re.sub('<.*?>', '', source[i])
print(source[i])
p_date = '<span class="c-color-gray2 c-font-normal">(.*?)</span>'
date = re.findall(p_date, res, re.S)
for j in range(len(date)):
print(date[j])
完整代码如下：
import requests
import re
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'}
url = 'https://www.baidu.com/s?rtt=1&bsst=1&cl=2&tn=news&rsv_dl=ns_pc&word=考察'
res = requests.get(url, headers=headers).text
p_href = '<h3 class="news-title_1YtI1">.*?<a href="(.*?)"'
p_title = '<h3 class="news-title_1YtI1">.*?>(.*?)</a>'
p_source = '<span class="c-color-gray c-font-normal c-gap-right">(.*?)</span>'
p_date = '<span class="c-color-gray2 c-font-normal">(.*?)</span>'
href = re.findall(p_href, res, re.S)
title = re.findall(p_title, res, re.S)
source = re.findall(p_source, res, re.S)
date = re.findall(p_date, res, re.S)
# 数据清洗及打印输出
for i in range(len(title)):
title[i] = title[i].strip()
title[i] = re.sub('<.*?>', '', title[i])
print(str(i+1) + '.' + title[i] + '(' + date[i] + '-' + source[i] + ')')
print(href[i])
本人是网络爬虫新手，拿百度新闻做了一个测试，代码中还有需要改进的地方，请指正，谢谢！
网友评论