python2.7用requests和xpath抓取煎蛋网段子

这里我简单的爬取了煎蛋网的段子，煎蛋网有些段子会被屏蔽的现象产生，所以要对这块东西进行处理。

屏蔽段子处理

下面就是按常规去处理，附上具体代码

import requests

froml xml import etree

url='http://jandan.net/duan'

headers={

'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',

'Accept-Encoding':'gzip, deflate',

'Accept-Language':'zh-CN,zh;q=0.8',

'Cache-Control':'no-cache',

'Connection':'keep-alive',

'Host':'jandan.net',

'Pragma':'no-cache',

'Referer':'http://jandan.net/qa',

'Upgrade-Insecure-Requests':'1',

'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3100.0 Safari/537.36',

}

html=requests.get(url,headers=headers);

html.encoding="utf-8"

root=etree.HTML(html.text)

result=root.xpath("//div[@class='row']")

for i in range(len(result)):

author=result[i].xpath(".//div[@class='author']/strong/text()")

text=re sult[i].xpath(".//div[@class='text']")[0]

if(text.xpath("./p[@class='bad_content']")):

text=result[i].xpath(".//div[@class='text']/p[2]/text()")

else:

text=result[i].xpath(".//div[@class='text']/p/text()")

print '作者',author[0],'内容',text[0]

上面的xpath上的.//div[@class='author']/strong/text()解释，就是在class为row的div下找到class为author的div，再在strong标签下，得到标签中的字。