(七)Requests+正则表达式爬取猫眼电影Top100(一)

最近更新：2018-02-08

1.目标站点分析
2.流程框架
3.最终的代码
4.显示的结果

1.目标站点分析

1.1爬取的内容

爬取的链接:猫眼电影
需要爬取的是top100电影中的每一个电影包含电影名称/主演/上映时间/评分

1.2爬取的代码分析

1.3翻页链接的分析

第一页:http://maoyan.com/board/4
第二页:http://maoyan.com/board/4?offset=10
第三页:http://maoyan.com/board/4?offset=20
第i页:http://maoyan.com/board/4?offset=10(i-1)

2.流程框架

3.最终的代码

import requests
from requests.exceptions import RequestException
import json
import re


def get_one_page(url):
    try:
        response =requests.get(url)
        if response.status_code ==200:
            return response.text
        return None
    except RequestException:
        return None
def parse_one_page(html):
   pattern=re.compile('<dd>.*?board-index.*?>(\d+)</i>.*?data-src="(.*?)".*?name"><a'
                        +'.*?>(.*?)</a>.*?star">(.*?)</p>.*?releasetime">(.*?)</p>'
                        +'.*?integer">(.*?)</i>.*?fraction">(.*?)</i>.*?</dd>', re.S)
   items=re.findall(pattern,html)
   for item in items:
       yield{
           "index":item[0],
           "image":item[1],
           "title":item[2],
           "actor":item[3].strip()[3:],#把主演:这个切掉
           "time":item[4].strip()[5:],
           "score":item[5]+item[6]
        }

def write_to_file(content):
    with open("result.txt","a",encoding="utf-8") as f:
        f.write(json.dumps(content,ensure_ascii=False) + "\n")
        f.close()

def main(offset):
    url="http://maoyan.com/board/4?offset="+str(offset)
    html=get_one_page(url)
    for item in parse_one_page(html):
        print(item)
        write_to_file(item)

if __name__ == '__main__':
    for i in range(10):
        main(i*10)

4.显示的结果