Python 爬虫：把廖雪峰的教程转换成 PDF 电子书

作者: liuzhijun | 来源:发表于2017-02-17 21:36 被阅读2313次

原文：https://foofish.net/python-crawler-html2pdf.html

写爬虫似乎没有比用 Python 更合适了，Python 社区提供的爬虫工具多得让你眼花缭乱，各种拿来就可以直接用的 library 分分钟就可以写出一个爬虫出来，今天就琢磨着写一个爬虫，将廖雪峰的 Python 教程爬下来做成 PDF 电子书方便大家离线阅读。

开始写爬虫前，我们先来分析一下该网站1的页面结构，网页的左侧是教程的目录大纲，每个 URL 对应到右边的一篇文章，右侧上方是文章的标题，中间是文章的正文部分，正文内容是我们关心的重点，我们要爬的数据就是所有网页的正文部分，下方是用户的评论区，评论区对我们没什么用，所以可以忽略它。

工具准备

弄清楚了网站的基本结构后就可以开始准备爬虫所依赖的工具包了。requests、beautifulsoup 是爬虫两大神器，reuqests 用于网络请求，beautifusoup 用于操作 html 数据。有了这两把梭子，干起活来利索，scrapy 这样的爬虫框架我们就不用了，小程序派上它有点杀鸡用牛刀的意思。此外，既然是把 html 文件转为 pdf，那么也要有相应的库支持， wkhtmltopdf 就是一个非常好的工具，它可以用适用于多平台的 html 到 pdf 的转换，pdfkit 是 wkhtmltopdf 的Python封装包。首先安装好下面的依赖包，接着安装 wkhtmltopdf

pip install requests
pip install beautifulsoup
pip install pdfkit

安装 wkhtmltopdf

Windows平台直接在 wkhtmltopdf 官网2下载稳定版的进行安装，安装完成之后把该程序的执行路径加入到系统环境 $PATH 变量中，否则 pdfkit 找不到 wkhtmltopdf 就出现错误 “No wkhtmltopdf executable found”。Ubuntu 和 CentOS 可以直接用命令行进行安装

$ sudo apt-get install wkhtmltopdf  # ubuntu
$ sudo yum intsall wkhtmltopdf      # centos

爬虫实现

一切准备就绪后就可以上代码了，不过写代码之前还是先整理一下思绪。程序的目的是要把所有 URL 对应的 html 正文部分保存到本地，然后利用 pdfkit 把这些文件转换成一个 pdf 文件。我们把任务拆分一下，首先是把某一个 URL 对应的 html 正文保存到本地，然后找到所有的 URL 执行相同的操作。

用 Chrome 浏览器找到页面正文部分的标签，按 F12 找到正文对应的 div 标签： <div class="x-wiki-content">，该 div 是网页的正文内容。用 requests 把整个页面加载到本地后，就可以使用 beautifulsoup 操作 HTML 的 dom 元素来提取正文内容了。

具体的实现代码如下：用 soup.find_all 函数找到正文标签，然后把正文部分的内容保存到 a.html 文件中。

def parse_url_to_html(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html5lib")
    body = soup.find_all(class_="x-wiki-content")[0]
    html = str(body)
    with open("a.html", 'wb') as f:
        f.write(html)

第二步就是把页面左侧所有 URL 解析出来。采用同样的方式，找到左侧菜单标签 <ul class="uk-nav uk-nav-side">

具体代码实现逻辑：因为页面上有两个uk-nav uk-nav-side的 class 属性，而真正的目录列表是第二个。所有的 url 获取了，url 转 html 的函数在第一步也写好了。

def get_url_list():
    """
    获取所有URL目录列表
    """
    response = requests.get("http://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000")
    soup = BeautifulSoup(response.content, "html5lib")
    menu_tag = soup.find_all(class_="uk-nav uk-nav-side")[1]
    urls = []
    for li in menu_tag.find_all("li"):
        url = "http://www.liaoxuefeng.com" + li.a.get('href')
        urls.append(url)
    return urls

最后一步就是把 html 转换成pdf文件了。转换成 pdf 文件非常简单，因为 pdfkit 把所有的逻辑都封装好了，你只需要调用函数 pdfkit.from_file

def save_pdf(htmls):
    """
    把所有html文件转换成pdf文件
    """
    options = {
        'page-size': 'Letter',
        'encoding': "UTF-8",
        'custom-header': [
            ('Accept-Encoding', 'gzip')
        ]
    }
    pdfkit.from_file(htmls, file_name, options=options)

执行 save_pdf 函数，电子书 pdf 文件就生成了，效果图：

总结

总共代码量加起来不到50行，不过，且慢，其实上面给出的代码省略了一些细节，比如，如何获取文章的标题，正文内容的 img 标签使用的是相对路径，如果要想在 pdf 中正常显示图片就需要将相对路径改为绝对路径，还有保存下来的 html 临时文件都要删除，这些细节末叶都放在github上。

完整代码和生成的电子书可以在github 3 下载，代码在 Windows 平台亲测有效，欢迎 fork 下载自己改进。

网友评论

鸭梨山大哎:list index out of range

5d35e4b5f6b8:wkhtmltopdf 安装完成后把该程序的执行路径加入到系统环境 $PATH 变量中，怎么把他加入到系统环境？这个路径是不是你安装request ，pdfkit的目录？
还有下面怎么在windows系统下用命令安装ubuntu和centos

faf7a1443fbb:好

东方越迁:发你爬下来PDF电子书百度云链接可以不？

liuzhijun:在公众号回复 pdf

e556fc01286f:python crawler.py
ERROR:root:解析错误
Traceback (most recent call last):
File "crawler.py", line 57, in parse_url_to_html
html = html.encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 130: ordinal not in range(128)
ERROR:root:解析错误。。。。。。。。。。。。。。

e556fc01286f: @MiracleWong 非常感谢

MiracleWong:看这个地方的解决方法：http://blog.csdn.net/mindmb/article/details/7898528

风逝流沙:ERROR:root:解析错误
Traceback (most recent call last):
File "f:\python\crawler\crawler_html2pdf.py", line 29, in parse_url_to_html
response=requests.get(url)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\sessions.py", line 474, in request
prep = self.prepare_request(req)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\sessions.py", line 407, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\models.py", line 302, in prepare
self.prepare_url(url, params)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\models.py", line 382, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '0': No schema supplied. Perhaps you meant http://0?
楼主有没有遇到这样的问题，猜测可能是解析url不对，应该拿到<a href的链接，可能拿到的是li的id，代码相同的，求解答。

liuzhijun:@风逝流沙你要调试一下，打印的url是什么？有可能网站结构有更新或者其他原因

风逝流沙:@liuzhijun 用的是相同的初始url，把github上的代码保存下来之后再运行，也是这个错误

liuzhijun:@风逝流沙 URL错了吧

鸭梨山大哎:2.7.12报错
ImportError Traceback (most recent call last)
<ipython-input-3-66edfb667748> in <module>()
3 import re
4 import time
----> 5 from urllib.parse import urlparse
6 from bs4 import BeautifulSoup
7 import pdfkit

ImportError: No module named parse

In [4]: !pip install parse
Collecting parse
Downloading parse-1.6.6-py2-none-any.whl
Installing collected packages: parse
Successfully installed parse-1.6.6

之后还报同样的错

liuzhijun:from urllib.parse import urlparse
这行用来干嘛的？

OMG_markle:思路清晰，感谢笔者分享

8f26b385cab9:🤗

e556fc01286f:ubuntu下pip install pdfkit时出现错误
Downloading/unpacking pdfkit
Downloading pdfkit-0.6.1-py2-none-any.whl
Installing collected packages: pdfkit
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1436, in install
requirement.install(install_options, global_options, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 672, in install
self.move_wheel_files(self.source_dir, root=root)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 902, in move_wheel_files
pycompile=self.pycompile,
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 206, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 193, in clobber
os.makedirs(destsubdir)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pdfkit'

Storing debug log for failure in /home/henry/.pip/pip.log

liuzhijun:Permission denied 是权限不够啊，用 sudo pip install

5d35e4b5f6b8:正在学习，非常实用

e556fc01286f:pip install beautifulsoup 时出现错误
Collecting beautifulsoup
Downloading BeautifulSoup-3.2.1.tar.gz
Exception:
Traceback (most recent call last):
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 17: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\commands\install.py", line 324, in run
requirement_set.prepare_files(finder)
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
command_desc='python setup.py egg_info')
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
line = console_to_str(proc.stdout.readline())
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 17: invalid start byte

e556fc01286f:ubuntu下错误
error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/BeautifulSoup.py'

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_henry/beautifulsoup/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-2XDgzk-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_henry/beautifulsoup
Storing debug log for failure in /home/henry/.pip/pip.log

e556fc01286f:这个是win下出现的问题

liuzhijun:@Hereyon pip install beautifulsoup4 试试

16b3398de041:一只跟着廖老师的学习笔记学python，倒是不知道出pdf了😂

鸭梨山大哎:list index out of range
5d35e4b5f6b8:wkhtmltopdf 安装完成后把该程序的执行路径加入到系统环境 $PATH 变量中，怎么把他加入到系统环境？这个路径是不是你安装request ，pdfkit的目录？
还有下面怎么在windows系统下用命令安装ubuntu和centos
faf7a1443fbb:好
东方越迁:发你爬下来PDF电子书百度云链接可以不？
liuzhijun:在公众号回复 pdf
e556fc01286f:python crawler.py
ERROR:root:解析错误
Traceback (most recent call last):
File "crawler.py", line 57, in parse_url_to_html
html = html.encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 130: ordinal not in range(128)
ERROR:root:解析错误。。。。。。。。。。。。。。
e556fc01286f: @MiracleWong 非常感谢
MiracleWong:看这个地方的解决方法：http://blog.csdn.net/mindmb/article/details/7898528
风逝流沙:ERROR:root:解析错误
Traceback (most recent call last):
File "f:\python\crawler\crawler_html2pdf.py", line 29, in parse_url_to_html
response=requests.get(url)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\sessions.py", line 474, in request
prep = self.prepare_request(req)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\sessions.py", line 407, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\models.py", line 302, in prepare
self.prepare_url(url, params)
File "D:\Program Files\Python35\lib\site-packages\requests-2.13.0-py3.5.egg\requests\models.py", line 382, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '0': No schema supplied. Perhaps you meant http://0?
楼主有没有遇到这样的问题，猜测可能是解析url不对，应该拿到<a href的链接，可能拿到的是li的id，代码相同的，求解答。
liuzhijun:@风逝流沙你要调试一下，打印的url是什么？有可能网站结构有更新或者其他原因
风逝流沙:@liuzhijun 用的是相同的初始url，把github上的代码保存下来之后再运行，也是这个错误
liuzhijun:@风逝流沙 URL错了吧
鸭梨山大哎:2.7.12报错
ImportError Traceback (most recent call last)
<ipython-input-3-66edfb667748> in <module>()
3 import re
4 import time
----> 5 from urllib.parse import urlparse
6 from bs4 import BeautifulSoup
7 import pdfkit

ImportError: No module named parse

In [4]: !pip install parse
Collecting parse
Downloading parse-1.6.6-py2-none-any.whl
Installing collected packages: parse
Successfully installed parse-1.6.6

之后还报同样的错
liuzhijun:from urllib.parse import urlparse
这行用来干嘛的？
OMG_markle:思路清晰，感谢笔者分享
8f26b385cab9:🤗
e556fc01286f:ubuntu下pip install pdfkit时出现错误
Downloading/unpacking pdfkit
Downloading pdfkit-0.6.1-py2-none-any.whl
Installing collected packages: pdfkit
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1436, in install
requirement.install(install_options, global_options, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 672, in install
self.move_wheel_files(self.source_dir, root=root)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 902, in move_wheel_files
pycompile=self.pycompile,
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 206, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 193, in clobber
os.makedirs(destsubdir)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pdfkit'

Storing debug log for failure in /home/henry/.pip/pip.log
liuzhijun:Permission denied 是权限不够啊，用 sudo pip install
5d35e4b5f6b8:正在学习，非常实用
e556fc01286f:pip install beautifulsoup 时出现错误
Collecting beautifulsoup
Downloading BeautifulSoup-3.2.1.tar.gz
Exception:
Traceback (most recent call last):
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 17: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\basecommand.py", line 215, in main
status = self.run(options, args)
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\commands\install.py", line 324, in run
requirement_set.prepare_files(finder)
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
command_desc='python setup.py egg_info')
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
line = console_to_str(proc.stdout.readline())
File "C:\Users\**\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 17: invalid start byte
e556fc01286f:ubuntu下错误
error: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/BeautifulSoup.py'

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_henry/beautifulsoup/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-2XDgzk-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_henry/beautifulsoup
Storing debug log for failure in /home/henry/.pip/pip.log
e556fc01286f:这个是win下出现的问题
liuzhijun:@Hereyon pip install beautifulsoup4 试试
16b3398de041:一只跟着廖老师的学习笔记学python，倒是不知道出pdf了😂

Python 爬虫：把廖雪峰的教程转换成 PDF 电子书

工具准备

安装 wkhtmltopdf

爬虫实现

总结

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

资料

Python

Linux+Python+Java

python开发

爬虫

爬虫

专注python学习100年

程序员