前言

实战之前请先参看这篇文章安装好虚拟环境：Python学习二之虚拟环境的安装（基于CentOS6.8）。

一、 安装scrapy

1-1、 在虚拟环境下安装scrapy

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ pip install Scrapy

1-2、报错

Could not find a version that satisfies the requirement Twisted>=13.1.0 (from Scrapy) (from versions: )
No matching distribution found for Twisted>=13.1.0 (from Scrapy)

1-3、解决报错

升级Twisted版本

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ wget https://pypi.python.org/packages/source/T/Twisted/Twisted-15.2.1.tar.bz2

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ tar -jxvf Twisted-15.2.1.tar.bz2
(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ cd Twisted-15.2.1
(py3_scrapy) [dev@dc-o-scrapy-03 Twisted-15.2.1]$ python setup.py install

1-4、再次安装

(py3_scrapy) [dev@dc-o-scrapy-03 Twisted-15.2.1]$ cd ..
(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ pip install Scrapy

安装成功！！！

1-5、测试

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] python
>>> import scrapy

1-6、导入import scrapy 报错

>>> ModuleNotFoundError: No module named 'twisted.persisted'

1-6、解决报错

卸载旧版，重新安装

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ pip uninstall twisted 卸载旧版
(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ wget https://twistedmatrix.com/Releases/Twisted/17.9/Twisted-17.9.0.tar.bz2
(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ tar -jxvf Twisted-17.9.0.tar.bz2
(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ cd Twisted-17.9.0
(py3_scrapy) [dev@dc-o-scrapy-03 Twisted-17.9.0]$ python setup.py install
(py3_scrapy) [dev@dc-o-scrapy-03 Twisted-17.9.0]$ cd ..

1-7、再次测试

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] python
>>> import scrapy

1-8、报错

ImportError: No module named incremental

1-9、解决错误

安装incremental

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider]$ pip install incremental

1-10、再次测试

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] python
>>> import scrapy

成功！！！

二、 安装scrapy-redis

2-1、 在虚拟环境下安装scrapy-redis

安装scrapy-redis

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] pip install scrapy-redis

安装pymongo

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] pip install pymongo

安装pymysql

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] pip install pymysql

安装redis：安装scrapy-redis时会默认安装redis，所以下面命令不需执行

(py3_scrapy) [dev@dc-o-scrapy-03 ffspider] pip install redis

三、 安装scrapyd

3-1、 在虚拟环境下安装scrapyd

安装scrapyd

(py3_scrapy) [dev@dc-o-scrapy-01 ffspider] pip install scrapyd

3-2、配置scrapyd远程访问

找到虚拟环境下的scrapyd配置文件

(py3_scrapy) [dev@dc-o-scrapy-01 ffspider]$ find /home/dev/ -name "default_scrapyd.conf"
/home/dev/.virtualenvs/py3_scrapy/lib/python3.6/site-packages/scrapyd/default_scrapyd.conf

修改default_scrapyd.conf配置文件

(py3_scrapy) [dev@dc-o-scrapy-01 ffspider]$ vim /home/dev/.virtualenvs/py3_scrapy/lib/python3.6/site-packages/scrapyd/default_scrapyd.conf

测试

(py3_scrapy) [dev@dc-o-scrapy-01 ffspider]$ mkdir scrapyd
(py3_scrapy) [dev@dc-o-scrapy-01 ffspider]$ 
(py3_scrapy) [dev@dc-o-scrapy-01 ffspider]$ cd scrapyd/