Virtualenv

安装virtualenv

$ cd ~/Python/
$ mkdir Envs
$ cd Envs
$ virtualenv env_data
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/wupeng/Python/Envs/env_data/bin/python3.5
Also creating executable in /Users/wupeng/Python/Envs/env_data/bin/python
Installing setuptools, pip, wheel...done.
$ source env_data/bin/activate
(env_data) localhost:Envs wupeng$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
appdirs (1.4.3)
packaging (16.8)
pip (9.0.1)
pyparsing (2.2.0)
setuptools (34.3.2)
six (1.10.0)
wheel (0.30.0a0)

安装virtualenvwrapper

$ pip install virtualenvwrapper
Collecting virtualenvwrapper
  Using cached virtualenvwrapper-4.7.2.tar.gz
Collecting virtualenv (from virtualenvwrapper)
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Collecting virtualenv-clone (from virtualenvwrapper)
  Using cached virtualenv-clone-0.2.6.tar.gz
Collecting stevedore (from virtualenvwrapper)
  Downloading stevedore-1.21.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9.0 in /Users/wupeng/Envs/py3/lib/python3.5/site-packages (from stevedore->virtualenvwrapper)
Collecting pbr>=2.0.0 (from stevedore->virtualenvwrapper)
  Downloading pbr-2.0.0-py2.py3-none-any.whl (98kB)
    100% |████████████████████████████████| 102kB 986kB/s
Building wheels for collected packages: virtualenvwrapper, virtualenv-clone
  Running setup.py bdist_wheel for virtualenvwrapper ... done
  Stored in directory: /Users/wupeng/Library/Caches/pip/wheels/3e/7e/eb/31f2187dde819aa4f67ade0ac8401da47257f44c213f153ca2
  Running setup.py bdist_wheel for virtualenv-clone ... done
  Stored in directory: /Users/wupeng/Library/Caches/pip/wheels/24/51/ef/93120d304d240b4b6c2066454250a1626e04f73d34417b956d
Successfully built virtualenvwrapper virtualenv-clone
Installing collected packages: virtualenv, virtualenv-clone, pbr, stevedore, virtualenvwrapper
Successfully installed pbr-2.0.0 stevedore-1.21.0 virtualenv-15.1.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.2
$ find / -name virtualenvwrapper.sh
/Library/Frameworks/Python.framework/Versions/3.5/bin/virtualenvwrapper.sh
$ vi ~/.bash_profile
# Setting virtualenvwrapper
export WORKON_HOME=$HOME/Envs
source /Library/Frameworks/Python.framework/Versions/3.5/bin/virtualenvwrapper.sh
$ source .bash_profile
$ workon
$ mkvirtualenv py3
$ deactivate
$ mkvirtualenv --python=/Library/Frameworks/Python.framework/Versions/2.7/bin/python py27
Running virtualenv with interpreter /Library/Frameworks/Python.framework/Versions/2.7/bin/python
New python executable in /Users/wupeng/Envs/py27/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/py27/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/py27/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/py27/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/py27/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/py27/bin/get_env_details
(py27) localhost:~ wupeng$

安装scrapy

localhost:~ wupeng$ mkvirtualenv article_spider
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/wupeng/Envs/article_spider/bin/python3.5
Also creating executable in /Users/wupeng/Envs/article_spider/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/article_spider/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/article_spider/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/article_spider/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/article_spider/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/wupeng/Envs/article_spider/bin/get_env_details
(article_spider) localhost:~ wupeng$ echo $PATH
/Users/wupeng/Envs/article_spider/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
(article_spider) localhost:~ wupeng$ PYTHON
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
KeyboardInterrupt
>>> ^D
(article_spider) localhost:~ wupeng$ pip install -i https://pypi.douban.com/simple/ scrapy

创建爬虫

(article_spider) localhost:PycharmProjects wupeng$ cd ~/PycharmProjects/
(article_spider) localhost:PycharmProjects wupeng$ scrapy startproject ArticleSpider
New Scrapy project 'ArticleSpider', using template directory '/Users/wupeng/Envs/article_spider/lib/python3.5/site-packages/scrapy/templates/project', created in:
    /Users/wupeng/PycharmProjects/ArticleSpider

You can start your first spider with:
    cd ArticleSpider
    scrapy genspider example example.com
(article_spider) localhost:PycharmProjects wupeng$ cd ArticleSpider/
(article_spider) localhost:ArticleSpider wupeng$ scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic' in module:
  ArticleSpider.spiders.jobbole
...
2017-04-01 12:47:30 [scrapy.core.engine] INFO: Spider closed (finished)
(article_spider) localhost:ArticleSpider wupeng$ scrapy shell http://blog.jobbole.com/110287/