pyspider 启动失败

pyspider 启动失败

(python36) [root@centos-7 ~]# pyspider
[W 181204 14:35:01 run:413] phantomjs not found, continue running without it.
[I 181204 14:35:03 result_worker:49] result_worker starting...
[I 181204 14:35:03 processor:211] processor starting...
[I 181204 14:35:03 scheduler:647] scheduler starting...
Process Process-4:
Traceback (most recent call last):
  File "/opt/conda3/envs/python36/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda3/envs/python36/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 236, in fetcher
    Fetcher = load_cls(None, None, fetcher_cls)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 48, in load_cls
    return utils.load_object(value)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/libs/utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/fetcher/__init__.py", line 1, in <module>
    from .tornado_fetcher import Fetcher
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in <module>
    from tornado.curl_httpclient import CurlAsyncHTTPClient
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/tornado/curl_httpclient.py", line 24, in <module>
    import pycurl  # type: ignore
ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.61.0)
[I 181204 14:35:03 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 181204 14:35:03 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
Traceback (most recent call last):
  File "/opt/conda3/envs/python36/bin/pyspider", line 11, in <module>
    sys.exit(main())
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 754, in main
    cli()
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli
    ctx.invoke(all)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 333, in webui
    app = load_cls(None, None, webui_instance)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/run.py", line 48, in load_cls
    return utils.load_object(value)
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/libs/utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/webui/__init__.py", line 8, in <module>
    from . import app, index, debug, task, result, login
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/webui/app.py", line 17, in <module>
    from pyspider.fetcher import tornado_fetcher
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/fetcher/__init__.py", line 1, in <module>
    from .tornado_fetcher import Fetcher
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/pyspider/fetcher/tornado_fetcher.py", line 30, in <module>
    from tornado.curl_httpclient import CurlAsyncHTTPClient
  File "/opt/conda3/envs/python36/lib/python3.6/site-packages/tornado/curl_httpclient.py", line 24, in <module>
    import pycurl  # type: ignore
ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.61.0)
(python36) [root@centos-7 ~]#

解决方案：

(python36) [root@centos-7 ~]# conda install pycurl

再次启动既可以了

(python36) [root@centos-7 lib]# pyspider
[W 181204 14:39:02 run:413] phantomjs not found, continue running without it.
[I 181204 14:39:04 result_worker:49] result_worker starting...
[I 181204 14:39:04 processor:211] processor starting...
[I 181204 14:39:04 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 181204 14:39:04 scheduler:647] scheduler starting...
[I 181204 14:39:04 tornado_fetcher:638] fetcher starting...
[I 181204 14:39:04 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 181204 14:39:04 app:76] webui running on 0.0.0.0:5000
[I 181204 14:40:04 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0