从头开始配置编译python3.6版本的caffe整整花了10天时间,期间经历了很多事,所以状态一直很差,真正的配置时间应该在五天左右.
目前搜索到的caffe配置版本多数是2.7和3.5,关于python3.6的配置基本没有.但是目前anaconda3用的python版本为3.6,tensorflow最新使用的都是3.6,所以我电脑里存在的就是3.6版本.所以我决定配置python3.6版本的caffe.
step by step!
1,配置opencv 3.4.1
我们需要达到的目标是在python3中可以直接使用import cv2,其中关于编译这方面的资料很多.主要分为编译本体库,以及编译一个附加下载的库.但是在编译完成之后发现,依然无法成功import cv2.程序中的Import cv2提示no modules named cv2错误
解决方法:sudo pip3 install opencv-python.注意这里一定要用pip3,之前一直有资料说用pip,这样是无法成功的.使用pip3后,直接成功.
2,编译caffe
这里的大前提是已经编译配置好anaconda3,python的版本为3.6.python3.6的bin和lib文件全部在anaconda3中.
这里主要参考的是这篇博文.https://blog.csdn.net/isuccess88/article/details/70165726,但也有很多不同,他那篇是针对python3.5的.
2.1 安装依赖
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install libhdf5-serial-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
然后把CAFFE的源代码下载下来: git clone https://github.com/BVLC/caffe.git
(当然没有安装GIT的得先安装一下)
下载完成之后,进入CAFFE文件夹, 进入里面的PYTHON文件夹,然后输入
for req in $(cat requirements.txt); do pip3 install $req; done
(PIP如果没有安装得先安装一下:sudo apt install python-pip)
注意的地方是:pip3安装的,第二点是sudo apt-get install --no-install-recommends libboost-all-dev这一条安装编译的并没有关于python3.6版本的libboost,我们需要重新编译.否则caffe无法调用成功.(怎么编译后面说)
2.2 安装caffe
到CAFFE文件夹, 使用模板写个Makefile.config. 具体就是先复制一下模板, 再改一些内容(我喜欢用EMACS).
cp Makefile.config.example Makefile.config
-因为CPU MODE, 所以在CPU_ONLY := 1前面的#要去掉.
-两个路径要改成这样:(添加后面的两个hdf5的路径, 否则编译时报hdf5错误)
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include/usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib/usr/lib/x86_64-linux-gnu/hdf5/serial
一份完整的python3.6配置文件.（这里黑色加粗字体的位置就是我们要做修改注意的地方）
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!
# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1
# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1#（因为我们要配置cpu版本的，所以这里把引号去掉）
# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
# You should not set this flag if you will be reading LMDBs with any
# possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1
# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3#（我们的opencv是3版本的，所以这里我们把引号去掉，之前opencv3已经安装并编译成功）
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
-gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_60,code=sm_60 \
-gencode arch=compute_61,code=sm_61 \
-gencode arch=compute_61,code=compute_61
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas
# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
#PYTHON_INCLUDE := /usr/include/python2.7 \
# /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda3#（我们使用的python3.6版本在anaconda3中，所以这里要取消注释，修改为你的根目录位置）
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python3.6m \#（这两个目录要看具体anaconda3中目录的名字，我做了对应修改）
$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include
# Uncomment to use Python 3 (default is Python 2)
PYTHON_LIBRARIES := boost_python3 python3.6m#（这里我取消了注释，并且修改了内容，重新编译boost python3，后面讲）
# PYTHON_INCLUDE := /usr/include/python3.5m \
# /usr/lib/python3.5/dist-packages/numpy/core/include
# We need to be able to find libpythonX.X.so or .dylib.
#PYTHON_LIB := /usr/lib#（把这里注释掉）
PYTHON_LIB := $(ANACONDA_HOME)/lib#（修改为我这里的位置）
# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib
# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1
# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1
# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1
# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
# enable pretty build (comment to see full commands)
Q ?= @
LINKFLAGS := -Wl,-rpath,$(HOME)/anaconda3/lib#（这里是由于会出现一个关于png的错误，我后面讲）
按照流程接下来的配置应该是：
准备好了.
make pycaffe -j 4
另外, 这个make默认是用CPU单核运算,如果想要快一点, 比如我想使用四核, 在make后面加上-j4标签.
如果上面4行某一行报错之后想要重试,建议先make clean再重新开始.
如果这里出现问题，马上 make clean.
配置好了之后 ，make all
全部成功后才算是成功。
2.3 配置环境变量
去到CAFFE文件夹里面的python文件夹, 把当前路径记录下来(pwd). 然后输入以下命令(把记下的路径放在相应地方)
export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH
2.4 大功告成
命令行中输入 python， 然后 import caffe ,基本是错的。接下来讲在编译caffe以及import caffe中出现的各种问题。
3.问题以及解决
3.1 编译时候，cannot find -lboost_python3, python3.6m
原因是这一行中，PYTHON_LIBRARIES := boost_python3 python3.6m，首先我们boost并没有编译3.6版本，其次python3.6m这个link不在 /usr/lib 。
1)，sudo ln -s /home/horsetif/anaconda3/bin/python3 /usr/lib/python3.6m(这里是建立一个软链接，把我本地目录下的python3.6加到usr/lib/中去。
2)，编译3.6版本的libboost_python3.so
首先去/usr/lib/x86_64-linux-gnu目录下查看是否有python3版本的libboost，如果有类似libboost_python35.so但是没有libboost_python3.so则需要手动建立连接。
方法为：
sudo ln-slibboost_python-py35.so libboost_python3.so
这个方法只是针对与3.5版本，现在我们重新编译，然后把编译好的结果的软链接加到/usr/lib/x86_64-linux-gnu这个目录里面。
1.首先下载安装包，然后解压， 切换目录
wget -O boost_1_67_0.tar.gz http://sourceforge.net/projects/boost/files/boost/1.67.0/boost_1_67_0.tar.gz/download
tar xzvf boost_1_67_0.tar.gz
cd boost_1_67_0/
安装包也可以在http://www.boost.org/users/history/version_1_67_0.html下载
2.安装附加依赖库
sudo apt-get update
sudo apt-get install build-essential g++ python-dev autotools-dev libicu-dev build-essential libbz2-dev libboost-all-dev
linux系统自带python2.7，boost编译后会生成libboost_python27.so，但有些应用需要libboost_python3，下面我们就来编译。
进入boost源文件目录boost_1_67_0，配置仅仅编译python，python路径指向anaconda3中的python3.6。
./bootstrap --with-libraries=python --with-toolset=gcc
./b2 --with-python include="/home/horsetif/anaconda3/include/python3.6m/"
sudo ./b2 install
编译安装成功后，/usr/local/lib下会有libboost_python36.so和libboost_python36.a，有些应用link时需要的是libboost_python3.so或者libboost_python3.a，我们建个软链：
cd /usr/local/lib
sudo ln -s libboost_python-py36.so libboost_python3.so
sudo ln -s libboost_python-py36.a libboost_python3.a
这时，我们要把/usr/local/lib 中，相关文件，建立相对于名称的软链接到 /usr/lib/x86_64-linux-gnu中。
sudo cp /usr/local/lib/libboost_python36.a /usr/lib/x86_64-linux-gnu/libboost_python_python36.a
sudo cp /usr/local/lib/libboost_python36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python3.so
3.2 多个opencv问题
错误：make pycaffe可以通过，但是make all时出现错误，具体错误我没有记录，但是好像是因为系统装了多个opencv的原因，不管怎么说，至少可以在python环境下import caffe了
make all时出现的错误：
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector >&, std::vector > const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
collect2: error: ld returned 1 exit status
Makefile:625: recipe for target '.build_release/tools/upgrade_net_proto_text.bin' failed
make: *** [.build_release/tools/upgrade_net_proto_text.bin] Error 1
make: *** Waiting for unfinished jobs....
解决办法：也是，的确在这之前我分别在python 3.5和python 2.7下面装了opencv，而且该教程前面又装了libopencv-dev，所以我就怀疑是这个，但是我尝试着卸载了opencv还是import出现同样的错误，最后不断找解决方案，也尝试了他们的各种解决方法，都不知道make clean然后又make all了多少次，还是不管用。最后在网站https://github.com/BVLC/caffe/issues/2348#issuecomment-95156848看到有人说”Uncomment
if you're using OpenCV 3”于是我在配置文件中去掉 ”OPENCV_ VERSION :=3“的注释，我又make clean，接着make all,奇迹终于出现了，上面的错误消失了,居然make all成功了。
3.3 matplotlib 问题
3、错误：现在可以import caffe了，但是又出现了以下错误：
>>> import caffe
Traceback (most recent call last):
File "", line 1, in
File "/home/.../Downloads/caffe-master/python/caffe/__init__.py", line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer
File "/home/.../Downloads/caffe-master/python/caffe/pycaffe.py", line 15, in
import caffe.io
File "/home/..../Downloads/caffe-master/python/caffe/io.py", line 2, in
import skimage.io
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/__init__.py", line 15, in
reset_plugins()
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 93, in reset_plugins
_load_preferred_plugins()
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 73, in _load_preferred_plugins
_set_plugin(p_type, preferred_plugins['all'])
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 85, in _set_plugin
use_plugin(plugin, kind=plugin_type)
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 255, in use_plugin
_load(name)
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 299, in _load
fromlist=[modname])
File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/_plugins/matplotlib_plugin.py", line 3, in
import matplotlib.pyplot as plt
File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/pyplot.py", line 36, in
from matplotlib.figure import Figure, figaspect
File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/figure.py", line 40, in
from matplotlib.axes import Axes, SubplotBase, subplot_class_factory
File "/home/..../Downloads/yes/lib/python3.5/site-packages/matplotlib/axes/__init__.py", line 4, in
from ._subplots import *
File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 10, in
from matplotlib.axes._axes import Axes
File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 23, in
import matplotlib.dates as _ # <-registers a date unit converter
File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/dates.py", line 126, in
from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, MONTHLY, WEEKLY, DAILY, HOURLY, MINUTELY, SECONDLY)
File "/home/.../Downloads/yes/lib/python3.5/site-packages/dateutil/rrule.py", line 55
raise ValueError, "Can't create weekday with n == 0"
^
SyntaxError: invalid syntax
错误解决办法：从上面可以看出也就是在 import matplotlib.pyplot as plt 时出现错误，百度了以下，尝试了几种解决方案，原来是matplotlib版本太低了，需要升级，于是我pip3 install matplotlib --upgrade 了以下，果然解决了，但是又出现了下面的问题。
3.4 numpy 问题
错误：Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
解决方案：继续百度了下，哈哈，原来是因为numpy版本过低导致，于是我 conda update numpy ，哈哈又成功了。。所以，我的anaconda里边的包必须得升级啊，里边的包都还是安装anaconda时系统自带的，太老了。
3.5 错误信息
//home/idc/anaconda3/lib/libpng16.so.16: undefined reference to `inflateValidate@ZLIB_1.2.9'
后面还有几行，主要是这个。
我的解决方法是在Makefile.config加入下列信息：
LINKFLAGS := -Wl,-rpath,$(HOME)/anaconda3/lib
3.6 错误信息
可能会出现下面的错误：
Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "", line 1, in
File "/home/david/download/caffe-master/python/caffe/__init__.py", line 4, in
from .proto.caffe_pb2 import TRAIN, TEST
File "/home/david/download/caffe-master/python/caffe/proto/caffe_pb2.py", line 7, in
from google.protobuf import reflection as _reflection
File "/home/david/opt/anaconda3/lib/python3.5/site-packages/google/protobuf/reflection.py", line 68, in
from google.protobuf.internal import python_message
File "/home/david/opt/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 848
except struct.error, e:
^
SyntaxError: invalid syntax
这是因为 protobuf 不支持 python3，解决方案是安装pip install protobuf-py3, 一个 python3.x 版本的替代包。‘’
4.错误代码总汇：
opencv 配置问题
make all
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector>&, std::vector > const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
collect2: error: ld returned 1 exit status
Makefile:625: recipe for target '.build_release/tools/compute_image_mean.bin' failed
make: *** [.build_release/tools/compute_image_mean.bin] Error 1
python版本与编译caffe版本问题
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: /home/horsetif/Software/caffe/python/caffe/../../build/lib/libcaffe.so.1.0.0: undefined symbol: _ZN2cv6imreadERKNS_6StringEi
File "/home/horsetif/Software/caffe/python/caffe/pycaffe.py", line 13, in from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: dynamic module does not define module export function (PyInit__caffe)
boost 配置问题
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
/usr/bin/ld: cannot find -lboost_python3
collect2: error: ld returned 1 exit status
Makefile:507: recipe for target 'python/caffe/_caffe.so' failed
make: *** [python/caffe/_caffe.so] Error 1
boost修改好了之后没有放对位置
File "/home/horsetif/Software/caffe/python/caffe/pycaffe.py", line 13, in from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: libboost_python36.so.1.67.0: cannot open shared object file: No such file or directory
matplotlib 问题
from matplotlib.axes._axes import Axes File "/home/horsetif/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 23, inimport matplotlib.dates as _ # <-registers a date unit converter File "/home/horsetif/anaconda3/lib/python3.6/site-packages/matplotlib/dates.py", line 148, in from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,
File "/home/horsetif/anaconda3/lib/python3.6/site-packages/dateutil/rrule.py", line 55
raise ValueError, "Can't create weekday with n == 0"
^
SyntaxError: invalid syntax
网友评论