美文网首页
CentOS 7 深度学习平台新用户导引

CentOS 7 深度学习平台新用户导引

作者: izhangxm | 来源:发表于2017-05-09 21:53 被阅读0次

    版本更新说明

    v1.3 20170630

    • 增加Matlab2012b、Matlab2014b、Matlab2016b菜单启动快捷方式
    • 安装PyCharm专业版并附有激活服务器,菜单栏可启动
    • 激活服务器可激活Jetbrain公司旗下所有所有专业产品

    v1.2 20170523
    增加Matlab2012,Matlab2014支持,现在已经安装Matlab2012、2014、2016三个版本

    v1.1 20170511
    增加Torch环境的支持

    v1.0 20170509
    主要内容有新用户必读,基础设置,caffe的搭建过程,TensorFlow搭建等

    前段时间在机房服务器搭建了基础深度学习平台,搭建全过程见CentOS 7.3 Caffe搭建全过程记录

    很多环境与软件已经安装完毕,ssh登录后就会显示相应的信息,如下图所示

    系统信息

    使用之前的几点重要说明:
    用户权限问题
    为了系统正常运行和其他用户环境的稳定性,不提供root权限给普通用户,如果你需要root权限安装系统级别软件,请联系我或者自己再搭建一台服务器

    支持的环境
    目前可以为Caffe和TensorFlow学习框架提供支持,两个框架均可以在不需要root权限条件下编译成功并运行,这当然要讲究点技巧和使用规范,下面会有说明。更多框架的支持请留意文章开头的更新说明

    软件版本问题
    Python目前有2.7.13(默认)和3.6.1两种版本存在,视个人需求而定,这个在创建python虚拟环境是有了更多的选择。Matlab为2016b,只有一个版本,敬请谅解,后期可能会增加2014版,并解决版本冲突问题。

    Xmanager可用
    大部分同学都需要使用图形话界面。服务器已经配置好,在windows端可直接连接。

    关于环境的说明
    初始环境已经配置好了PIP的镜像及virtualenvwrapper,在你的桌面上已经放好了最新的Caffe,接下来就要编译它。

    如果使用到了python环境,在开始之前你需要移步Python Virtual Environments虚拟环境导引,找另外一台练习机学会以后再往下读,这点很重要,请务必学会,请务必学会,请务必学会

    好,假设你已经会使用python的virtual env 了,那么请往下看
    用户以simon为例,已经创建好

    连接到服务器并配置基本环境

    在windows端,打开xshell(推荐你使用)或XManager连接服务器后打开shell

    登录后会显示系统信息,并要求你更改密码,这个密码必须比较复杂才能被设置。密码更改成功后会强制退出,再次使用ssh登录即可。
    显示显示系统信息,紧接着,virtualenvwrapper初始化

    • 配置上网代理
      首先我们需要在windows端使用ccproxy这个软件搭建起代理服务器,记住代理服务器的地址和端口号,别忘记设置客户端访问权限。接下来设置远程服务器的代理
    vi ~/.bashrc
    #修改export http_proxy https_proxy ftp_proxy 为你自己的代理服务器地址和端口
    
    • 创建virtualenvironment ,名字为caffe-master
    mkvirtualenv caffe-master # 默认使用2.7版本
    

    好了,我们已经在虚拟环境中了,如下图所示。

    image.png

    PyCharm的使用

    管理员已经为所有用户安装了最新的PyCharm专业版(2017.1.4),可从开始菜单快捷启动
    要求激活时,选择服务器激活

    image.png

    点击激活即可
    需要注意的是,激活程序切忌不可商用盈利
    实验室的同学可使用地址

    http://192.168.69.164:9714
    http://192.168.69.68:9714
    

    激活服务器可激活Jetbrain公司旗下的所有收费产品,包括PyCharm、PhpStorm、Intelj Idea、WebStorm等,其他产品及详细信息https://www.jetbrains.com/products.html

    激活服务器由Lanyu发布,由于涉及到版权及商业利益,暂不公开发布,如果你需要的话,请私信我。

    PyCharm的配置过程有兴趣的同学请移步本人博客CentOS 7 为所有用户安装PyCharm并激活

    编译caffe

    如果你有使用caffe的需求,请参照一下步骤,否则请跳过
    解压并修改caffe

    cd ~/Desktop
    unzip caffe-master.zip
    

    如果你的虚拟环境的python版本是2.7,请执行下面的操作

    cd caffe-master/python
    vi requirements.txt
    #修改ipython>=3.0.0 为 ipython==5.3.0
    

    安装python依赖

    cd caffe-master/python #进入python目录
    pip install -r requirements.txt
    

    修改Makefile.config文件

    cd caffe-master
    cp Makefile.config.example Makefile.config
    vi Makefile.config
    #修改内容如下
    
    ## Refer to http://caffe.berkeleyvision.org/installation.html
    # Contributions simplifying and improving our build system are welcome!
    
    # cuDNN acceleration switch (uncomment to build with cuDNN).
    USE_CUDNN := 1  ##使用CUDNN
    
    # CPU-only switch (uncomment to build without GPU support).
    # CPU_ONLY := 1
    
    # uncomment to disable IO dependencies and corresponding data layers
    # USE_OPENCV := 0
    # USE_LEVELDB := 0
    # USE_LMDB := 0
    
    # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
    #   You should not set this flag if you will be reading LMDBs with any
    #   possibility of simultaneous read and write
    # ALLOW_LMDB_NOLOCK := 1
    
    # Uncomment if you're using OpenCV 3
    OPENCV_VERSION := 3 ##指定opencv的版本
    
    # To customize your choice of compiler, uncomment and set the following.
    # N.B. the default for Linux is g++ and the default for OSX is clang++
    # CUSTOM_CXX := g++
    
    # CUDA directory contains bin/ and lib/ directories that we need.
    CUDA_DIR := /usr/local/cuda
    # On Ubuntu 14.04, if cuda tools are installed via
    # "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
    # CUDA_DIR := /usr
    
    # CUDA architecture setting: going with all of them.
    # For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
    # For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
    ##删除多余的架构
    CUDA_ARCH := -gencode arch=compute_30,code=sm_30 
            -gencode arch=compute_35,code=sm_35 \
            -gencode arch=compute_50,code=sm_50 \
            -gencode arch=compute_52,code=sm_52 \
            -gencode arch=compute_60,code=sm_60 \
            -gencode arch=compute_61,code=sm_61 \
            -gencode arch=compute_61,code=compute_61
    
    # BLAS choice:
    # atlas for ATLAS (default)
    # mkl for MKL
    # open for OpenBlas
    BLAS := open ##使用open-blas
    # Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
    # Leave commented to accept the defaults for your choice of BLAS
    # (which should work)!
    BLAS_INCLUDE := /usr/include/openblas ##指定blas的头文件路径
    # BLAS_LIB := /path/to/your/blas
    
    # Homebrew puts openblas in a directory that is not on the standard search path
    # BLAS_INCLUDE := $(shell brew --prefix openblas)/include
    # BLAS_LIB := $(shell brew --prefix openblas)/lib
    
    # This is required only if you will compile the matlab interface.
    # MATLAB directory should contain the mex binary in /bin.
    MATLAB_DIR := /usr/local/MATLAB/R2016b ## Matlab的路径
    # MATLAB_DIR := /Applications/MATLAB_R2012b.app 
    
    # NOTE: this is required only if you will compile the python interface.
    # We need to be able to find Python.h and numpy/arrayobject.h.
    #PYTHON_INCLUDE := /usr/include/python2.7 /usr/lib/python2.7/dist-packages/numpy/core/include
    ## 修改为你的自己的路径,基本上就是用户名那里变一下
    PYTHON_INCLUDE := /home/simon/Envs/caffe-master/include/python2.7 /home/simon/Envs/caffe-master/lib/python2.7/site-packages/numpy/core/include
    # Anaconda Python distribution is quite popular. Include path:
    # Verify anaconda location, sometimes it's in root.
    # ANACONDA_HOME := $(HOME)/anaconda
    # PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
            # $(ANACONDA_HOME)/include/python2.7 \
            # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include
    
    # Uncomment to use Python 3 (default is Python 2)
    # PYTHON_LIBRARIES := boost_python3 python3.5m
    # PYTHON_INCLUDE := /usr/include/python3.5m \
    #                 /usr/lib/python3.5/dist-packages/numpy/core/include
    
    # We need to be able to find libpythonX.X.so or .dylib.
    PYTHON_LIB := /usr/lib
    # PYTHON_LIB := $(ANACONDA_HOME)/lib
    
    # Homebrew installs numpy in a non standard path (keg only)
    # PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
    # PYTHON_LIB += $(shell brew --prefix numpy)/lib
    
    # Uncomment to support layers written in Python (will link against Python libs)
    WITH_PYTHON_LAYER := 1 ## 打开pyhton层支持
    
    # Whatever else you find you need goes here.
    INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
    LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
    
    # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
    # INCLUDE_DIRS += $(shell brew --prefix)/include
    # LIBRARY_DIRS += $(shell brew --prefix)/lib
    
    # NCCL acceleration switch (uncomment to build with NCCL)
    # https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
    # USE_NCCL := 1
    
    # Uncomment to use `pkg-config` to specify OpenCV library paths.
    # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
    # USE_PKG_CONFIG := 1
    
    # N.B. both build and distribute dirs are cleared on `make clean`
    BUILD_DIR := build
    DISTRIBUTE_DIR := distribute
    
    # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
    # DEBUG := 1
    
    # The ID of the GPU that 'make runtest' will use to run unit tests.
    TEST_GPUID := 0
    
    # enable pretty build (comment to see full commands)
    Q ?= @
    
    

    为protoc打补丁

    cd caffe-master
    protoc src/caffe/proto/caffe.proto --cpp_out=.
    mkdir include/caffe/proto
    mv src/caffe/proto/caffe.pb.h include/caffe/proto
    

    编译

    make -j24
    make pycaffe
    make matcaffe
    make test -j24 #可选操作
    make runtest -j8 #可选操作
    

    成功后如下图所示

    image.png
    验证
    cd caffe-master/python
    ipython
    import caffe #如果没问题就成功了
    
    验证caffe

    安装TensorFlow(教程太老,已废弃)

    TensorFlow是作为python的第三方包发布的,利用包管理工具pip即可完成安装,只是需要注意版本的选择就可以了。移步CentOS 7 搭建Tensorflow环境 获取更详细的信息
    以TensorFlow r0.12版本为例进行说明

    去官网查询安装地址

    • 官网
    • 选择版本号


      选择版本号
    • 找到地址


      就是图中的地址啦

    安装
    注意确保你在虚拟环境中

    pip install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.0rc1-cp27-none-linux_x86_64.whl
    

    验证

    ipython
    import tensorflow
    

    安装Torch

    一定要确保你在自己创建的Python虚拟环境中,并且已经安装了ipython

    安装

    # in a terminal, run the commands WITHOUT sudo
    git clone https://github.com/torch/distro.git ~/torch --recursive
    cd ~/torch; 
    ./install.sh
    source ~/.bashrc
    

    官网网站的指导中还有bash install-deps;,这句命令的作用是安装Torch7所需要的依赖,由于管理员已经安装过,所以,普通用户就不用安装了

    验证

    (simon_main) [zhangxinming@localhost torch]$ th
     
      ______             __   |  Torch7 
     /_  __/__  ________/ /   |  Scientific computing for Lua. 
      / / / _ \/ __/ __/ _ \  |  Type ? for help 
     /_/  \___/_/  \__/_//_/  |  https://github.com/torch 
                              |  http://torch.ch 
        
    th> torch.Tensor{1,2,3}
     1
     2
     3
    [torch.DoubleTensor of size 3]
    th> 
    

    更多Torch的用法及知识请移步官方教程

    Matlab的使用

    启动Matlab非常简单,在图形环境中打开终端,输入

    matlab #默认启动2016b版本
    
    matlab2012 #启动2012版本
    matlab2014 #启动2014版本
    matlab2016 #启动2016版本
    

    v1.3版本后,增加了菜单栏启动的方式

    image.png

    相关文章

      网友评论

          本文标题:CentOS 7 深度学习平台新用户导引

          本文链接:https://www.haomeiwen.com/subject/pvwitxtx.html