ubuntu编译tensorflow2.1-gpu

Setup

环境：Ubuntu 18.04.3 LTS, CUDA 10.2, cuDNN 7.5, Python 3.6.9, Bazel 0.29.1

# 创建虚拟环境
$ python3 -m build-tf
$ source build-tf/bin/activate

# 安装一些依赖包(这里有些坑)
(build-tf) $ pip install -U pip numpy wheel
(build-tf) $ pip install -U keras_preprocessing --no-deps

Install Bazel

# 使用[bazelisk](https://github.com/bazelbuild/bazelisk)
(build-tf) $ wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-amd64
(build-tf) $ chmod +x /usr/local/bin/bazel
(build-tf) $ export USE_BAZEL_VERSION=0.29.1
(build-tf) $ export TMP=/tmp

Configure the build

# 用国内的gitee镜像仓库，github基本没可能能拉下来
(build-tf) $ git clone https://gitee.com/mirrors/tensorflow.git
(build-tf) $ cd tensorflow
# 切换到tag v2.1.0
(build-tf) $ git checkout v2.1.0
(build-tf) $ python configure.py
# CUDA support选Y，其他回车跳过
Please specify the location of python. [Default is /usr/bin/python3]: 


Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.6/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: 
No OpenCL SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with ROCm support? [y/N]: 
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: Y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: 
No TensorRT support will be enabled for TensorFlow.

Found CUDA 10.1 in:
    /usr/local/cuda-10.1/targets/x86_64-linux/lib
    /usr/local/cuda-10.1/targets/x86_64-linux/include
Found cuDNN 7 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 6.1


Do you want to use clang as CUDA compiler? [y/N]: 
nvcc will be used as CUDA compiler.

Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: 


Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]: 


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: 
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
    --config=mkl            # Build with MKL support.
    --config=monolithic     # Config for mostly static monolithic build.
    --config=ngraph         # Build with Intel nGraph support.
    --config=numa           # Build with NUMA support.
    --config=dynamic_kernels    # (Experimental) Build kernels into separate shared objects.
    --config=v2             # Build TensorFlow 2.x instead of 1.x.
Preconfigured Bazel build configs to DISABLE default on features:
    --config=noaws          # Disable AWS S3 filesystem support.
    --config=nogcp          # Disable GCP support.
    --config=nohdfs         # Disable HDFS support.
    --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished

Build the pip package

### 开始编译
(build-tf) $ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Build and Install the package

(build-tf) $ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
(build-tf) $ /tmp/tensorflow_pkg/tensorflow-2.1.0-cp36-cp36m-linux_x86_64.whl

一些巨坑

• 科学上网和git代理，编译过程中一直在git fetch，没有外网环境几乎寸步难行
• 有些包就是拉不下来，比如mkl和llvm。拿到url之后用电脑或者手机想尽办法下载下来，然后搭个文件服务器放上去，然后把tensorflow/tensorflow/workspace.bzl中对应repository的urls加上自己文件服务器的链接，比如：

    mkl_repository(
        name = "mkl_linux",
        build_file = clean_dep("//third_party/mkl:mkl.BUILD"),
        sha256 = "a936d6b277a33d2a027a024ea8e65df62bd2e162c7ca52c48486ed9d5dc27160",
        strip_prefix = "mklml_lnx_2019.0.5.20190502",
        urls = [
            "http://47.96.129.56/mklml_lnx_2019.0.5.20190502.tgz",
            "https://storage.googleapis.com/mirror.tensorflow.org/github.com/intel/mkl-dnn/releases/download/v0.21/mklml_lnx_2019.0.5.20190502.tgz",
            "https://github.com/intel/mkl-dnn/releases/download/v0.21/mklml_lnx_2019.0.5.20190502.tgz",
        ],
    )

• C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed (Exit 1)错误，参考issue#41061，可能是一些依赖包的问题。

(build-tf) $ pip install 'numpy<1.19.0'
(build-tf) $ pip install -U pip six wheel mock
(build-tf) $ pip install future
(build-tf) $ pip install keras_applications==1.0.8 --no-deps
(build-tf) $ pip install keras_preprocessing==1.1.2 --no-deps
(build-tf) $ pip install pandas

编译成功界面

tf编译成功纪念.png