Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network

This paper is published in CVPR 2017
Paper link
Github repository

Install code

** Follow the instructions on the github repo readme page. ↓↓↓↓↓ **

//Green letters and comments are my own words, others are all from README.MD

Data requirements

Before running the code, please, make sure to have all the required data in the following specific folder:

Download our CNN and move the CNN model (3 files: 3dmm_cnn_resnet_101.caffemodel,deploy_network.prototxt,mean.binaryproto) into the CNN folder
Download the Basel Face Model and move 01_MorphableModel.mat into the 3DMM_model folder
Acquire 3DDFA Expression Model, run its code to generate Model_Expression.mat and move this file the 3DMM_model folder

//After running the code, I only got Model_Shape.mat, but I found a Model_Expression.mat in the folder where it should be generated. It seems that this file comes with the original download. So I upload that one to the 3DMM_model folder. (Anyway, this file seems unnecessary to the demo code.)

Go into 3DMM_model folder. Run the script python trimBaselFace.py. This should output 2 files BaselFaceModel_mod.mat and BaselFaceModel_mod.h5.
Download dlib face prediction model and move the .dat file into the dlib_model folder.

Installation

If you don't have apt-get install, use yum install instead.

Install cmake:

$ apt-get install cmake
$ #yum install cmake

Install opencv (2.4.6 or higher is recommended):

(http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html)

Install libboost (1.5 or higher is recommended):

$ apt-get install libboost-all-dev
$ #yum install boost-devel

Install OpenGL, freeglut, and glew

$ sudo apt-get install freeglut3-dev
$ #yum install freeglut-devel

$ sudo apt-get install libglew-dev
$ #yum install glew-devel   #not sure, may need to build from source

Install libhdf5-dev library

$ sudo apt-get install libhdf5-dev
$ #yum install hdf5-devel

Install Dlib C++ library

$ (http://dlib.net/)

To get libdlib.so, do not follow the instructions on the offical website for Dlib. Use the following commands, which are originally from stackoverflow.

$ cd dlib
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install

libdlib.so will appear in build foler.

Update Dlib directory paths (DLIB_INCLUDE_DIR and DLIB_LIB_DIR) in CMakeLists.txt

DLIB_INCLUDE_DIR = /path/to/dlib-19.4
DLIB_LIB_DIR = /path/to/dlib-19.4/dlib/build

Make build directory (temporary). Make & install to bin folder

$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=../bin ..
$ make
$ make install

This code should generate TestVisualization in bin folder

Usage

3DMM fitting on a set of input images

Go into demoCode folder. The demo script can be used from the command line with the following syntax:

$ Usage: python testBatchModel.py <inputList> <outputDir> <needCrop> <useLM>

where the parameters are the following:

<inputList> is a text file containing the paths to each of the input images, one in each line.
<outputDir> is the path to the output directory, where ply files are stored.
<needCrop> tells the demo if the images need cropping (1) or not (0). Default 1. If your input image size is equal (square) and has a CASIA-like [2] bounding box, you can set <needCrop> as 0. Otherwise, you have to set it as 1.
<useLM> is an option to refine the bounding box using detected landmarks (1) or not (0). Default 1.

Example for <inputList>:
<pre>
data/1.jpg
data/2.jpg
....
</pre>

Other errors I have encountered

import caffe error: add caffe path to PYTHONPATH

$ export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH

"xxx" is not a member of cv: add corresponding header files

#include "opencv2/calib3d/calib3d.hpp"
#include "opencv2/imgproc/imgproc.hpp"

//my markdown doesn't support cpp :(

import dlib error: Go to the base folder of the dlib repository and run python setup.py install or pip install dlib. I don't know which one works.
libcudart.so.8.0: cannot open shared object file: No such file or directory: I failed on this. _(:3」∠)_

After words: I finally managed to run the demo with two servers. I generated TestVisualization on a RedHat server. But I can't run the demo code. I copied TestVisualization to a Ubuntu server, with some other preparations, it works, miraculously!