深度学习分布式训练 - 以FireCaffe为例

这篇文章写于2015年，最近翻出来再看，其实江山哪曾变一分。

论文

Forrest N. Iandola etc., FireCaffe: near-linear acceleration of deep neural network training on computer clusters 2016.1

Problem statements from data scientists

4 key pain points summarized by Jeff Dean from Google:

• DNN researchers and users want results of experiments quickly.
• There is a "patience threshold": No one wants to wait more than a few days or a week for result.
• This significantly affects scale of problems that can be tackled.

• We sometimes optimize for experiment turnaround time, rather than absolute minimal system resources for performing the experiments

  
  Turn-around time impacts for DL projects

Problem analysis

The speed and scalability of distributed algorithm are almost always limited by the overhead of communicating between servers; DNN training is not an exception to this rule.
So the design focuses on the communication enhancement, including:

• Upgrade to high throughput interconnects, i.e. use high throughput interconnects like IB etc.
• Decrease the data transmission volume while training, which includes:
  • Balance carefully between data parallelism and model parallelism
  • Increase batch size to reduce communication quantity. And identify hyperparameters suitable for large batch size.
  • Communication data quantity balance among nodes to avoid single point dependency.

Key take-aways

Parallelism Scheme: Model parallelism or Data Parallelism

Model parallelism

model parallelism
Each worker gets a subset of the model parameters, and the workers communication by exchanging data gradients and exchanging activations . and data quantity is:

Data parallelism

data parallelism

Each worker gets a subset of the batch, and then the workers communicate by exchanging weight gradient updates , where and data quantity is:

Convolution layer and fully connection layer have different characteristics in data/weight ratio. So they can use different parallelism schemes.

Different model prefers different parallelism scheme

So a basic conclusion is: convolution layers can be fitted into data parallelism, and fc layers can be fitted into model parallelism.
Further more, for more advanced CNNs like GoogLeNet and ResNet etc., we can directly use data parallelism, as this paper is using.

Gradient Aggregation Scheme: Parameter Server or Reduction Tree

One picture to show how parameter server and reduction tree work in data parallelism.

gradient aggregation scheme

Parameter Server

Parameter communication time with regard to worker number in parameter server scheme.

The communication time scales linearly as we increase the number of workers. single parameter server becomes scalability bottleneck.
Microsoft Adam and Google DistBelief relief this issue by defining a poll of nodes taht colelctively behave as a parameter server. The bigger the parameter server hierarchy gets, the more it looks like a reduction tree.

Reduction Tree

The idea is same as allreduce in message passing model. Parameter communication time with regard to worker number in reduction tree scheme.

It scales logrithmatically as the number of workers.

reduce tree scalability

Batch size selection

Larger batch size lead to less frequent communication and therefore enable more scalability in a distributed setting. But for larger batch size, we need identify a suitable hyperparameter setting to maintain the speed and accuracy produced in DNN training.
Hyperparameters includes:

• Initial learning rate
• learning rate update scheme
• weight delay
• momentum
  Weight update rule used, here means iteration index:
  

Learning rate update rule:

On how to get hyperparameters according to batch size, I will write another article for this.

Results

Final results on GPU cluster w/ GoogleNet.

results

More thinkings

• 以上方案基本上是无损的，为了更进一步减少通信开销，大家开始尝试有损的方案，在训练速度和准确度之间进行折衷。典型的有:
  1. Reduce parameter size using 16-bit floating-point - Google
  2. Use 16-bit weights and 8-bit activations.
  3. 1-bit gradients backpropagation - Microsoft
  4. Discard gradients whose numerical values fall below a certain threshold - Amazon
  5. Compress(e.g. using PCA) weights before transmitting
  6. Network pruning/encoding/quantization - Intel, DeePhi
• 使用新的底层技术来减少通信开销
  1. RDMA rather than traditional TCP/IP?

References

1. https://www.slideshare.net/AIFrontiers/jeff-dean-trends-and-developments-in-deep-learning-research