训练笔记 Fine tune my first semantic

正如Deeplab-v2所做的一样，每经过一定的iteration对learning rate进行调整，无论是使用step policy还是poly policy，都需要由当前迭代次数来决定。翻了各种文档和issue，总结出在mxnet中，目前有两种方式可以实现。
参考:
https://github.com/dmlc/mxnet/issues/540
https://github.com/dmlc/mxnet/issues/625

• 1 使用mxnet/python/mxnet/lr_scheduler下提供的方法：
  具体可以参考example/image_classification下的fit.py中，在mx.model.Feedforward中给出lr_scheduler参数。关于如何实现lr_scheduler的例子：

def _get_lr_scheduler(args, kv):
    if 'lr_factor' not in args or args.lr_factor >= 1:        
      return (args.lr, None)    
    epoch_size = args.num_examples / args.batch_size    
    if 'dist' in args.kv_store:        
      epoch_size /= kv.num_workers    
    begin_epoch = args.load_epoch if args.load_epoch else 0    
    step_epochs = [int(l) for l in args.lr_step_epochs.split(',')]    
    lr = args.lr    
    for s in step_epochs:        
      if begin_epoch >= s:            
        lr *= args.lr_factor    
    if lr != args.lr:        
      logging.info('Adjust learning rate to %e for epoch %d' %(lr, begin_epoch))    
  
    steps = [epoch_size * (x-begin_epoch) for x in step_epochs if x-begin_epoch > 0]    
    return (lr, mx.lr_scheduler.MultiFactorScheduler(step=steps, factor=args.lr_factor))

如何调用：

  lr, lr_scheduler = _get_lr_scheduler(args, kv)

  # create model
  model = mx.model.FeedForward(    
          ......
          learning_rate = lr,    lr_scheduler  = lr_scheduler,    
          ......  )

或者在lenet中：

model = mx.model.FeedForward(ctx=dev, symbol=lenet, num_epoch=20,  
                                                    learning_rate=0.05, momentum=0.9, wd=0.00001, 
                                                    lr_scheduler=mx.misc.FactorScheduler(step=5))

但是由于我使用的不是feedforward方法，而是bind和fit，且feedforward对应的是single data-single output的结构，所以这种方法并不适用于我。

• 2 直接在fit函数中自己写一个epoch_end_callback：

  
  Learning rate policy adopted by Deeplab-v2

  # my code
  def lr_callback(epoch, net, net_args, net_auxs):
    opt.lr = args.base_lr * (1 - float(epoch)/num_epoch)**0.9

  mod.fit(train, val,
        batch_end_callback =   mx.callback.Speedometer(batch_size, 30)
        epoch_end_callback = [checkpoint, lr_callback],
        optimizer = opt, num_epoch = num_epoch)

• 3 在optimizer中给出lr_scheduler，我的实现如下：

opt = mx.optimizer.create('sgd', learning_rate=args.base_lr,
                                          momentum=0.9, wd=0.0005,
                                          rescale_grad=float(1)/len(contexts),
                                          lr_scheduler=mx.lr_scheduler.MultiFactorScheduler([10,20,30], 0.9))
# the first parameter of scheduler is the list of all those changing point

lr_scheduler.MultiFactorScheduler

在step那里需要用np.linspace()的话记得要转成list()，函数不接受ndarray的类型。如果需要更灵活的话，可以自己在lr_scale.py写一个满足需求的scheduler。