How to implement "random mini-ba

def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):
    """
    Creates a list of random minibatches from (X, Y)
    
    Arguments:
    X -- input data, of shape (input size, number of examples) (m, Hi, Wi, Ci)
    Y -- true "label" vector (containing 0 if cat, 1 if non-cat), of shape (1, number of examples) (m, n_y)
    mini_batch_size - size of the mini-batches, integer
    seed -- this is only for the purpose of grading, so that you're "random minibatches are the same as ours.
    
    Returns:
    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
    """
    
    m = X.shape[0]                  # number of training examples
    mini_batches = []
    np.random.seed(seed)
    
    # Step 1: Shuffle (X, Y)
    permutation = list(np.random.permutation(m))
    shuffled_X = X[permutation,:,:,:]
    shuffled_Y = Y[permutation,:]

    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
    # number of mini batches of size mini_batch_size in your partitionning
    num_complete_minibatches = math.floor(m/mini_batch_size) 
    for k in range(0, num_complete_minibatches):
        mini_batch_X = shuffled_X[k * mini_batch_size : k * mini_batch_size + mini_batch_size,:,:,:]
        mini_batch_Y = shuffled_Y[k * mini_batch_size : k * mini_batch_size + mini_batch_size,:]
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
    
    # Handling the end case (last mini-batch < mini_batch_size)
    if m % mini_batch_size != 0:
        mini_batch_X = shuffled_X[num_complete_minibatches * mini_batch_size : m,:,:,:]
        mini_batch_Y = shuffled_Y[num_complete_minibatches * mini_batch_size : m,:]
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
    
    return mini_batches

Before getting into the idea of what is and how to mini-batch, we need to first understand what is stochastic gradient descent, batch gradient descent and finally min-batch gradient descent.

• Stochastic gradient descent, often abbreviated SGD, is a variation of the gradient descent algorithm that calculates the error and updates the model for each example in the training dataset.

  • Upsides
  1. The frequent updates immediately give an insight into the performance of the model and the rate of improvement.
  2. This variant of gradient descent may be the simplest to understand and implement, especially for beginners.
  3. The increased model update frequency can result in faster learning on some problems.
  4. The noisy update process can allow the model to avoid local minima (e.g. premature convergence).
  • Downsides
  1. Updating the model so frequently is more computationally expensive than other configurations of gradient descent, taking significantly longer to train models on large datasets.
  2. The frequent updates can result in a noisy gradient signal, which may cause the model parameters and in turn the model error to jump around (have a higher variance over training epochs).
  3. The noisy learning process down the error gradient can also make it hard for the algorithm to settle on an error minimum for the model.

• Batch gradient descent is a variation of the gradient descent algorithm that calculates the error for each example in the training dataset, but only updates the model after all training examples have been evaluated.

  One cycle through the entire training dataset is called a training epoch. Therefore, it is often said that batch gradient descent performs model updates at the end of each training epoch.

  • Upsides
  1. Fewer updates to the model means this variant of gradient descent is more computationally efficient than stochastic gradient descent.
  2. The decreased update frequency results in a more stable error gradient and may result in a more stable convergence on some problems.
  3. The separation of the calculation of prediction errors and the model update lends the algorithm to parallel processing based implementations.
  • Downsides
  1. The more stable error gradient may result in premature convergence of the model to a less optimal set of parameters.
  2. The updates at the end of the training epoch require the additional complexity of accumulating prediction errors across all training examples.
  3. Commonly, batch gradient descent is implemented in such a way that it requires the entire training dataset in memory and available to the algorithm.
  4. Model updates, and in turn training speed, may become very slow for large datasets.

• Finally, we have mini-batch gradient descent

• Mini-batch gradient descent is a variation of the gradient descent algorithm that splits the training dataset into small batches that are used to calculate model error and update model coefficients.

  Implementations may choose to sum the gradient over the mini-batch or take the average of the gradient which further reduces the variance of the gradient.

  Mini-batch gradient descent seeks to find a balance between the robustness of stochastic gradient descent and the efficiency of batch gradient descent. It is the most common implementation of gradient descent used in the field of deep learning.

  • Upsides
    1. The model update frequency is higher than batch gradient descent which allows for a more robust convergence, avoiding local minima.
    2. The batched updates provide a computationally more efficient process than stochastic gradient descent.
    3. The batching allows both the efficiency of not having all training data in memory and algorithm implementations.
    • Downsides
    1. Mini-batch requires the configuration of an additional “mini-batch size” hyperparameter for the learning algorithm.
    2. Error information must be accumulated across mini-batches of training examples like batch gradient descent.

• 简单来说，在训练神经网络的时候我们经常涉及到如何安排数据输入及梯度下降运算，常用的方法有随机梯度，批次梯度和迷你批次梯度等。

  • 随机梯度和批次梯度的方法对于实现来说较为简单并容易理解，主要方法是随机梯度（每次输入一个训练样本然后立即计算失误率并更新参数），批次梯度（每次输入一个训练样本然后立即计算失误率但是并不更新参数，而是等所有的训练样子都完整输入并计算失误后再对参数进行求解和更新）。

    前者实现较为容易，但是计算曲线会呈现杂乱跳动的现象，因为每次只计算一个样本，每个样本之间存在的差异变化不同，有时非常巨大，有时非常相近，所以训练时计算成本偏高，收敛慢。

  • 迷你批次梯度的计算方法较为复杂，首先需要将所有的输入数据细分为相对应的份数，然后每次输入其中一份进行失误率和梯度的计算并更新参数，难点在于如何划分输入数据，需要注意无法完整划分的问题及多余数据的处理 见以下代码片。

num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
    for k in range(0, num_complete_minibatches):
        mini_batch_X = shuffled_X[k * mini_batch_size : k * mini_batch_size + mini_batch_size,:,:,:]
        mini_batch_Y = shuffled_Y[k * mini_batch_size : k * mini_batch_size + mini_batch_size,:]
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
    
    # Handling the end case (last mini-batch < mini_batch_size)
    if m % mini_batch_size != 0:
        mini_batch_X = shuffled_X[num_complete_minibatches * mini_batch_size : m,:,:,:]
        mini_batch_Y = shuffled_Y[num_complete_minibatches * mini_batch_size : m,:]
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)

虽然迷你批次梯度处理过程比较复杂但是计算效率比其他方法高，所以我们在处理数据输入时常用这种方法。