cs230 深度学习 Lecture 2 编程作业： Logis

本文结构：

1. 将 Logistic 表达为 神经网络 的形式
2. 构建模型
   1. 导入包
   2. 获得数据
   3. 并进行预处理： 格式转换，归一化
   4. 整合模型：
      • A. 构建模型
        • a. 初始化参数：w 和 b 为 0
        • b. 前向传播：计算当前的损失
        • c. 反向更新：计算当前的梯度
      • B. 梯度更新求模型参数
      • C. 进行预测
   5. 绘制学习曲线

---

1. 将 Logistic 表达为 神经网络 的形式

本文的目的是要用神经网络的思想实现 Logistic Regression，输入一张图片就可以判断该图片是不是猫。

那么什么是神经网络呢？
可以看我之前写的这篇文章：

什么是神经网络

其中一个很重要的概念，神经元：

再来看 Logistic 模型的表达：

那么把 Logistic 表达为 神经网络 的形式为：

（关于 Logistic 可以看这两篇文章：
Logistic Regression 为什么用极大似然函数
Logistic regression 为什么用 sigmoid ？）

接下来就可以构建模型:

---

2. 构建模型

我们的目的是学习 w 和 b 使 cost function J 达到最小，

方法就是：

• 通过前向传播 (forward propagation) 计算当前的损失，
• 通过反向传播 (backward propagation) 计算当前的梯度，
• 再用梯度下降法对参数进行优化更新 (gradient descent)

关于反向传播可以看这两篇文章：
手写，纯享版反向传播算法公式推导

构建模型，训练模型，并进行预测，包含下面几步：

1. 导入包
2. 获得数据
3. 并进行预处理： 格式转换，归一化
4. 整合模型：
   • A. 构建模型
     • a. 初始化参数：w 和 b 为 0
     • b. 前向传播：计算当前的损失
     • c. 反向更新：计算当前的梯度
   • B. 梯度更新求模型参数
   • C. 进行预测
5. 绘制学习曲线

下面进入详细代码：

---

1. 导入包

引入需要的 packages，
其中，
h5py 是 python 中用于处理 H5 文件的接口，
PIL 和 scipy 在本文是用自己的图片来测试训练好的模型，
load_dataset 读取数据

import numpy as np
import matplotlib.pyplot as plt
import h5py
import scipy
from PIL import Image
from scipy import ndimage
from lr_utils import load_dataset

%matplotlib inline

其中 lr_utils.py 如下，是对 H5 文件进行解析 ：

#lr_utils.py  

import numpy as np  
import h5py  
          
def load_dataset():  
    train_dataset = h5py.File('datasets/train_catvnoncat.h5', "r")  
    train_set_x_orig = np.array(train_dataset["train_set_x"][:]) # your train set features  
    train_set_y_orig = np.array(train_dataset["train_set_y"][:]) # your train set labels  
  
    test_dataset = h5py.File('datasets/test_catvnoncat.h5', "r")  
    test_set_x_orig = np.array(test_dataset["test_set_x"][:]) # your test set features  
    test_set_y_orig = np.array(test_dataset["test_set_y"][:]) # your test set labels  
  
    classes = np.array(test_dataset["list_classes"][:]) # the list of classes  
      
    train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))  
    test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))  
      
    return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes 

---

2. 获得数据

# Loading the data (cat/non-cat)
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()

可以看一下图片的例子：

# Example of a picture
index = 25
plt.imshow(train_set_x_orig[index])
print ("y = " + str(train_set_y[:,index]) + ", it's a '" + classes[np.squeeze(train_set_y[:,index])].decode("utf-8") +  "' picture.")

---

3. 进行预处理： 格式转换，归一化

这时需要获得下面几个值：

• m_train (训练样本数量)
• m_test (测试样本数量)
• num_px (训练数据集的长和宽)

### START CODE HERE ### (≈ 3 lines of code)### STA 
m_train = train_set_y.shape[1]
m_test = test_set_y.shape[1]
num_px = train_set_x_orig.shape[1]
### END CODE HERE ###

print ("Number of training examples: m_train = " + str(m_train))
print ("Number of testing examples: m_test = " + str(m_test))
print ("Height/Width of each image: num_px = " + str(num_px))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_set_x shape: " + str(train_set_x_orig.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x shape: " + str(test_set_x_orig.shape))
print ("test_set_y shape: " + str(test_set_y.shape))

图像需要进行 reshape，原本是 (num_px, num_px, 3)，要扁平化为一个向量 (num_px * num_px * 3, 1)，
将 (a, b, c, d) 维的矩阵转换为 (b∗c∗d, a) 可以用： X_flatten = X.reshape(X.shape[0], -1).T

# Reshape the training and test examples

### START CODE HERE ### (≈ 2 lines of code)
train_set_x_flatten = train_set_x_orig.reshape(train_set_x_orig.shape[0], -1).T
test_set_x_flatten = test_set_x_orig.reshape(test_set_x_orig.shape[0], -1).T
### END CODE HERE ###

print ("train_set_x_flatten shape: " + str(train_set_x_flatten.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x_flatten shape: " + str(test_set_x_flatten.shape))
print ("test_set_y shape: " + str(test_set_y.shape))
print ("sanity check after reshaping: " + str(train_set_x_flatten[0:5,0]))

预处理还常常包括对数据进行中心化和标准化，图像数据的话，可以简单除以最大的像素值：

train_set_x = train_set_x_flatten / 255.
test_set_x = test_set_x_flatten / 255.

---

4. 整合模型

- A. 构建模型
    - a. 初始化参数：w 和 b 为 0
    - b. 前向传播：计算当前的损失
    - c. 反向更新：计算当前的梯度
- B. 梯度更新求模型参数
- C. 进行预测

---

先来 A. 构建模型

按照前面提到的三步：
初始化参数：w 和 b 为 0
前向传播：计算当前的损失
反向更新：计算当前的梯度

首先需要一个辅助函数 sigmoid( w^T x + b)

# GRADED FUNCTION: sigmoid

def sigmoid(z):
    """
    Compute the sigmoid of z

    Arguments:
    x -- A scalar or numpy array of any size.

    Return:
    s -- sigmoid(z)
    """

    ### START CODE HERE ### (≈ 1 line of code)
    s = 1 / (1 + np.exp(-z))
    ### END CODE HERE ###
    
    return s

a. 初始化参数：w 和 b 为 0

# GRADED FUNCTION: initialize_with_zeros

def initialize_with_zeros(dim):
    """
    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.
    
    Argument:
    dim -- size of the w vector we want (or number of parameters in this case)
    
    Returns:
    w -- initialized vector of shape (dim, 1)
    b -- initialized scalar (corresponds to the bias)
    """
    
    ### START CODE HERE ### (≈ 1 line of code)
    w = np.zeros(shape=(dim, 1))
    b = 0
    ### END CODE HERE ###

    assert(w.shape == (dim, 1))
    assert(isinstance(b, float) or isinstance(b, int))
    
    return w, b

b. 前向传播：计算当前的损失
c. 反向更新：计算当前的梯度

# GRADED FUNCTION: propagate

def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b
    
    Tips:
    - Write your code step by step for the propagation
    """
    
    m = X.shape[1]
    
    # FORWARD PROPAGATION (FROM X TO COST)
    ### START CODE HERE ### (≈ 2 lines of code)
    A = sigmoid(np.dot(w.T, X) + b)  # compute activation
    cost = (- 1 / m) * np.sum(Y * np.log(A) + (1 - Y) * (np.log(1 - A)))  # compute cost
    ### END CODE HERE ###
    
    # BACKWARD PROPAGATION (TO FIND GRAD)
    ### START CODE HERE ### (≈ 2 lines of code)
    dw = (1 / m) * np.dot(X, (A - Y).T)
    db = (1 / m) * np.sum(A - Y)
    ### END CODE HERE ###

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())
    
    grads = {"dw": dw,
             "db": db}
    
    return grads, cost

---

B. 梯度更新求模型参数

这一步 optimize 的目的是要学习 w 和 b 使 cost function J 达到最小，
用到的方法是梯度下降 \theta = \theta - \alpha \text{ } d\theta，

# GRADED FUNCTION: optimize

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm
    
    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps
    
    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
    
    Tips:
    You basically need to write down two steps and iterate through them:
        1) Calculate the cost and the gradient for the current parameters. Use propagate().
        2) Update the parameters using gradient descent rule for w and b.
    """
    
    costs = []
    
    for i in range(num_iterations):
        
        
        # Cost and gradient calculation (≈ 1-4 lines of code)
        ### START CODE HERE ### 
        grads, cost = propagate(w, b, X, Y)
        ### END CODE HERE ###
        
        # Retrieve derivatives from grads
        dw = grads["dw"]
        db = grads["db"]
        
        # update rule (≈ 2 lines of code)
        ### START CODE HERE ###
        w = w - learning_rate * dw  # need to broadcast
        b = b - learning_rate * db
        ### END CODE HERE ###
        
        # Record the costs
        if i % 100 == 0:
            costs.append(cost)
        
        # Print the cost every 100 training examples
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" % (i, cost))
    
    params = {"w": w,
              "b": b}
    
    grads = {"dw": dw,
             "db": db}
    
    return params, grads, costs

---

C. 进行预测

# GRADED FUNCTION: predict

def predict(w, b, X):
    '''
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)
    
    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    
    Returns:
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    '''
    
    m = X.shape[1]
    Y_prediction = np.zeros((1, m))
    w = w.reshape(X.shape[0], 1)
    
    # Compute vector "A" predicting the probabilities of a cat being present in the picture
    ### START CODE HERE ### (≈ 1 line of code)
    A = sigmoid(np.dot(w.T, X) + b)
    ### END CODE HERE ###
    
    for i in range(A.shape[1]):
        # Convert probabilities a[0,i] to actual predictions p[0,i]
        ### START CODE HERE ### (≈ 4 lines of code)
        Y_prediction[0, i] = 1 if A[0, i] > 0.5 else 0
        ### END CODE HERE ###
    
    assert(Y_prediction.shape == (1, m))
    
    return Y_prediction

---

下面为整合的逻辑回归模型：

将参数初始化，优化求参，预测整合在一起，

输入为 训练集，测试集，迭代次数，学习速率，是否打印中间损失
打印 test 和 train 集的预测准确率
返回的 d 含有 参数 w，b，还有 test train 集上面的预测值，

# GRADED FUNCTION: model

def model(X_train, Y_train, X_test, Y_test, num_iterations=2000, learning_rate=0.5, print_cost=False):
    """
    Builds the logistic regression model by calling the function you've implemented previously
    
    Arguments:
    X_train -- training set represented by a numpy array of shape (num_px * num_px * 3, m_train)
    Y_train -- training labels represented by a numpy array (vector) of shape (1, m_train)
    X_test -- test set represented by a numpy array of shape (num_px * num_px * 3, m_test)
    Y_test -- test labels represented by a numpy array (vector) of shape (1, m_test)
    num_iterations -- hyperparameter representing the number of iterations to optimize the parameters
    learning_rate -- hyperparameter representing the learning rate used in the update rule of optimize()
    print_cost -- Set to true to print the cost every 100 iterations
    
    Returns:
    d -- dictionary containing information about the model.
    """
    
    ### START CODE HERE ###
    # initialize parameters with zeros (≈ 1 line of code)
    w, b = initialize_with_zeros(X_train.shape[0])

    # Gradient descent (≈ 1 line of code)
    parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost)
    
    # Retrieve parameters w and b from dictionary "parameters"
    w = parameters["w"]
    b = parameters["b"]
    
    # Predict test/train set examples (≈ 2 lines of code)
    Y_prediction_test = predict(w, b, X_test)
    Y_prediction_train = predict(w, b, X_train)

    ### END CODE HERE ###

    # Print train/test Errors
    print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))
    print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))

    
    d = {"costs": costs,
         "Y_prediction_test": Y_prediction_test, 
         "Y_prediction_train" : Y_prediction_train, 
         "w" : w, 
         "b" : b,
         "learning_rate" : learning_rate,
         "num_iterations": num_iterations}
    
    return d

下面代码进行模型训练：

d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)

结果：

Cost after iteration 0: 0.693147
Cost after iteration 100: 0.584508
Cost after iteration 200: 0.466949
Cost after iteration 300: 0.376007
Cost after iteration 400: 0.331463
Cost after iteration 500: 0.303273
Cost after iteration 600: 0.279880
Cost after iteration 700: 0.260042
Cost after iteration 800: 0.242941
Cost after iteration 900: 0.228004
Cost after iteration 1000: 0.214820
Cost after iteration 1100: 0.203078
Cost after iteration 1200: 0.192544
Cost after iteration 1300: 0.183033
Cost after iteration 1400: 0.174399
Cost after iteration 1500: 0.166521
Cost after iteration 1600: 0.159305
Cost after iteration 1700: 0.152667
Cost after iteration 1800: 0.146542
Cost after iteration 1900: 0.140872
train accuracy: 99.04306220095694 %
test accuracy: 70.0 %

---

得到模型后可以看指定 index 所代表图片的预测值：

# Example of a picture that was wrongly classified.# Exampl 
index = 5
plt.imshow(test_set_x[:,index].reshape((num_px, num_px, 3)))
print ("y = " + str(test_set_y[0, index]) + ", you predicted that it is a \"" + classes[d["Y_prediction_test"][0, index]].decode("utf-8") +  "\" picture.")

---

5. 绘制学习曲线

# Plot learning curve (with costs)# Plot l 
costs = np.squeeze(d['costs'])
plt.plot(costs)
plt.ylabel('cost')
plt.xlabel('iterations (per hundreds)')
plt.title("Learning rate =" + str(d["learning_rate"]))
plt.show()

可以看出 costs 是在下降的，如果增加迭代次数，那么训练数据的准确率会进一步提高，但是测试数据集的准确率可能会明显下降，这就是由于过拟合造成的。

还可以对比下不同学习率对应下的学习效果：

learning_rates = [0.01, 0.001, 0.0001]
models = {}
for i in learning_rates:
    print ("learning rate is: " + str(i))
    models[str(i)] = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 1500, learning_rate = i, print_cost = False)
    print ('\n' + "-------------------------------------------------------" + '\n')

for i in learning_rates:
    plt.plot(np.squeeze(models[str(i)]["costs"]), label= str(models[str(i)]["learning_rate"]))

plt.ylabel('cost')
plt.xlabel('iterations')

legend = plt.legend(loc='upper center', shadow=True)
frame = legend.get_frame()
frame.set_facecolor('0.90')
plt.show()

当学习率过大 (例 0.01) 时，costs 出现上下震荡，甚至可能偏离，不过这里 0.01 最终幸运地收敛到了一个比较好的值。

---

推荐阅读 历史技术博文链接汇总
http://www.jianshu.com/p/28f02bb59fe5
也许可以找到你想要的：
[入门问题][TensorFlow][深度学习][强化学习][神经网络][机器学习][自然语言处理][聊天机器人]