Linear regression with multiple

Linear-Regression - Shuai-Xie -Github

多变量线性回归 预测房价

ex1data2.txt（房屋尺寸，卧室数量，房屋价格）

2104,3,399900
1600,3,329900
2400,3,369000
1416,2,232000
3000,4,539900
1985,4,299900
……

Part 1: Feature Normalization

对于多维特征，要使用特征缩放保证其具有相近尺度，这样梯度下降算法会更快收敛。这个例子中，房屋尺寸是卧室数量接近1000倍。

%% ================ Part 1: Feature Normalization ================
fprintf('Loading data ...\n');

data = load('ex1data2.txt');
X = data(:, 1:2); % 房屋特征2个 (m*2)
y = data(:, 3); % 价格 (m*1)

m = length(y); % 样本数量

% 输出10个样本点
fprintf('First 10 examples from the dataset: \n');

% X 2个特征，y 1个价格
% 因为fprintf按列输出矩阵，所以要将矩阵先转置，使得每一列对应一个实例
fprintf(' x = [%.0f %.0f], y = %.0f \n', [X(1:10,:) y(1:10,:)]');
% fprintf(' x = [%.0f %.0f], y = %.0f \n', data(1:10, :)'); % 也可以

fprintf('Program paused. Press enter to continue.\n');
pause;

% 特征缩放
fprintf('Normalizing Features ...\n');
% X是特征缩放后的特征矩阵，mu是平均值矩阵，sigma是方差矩阵
[X, mu, sigma] = featureNormalize(X);
X = [ones(m, 1), X]; % X 补充 theta0

% 格式化输出特征缩放后的矩阵
fprintf(' x = [%.6f %.6f %.6f], y = %.0f \n', [X(1:10,:) y(1:10,:)]');
fprintf('Program paused. Press enter to continue.\n');
pause;

Loading data ...
First 10 examples from the dataset: 
 x = [2104 3], y = 399900 
 x = [1600 3], y = 329900 
 x = [2400 3], y = 369000 
 x = [1416 2], y = 232000 
 x = [3000 4], y = 539900 
 x = [1985 4], y = 299900 
 x = [1534 3], y = 314900 
 x = [1427 3], y = 198999 
 x = [1380 3], y = 212000 
 x = [1494 3], y = 242500 
Program paused. Press enter to continue.
Normalizing Features ...
 x = [1.000000 0.130010 -0.223675], y = 399900 
 x = [1.000000 -0.504190 -0.223675], y = 329900 
 x = [1.000000 0.502476 -0.223675], y = 369000 
 x = [1.000000 -0.735723 -1.537767], y = 232000 
 x = [1.000000 1.257476 1.090417], y = 539900 
 x = [1.000000 -0.019732 1.090417], y = 299900 
 x = [1.000000 -0.587240 -0.223675], y = 314900 
 x = [1.000000 -0.721881 -0.223675], y = 198999 
 x = [1.000000 -0.781023 -0.223675], y = 212000 
 x = [1.000000 -0.637573 -0.223675], y = 242500 
Program paused. Press enter to continue.

featureNormalize 函数

function [X_norm, mu, sigma] = featureNormalize(X)
% 标准化 X 的特征到[0,1]之间

% 初始化
X_norm = X; % X(47*2)

% 利用matlab自带函数mean，std求平均值和标准差
mu = mean(X); % mu(1*2) a row vector containing the mean value of each column. 
sigma = std(X); % sigma(1*2) 

% size(X, 1) 获得 X 矩阵的行数，即样本数 = 47
% size(X, 2) 获得 X 矩阵的列数，即特征数 = 2
m = size(X, 1);

% 缩放 X 的 2 个特征
for i = 1:m
    X_norm(i, 1) = ( X_norm(i, 1) - mu(1) ) / sigma(1);
    X_norm(i, 2) = ( X_norm(i, 2) - mu(2) ) / sigma(2);
end

end

特征缩放

Part 2: Gradient Descent

%% ================ Part 2: Gradient Descent ================
fprintf('Running gradient descent ...\n');

% Choose some alpha value 3个学习率
alpha = 0.01; % 学习率
alpha1 = 0.03;
alpha2 = 0.1;
num_iters = 400; % 迭代次数

% Init Theta and Run Gradient Descent
theta = zeros(3, 1); % 3个特征
theta1 = zeros(3, 1);
theta2 = zeros(3, 1);
[theta, J_history] = gradientDescentMulti(X, y, theta, alpha, num_iters); % 多变量梯度下降
[theta1, J_history1] = gradientDescentMulti(X, y, theta1, alpha1, num_iters);
[theta2, J_history2] = gradientDescentMulti(X, y, theta2, alpha2, num_iters);

% Plot the convergence graph
figure;
plot(1:numel(J_history), J_history, '-b', 'LineWidth', 2);
hold on; % 继续画图
plot(1:numel(J_history1), J_history1, '-r', 'LineWidth', 2);
plot(1:numel(J_history2), J_history2, '-k', 'LineWidth', 2);
% numel: number of elements 返回元素个数，作为x轴迭代次数
xlabel('Number of iterations');
ylabel('Cost J');
legend('alpha = 0.01', 'alpha = 0.03', 'alpha = 0.1'); % 图例

% 显示theta值
fprintf('Theta computed from gradient descent: \n');
fprintf(' %f \n', theta); % 3*1

% 估计房屋价格 1650 sq-ft, 3 br house
% x1不需要特征缩放为1，将后面的x2，x3特征缩放
% price = 0; % You should change this
price = [1, (1650 - mu(1,1)) / sigma(1,1), (3 - mu(1,2)) / sigma(1,2)] * theta;

fprintf(['Predicted price of a 1650 sq-ft, 3 br house ' ...
    '(using gradient descent): $%f\n'], price);

fprintf('Program paused. Press enter to continue.\n');
pause;

Running gradient descent ...
Theta computed from gradient descent: 
 334302.063993 
 100087.116006 
 3673.548451 
Predicted price of a 1650 sq-ft, 3 br house (using gradient descent): $289314.620338
Program paused. Press enter to continue.

不同学习率下的收敛曲线

gradientDescentMulti 函数

function [theta, J_history] = gradientDescentMulti(X, y, theta, alpha, num_iters)

% Initialize some useful values
J_history = zeros(num_iters, 1);
m = length(y);

% 迭代
for iter = 1:num_iters
    
    tmp = (X' * ( X * theta - y )); % 向量运算 求和项 (3*m)*(m*1)=(3*1)
    theta = theta - alpha / m * tmp;
    
    % Save the cost J in every iteration
    J_history(iter) = computeCostMulti(X, y, theta); % 根据theta求J
    
end

computeCostMulti 函数

function J = computeCostMulti(X, y, theta)

% Initialize some useful values
m = length(y); % number of training examples

E = X * theta - y; % 偏差矩阵(m*1)
J = (E' * E) / (2 * m); % (E'*E) 方差和

end

Part 3: Normal Equations

正规方程求 θ

%% ================ Part 3: Normal Equations ================
fprintf('Solving with normal equations...\n');

% Load Data
data = csvread('ex1data2.txt'); % load也可以
X = data(:, 1:2);
y = data(:, 3);
m = length(y);

% Add intercept term to X
X = [ones(m, 1) X];

% Calculate the parameters from the normal equation
theta = normalEqn(X, y);

% Display normal equation result
fprintf('Theta computed from the normal equations: \n');
fprintf(' %f \n', theta);

% Estimate the price of a 1650 sq-ft, 3 br house
% 不需要特征缩放
price = [1, 1650, 3] * theta;

fprintf(['Predicted price of a 1650 sq-ft, 3 br house ' ...
    '(using normal equations): $%f\n'], price);

预测结果与梯度下降法有出入，正规预测的是准确值，梯度下降是递进值。

Solving with normal equations...
Theta computed from the normal equations: 
 89597.909544 
 139.210674 
 -8738.019113 
Predicted price of a 1650 sq-ft, 3 br house (using normal equations): $293081.464335

normalEqn 函数

function [theta] = normalEqn(X, y)

theta = pinv(X' * X) * X' * y; % 调用方程

end