Learning Torch

作者: 王拓 | 来源:发表于2018-12-24 12:55 被阅读0次

Learning Torch
安装torch7
pytorch学习率的优化方法
torch、tensorflow-cpu
crf
Pytorch错误：Torch not compiled wit
torch.randn和torch.rand区别
pytorch
Python OSError: libcusparse.so.1
Pytorch的第二步：(1) torch.nn.functio

Learning PyTorch

Autograd

autograd.Variable

<u>新版本的pytorch Tensor和Variab合并了</u>

Variable主要包含三个属性

data: 保存Variable所含的Tensor
grad: 保存data对应的梯度，grad也是一个Variable，不是Tensor
grad_fn: 指向一个Function对象，用来反向传播计算输入的梯度

x = Variable(t.ones(3), requires_grad=True) # Tensor初始化的时候不能用requires_grad初始化，需要单独x.requires_grad=True改变该属性
y = t.sum(x)
y.backward() # 反向传播
x.grad # 得到y对x的导数
# !! x.grad在反向传播过程中是累加，每次运行反向传播，梯度都会累加之前的梯度，所以反向传播需把之前梯度清零
x.grad.data.zero_() # 带_操作是inplace操作

保存和加载模型

保存和提取主要使用torch.save和torch.load方法实现保存和提取

test_data = torch.FloatTensor(2,3)
torch.save(test_data, "test_data.pkl")
test_data_load = torch.load("test_data.pkl")

只保存和加载模型参数

net = torch.nn.Sequential(
        torch.nn.Linear(1,10),
        torch.nn.ReLU(),
        torch.nn.Linear(10,1)
    )
torch.save(net.state_dict(), "net_params.pkl") # 保存模型参数

net.load_state_dict(torch.load('net_params.pkl')) # 将保存的参数复制到net，这种方式会提取所有参数，放到现有的网路

保存和加载整个模型

torch.save(net, "net_total.pkl")
net = torch.load('net_total.pkl') # 提取整个网络，网络大的时候费时

<u>同时保存epoch和模型</u>

torch.save({'epoch':epoch, 'state_dict':net.state_dict()}, 'checkpoint.tar')
checkpoint = torch.load('checkpoint.tar')
epoch = checkpoint['epoch']
net.load_state_dict(checkpoint['state_dict'])

nn

nn.Conv2d

nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias)

padding=0表示四周不进行零填充， padding=1表示四周进行1个像素点的零填充
bias是一个布尔值，默认bias=True，表示使用偏置
groups表示输出数据体深度上和输入数据体深度上的联系，默认groups=1，也就是所有的输入和输出都是相关联的，如果groups=2，这表示输入的深度被分割成两份，输入的深度被分成两份，输出的深度也被分成两份，它们之间分别对应起来，要求输入输出的深度都能被groups整除
dilation表示卷积对输入数据体的空间间隔（空洞卷积）

nn.MaxPool2d/AvgPool2d

nn.MaxPool2d(kernel_size, stride, padding, dilation, return_indices, ceil_mode)
nn.AvgPool2d(kernel_size, stride, padding, dilation, return_indices, ceil_mode， count_include_pad=True)

stride 默认和kernel_size一样
return_indices表示是否返回最大值所在处的下标，默认为False
ceil_mode ceil模式就是会把不足square_size的边给保留下来，单独另算，或者也可以理解为在原来的数据上补充了值为-NAN的边。而floor模式则是直接把不足square_size的边给舍弃了。
count_include_pad表示计算均值时是否包含零填充

nn.ZeroPad2d

在输入的数据周围做zero-padding，参数可以是一个int或者一个tuple，tuple的顺序是（左，右，上，下）

>>> m = nn.ZeroPad2d(2)
>>> input = torch.randn(1, 1, 3, 3)
>>> input
tensor([[[[-0.1678, -0.4418,  1.9466],
          [ 0.9604, -0.4219, -0.5241],
          [-0.9162, -0.5436, -0.6446]]]])
>>> m(input)
tensor([[[[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [ 0.0000,  0.0000, -0.1678, -0.4418,  1.9466,  0.0000,  0.0000],
          [ 0.0000,  0.0000,  0.9604, -0.4219, -0.5241,  0.0000,  0.0000],
          [ 0.0000,  0.0000, -0.9162, -0.5436, -0.6446,  0.0000,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000]]]])
>>> # using different paddings for different sides
>>> m = nn.ZeroPad2d((1, 1, 2, 0))
>>> m(input)
tensor([[[[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [ 0.0000, -0.1678, -0.4418,  1.9466,  0.0000],
          [ 0.0000,  0.9604, -0.4219, -0.5241,  0.0000],
          [ 0.0000, -0.9162, -0.5436, -0.6446,  0.0000]]]])

nn.CrossEntropyLoss

$-\log\left(\frac{\exp(x[class])}{\sum_j \exp(x[j])}\right)$

输出比输入少一个维度

x = torch.Tensor([[0.3, 0.7]])
y = torch.LongTensor([1])
loss = torch.nn.CrossEntropyLoss()
loss(x, y)
>>> tensor(0.5130)

nn.LogSoftmax

$log(\frac{\exp(x_i)}{\sum_j \exp(x_j)})$

输入输出维度相同

x = torch.Tensor([0.3, 0.7])
sm = torch.nn.LogSoftmax()
sm(x)
>>> tensor([-0.9130, -0.5130])

nn.NLLLoss

x = torch.Tensor([[0.3, 0.7]])
y = torch.LongTensor([1])
loss = torch.nn.NLLLoss()
loss(x,y)
>>> tensor(-0.7000)

nn.BCELoss

x = torch.Tensor([0.3, 0.7])
y = torch.Tensor([0,1])
loss = torch.nn.BCELoss(reduce=False)
loss(x,y)
>>> tensor([0.3567, 0.3567]) # 0*ln(0.3) + 1*ln(1-0.3) + 1*ln(0.7)

nn.Module

detach

截断反向传播的梯度流，返回一个不再计算梯度的新Variable。在GAN中会用到。

utils

utils.data

制作自己的数据集

x = torch.linspace(1,10,10)
y = torch.linspace(10,1,10)

torch_dataset = data.TensorDataset(data_tensor=x, target_tensor=y)

loader = data.DataLoader(
    dataset=torch_dataset,
    batch_size=BATCH_SIZE,
    shuffle=True,
    num_workers=2 # 多线程来读取数据
)

Others

add

x.add(y) # 返回新的tensor
x.add_(y) # 改变x

result = t.Tensor(5,3)
.add(x,y, out=result) # 输入到result

numpy

a = np.ones(3)
b = t.from_numpy(a)

b.add_(1)
# a 的数值也跟着+1， Tensor和numpy对象共享内存，其中一个变了，另外一个也跟着变了

randperm

产生长度为10的随机排列区间在[0,9]

t.randperm(10)

unsqueeze

b.unsqueeze(1) # 在第1维（下标从0开始）增加1个维度

LeNet

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        # 1：输入图片为单通道 6：输出通道  5：卷积核5*5
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84,10)
    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x), (2,2)))
        x = F.max_pool2d(F.relu(self.conv2(x), 2))
        x = x.view(x.size()[0], -1) # 拉直
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    
net = Net()
print(net)

for name, param in net.named_parameters():
    print(name, ":", parameters.size())

input = Variable(t.randn(1,1,32,32))
out = net(input)
out.size()

net.zero_grad() # 所有参数梯度清零
out.backward()

只要在nn.Module的子类中定义了forward，backward函数会自动实现。在forward函数中可以使用任何Variable支持的函数。

网络的可学习参数通过net.parameters()返回，net.named_parameters可同时返回可学习的参数及名称

SPP_Net

image.png

计算公式
$Kernel = \frac{height_{in}} {height_{out}}=ceil(\frac{height_{in}}{height_{out}}) \\ Sride = \frac{height_{in}}{height_{out}}=ceil(\frac{height_{in}}{height_{out}}) \\ padding = \frac{kernel*height_{out}-height_{in}+1}{2}=floor(\frac{kernel*height_{out}-height_{in}+1}{2}) \\ height_{new} = 2*padding +height_{in} \\$


class SPP(nn.Module):
    def __init__(self, num_levels, pool_type='max_pool'):
        super(SPP, self).__init__()
        self.num_levels = num_levels
        self.pool_type = pool_type

    def forward(self, x):
        num, c, h, w = x.size()
        for i in range(self.num_levels):
            level = i+1
            
            # 当层数spp层数多的时候可能出现padding大于kernel_size/2的情况，此时torch的max_pooling2d会报错，所以使用
            kernel_size = (math.ceil(h / level), math.ceil(w / level))
            stride = (math.floor(h / level), math.floor(w / level))
            padding = (math.floor((kernel_size[0]*level-h+1)/2), math.floor((kernel_size[1]*level-w+1)/2))
            # update input data with padding
            zero_pad = torch.nn.ZeroPad2d((padding[1],padding[1],padding[0],padding[0])) #left,right,top,botton
            x_new = zero_pad(x)
            # update kernel and stride
            h_new = 2*padding[0] + h
            w_new = 2*padding[1] + w            
            kernel_size = (math.ceil(h_new / level), math.ceil(w_new / level))
            stride = (math.floor(h_new / level), math.floor(w_new / level))

            #kernel_size = (math.ceil(h/level), math.ceil(w/level))
            #stride = (math.ceil(h/level), math.ceil(w/level))
            #padding = (math.floor((kernel_size[0]*level-h+1)/2), math.floor((kernel_size[1]*level-w+1)/2))
            # 如果不自己调用ZeroPad2d，还需在调用max_pool2d时传入padding=padding


            if self.pool_type == 'max_pool':
                tensor = F.max_pool2d(x, kernel_size=kernel_size, stride=stride).view(num, -1)
            else:
                tensor = F.avg_pool2d(x, kernel_size=kernel_size, stride=stride).view(num, -1)

            if(i == 0):
                x_flatten = tensor.view(num, -1)
            else:
                x_flatten = torch.cat((x_flatten, tensor), 1)
        return x_flatten

spp = SPP(5) # 1*1 + 2*2 + 3*3 + 4*4 + 5*5 = 55 
#print(list(spp.parameters()))
for name, parameters in list(spp.named_parameters()):
    print(name, ":", parameters.size())

input = Variable(torch.randn(1,1,7,9))
out = spp(input)
print(out.size()) # (1,55) output size isn't determined by input shape

Learning Torch
Learning PyTorch Autograd autograd.Variable 新版本的pytorch T...
安装torch7
Installing Torch 7 deep learning on Ubuntu 16.04 ubuntu 1...
pytorch学习率的优化方法
torch.optim.lr_scheduler模块提供了一些根据epoch训练次数来调整学习率（learning...
torch、tensorflow-cpu
import torch print(torch.__version__) print(torch) pyana3...
crf
import torch import torch.nn as nn import torch.optim as ...
Pytorch错误：Torch not compiled wit
先测试： import torch print(torch.__version__) print(torch.cu...
torch.randn和torch.rand区别
torch.rand是均匀分布torch.randn是标准正态分布 torch.rand 均匀分布 torch.r...
pytorch
1.numpy and torch numpy中是array torch中是tensor torch.from_n...
Python OSError: libcusparse.so.1
背景提要：刚刚安装了pytorch，torch-geometric、torch_scatter和torch_spa...
Pytorch的第二步：(1) torch.nn.functio
torch.nn.functional 涉及了所有 torch.nn 需要类和方法，torch.nn 构建...