损失函数(一)
损失函数概念
损失函数是衡量模型输出与真实标签的差异
在我们讨论损失函数时,经常会出现以下概念:损失函数(Loss Function)、代价函数(Cost Function)、目标函数(Objective Function)。这三者有什么区别及联系呢?
Loss Function是计算一个样本的差异,
代价函数是计算整个样本集的差异的平均值:
目标函数是更广泛的概念,通常目标函数包括cost和regularization,
pytorch中Loss:
class _Loss(Module):
def __init__(self,size_average=None,reduce=None,reduction="mean"):
super(_Loss,self).__init__()
if size_average is not None or reduce is not None:
self.reduction = Reduction.legacy_get_string(size_average,reduce)
else:
self.reduction = reduction
Loss函数继承了Module,相当于一个网络层,它有三个参数,其中size_average与reduce参数即将被舍弃,他们的功能可以在reduction中实现。
交叉熵损失函数
功能:nn.LogSoftmax()与nn.NLLLoss()结合,进行交叉熵计算
nn.CrossEntropyLoss(weight=None,size_average=None,ignore_index=-100,reduce=None,reduction="mean")
主要参数:
- weight:各类别的loss设置权值
- ingnore_index:忽略某个类别
- reduction:计算模式,可为none/sum/mean
- none:逐个元素计算
- sum:所有元素求和,返回标量
- mean:加权平均,返回标量
交叉熵 = 信息熵 + 相对熵
熵:用来描述一个事件的不确定性,一个事件越不确定,它的熵越大,比如明天下雨与明天太阳升起的熵大很多,因为明天是否下雨,不确定性很大,但不论明天下不下雨,太阳一定会升起。熵是自信息的一个期望。
自信息:它是衡量单个输出、单个事件的不确定性,
指一个事件的概率
相对熵:也称KL散度,它是用来衡量两个分布之间的差异也就是两个分布之间的距离,但它不是一个距离函数,因为两个分布之间的距离不具有对称性
其中
为真实的分布,
为模型输出的一个分布,这里我们需要用
去拟合、逼近
的分布,所以其不具有对称性。
交叉熵:
故交叉熵为
。因此,最优化交叉熵也就是最优化相对熵,因为熵
,
是样本的真实分布,而
为模型输出分布,由于训练集是固定的,所以
为一个常数,做优化时可以忽略。
下面用代码进行说明:
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
# fake data
inputs = torch.tensor([[1, 2], [1, 3], [1, 3]], dtype=torch.float)
target = torch.tensor([0, 1, 1], dtype=torch.long)
flag = 1
if flag:
# def loss function
loss_f_none = nn.CrossEntropyLoss(weight=None, reduction='none')
loss_f_sum = nn.CrossEntropyLoss(weight=None, reduction='sum')
loss_f_mean = nn.CrossEntropyLoss(weight=None, reduction='mean')
# forward
loss_none = loss_f_none(inputs, target)
loss_sum = loss_f_sum(inputs, target)
loss_mean = loss_f_mean(inputs, target)
# view
print("Cross Entropy Loss:\n ", loss_none, loss_sum, loss_mean)
上面是三种不同模式计算出的loss,第一个逐个元素计算,第二个是所有元素求和,第三个是求平均。
下面用手算的方式进行检测(只计算第一个样本):
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
idx = 0
input_1 = inputs.detach().numpy()[idx] # [1, 2]
target_1 = target.numpy()[idx] # [0]
# 第一项
x_class = input_1[target_1]
# 第二项
sigma_exp_x = np.sum(list(map(np.exp, input_1)))
log_sigma_exp_x = np.log(sigma_exp_x)
# 输出loss
loss_1 = -x_class + log_sigma_exp_x
print("第一个样本loss为: ", loss_1)
下面是weight,weight是向量形式,有多少个类别就有多少个元素,每个类别都要设置它的位置
# ----------------------------------- weight -----------------------------------
# flag = 0
flag = 1
if flag:
# def loss function
weights = torch.tensor([1, 2], dtype=torch.float)
# weights = torch.tensor([0.7, 0.3], dtype=torch.float)
loss_f_none_w = nn.CrossEntropyLoss(weight=weights, reduction='none')
loss_f_sum = nn.CrossEntropyLoss(weight=weights, reduction='sum')
loss_f_mean = nn.CrossEntropyLoss(weight=weights, reduction='mean')
# forward
loss_none_w = loss_f_none_w(inputs, target)
loss_sum = loss_f_sum(inputs, target)
loss_mean = loss_f_mean(inputs, target)
# view
print("\nweights: ", weights)
print(loss_none_w, loss_sum, loss_mean)
上面的结果是不带weight的loss结果,下面是带有weight的loss结果;第一个类别的weight是1,所以其loss没有变化,而第二个类别的weight为2,所以其loss发生了变化。其中取平均都是加权值的,1.8210 = 1.3133+0.2539+0.2539,0.3642=1.8210/5。故带权重的均值是不再是除以样本总数,而是除以权值的份数。
下面通过手算代码理解权值的份数:
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
weights = torch.tensor([1, 2], dtype=torch.float)
weights_all = np.sum(list(map(lambda x: weights.numpy()[x], target.numpy()))) # [0, 1, 1] # [1 2 2]
mean = 0
loss_sep = loss_none.detach().numpy()
for i in range(target.shape[0]):
x_class = target.numpy()[i]
tmp = loss_sep[i] * (weights.numpy()[x_class] / weights_all)
mean += tmp
print(mean)
我们通过debug模式,观测weight_all的模式:
其中
通过手算方式得到的均值也为0.3642。
pytorch中的第二个损失函数:
nn.NLLLoss(weight=None,size_average=None,ignore_index=-100,reduce=None,reduction="mean")
功能:实现负对数似然函数中的负号功能
主要参数:
- weight:各类别的loss设置权值
- ingnore_index:忽略某个类别
- reduction:计算模式,可为none/sum/mean
- none:逐个元素计算
- sum:所有元素求和,返回标量
- mean:加权平均,返回标量
其计算公式为:
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
# fake data
inputs = torch.tensor([[1, 2], [1, 3], [1, 3]], dtype=torch.float)
target = torch.tensor([0, 1, 1], dtype=torch.long)
# ----------------------------------- 2 NLLLoss -----------------------------------
# flag = 0
flag = 1
if flag:
weights = torch.tensor([1, 1], dtype=torch.float)
loss_f_none_w = nn.NLLLoss(weight=weights, reduction='none')
loss_f_sum = nn.NLLLoss(weight=weights, reduction='sum')
loss_f_mean = nn.NLLLoss(weight=weights, reduction='mean')
# forward
loss_none_w = loss_f_none_w(inputs, target)
loss_sum = loss_f_sum(inputs, target)
loss_mean = loss_f_mean(inputs, target)
# view
print("\nweights: ", weights)
print("NLL Loss", loss_none_w, loss_sum, loss_mean)
该公式只是实现了一个负号功能。
第三个损失函数:
nn.BCELoss(weight=None,size_average=None,reduce=None,reduction="mean")
功能:二分类交叉熵。注:输入值取值在[0,1]
主要参数:
- weight:各类别的loss设置权值
- ingnore_index:忽略某个类别
- reduction:计算模式,可为none/sum/mean
- none:逐个元素计算
- sum:所有元素求和,返回标量
- mean:加权平均,返回标量
它是交叉熵损失函数的一个特例,是二分类交叉熵损失函数,其计算公式为:
它是每个神经元一一对应地计算loss,而不是一整个神经元去计算loss
# ----------------------------------- 3 BCE Loss -----------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[1, 2], [2, 2], [3, 4], [4, 5]], dtype=torch.float)
target = torch.tensor([[1, 0], [1, 0], [0, 1], [0, 1]], dtype=torch.float)
target_bce = target
# itarget
inputs = torch.sigmoid(inputs)
weights = torch.tensor([1, 1], dtype=torch.float)
loss_f_none_w = nn.BCELoss(weight=weights, reduction='none')
loss_f_sum = nn.BCELoss(weight=weights, reduction='sum')
loss_f_mean = nn.BCELoss(weight=weights, reduction='mean')
# forward
loss_none_w = loss_f_none_w(inputs, target_bce)
loss_sum = loss_f_sum(inputs, target_bce)
loss_mean = loss_f_mean(inputs, target_bce)
# view
print("\nweights: ", weights)
print("BCE Loss", loss_none_w, loss_sum, loss_mean)
上诉报错是说输入必须是在[0,1]之间,而我们输入中有大于1的数,因此我们必须对输入进行sigmoid处理
我们看到有四个样本,每个样本有两个神经元,因此有8个loss,也就如之前所说每个神经元一一地计算其loss,然后对所有loss求和,求平均。
下面通过手段的模式计算第一个神经元的loss
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
idx = 0
x_i = inputs.detach().numpy()[idx, idx]
y_i = target.numpy()[idx, idx] #
# loss
# l_i = -[ y_i * np.log(x_i) + (1-y_i) * np.log(1-y_i) ] # np.log(0) = nan
l_i = -y_i * np.log(x_i) if y_i else -(1-y_i) * np.log(1-x_i)
# 输出loss
print("BCE inputs: ", inputs)
print("第一个loss为: ", l_i)
第四个损失函数:
nn.BCEWithLogitsLoss(weight=None,size_average=None,reduce=None,reduction="mean",pos_weight=None)
功能:结合sigmoid与二分类交叉熵。注:网络最后不加sigmoid函数
主要参数:
- pos_weight:正样本的权值
- weight:各类别的loss设置权值
- ingnore_index:忽略某个类别
- reduction:计算模式,可为none/sum/mean
- none:逐个元素计算
- sum:所有元素求和,返回标量
- mean:加权平均,返回标量
其计算公式为:
,其中
为sigmoid函数。
# ----------------------------------- 4 BCE with Logis Loss -----------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[1, 2], [2, 2], [3, 4], [4, 5]], dtype=torch.float)
target = torch.tensor([[1, 0], [1, 0], [0, 1], [0, 1]], dtype=torch.float)
target_bce = target
# inputs = torch.sigmoid(inputs)
weights = torch.tensor([1, 1], dtype=torch.float)
loss_f_none_w = nn.BCEWithLogitsLoss(weight=weights, reduction='none')
loss_f_sum = nn.BCEWithLogitsLoss(weight=weights, reduction='sum')
loss_f_mean = nn.BCEWithLogitsLoss(weight=weights, reduction='mean')
# forward
loss_none_w = loss_f_none_w(inputs, target_bce)
loss_sum = loss_f_sum(inputs, target_bce)
loss_mean = loss_f_mean(inputs, target_bce)
# view
print("\nweights: ", weights)
print(loss_none_w, loss_sum, loss_mean)
若对输入再进行一个sigmoid处理后,结果为:
从上可以看出,loss缩小了,出现了偏差。
下面是加入了pos_weight参数:
# --------------------------------- pos weight
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[1, 2], [2, 2], [3, 4], [4, 5]], dtype=torch.float)
target = torch.tensor([[1, 0], [1, 0], [0, 1], [0, 1]], dtype=torch.float)
target_bce = target
# itarget
# inputs = torch.sigmoid(inputs)
weights = torch.tensor([1], dtype=torch.float)
pos_w = torch.tensor([3], dtype=torch.float) # 3
loss_f_none_w = nn.BCEWithLogitsLoss(weight=weights, reduction='none', pos_weight=pos_w)
loss_f_sum = nn.BCEWithLogitsLoss(weight=weights, reduction='sum', pos_weight=pos_w)
loss_f_mean = nn.BCEWithLogitsLoss(weight=weights, reduction='mean', pos_weight=pos_w)
# forward
loss_none_w = loss_f_none_w(inputs, target_bce)
loss_sum = loss_f_sum(inputs, target_bce)
loss_mean = loss_f_mean(inputs, target_bce)
# view
print("\npos_weights: ", pos_w)
print(loss_none_w, loss_sum, loss_mean)
对正样本乘以pos_weight。
5. nn.L1Loss(size_average=None,reduce=None,reduction="mean")
功能:计算inputs与target之差的绝对值
6. nn.MSELoss(size_average=None,reduce=None,reduction="mean")
功能:计算inputs与target之差的平方
主要参数:
- reduction:计算模式,可为none/sum/mean
- none:逐个元素计算
- sum:所有元素求和,返回标量
- mean:加权平均,返回标量
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import numpy as np
from tools.common_tools import set_seed
set_seed(1) # 设置随机种子
# ------------------------------------------------- 5 L1 loss ----------------------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.ones((2, 2))
target = torch.ones((2, 2)) * 3
loss_f = nn.L1Loss(reduction='none')
loss = loss_f(inputs, target)
print("input:{}\ntarget:{}\nL1 loss:{}".format(inputs, target, loss))# ------------------------------------------------- 6 MSE loss ----------------------------------------------
loss_f_mse = nn.MSELoss(reduction='none')
loss_mse = loss_f_mse(inputs, target)
print("MSE loss:{}".format(loss_mse))
7. SmoothL1Loss(size_average=None,reduce=None,reduction="mean")
功能:平滑L1Loss
主要参数:
- reduction:计算模式,可为none/sum/mean
- none:逐个元素计算
- sum:所有元素求和,返回标量
- mean:加权平均,返回标量
# ------------------------------------------------- 7 Smooth L1 loss ----------------------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.linspace(-3, 3, steps=500)
target = torch.zeros_like(inputs)
loss_f = nn.SmoothL1Loss(reduction='none')
loss_smooth = loss_f(inputs, target)
loss_l1 = np.abs(inputs.numpy())
plt.plot(inputs.numpy(), loss_smooth.numpy(), label='Smooth L1 Loss')
plt.plot(inputs.numpy(), loss_l1, label='L1 loss')
plt.xlabel('x_i - y_i')
plt.ylabel('loss value')
plt.legend()
plt.grid()
plt.show()
8. PoissonNLLLoss(log_input=True,full=False,size_average=None,eps=1e-8,reduce=None,reduction="mean")
功能:泊松分布的负对数似然损失函数
log_input = True ,loss(input,target) = exp(input) - target*input
log_input = False, loss(input,target) = input - target*log(input+eps)
主要参数:
- log_input:输入是否为对数形式,决定计算公式
- full:计算所有loss,默认为False
- eps:修正项,避免log(input)为nan
# ------------------------------------------------- 8 Poisson NLL Loss ----------------------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.randn((2, 2))
target = torch.randn((2, 2))
loss_f = nn.PoissonNLLLoss(log_input=True, full=False, reduction='none')
loss = loss_f(inputs, target)
print("input:{}\ntarget:{}\nPoisson NLL loss:{}".format(inputs, target, loss))
下面通过手动计算第一个神经元的loss:
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
idx = 0
loss_1 = torch.exp(inputs[idx, idx]) - target[idx, idx]*inputs[idx, idx]
print("第一个元素loss:", loss_1)
9. nn.KLDivLoss(size_average=None,reduce=None,reduction="mean")
功能:计算KLD(divergence),KL散度,相对熵
注:需提前将输入计算log-probabilities,如通过nn.logsoftmax()
,这是对一个样本计算loss
主要参数:
- reduction:计算模式,可为none/sum/mean/batchmean
- batchmean:batchsize维度求平均值
- mean:加权平均,返回标量
- sum:所有元素求和,返回标量
- none:逐个元素计算
# ------------------------------ 9 KL Divergence Loss --------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[0.5, 0.3, 0.2], [0.2, 0.3, 0.5]])
inputs_log = torch.log(inputs)
target = torch.tensor([[0.9, 0.05, 0.05], [0.1, 0.7, 0.2]], dtype=torch.float)
loss_f_none = nn.KLDivLoss(reduction='none')
loss_f_mean = nn.KLDivLoss(reduction='mean')
loss_f_bs_mean = nn.KLDivLoss(reduction='batchmean')
loss_none = loss_f_none(inputs, target)
loss_mean = loss_f_mean(inputs, target)
loss_bs_mean = loss_f_bs_mean(inputs, target)
print("loss_none:\n{}\nloss_mean:\n{}\nloss_bs_mean:\n{}".format(loss_none, loss_mean, loss_bs_mean))
上面loss_mean是所有loss求和后,除以6,而batchmean则是所以loss求和后除以2。下面通过手动计算第一个神经元的loss进行验证:
# --------------------------------- compute by hand------------------
# flag = 0
flag = 1
if flag:
idx = 0
loss_1 = target[idx, idx] * (torch.log(target[idx, idx]) - inputs[idx, idx])
print("第一个元素loss:", loss_1)
10. nn.MarginRankingLoss(margin=0.0,size_average=None,reduce=None,reduction="mean")
功能:计算两个向量之间的相似度,用于排序任务
特别说明:该方法计算两组数据之间的差异,返回一个n*n的loss矩阵
主要参数:
- margin:边界值,x1与x2之间的差异值
- reduction:计算模式,可为none/sum/mean
- y = 1时,希望x1比x2大,当x1>x2时,不产生loss
- y = -1时,希望x2比x1大,当x2>x1时,不产生loss
# ------------------------------ 10 Margin Ranking Loss -----------------------------------
# flag = 0
flag = 1
if flag:
x1 = torch.tensor([[1], [2], [3]], dtype=torch.float)
x2 = torch.tensor([[2], [2], [2]], dtype=torch.float)
target = torch.tensor([1, 1, -1], dtype=torch.float)
loss_f_none = nn.MarginRankingLoss(margin=0, reduction='none')
loss = loss_f_none(x1, x2, target)
print(loss)
11. nn.MultiLabelMarginLoss(size_average=None,reduce=NOne,reduction="mean")
功能:多标签边界损失函数
举例:时分类任务,样本x属于0类和3类,标签:[0,3,-1,-1],不是[1,0,0,1]
主要参数:
- reduction:计算模式,可为none/sum/mean
;分母是神经元的个数,分子求max(标签所在的神经元减去标签非在的神经元)
where i == 0 to x.size(0), j == 0 to y.size(0) , y[j]>= 0, and i 不等于 y[j] for all i and j .
# ---------------------------------------------- 11 Multi Label Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
x = torch.tensor([[0.1, 0.2, 0.4, 0.8]])
y = torch.tensor([[0, 3, -1, -1]], dtype=torch.long)
loss_f = nn.MultiLabelMarginLoss(reduction='none')
loss = loss_f(x, y)
print(loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
x = x[0]
item_1 = (1-(x[0] - x[1])) + (1 - (x[0] - x[2])) # [0]
item_2 = (1-(x[3] - x[1])) + (1 - (x[3] - x[2])) # [3]
loss_h = (item_1 + item_2) / x.shape[0]
print(loss_h)
12. nn.SoftMarginLoss(size_average=None,reduce=None,reduction="mean")
功能:计算二分类的logistic损失
主要参数:
- reduction:计算模式,可为none/sum/mean
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[0.3, 0.7], [0.5, 0.5]])
target = torch.tensor([[-1, 1], [1, -1]], dtype=torch.float)
loss_f = nn.SoftMarginLoss(reduction='none')
loss = loss_f(inputs, target)
print("SoftMargin: ", loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
idx = 0
inputs_i = inputs[idx, idx]
target_i = target[idx, idx]
loss_h = np.log(1 + np.exp(-target_i * inputs_i))
print(loss_h)
13. nn.MultiLabelSoftMarginLoss(weight=None,size_average=None,reduce=None,reduction="mean")
功能:SoftMarginLoss多标签版本
主要参数:
- weight:各类别的loss设置权值
- reduction:计算模式,可为none/sum/mean
C:标签的数量
# ---------------------------------------------- 13 MultiLabel SoftMargin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[0.3, 0.7, 0.8]])
target = torch.tensor([[0, 1, 1]], dtype=torch.float)
loss_f = nn.MultiLabelSoftMarginLoss(reduction='none')
loss = loss_f(inputs, target)
print("MultiLabel SoftMargin: ", loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
i_0 = torch.log(torch.exp(-inputs[0, 0]) / (1 + torch.exp(-inputs[0, 0])))
i_1 = torch.log(1 / (1 + torch.exp(-inputs[0, 1])))
i_2 = torch.log(1 / (1 + torch.exp(-inputs[0, 2])))
loss_h = (i_0 + i_1 + i_2) / -3
print(loss_h)
14. nn.MultiMarginLoss(p=1,margin=1.0,weight=None,size_average=None,reduce=None,reduction="mean")
功能:计算多分类的折页损失
主要参数:
- p:可选1或2
- weight:各类别的loss设置权值
- margin:边界值
- reduction:计算模式,可为none/sum/mean
where x
{0,...,x.size(0)-1} , y
{0,...,y.size(0)-1},0=< y[j] =< x.size(0)-1, and i 不等于 y[j] for all i and j.
# ---------------------------------------------- 14 Multi Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
x = torch.tensor([[0.1, 0.2, 0.7], [0.2, 0.5, 0.3]])
y = torch.tensor([1, 2], dtype=torch.long)
loss_f = nn.MultiMarginLoss(reduction='none')
loss = loss_f(x, y)
print("Multi Margin Loss: ", loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
x = x[0]
margin = 1
i_0 = margin - (x[1] - x[0])
# i_1 = margin - (x[1] - x[1])
i_2 = margin - (x[1] - x[2])
loss_h = (i_0 + i_2) / x.shape[0]
print(loss_h)
15. nn.TripletMarginLoss(margin=1.0,p=2.0,eps=1e-06,swap=False,size_average=None,reduce=None,reduction="mean")
功能:计算三元组损失,人脸验证中常用
主要参数:
p:范数的阶,默认为2
margin:边界值
reduction:计算模式,可为none/sum/mean
# ---------------------------------------------- 15 Triplet Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
anchor = torch.tensor([[1.]])
pos = torch.tensor([[2.]])
neg = torch.tensor([[0.5]])
loss_f = nn.TripletMarginLoss(margin=1.0, p=1)
loss = loss_f(anchor, pos, neg)
print("Triplet Margin Loss", loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
margin = 1
a, p, n = anchor[0], pos[0], neg[0]
d_ap = torch.abs(a-p)
d_an = torch.abs(a-n)
loss = d_ap - d_an + margin
print(loss)
16. nn.HingeEmbeddingLoss(margin=1.0,size_average=None,reduce=None,reduction="mean")
功能:计算两个输入的相似性,常用于非线性embedding和半监督学习
特别注意:输入x应为两个输入之差的绝对值
主要参数:
- margin:边界值
- reduction:计算模式,可为none/sum/mean
# ---------------------------------------------- 16 Hinge Embedding Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
inputs = torch.tensor([[1., 0.8, 0.5]])
target = torch.tensor([[1, 1, -1]])
loss_f = nn.HingeEmbeddingLoss(margin=1, reduction='none')
loss = loss_f(inputs, target)
print("Hinge Embedding Loss", loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
margin = 1.
loss = max(0, margin - inputs.numpy()[0, 2])
print(loss)
17. nn.CosineEmbeddingLoss(margin=0.0,size_average=None,reduce=None,reduction="mean")
功能:采用余弦相似度计算两个输入的相似性
主要参数:
- margin:可取值[-1,1],推荐为[0,0.5]
- reduction:计算模式,可为none/sum/mean
# ---------------------------------------------- 17 Cosine Embedding Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
x1 = torch.tensor([[0.3, 0.5, 0.7], [0.3, 0.5, 0.7]])
x2 = torch.tensor([[0.1, 0.3, 0.5], [0.1, 0.3, 0.5]])
target = torch.tensor([[1, -1]], dtype=torch.float)
loss_f = nn.CosineEmbeddingLoss(margin=0., reduction='none')
loss = loss_f(x1, x2, target)
print("Cosine Embedding Loss", loss)
# --------------------------------- compute by hand
# flag = 0
flag = 1
if flag:
margin = 0.
def cosine(a, b):
numerator = torch.dot(a, b)
denominator = torch.norm(a, 2) * torch.norm(b, 2)
return float(numerator/denominator)
l_1 = 1 - (cosine(x1[0], x2[0]))
l_2 = max(0, cosine(x1[0], x2[0]))
print(l_1, l_2)
18. nn.CTCLoss(blank=0,reduction="mean",zero_infinity=False)
功能:计算CTC损失,解决时序类数据的分类 Connectionist Temporal Classification
主要参数:
- blank:blank label
- zero_infinity:无穷大的值或梯度值0
- reduction:计算模式,可为None/sum/mean
# ---------------------------------------------- 18 CTC Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
T = 50 # Input sequence length
C = 20 # Number of classes (including blank)
N = 16 # Batch size
S = 30 # Target sequence length of longest target in batch
S_min = 10 # Minimum target length, for demonstration purposes
# Initialize random batch of input vectors, for *size = (T,N,C)
inputs = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
# Initialize random batch of targets (0 = blank, 1:C = classes)
target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long)
input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long)
ctc_loss = nn.CTCLoss()
loss = ctc_loss(inputs, target, input_lengths, target_lengths)
print("CTC loss: ", loss)
18种损失函数