python 中文 情感 分析 pytorch 中文情感分析

第四节 情感分析

模型从简单到复杂，依次构建：

• Word Averaging模型
• RNN/LSTM模型
• CNN模型

准备数据

• TorchText中的一个重要概念是Field。Field决定了你的数据会被怎样处理。在我们的情感分类任务中，我们所需要接触到的数据有文本字符串和两种情感，“pos"或者"neg”。
• Field的参数制定了数据会被怎样处理。
• 我们使用TEXT field来定义如何处理电影评论，使用LABEL field来处理两个情感类别。
• 我们的TEXT field带有tokenize='spacy'，这表示我们会用spaCy tokenizer来tokenize英文句子。如果我们不特别声明tokenize这个参数，那么默认的分词方法是使用空格。
• 安装spaCy

pip install -U spacy
python -m spacy download en

• LABEL由LabelField定义。这是一种特别的用来处理label的Field。我们后面会解释dtype。

import torch
from torchtext import data

SEED = 1234

torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

TEXT = data.Field(tokenize='spacy')
LABEL = data.LabelField(dtype=torch.float)

• TorchText支持很多常见的自然语言处理数据集。
• 下面的代码会自动下载IMDb数据集，然后分成train/test两个torchtext.datasets类别。数据被前面的Fields处理。IMDb数据集一共有50000电影评论，每个评论都被标注为正面的或负面的。

In [115]:

from torchtext import datasets
train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)

查看每个数据split有多少条数据。

In [116]:

print(f'Number of training examples: {len(train_data)}')
print(f'Number of testing examples: {len(test_data)}')
Number of training examples: 25000
Number of testing examples: 25000

• 由于我们现在只有train/test这两个分类，所以我们需要创建一个新的validation set。我们可以使用.split()创建新的分类。
• 默认的数据分割是 70、30，如果我们声明split_ratio，可以改变split之间的比例，split_ratio=0.8表示80%的数据是训练集，20%是验证集。
• 我们还声明random_state这个参数，确保我们每次分割的数据集都是一样的。

import random
train_data, valid_data = train_data.split(random_state=random.seed(SEED))

print(f'Number of training examples: {len(train_data)}')
print(f'Number of validation examples: {len(valid_data)}')
print(f'Number of testing examples: {len(test_data)}')

• 下一步我们需要创建 vocabulary 。vocabulary 就是把每个单词一一映射到一个数字。[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-42BYGRrZ-1625301850365)(assets/sentiment5.png)]
• 我们使用最常见的25k个单词来构建我们的单词表，用max_size这个参数可以做到这一点。
• 所有其他的单词都用<unk>来表示。

In [120]:

# TEXT.build_vocab(train_data, max_size=25000)
# LABEL.build_vocab(train_data)
TEXT.build_vocab(train_data, max_size=25000, vectors="glove.6B.100d", unk_init=torch.Tensor.normal_)
LABEL.build_vocab(train_data)

In [121]:

print(f"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}")
print(f"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}")
Unique tokens in TEXT vocabulary: 25002
Unique tokens in LABEL vocabulary: 2

• 当我们把句子传进模型的时候，我们是按照一个个 batch 穿进去的，也就是说，我们一次传入了好几个句子，而且每个batch中的句子必须是相同的长度。为了确保句子的长度相同，TorchText会把短的句子pad到和最长的句子等长
• 下面我们来看看训练数据集中最常见的单词。

print(TEXT.vocab.freqs.most_common(20))

[('the', 201455), (',', 192552), ('.', 164402), ('a', 108963), ('and', 108649), ('of', 100010), ('to', 92873), ('is', 76046), ('in', 60904), ('I', 54486), ('it', 53405), ('that', 49155), ('"', 43890), ("'s", 43151), ('this', 42454), ('-', 36769), ('/><br', 35511), ('was', 34990), ('as', 30324), ('with', 29691)]

我们可以直接用 stoi(string to int) 或者 itos (int to string) 来查看我们的单词表。

print(TEXT.vocab.itos[:10])

['<unk>', '<pad>', 'the', ',', '.', 'a', 'and', 'of', 'to', 'is']

查看labels。

print(LABEL.vocab.stoi)

defaultdict(<function _default_unk_index at 0x7fbec39a79d8>, {'neg': 0, 'pos': 1})

• 最后一步数据的准备是创建iterators。每个itartion都会返回一个batch的examples。
• 我们会使用BucketIterator。BucketIterator会把长度差不多的句子放到同一个batch中，确保每个batch中不出现太多的padding。
• 严格来说，我们这份notebook中的模型代码都有一个问题，也就是我们把<pad>也当做了模型的输入进行训练。更好的做法是在模型中把由<pad>产生的输出给消除掉。在这节课中我们简单处理，直接把<pad>也用作模型输入了。由于<pad>数量不多，模型的效果也不差。
• 如果我们有GPU，还可以指定每个iteration返回的tensor都在GPU上。

BATCH_SIZE = 64

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data), 
    batch_size=BATCH_SIZE,
    device=device)

Word Averaging模型

• 我们首先介绍一个简单的Word Averaging模型。这个模型非常简单，我们把每个单词都通过Embedding层投射成word embedding vector，然后把一句话中的所有word vector做个平均，就是整个句子的vector表示了。接下来把这个sentence vector传入一个Linear层，做分类即可。
• 我们使用avg_pool2d来做average pooling。我们的目标是把sentence length那个维度平均成1，然后保留embedding这个维度。
• avg_pool2d的kernel siz是 (embedded.shape[1], 1)，所以句子长度的那个维度会被压扁

import torch.nn as nn
import torch.nn.functional as F

class WordAVGModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, output_dim, pad_idx):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.fc = nn.Linear(embedding_dim, output_dim)
        
    def forward(self, text):
        embedded = self.embedding(text) # [sent len, batch size, emb dim]
        embedded = embedded.permute(1, 0, 2) # [batch size, sent len, emb dim]
        pooled = F.avg_pool2d(embedded, (embedded.shape[1], 1)).squeeze(1) # [batch size, embedding_dim]
        return self.fc(pooled)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
OUTPUT_DIM = 1
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = WordAVGModel(INPUT_DIM, EMBEDDING_DIM, OUTPUT_DIM, PAD_IDX)

In [128]:

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')
The model has 2,500,301 trainable parameters

In [129]:

pretrained_embeddings = TEXT.vocab.vectors
model.embedding.weight.data.copy_(pretrained_embeddings)

Out[129]:

tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],
        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]])

In [130]:

UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

训练模型

In [131]:

import torch.optim as optim

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
model = model.to(device)
criterion = criterion.to(device)

计算预测的准确率

In [132]:

def binary_accuracy(preds, y):
    """
    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8
    """

    #round predictions to the closest integer
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float() #convert into float for division 
    acc = correct.sum()/len(correct)
    return acc

In [133]:

def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    
    for batch in iterator:
        optimizer.zero_grad()
        predictions = model(batch.text).squeeze(1)
        loss = criterion(predictions, batch.label)
        acc = binary_accuracy(predictions, batch.label)
        loss.backward()
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

In [135]:

def evaluate(model, iterator, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    model.eval（)
    
    with torch.no_grad():
        for batch in iterator:
            predictions = model(batch.text).squeeze(1)
            loss = criterion(predictions, batch.label)
            acc = binary_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

In [136]:

import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

In [137]:

N_EPOCHS = 10

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'wordavg-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

import spacy
nlp = spacy.load('en')

def predict_sentiment(sentence):
    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
    indexed = [TEXT.vocab.stoi[t] for t in tokenized]
    tensor = torch.LongTensor(indexed).to(device)
    tensor = tensor.unsqueeze(1)
    prediction = torch.sigmoid(model(tensor))
    return prediction.item()

In [141]:

predict_sentiment("This film is terrible")

Out[141]:

5.568591932965664e-26

In [142]:

predict_sentiment("This film is great")

Out[142]:

1.0

RNN模型

• 下面我们尝试把模型换成一个
  recurrent neural network
  (RNN)。RNN经常会被用来encode一个sequence
  ht=RNN(xt,ht−1)ht=RNN(xt,ht−1)
• 我们使用最后一个hidden state hThT来表示整个句子。
• 然后我们把hThT通过一个线性变换ff，然后用来预测句子的情感。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-AEtanpHE-1625301850367)(assets/sentiment1.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-opCNLBsk-1625301850368)(assets/sentiment7.png)]

In [149]:

class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, 
                 n_layers, bidirectional, dropout, pad_idx):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers=n_layers, 
                           bidirectional=bidirectional, dropout=dropout)
        self.fc = nn.Linear(hidden_dim*2, output_dim)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        embedded = self.dropout(self.embedding(text)) #[sent len, batch size, emb dim]
        output, (hidden, cell) = self.rnn(embedded)
        #output = [sent len, batch size, hid dim * num directions]
        #hidden = [num layers * num directions, batch size, hid dim]
        #cell = [num layers * num directions, batch size, hid dim]
        
        #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
        #and apply dropout
        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1)) # [batch size, hid dim * num directions]
        return self.fc(hidden.squeeze(0))

In [150]:

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, 
            N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)

In [151]:

print(f'The model has {count_parameters(model):,} trainable parameters')
The model has 4,810,857 trainable parameters

In [155]:

model.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

print(model.embedding.weight.data)
tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]],
       device='cuda:0')

训练RNN模型

In [156]:

optimizer = optim.Adam(model.parameters())
model = model.to(device)

In [158]:

N_EPOCHS = 5
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'lstm-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')
Epoch: 01 | Epoch Time: 1m 29s
	Train Loss: 0.676 | Train Acc: 57.69%
	 Val. Loss: 0.694 |  Val. Acc: 53.40%
Epoch: 02 | Epoch Time: 1m 29s
	Train Loss: 0.641 | Train Acc: 63.77%
	 Val. Loss: 0.744 |  Val. Acc: 49.22%
Epoch: 03 | Epoch Time: 1m 29s
	Train Loss: 0.618 | Train Acc: 65.77%
	 Val. Loss: 0.534 |  Val. Acc: 73.72%
Epoch: 04 | Epoch Time: 1m 30s
	Train Loss: 0.634 | Train Acc: 63.79%
	 Val. Loss: 0.619 |  Val. Acc: 66.85%
Epoch: 05 | Epoch Time: 1m 29s
	Train Loss: 0.448 | Train Acc: 79.19%
	 Val. Loss: 0.340 |  Val. Acc: 86.63%

You may have noticed the loss is not really decreasing and the accuracy is poor. This is due to several issues with the model which we’ll improve in the next notebook.

Finally, the metric we actually care about, the test loss and accuracy, which we get from our parameters that gave us the best validation loss.

In [ ]:

model.load_state_dict(torch.load('lstm-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')

CNN模型

In [159]:

class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, 
                 filter_sizes, output_dim, dropout, pad_idx):
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        text = text.permute(1, 0) # [batch size, sent len]
        embedded = self.embedding(text) # [batch size, sent len, emb dim]
        embedded = embedded.unsqueeze(1) # [batch size, 1, sent len, emb dim]
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
            
        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]
        
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
        
        #pooled_n = [batch size, n_filters]
        
        cat = self.dropout(torch.cat(pooled, dim=1))

        #cat = [batch size, n_filters * len(filter_sizes)]
            
        return self.fc(cat)

In [160]:

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
N_FILTERS = 100
FILTER_SIZES = [3,4,5]
OUTPUT_DIM = 1
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]


model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)
model.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)
model = model.to(device)

In [161]:

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
criterion = criterion.to(device)

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'CNN-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')
Epoch: 01 | Epoch Time: 0m 11s
	Train Loss: 0.645 | Train Acc: 62.12%
	 Val. Loss: 0.485 |  Val. Acc: 79.61%
Epoch: 02 | Epoch Time: 0m 11s
	Train Loss: 0.423 | Train Acc: 80.59%
	 Val. Loss: 0.360 |  Val. Acc: 84.63%
Epoch: 03 | Epoch Time: 0m 11s
	Train Loss: 0.302 | Train Acc: 87.33%
	 Val. Loss: 0.320 |  Val. Acc: 86.59%
Epoch: 04 | Epoch Time: 0m 11s
	Train Loss: 0.222 | Train Acc: 91.20%
	 Val. Loss: 0.306 |  Val. Acc: 87.17%
Epoch: 05 | Epoch Time: 0m 11s
	Train Loss: 0.161 | Train Acc: 93.99%
	 Val. Loss: 0.325 |  Val. Acc: 86.82%

In [162]:

model.load_state_dict(torch.load('CNN-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')
Test Loss: 0.336 | Test Acc: 85.66%
Val. Loss: 0.360 |  Val. Acc: 84.63%
Epoch: 03 | Epoch Time: 0m 11s
	Train Loss: 0.302 | Train Acc: 87.33%
	 Val. Loss: 0.320 |  Val. Acc: 86.59%
Epoch: 04 | Epoch Time: 0m 11s
	Train Loss: 0.222 | Train Acc: 91.20%
	 Val. Loss: 0.306 |  Val. Acc: 87.17%
Epoch: 05 | Epoch Time: 0m 11s
	Train Loss: 0.161 | Train Acc: 93.99%
	 Val. Loss: 0.325 |  Val. Acc: 86.82%

In [162]:

model.load_state_dict(torch.load('CNN-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')
Test Loss: 0.336 | Test Acc: 85.66%