语料获取

本示例所用语料为某外卖平台用户的评论，共11987条样本数据， 其中正向评论样本4000条， 负向评论样本7987条， 点击进入下载地址

文本预处理——分词并去除停用词

分词工具采用 jieba 分词 pip install jieba
停用词表为中文停用词表, 哈工大停用词表, 百度停用词表, 四川大学机器智能实验室停用词库四份停用词表进行了合并去重，下载地址

import pandas as pd
import jieba


def load_stop_words():
    """加载停用词"""
    with open("../data_set/stop_words") as fr:
        stop_words = set([word.strip() for word in fr])
    return stop_words


if __name__ == '__main__':
    # 加载停用词
    stop_words = load_stop_words()
    # 读取文件
    df = pd.read_csv("../data_set/waimai_10k.csv")
    # 切词并过滤调停用词
    df["review"] = df["review"].map(lambda x: " ".join([i for i in jieba.cut(x) if i not in stop_words]))
    # 保存处理好的文本
    df.to_csv("./waimai.csv", index=False, header=False, columns=["review"])

处理后的文件如下

很快 好吃 味道 足 量
送水 送水 送水
快 态度
快捷 味道 可口 快 递给 力
菜 味道 很棒 送餐
师傅 手抖 微辣 辣
送餐 快 态度 特别 辛苦 谢谢
超级 快 送到 冷 天气 骑士 辛苦 谢谢你们 麻辣 香锅 依然 好吃
上次 晚 小时 超级 快 20 分钟 送到 … …
五分钟 订 卖家 特别 接单 谢谢

词向量训练

词向量训练采用 gensim 模块
pip install gensim

import pandas as pd
import gensim


if __name__ == '__main__':
    df = pd.read_csv("./waimai.csv", header=None)
    sentences = df.iloc[:, 1].astype("str").map(lambda x: x.split(" "))
    model = gensim.models.Word2Vec(sentences, size=128, workers=4, min_count=0)
    model.wv.save_word2vec_format('./word_vec.txt', binary=False)

一秒训练完。。。

10246 128
味道 -0.34176767 0.1422865 0.5661937 0.6358043 0.99514866 0.4597006 0.83270687 0.67648256 -0.16697478 -0.39428547 1.0036861 -0.11081296 0.46511438 -0.13658749 0.1482188 -0.73044026 -0.436237 0.49267843 -0.4390354 -0.18539704 0.65958226 -0.16079785 -0.274373 0.5363726 0.30086786 0.8065682 -0.1525818 -0.22661807 -0.22201005 0.4886867 0.17151305 0.6941031 -0.108169384 0.5792473 -0.32802448 -0.55200815 0.25212976 -0.48502052 -0.54447377 -0.6231095 0.074580014 -0.39396325 -0.3055023 0.26453292 -0.22691658 -0.0020165953 -0.61628485 -0.6484135 0.09854976 1.1460947 0.5731212 0.34008676 0.22624384 0.25767615 0.018547993 -0.98884386 0.3754116 0.6716524 0.6616562 0.04034343 -0.76667696 0.4891913 0.7732257 -0.35860583 0.3487415 -0.35146204 -0.22601724 0.58054966 0.4355862 0.76702976 0.301465 -1.2691803 0.5356394 0.371402 0.54972774 0.6574773 0.9245965 0.115193315 -0.4758462 0.15970539 0.0103178015 -0.116095796 0.7121025 0.21703413 -0.38246158 0.20462309 -0.4325303 -0.106235646 -0.4519505 -0.09121104 -0.59677076 -0.07223956 0.5214413 0.02620219 0.27124628 0.4202872 -1.366464 0.12932953 0.24490091 -0.6196757 0.5437061 0.61052674 1.3073382 -0.7672843 -0.72003216 0.06251624 0.26925302 0.58113956 -0.45114592 -0.8383538 0.02999392 -0.1300691 -0.9095514 -0.79298264 0.70403606 0.32387486 0.35832337 0.9410294 0.19609398 -0.21290061 -0.22644106 0.92168874 0.22732252 0.19034281 0.75808144 0.9700917 -0.15570371 0.28114718
送餐 -0.34831476 0.16106193 0.5994892 0.64805424 1.0271257 0.46728194 0.8493655 0.6830762 -0.18260212 -0.3587068 1.01343 -0.11325885 0.47106928 -0.1680743 0.1278685 -0.7440699 -0.4252064 0.47813246 -0.41524065 -0.17440696 0.6049742 -0.15751494 -0.2982898 0.52218205 0.28473657 0.8560871 -0.14586104 -0.16491853 -0.15567955 0.4590574 0.17111456 0.6245679 -0.049771745 0.63952273 -0.36627942 -0.504943 0.24357647 -0.4964198 -0.5470333 -0.63577425 0.097736515 -0.35689566 -0.28207627 0.2810563 -0.2688453 -0.06852475 -0.5780539 -0.70400196 0.12267686 1.0992405 0.62311095 0.34455365 0.2872575 0.20026866 0.032420043 -1.0238267 0.41567177 0.57241106 0.7557455 0.06543916 -0.79624844 0.42394665 0.80884355 -0.2969704 0.3194532 -0.2706822 -0.19089109 0.56951344 0.38337457 0.79456556 0.27990746 -1.2773975 0.48038802 0.307269 0.46341935 0.5919445 0.89985734 0.07449208 -0.5155747 0.099851035 0.015074631 -0.11942169 0.6676198 0.21894054 -0.32572982 0.1891366 -0.5257631 -0.12457273 -0.37080353 -0.12841094 -0.6200164 -0.042225726 0.5027024 0.17725371 0.23022962 0.29081154 -1.42769 0.13530985 0.2771318 -0.5822854 0.6126383 0.62363005 1.2924049 -0.662307 -0.7142578 0.10473512 0.18142381 0.59187007 -0.4600846 -0.85942894 0.005593317 -0.147893 -0.9587728 -0.75739866 0.65089595 0.31477287 0.31118864 0.89044017 0.22873242 -0.2516947 -0.23799604 0.95505095 0.18526594 0.22436847 0.83456284 0.9753054 -0.11093519 0.27834952
好吃 -0.30725345 0.1283905 0.52813786 0.5933264 0.9303256 0.4126685 0.7771101 0.6270748 -0.15313117 -0.37562656 0.92349505 -0.11283011 0.41913372 -0.12276961 0.14269884 -0.6803193 -0.41086897 0.4431171 -0.40913734 -0.17537531 0.6322392 -0.14475387 -0.2530998 0.4914092 0.28248012 0.74353975 -0.13958585 -0.20634086 -0.1969838 0.44981313 0.17648831 0.6567505 -0.09571447 0.52662045 -0.31215164 -0.50666744 0.22952724 -0.4708712 -0.5190871 -0.57296234 0.06719357 -0.37752706 -0.26550692 0.251655 -0.21778527 0.0020325563 -0.5790934 -0.6029802 0.086034015 1.0809773 0.52037036 0.31693274 0.21899918 0.25061807 0.011746211 -0.93361104 0.36211205 0.6272974 0.62583774 0.03684908 -0.6984929 0.45416585 0.7099049 -0.33532396 0.30931497 -0.33459026 -0.20909707 0.5540794 0.41215074 0.721903 0.2852583 -1.1962199 0.50577945 0.3373197 0.50444156 0.6198687 0.86888 0.09832464 -0.43946823 0.15845151 0.004639616 -0.10199037 0.6669751 0.20029305 -0.34718072 0.18566431 -0.4217568 -0.09939404 -0.42297262 -0.07127523 -0.5585624 -0.07119543 0.48151365 0.012957705 0.24921264 0.40019986 -1.2703121 0.12616552 0.22909844 -0.58044565 0.5054929 0.5549034 1.2014142 -0.7182815 -0.68673414 0.06984787 0.26249847 0.5499739 -0.4183708 -0.79104775 0.018213468 -0.11408542 -0.8455202 -0.74249464 0.6450488 0.299841 0.3214686 0.87345284 0.19684848 -0.19231175 -0.21840167 0.8621186 0.2152602 0.1791741 0.7084485 0.9104159 -0.142168 0.25246274
吃 -0.317348 0.1756655 0.6776852 0.7761266 1.1800449 0.48290405 1.0057433 0.7316651 -0.15752426 -0.5104708 1.1479326 -0.13435195 0.5668409 -0.13615549 0.1672698 -0.8952324 -0.5223199 0.5390238 -0.5268473 -0.21923916 0.81747895 -0.18072836 -0.3433976 0.5930387 0.36616674 0.9447755 -0.19561037 -0.23575857 -0.22093017 0.55783033 0.2159247 0.8529153 -0.14994794 0.66850096 -0.40219787 -0.6253349 0.32207093 -0.55757445 -0.6221306 -0.68742085 0.06836425 -0.47764063 -0.34166995 0.27392557 -0.21961759 0.037943263 -0.75723386 -0.79410183 0.11590063 1.3745393 0.6318284 0.3727167 0.23632705 0.32733127 -0.016460735 -1.1486912 0.46428093 0.82637024 0.76288664 0.022743462 -0.8505736 0.57333946 0.8542382 -0.41087016 0.36125845 -0.4183659 -0.2612495 0.6776614 0.5016058 0.87514096 0.42352787 -1.5055473 0.63744575 0.45063123 0.64591473 0.7956222 1.0891579 0.12250588 -0.51378715 0.20177063 0.029593572 -0.1845762 0.8167282 0.25598404 -0.4376466 0.27361465 -0.48486224 -0.094121136 -0.5315548 -0.06680334 -0.66599524 -0.1096651 0.5843629 -0.011161809 0.32369217 0.5596714 -1.6154377 0.136895 0.30053017 -0.7115952 0.66305244 0.6674846 1.5191295 -0.9946287 -0.7906497 0.04872249 0.33399847 0.68079853 -0.5693448 -0.9646293 0.018165639 -0.14496508 -1.0399712 -0.9866209 0.7748378 0.39542305 0.45934528 1.1698129 0.24988888 -0.21097943 -0.31229407 1.0970247 0.29161188 0.24489474 0.87869185 1.1473607 -0.2317162 0.28112486
送 -0.3122476 0.1951807 0.63570726 0.7103084 1.1110979 0.46854934 0.9254026 0.6948979 -0.17032832 -0.4031734 1.0291259 -0.12722725 0.5409529 -0.18306918 0.11178081 -0.80404085 -0.4670057 0.4683021 -0.431196 -0.19963683 0.6596009 -0.18137656 -0.33063754 0.535615 0.2974652 0.921764 -0.1692971 -0.14714755 -0.114276186 0.50176394 0.18717885 0.68588567 -0.06989612 0.67240196 -0.3996118 -0.5202358 0.2958069 -0.53781116 -0.5413914 -0.6655814 0.1056774 -0.40086055 -0.30898282 0.28031486 -0.25067914 -0.026333455 -0.6328047 -0.77418697 0.12626144 1.175131 0.6298331 0.3529492 0.299159 0.19088757 0.03356706 -1.0693496 0.42886 0.6319119 0.78095907 0.050119836 -0.7968591 0.417419 0.80905426 -0.2909621 0.31810242 -0.2809425 -0.19401962 0.6181103 0.38274473 0.81851476 0.3080845 -1.3709657 0.50125974 0.32389018 0.47050905 0.64839256 0.9596635 0.066670604 -0.53640294 0.14316899 0.02482079 -0.16103546 0.712954 0.22898223 -0.34023425 0.22759342 -0.5249994 -0.1033441 -0.39438552 -0.13098606 -0.6477487 -0.039757647 0.5320035 0.17372783 0.24086252 0.36096773 -1.5161527 0.12145833 0.31156737 -0.5838143 0.6931728 0.62331647 1.3688667 -0.7866663 -0.739208 0.10475284 0.1719332 0.62416357 -0.5066214 -0.8924192 0.0045560496 -0.15892388 -1.0038497 -0.8304806 0.65664387 0.34376568 0.3370172 0.96435684 0.26185748 -0.2452078 -0.26580992 1.0299401 0.20957707 0.2636418 0.8989269 1.0241199 -0.14050174 0.23097765
...

第一行为词个数及向量维数

模型搭建

建立词索引

首先我们需要先导入这么多库, 后边会用到

import sys 
from collections import defaultdict

import jieba
import gensim
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from sklearn.model_selection import train_test_split

建立词索引前先把刚刚训练好的词向量加载进来

word_vec_model = gensim.models.KeyedVectors.load_word2vec_format("word_vec.txt", binary=False)

然后建立一个字典(词-索引的映射), 一个表(索引-词向量的矩阵)

def build_embeddings_matrix(word_vec_model):
    # 初始化词向量矩阵
    embeddings_matrix = np.random.random((len(word_vec_model.wv.vocab)+1, 128))
    # 初始化词索引字典
    word_index = defaultdict(dict)

    for index, word in enumerate(word_vec_model.index2word):
        word_index[word] = index + 1
        # 预留0行给查不到的词
        embeddings_matrix[index+1] = word_vec_model.get_vector(word)
    return word_index, embeddings_matrix

word_index 是这样的

{
  "味道": 1,
  "送餐": 2,
  "好吃": 3,
  "吃": 4,
  "送": 5,
  "不错": 6,
  "小时": 7,
  "没有": 8,
  "没": 9,
  "点": 10,
  "送到": 11,
  "说": 12,
  ...
  "满头大汗": 10234,
  "心疼": 10235,
  "单说": 10236,
  "螺号": 10237,
  "多装": 10238,
  "忍心": 10239,
  "羊血": 10240,
  "大不一样": 10241,
  "连双": 10242,
  "羊肉汤": 10243,
  "几元": 10244,
  "羊血要": 10245,
  "羊汤": 10246
}

就是给每个词一个编号

embeddings_matrix 是这样的

[[ 5.69546860e-01  7.94050459e-01  9.21687010e-01 ...  5.39988930e-01  8.41095603e-01  9.84990521e-01]
 [-3.41767669e-01  1.42286494e-01  5.66193700e-01 ...  9.70091701e-01  -1.55703709e-01  2.81147182e-01]
 [-3.48314762e-01  1.61061928e-01  5.99489212e-01 ...  9.75305378e-01  -1.10935189e-01  2.78349519e-01]
 ...
 [-5.99376392e-03  8.19191220e-04  9.59137175e-03 ...  1.21581573e-02  1.40984030e-03 -9.48444358e-04]
 [-1.57012604e-04 -2.43240129e-03  8.13777931e-03 ...  1.11063607e-02  -4.23950190e-03  7.17222691e-04]
 [-2.05256371e-03 -1.63255120e-03  4.44210222e-04 ...  6.44489145e-03  -1.32397411e-03 -2.05684011e-03]]

每个词的向量, 例如 好吃 这个词的index是3, 那么这个词的向量就是上表当中的第三个数组, 即

 [-3.48314762e-01  1.61061928e-01  5.99489212e-01 ...  9.75305378e-01  -1.10935189e-01  2.78349519e-01]

这个过程也可以用 keras 的 keras.preprocessing.text 模块实现

生成三组数据集(训练集, 验证集, 测试集)

def train_data(word_index):
    df = pd.read_csv("./waimai.csv", names=["label", "review"])
    df["word_index"] = df["review"].astype("str").map(lambda x: np.array([word_index.get(i, 0) for i in x.split(" ")]))
    # 填充及截断
    train = keras.preprocessing.sequence.pad_sequences(df["word_index"].values, maxlen=20, padding='post', truncating='post', dtype="float32")
    x_train, x_test, y_train, y_test = train_test_split(train, df["label"].values, test_size=0.2, random_state=1)
    # 从训练集上分出验证集
    x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.15)
    return x_train, x_val, x_test, y_train, y_val, y_test

这几个数据分别长这个样子

x_train x_val x_test

[[5.700e+01 3.790e+02 1.600e+02 ... 1.123e+03 3.320e+02 1.766e+03]
 [4.000e+00 1.800e+01 2.570e+02 ... 0.000e+00 0.000e+00 0.000e+00]
 [1.400e+02 4.010e+02 6.560e+02 ... 0.000e+00 0.000e+00 0.000e+00]
 ...
 [1.120e+02 2.500e+02 2.400e+01 ... 0.000e+00 0.000e+00 0.000e+00]
 [4.026e+03 5.930e+02 3.100e+02 ... 0.000e+00 0.000e+00 0.000e+00]
 [5.700e+02 5.646e+03 4.000e+00 ... 0.000e+00 0.000e+00 0.000e+00]]

其实就是每句话每个词的 index 集合, 至于为啥不是整数, 可能是 numpy 处理过程当中把它转为了浮点, 不过不影响, 单独取出一组数看是没问题的

[114.  14. 405. 100.   7.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]

根据词的index查 word_index 可以知道这组向量的原句是

[行 速度 好慢 一个多 小时]

后边的 0 向量是对句子的填充, 我们需要保证每组向量的维度为20, 多的截断, 少的用 0 填充

y_train y_val y_test

[0 0 0 ... 0 0 1]

这个就是 label, 1好评, 0差评

构建模型

def build_model(word_index, embeddings_matrix):
    model = keras.Sequential()
    model.add(keras.layers.Embedding(input_dim=len(word_index)+1, 
                                    output_dim=128, 
                                    weights=[embeddings_matrix],
                                    input_length=20,
                                    trainable=False))
    model.add(keras.layers.GlobalAveragePooling1D())
    model.add(keras.layers.Dense(32, activation=tf.nn.relu))
    model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))
    
    model.compile(optimizer=tf.train.AdamOptimizer(),
                    loss='binary_crossentropy',
                    metrics=['accuracy'])
    model.summary()
    return model

keras.layers.Embedding: embedding层

• input_dim: 词个数(+1是因为我们预留了第0位给查不到的词)
• output_dim: 词向量维度
• weights: 预训练的向量矩阵
• input_length: 截断长度
• trainable: 因为我们的词向量是预训练的, 所以训练过程当中不需要再对向量矩阵进行调整了

keras.layers.GlobalAveragePooling1D: 通过对序列维度求平均值，针对每个样本返回一个长度固定的输出向量。这样，模型便能够以尽可能简单的方式处理各种长度的输入。
keras.layers.Dense: 全连接 (Dense) 层（包含 32个隐藏单元）。
keras.layers.Dense: 同上, 输出层, 因为是二分类问题, 所以连1个隐藏单元就可以了, 激活函数选 sigmoid

model.compile 是为模型选择的优化方式, 可以看我的另一篇博客博客, 或者查阅TensorFlow Keras 官方文档

model.summary 输出模型各层信息, 如下

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 20, 128)           1311616   
_________________________________________________________________
global_average_pooling1d (Gl (None, 128)               0         
_________________________________________________________________
dense (Dense)                (None, 32)                4128      
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 33        
=================================================================
Total params: 1,315,777
Trainable params: 4,161
Non-trainable params: 1,311,616
_________________________________________________________________

训练

迭代50次

model.fit(x_train, y_train, epochs=100, validation_data=(x_val, y_val))

Train on 8150 samples, validate on 1439 samples
Epoch 1/50
OMP: Info #250: KMP_AFFINITY: pid 27275 tid 27289 thread 1 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 27275 tid 27288 thread 2 bound to OS proc set 0
8150/8150 [==============================] - 1s 111us/sample - loss: 0.6213 - acc: 0.6618 - val_loss: 0.6101 - val_acc: 0.6782
Epoch 2/50
8150/8150 [==============================] - 0s 61us/sample - loss: 0.6153 - acc: 0.6649 - val_loss: 0.6113 - val_acc: 0.6782
Epoch 3/50
8150/8150 [==============================] - 1s 63us/sample - loss: 0.6138 - acc: 0.6648 - val_loss: 0.6102 - val_acc: 0.6782
Epoch 4/50
8150/8150 [==============================] - 1s 62us/sample - loss: 0.6127 - acc: 0.6655 - val_loss: 0.6127 - val_acc: 0.6782
...
Epoch 46/50
8150/8150 [==============================] - 0s 60us/sample - loss: 0.5445 - acc: 0.7245 - val_loss: 0.5325 - val_acc: 0.7262
Epoch 47/50
8150/8150 [==============================] - 1s 62us/sample - loss: 0.5394 - acc: 0.7303 - val_loss: 0.5336 - val_acc: 0.7457
Epoch 48/50
8150/8150 [==============================] - 1s 63us/sample - loss: 0.5411 - acc: 0.7252 - val_loss: 0.5339 - val_acc: 0.7519
Epoch 49/50
8150/8150 [==============================] - 1s 64us/sample - loss: 0.5400 - acc: 0.7302 - val_loss: 0.5296 - val_acc: 0.7366
Epoch 50/50
8150/8150 [==============================] - 1s 65us/sample - loss: 0.5383 - acc: 0.7298 - val_loss: 0.5343 - val_acc: 0.7165

准确率在 70% 左右

评估

在测试集上进行效果评估

results = model.evaluate(x_test, y_test)
print(f"损失: {results[0]}, 准确率: {results[1]}")

2398/2398 [==============================] - 0s 36us/sample - loss: 0.5138 - acc: 0.7627
损失: 0.5137957387213115, 准确率: 0.762718915939331

模型保存

model.save_weights('./model/waimai_model')

目录结构如下

model
├── checkpoint
├── waimai_model.data-00000-of-00002
├── waimai_model.data-00001-of-00002
└── waimai_model.index

0 directories, 4 files

模型载入

import gensim

from train import build_model, build_embeddings_matrix


if __name__ == '__main__':
    word_vec_model = gensim.models.KeyedVectors.load_word2vec_format("word_vec.txt", binary=False)
    word_index, embeddings_matrix = build_embeddings_matrix(word_vec_model)
    model = build_model(word_index, embeddings_matrix)
    model.load_weights("./model/waimai_model")

载入模型时需保证模型结构与训练时结构一致, 所以需要调用 build_model 方法创建同样结构的模型

预测

import jieba
import gensim
import numpy as np
from tensorflow import keras

from train import build_model, build_embeddings_matrix
from text_preprocessing import load_stop_words


if __name__ == '__main__':
    word_vec_model = gensim.models.KeyedVectors.load_word2vec_format("word_vec.txt", binary=False)
    word_index, embeddings_matrix = build_embeddings_matrix(word_vec_model)
    model = build_model(word_index, embeddings_matrix)
    model.load_weights("./model/waimai_model")
    
    stop_words = load_stop_words()
    
    while True:
        text = input("请输入一句话：")
        text = [word_index.get(word, 0) for word in jieba.cut(text)]
        text = keras.preprocessing.sequence.pad_sequences([text], maxlen=20, padding='post', truncating='post', dtype="float32")

        res = model.predict(text)[0][0]
        if res >= 0.5:
            print(f"好评, 得分: {res*100}")
        else:
            print(f"差评，得分: {res*100}")

        print()

试一下

请输入一句话：特别好吃，量特大，而且送餐特别快，特别特别棒
好评, 得分: 71.35688066482544

请输入一句话：外送员很赞，商家能不能仔细看订单啊！点的干拌面送来的是汤面，说了粉汤羊血要多加辣椒送来的一点儿辣没有！！
差评，得分: 12.211278080940247

请输入一句话：肉夹馍肉太少
差评，得分: 45.60675919055939

请输入一句话：下了单说一个小时之后送，过了一个小时送餐员打电话又说晚15分钟，而且态度不好！味道也一般，跟小螺号比差远了，再也不点了
差评，得分: 4.753769561648369

请输入一句话：味道不错，份量很足，建议都点小份。红薯超好吃就是太烂了容易碎
差评，得分: 36.52855157852173

以上就是使用 tf.keras 实现的简单的文本二分类, 由于语料集大小的问题, 预训练的词向量表征不够好, 大家也可以使用更多的语料进行词向量的预训练, 或者使用网上开源的词向量, 项目的完整代码可以查看我的github