NLTK VADER lexicon Structure for

VADER lexicon基本结构

$:  -1.5    0.80623 [-1, -1, -1, -1, -3, -1, -3, -1, -2, -1]
%)  -0.4    1.0198  [-1, 0, -1, 0, 0, -2, -1, 2, -1, 0]
%-) -1.5    1.43178 [-2, 0, -2, -2, -1, 2, -2, -3, -2, -3]
&-: -0.4    1.42829 [-3, -1, 0, 0, -1, -1, -1, 2, -1, 2]
&:  -0.7    0.64031 [0, -1, -1, -1, 1, -1, -1, -1, -1, -1]
( '}{' )    1.6 0.66332 [1, 2, 2, 1, 1, 2, 2, 1, 3, 1]
worriedly   -2.0    0.44721 [-2, -2, -3, -2, -2, -2, -2, -1, -2, -2]
worrier -1.8    0.6 [-2, -2, -1, -2, -1, -3, -2, -2, -1, -2]
worriers    -1.7    0.45826 [-2, -1, -2, -2, -2, -2, -1, -2, -1, -2]
worries -1.8    0.6 [-2, -2, -1, -2, -1, -2, -2, -3, -1, -2]
worriment   -1.5    0.67082 [-1, -2, -1, -1, -1, -2, -1, -3, -1, -2]
worriments  -1.9    0.7 [-2, -1, -2, -3, -1, -2, -3, -1, -2, -2]
worrisome   -1.7    0.64031 [-1, -1, -1, -2, -1, -2, -3, -2, -2, -2]

vader_lexicon.txt文件是以tab键分割的四列字段组成的
第一列：单词或词组（token）
第二列：人类情感打分的均值
第三列：它是单词的标准偏差，假设它遵循正态分布
第四列：这是在实验中10个人对单词进行评分的列表。
实际代码或情感计算不使用第3列和第4列。因此，如果你想根据你的需求更新词典，你可以将最后两列留空，或者用一个随机数和一个列表填充。

VADER 源码如何加载lexicon?

def __init__(self, lexicon_file="sentiment/vader_lexicon.zip/vader_lexicon/vader_lexicon.txt"):
    self.lexicon_file = nltk.data.load(lexicon_file)
    self.lexicon = self.make_lex_dict()

def make_lex_dict(self):
    """
    Convert lexicon file to a dictionary
    """
    lex_dict = {}
    for line in self.lexicon_file.split('\n'):
        (word, measure) = line.strip().split('\t')[0:2]
        lex_dict[word] = float(measure)
    return lex_dict

print(self.lexicon)
{'guiltier': -2.0, 'proud': 2.1, 'freeholds': 1.0, 'madness': -1.9, 'unsecured': -1.6, 'wilco': 0.9, 'doom': -1.7, 'crazy': -1.4, '|o:': -0.9, 'faultlessness': 1.1, 'triumphs': 2.0, 'excruciatingly': -2.9, 'warsaws': -0.2, 'insecurely': -1.4, 'abusing': -2.0, 'confusions': -0.9, 'relieve': 1.5, 'futile': -1.9, 'stinkpots': -0.7, ...}

VADER lexicon更新使用

• 预判某个词的正负中极性

import nltk
from nltk.tokenize import word_tokenize, RegexpTokenizer
from nltk.sentiment.vader import SentimentIntensityAnalyzer

Analyzer = SentimentIntensityAnalyzer()

sentence = 'enter your text to test'

tokenized_sentence = nltk.word_tokenize(sentence)
pos_word_list=[]
neu_word_list=[]
neg_word_list=[]

for word in tokenized_sentence:
    if (Analyzer.polarity_scores(word)['compound']) >= 0.1:
        pos_word_list.append(word)
    elif (Analyzer.polarity_scores(word)['compound']) <= -0.1:
        neg_word_list.append(word)
    else:
        neu_word_list.append(word)                

print('Positive:',pos_word_list)
print('Neutral:',neu_word_list)
print('Negative:',neg_word_list) 
score = Analyzer.polarity_scores(sentence)
print('\nScores:', score)

对vader lexicon可以对其进行增添新领域的lexicon

• 增加更新前

sentence = 'stocks were volatile on Tuesday due to the recent calamities in the Chinese market'

Positive: []
Neutral: ['stocks', 'were', 'volatile', 'on', 'Tuesday', 'due', 'to', 'the', 'recent', 'calamities', 'in', 'the', 'Chinese', 'markets']
Negative: []
Scores: {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0}

• 增加更新后

Analyzer.lexicon.update(Financial_Lexicon)
sentence = 'stocks were volatile on Tuesday due to the recent calamities in the Chinese market'

Positive: []
Neutral: ['stocks', 'were', 'on', 'Tuesday', 'due', 'to', 'the', 'recent', 'in', 'the', 'Chinese', 'markets']
Negative: ['volatile', 'calamities']
Scores: {'neg': 0.294, 'neu': 0.706, 'pos': 0.0, 'compound': -0.6124}