RAKE算法
RAKE算法用来做关键词(key word)的提取,实际上提取的是关键的短语(phrase),并且倾向于较长的短语,在英文中,关键词通常包括多个单词,但很少包含标点符号和停用词,例如and,the,of等,以及其他不包含语义信息的单词。
算法思想
- RAKE算法首先使用标点符号(如半角的句号、问号、感叹号、逗号等)将一篇文档分成若干分句。
- 然后对于每一个分句,使用停用词作为分隔符将分句分为若干短语,这些短语作为最终提取出的关键词的候选词。
- 最后,每个短语可以再通过空格分为若干个单词,可以通过给每个单词赋予一个得分,通过累加得到每个短语的得分。
一个关键点在于将这个短语中每个单词的共现关系考虑进去。最终定义的公式是:
词共现矩阵的两种不同含义:
- 词共现矩阵的每一列的值即为该词的度deg(是一个网络中的概念,每与一个单词共现在一个短语中,度就加1,考虑该单词本身),每个词在文本中出现的次数即为频率freq。
- 通过统计一个事先指定大小的窗口内的word共现次数,以word周边的共现词的次数做为当前word的vector。具体来说,我们通过从大量的语料文本中构建一个共现矩阵来定义word representation。
对于第一解释存在以下示例:
共词矩阵对角线上元素赋值为它自身在所有文章出现次数。循环遍历特征词列表,构建全部两个词之间的组合,再遍历每一篇文章的切词结果,如果该两个词在同一片文章中出现,则该两词的权重+1,再将其存入共词矩阵的对应位置中。
实现可以参照[4]或[7]。
对于第二种解释有以下示例:
I like deep learning.
I like NLP.
I enjoy flying.
假设窗口大小为2,其词共现矩阵如下表所示:
表1 词共现矩阵
count | I | like | enjoy | deep | learning | NLP | flying | . |
I | 0 | 2 | 1 | 0 | 0 | 0 | 0 | 0 |
like | 2 | 0 | 0 | 1 | 0 | 1 | 0 | 0 |
enjoy | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
deep | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 |
learning | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 |
NLP | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
flying | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 |
. | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 |
其中(I,like)=2是因为窗口等于3,“I like deep”和“I like NLP”中各出现了一次,其他的依此类推。
附页
代码实现
import re
import operator
import argparse
import codecs
def isNumber(s):
try:
float(s) if '.' in s else int(s)
return True
except ValueError:
return False
class Rake:
def __init__(self, inputFilePath, stopwordsFilePath, outputFilePath, minPhraseChar, maxPhraseLength):
self.outputFilePath = outputFilePath
self.minPhraseChar = minPhraseChar
self.maxPhraseLength = maxPhraseLength
# read documents
self.docs = []
for document in codecs.open(inputFilePath, 'r', 'utf-8'):
self.docs.append(document)
# read stopwords
stopwords = []
for word in codecs.open(stopwordsFilePath, 'r', 'utf-8'):
stopwords.append(word.strip())
stopwordsRegex = []
for word in stopwords:
regex = r'\b' + word + r'(?![\w-])'
stopwordsRegex.append(regex)
self.stopwordsPattern = re.compile('|'.join(stopwordsRegex), re.IGNORECASE)
def separateWords(self, text):
splitter = re.compile('[^a-zA-Z0-9_\\+\\-/]')
words = []
for word in splitter.split(text):
word = word.strip().lower()
# leave numbers in phrase, but don't count as words, since they tend to invalidate scores of their phrases
if len(word) > 0 and word != '' and not isNumber(word):
words.append(word)
return words
def calculatePhraseScore(self, phrases):
# calculate wordFrequency and wordDegree
wordFrequency = {}
wordDegree = {}
for phrase in phrases:
wordList = self.separateWords(phrase)
wordListLength = len(wordList)
wordListDegree = wordListLength - 1
for word in wordList:
wordFrequency.setdefault(word, 0)
wordFrequency[word] += 1
wordDegree.setdefault(word, 0)
wordDegree[word] += wordListDegree
for item in wordFrequency:
wordDegree[item] = wordDegree[item] + wordFrequency[item]
# calculate wordScore = wordDegree(w)/wordFrequency(w)
wordScore = {}
for item in wordFrequency:
wordScore.setdefault(item, 0)
wordScore[item] = wordDegree[item] * 1.0 / wordFrequency[item]
# calculate phraseScore
phraseScore = {}
for phrase in phrases:
phraseScore.setdefault(phrase, 0)
wordList = self.separateWords(phrase)
candidateScore = 0
for word in wordList:
candidateScore += wordScore[word]
phraseScore[phrase] = candidateScore
return phraseScore
def execute(self):
file = codecs.open(self.outputFilePath,'w','utf-8')
for document in self.docs:
# split a document into sentences
sentenceDelimiters = re.compile(u'[.!?,;:\t\\\\"\\(\\)\\\'\u2019\u2013]|\\s\\-\\s')
sentences = sentenceDelimiters.split(document)
# generate all valid phrases
phrases = []
for s in sentences:
tmp = re.sub(self.stopwordsPattern, '|', s.strip())
phrasesOfSentence = tmp.split("|")
for phrase in phrasesOfSentence:
phrase = phrase.strip().lower()
if phrase != "" and len(phrase) >= self.minPhraseChar and len(phrase.split()) <= self.maxPhraseLength:
phrases.append(phrase)
# calculate phrase score
phraseScore = self.calculatePhraseScore(phrases)
keywords = sorted(phraseScore.items(), key = operator.itemgetter(1), reverse=True)
file.write(str(keywords[0:int(len(keywords)/3)]) + "\n")
file.close()
def readParamsFromCmd():
parser = argparse.ArgumentParser(description = "This is a python implementation of rake(rapid automatic keyword extraction).")
parser.add_argument('inputFilePath', help = 'The file path of input document(s). One line represents a document.')
parser.add_argument('stopwordsFilePath', help = 'The file path of stopwords, each line represents a word.')
parser.add_argument('-o', '--outputFilePath', help = 'The file path of output. (default output.txt in current dir).', default = 'output.txt')
parser.add_argument('-m', '--minPhraseChar', type = int, help = 'The minimum number of characters of a phrase.(default 1)', default = 1)
parser.add_argument('-a', '--maxPhraseLength', type = int, help = 'The maximum length of a phrase.(default 3)', default = 3)
return parser.parse_args()
params = readParamsFromCmd().__dict__
rake = Rake(params['inputFilePath'], params['stopwordsFilePath'], params['outputFilePath'], params['minPhraseChar'], params['maxPhraseLength'])
rake.execute()