onehot编码是一种稀疏编码方式,如果词语越多,维度也越大.会出现维数灾难.
针对one-hot编码,sklearn中实现如下.CountVectorizer类(计数向量)
先用英文举例.
它会针对每个单词计数,丢失位置信息

from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())

print(X.toarray())

下面这个是ngram为2的编码方式

vectorizer2 = CountVectorizer(analyzer='word', ngram_range=(2, 2))
X2 = vectorizer2.fit_transform(corpus)
print(vectorizer2.get_feature_names())