英文token预处理，用于将英文句子处理成单词

英文token预处理，用于将英文句子处理成单词

参考​​​使用

import tokenizationtokenizer = tokenization.BasicTokenizer(do_lower_case=True)f = open("sample_text.txt",mode="r",encoding="utf-8")lines = f.readlines()f2 = open("vocab.txt",mode="w",encoding="utf-8")f2.write("[PAD]")f2.write("\n")f2.write("[UNK]")f2.write("\n")f2.write("[CLS]")f2.write("\n")f2.write("[SEP]")f2.write("\n")f2.write("[MASK]")f2.write("\n")word_set = set()for line in lines: word_list = tokenizer.tokenize(line) for word in word_list: word_set.add(word)for word in list(word_set): if word!=" " and word!="": f2.write(word) f2.write("\n")f.close()f2.close()

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。