Python-jieba库

jieba库基本介绍

• jieba库概述

  jieba是优秀的中文分词第三方库

  • 中文文本需要通过分词获得单个的词语
  • jieba是优秀的中文分词第三方库，需要额外安装
  • jieba库提供三种分词模式，最简单只需掌握一个函数

• jieba库安装

pip install jieba

• jieba分词原理

  jieba分词依靠中文词库

  • 利用一个中文词库，确定汉字之间的关联概率
  • 汉字间概率大的组成词组，形成分词结果
  • 除了分词，用户还可以添加自定义的词组

jieba库使用说明

• jieba分词的三种模式

  精确模式、全模式、搜索引擎模式

  • 精确模式：把文本精确的切分开，不存在冗余单词
  • 全模式：把文本中所有可能的词语都扫描出来，有冗余
  • 搜索引擎模式：在精确模式基础上，对长词再次切分

• jieba库常用函数

实例 文本词频统计

• 英文文本： Hamet 分析词频
  https://python123.io/resources/pye/hamlet.txt
• 中文文本：《三国演义》 分析人物
  https://python123.io/resources/pye/threekingdoms.txt

"Hamlet英文词频统计"实例讲解

#CalHamletV1.pydef getText():txt = open("hamlet.txt", "r").read()txt = txt.lower()for ch in '!"#$%&()*+,-./:;<=>?@[\\]^_‘{|}~':txt = txt.replace(ch, " ")retu txthamletTxt = getText()words = hamletTxt.split()counts = {}for word in words:counts[word] = counts.get(word,0) + 1items = list(counts.items())items.sort(key=lambda x:x[1], reverse=True)for i in range(10):word, count = items[i]print("{0:<10}{1:>5}".format(word, count))

《三国演义》人物出场统计

#CalThreeKingdomsV2.pyimport jiebatxt = open("threekingdoms.txt", "r", encoding="utf-8").read()excludes = {"将军","却说","荆州","二人","不可","不能","如此"}words = jieba.lcut(txt)counts = {}for word in words:if len(word) == 1:continueelif word == "诸葛亮" or word == "孔明曰":rword = "孔明"elif word == "关公" or word == "云长":rword = "关羽"elif word == "玄德" or word == "玄德曰":rword = "刘备"elif word == "孟德" or word == "丞相":rword = "曹操"else:rword = wordcounts[rword] = counts.get(rword,0) + 1for word in excludes:del counts[word]items = list(counts.items())items.sort(key=lambda x:x[1], reverse=True)for i in range(10):word, count = items[i]print("{0:<10}{1:>5}".format(word, count))