{"id":15942112,"url":"https://github.com/discretetom/nlp-homework","last_synced_at":"2025-03-30T04:22:22.874Z","repository":{"id":114055085,"uuid":"163816026","full_name":"DiscreteTom/NLP-homework","owner":"DiscreteTom","description":"大三上-自然语言处理导论-作业","archived":false,"fork":false,"pushed_at":"2019-01-13T07:57:12.000Z","size":8304,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-05T06:38:58.189Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DiscreteTom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-02T08:43:30.000Z","updated_at":"2023-12-21T15:07:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"4753c7e1-c627-402d-87c3-3810cf5f0266","html_url":"https://github.com/DiscreteTom/NLP-homework","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2FNLP-homework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2FNLP-homework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2FNLP-homework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DiscreteTom%2FNLP-homework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DiscreteTom","download_url":"https://codeload.github.com/DiscreteTom/NLP-homework/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246274646,"owners_count":20751112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-07T07:40:21.269Z","updated_at":"2025-03-30T04:22:22.867Z","avatar_url":"https://github.com/DiscreteTom.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NLP-homework\n\n大三上-自然语言处理导论作业\n\n## 环境\n\npython3\n\n## Project-1 - Chinese word segmentation.\n\n### 任务定义\n\nThis task provides PKU data as training set and test set(e.g., you can use 80% data for model training and other 20% for testing), and you are free to use data learned or model trained from any resources.\n\nEvaluation Metrics:\n- Precision = (Number of words correctly segmented) / (Number of words segmented) * 100%\n- Recall = (Number of words correctly segmented) / (Number of words in the reference) * 100%\n- F-measure = 2 * P * R / (P + R)\n\n### 文件目录\n\n- project-1/\n  - 1.py - 源代码\n  - lcs.py - 基于动态规划算法的求最长公共子序列函数，用于在1.py中计算困惑矩阵\n  - play.py - 基于1.py改造而成的玩具程序，用于自定义输入的分词\n  - data/\n    - pku_training.utf8 - 北京大学中文分词训练词库，词与词之间使用两个空格分隔，段落之间使用换行符分隔\n    - pku_training_words.utf8 - 北京大学中文分词训练词库，词与词之间使用换行符分隔\n    - pku_test.utf8 - 北京大学中文分词测试词库。为未分词的文章\n    - pku_test_gold.utf8 - pku_test.utf8分词后的标准答案\n\n### 方法描述\n\n主要思想：**基于窗口的前向最大匹配**，即在最大前向匹配的基础上添加一个look ahead窗口，防止匹配`北京大学`时因为有`北京`这个词却没有`北京大`这个词而拆分`北京大学`为`北京`和`大学`这样的情况。\n\n由于look ahead窗口的可变性，把分词器设置为一个class - `Segmentor`，而look ahead窗口的大小设置在其构造函数中，作为参数`lookAhead`，默认窗口大小为1。\n\n`Segmentor`类的结构如下：\n- `wordSet`\n  - 词库，数据结构为`set`，存放所有中文词\n- `lookAhead`\n  - look ahead窗口大小，默认为1\n- `__init__(self, fileName = '', lookAhead = 1)`\n  - 构造函数。初始化成员变量。\n  - 初始化look ahead窗口大小。\n  - 如果给出了文件名则调用函数`self.openFile`。\n- `openFile(self, fileName)`\n  - 把指定文件作为训练文件导入`wordSet`\n- `parse(self, string)`\n  - 基于`wordSet`解析输入串\n\n`Segmentor.parse`函数代码逻辑如下：\n\n```python\ndef parse(self, string):\n'parse string using word set and return result as an array'\n\nresultWords = [] # 保存分词结果，是词的list\ncurrentLookAhead = 0 # 当前look ahead的情况，用来和self.lookAhead对比\nstartIndex = 0 # 指向当前词的词首\ncurrentIndex = 0 # 指向当前词的词尾\n\nwhile currentIndex \u003c len(string): # 当前词的词尾不是输入串尾\n\tif currentLookAhead \u003c self.lookAhead: # look ahead的字符数小于look ahead窗口\n\t\tcurrentLookAhead += 1 # 向前看+1\n\t\tif currentIndex + currentLookAhead \u003c len(string) \\\n\t\tand string[startIndex : currentIndex + currentLookAhead + 1] in self.wordSet:\n\t\t\t# 向前看未越界且匹配到了更长的词\n\t\t\tcurrentIndex += currentLookAhead # 刷新词尾指针\n\t\t\tcurrentLookAhead = 0 # 重置当前look ahead字符数\n\t\telif currentIndex + currentLookAhead \u003e= len(string): # 向前看越界\n\t\t\tresultWords.append(string[startIndex : currentIndex + 1]) # 把当前词加入结果\n\t\t\tstartIndex = currentIndex + 1 # 刷新词首指针为上一个词的词尾+1\n\t\t\tcurrentIndex += 1 # 刷新词尾指针为上一个词的词尾+1\n\t\t\tcurrentLookAhead = 0 # 重置look ahead字符数\n\telse: # look ahead超过窗口大小\n\t\tresultWords.append(string[startIndex : currentIndex + 1]) # 把当前词加入结果\n\t\tstartIndex = currentIndex + 1 # 刷新词首指针为上一个词的词尾+1\n\t\tcurrentIndex += 1 # 刷新此位置指针为上一个词的词尾+1\n\t\tcurrentLookAhead = 0 # 重置look ahead字符数\n\n# 词串解析完毕。startIndex和currentIndex可能还指着最后一个词\nif startIndex \u003c currentIndex: # 如果指向最后一个词\n\tresultWords.append(string[startIndex : currentIndex + 1]) # 把最后一个词加入结果\n\nreturn resultWords\n```\n\n### 程序工作流程\n\n```python\n# 定义分词器，指定训练文件和look ahead窗口尺寸\nsegmentor = Segmentor('data/pku_training_words.utf8', 5)\n\n# 为了计算困惑矩阵，定义一些计数器\nwordsSegmented = 0 # 分词器分出了多少词\ncorrectlySegmented = 0 # 分词器正确分出了多少词\nshouldBeSegmented = 0 # 分词器应该分出多少词\n\n# 考虑到需要最长公共子序列需要一个很大的二维matrix，所以对于输入的测试文件，一行一行分析\ntestFile = open('data/pku_test.utf8', 'r', encoding = 'utf-8')\ngoldFile = open('data/pku_test_gold.utf8', 'r', encoding = 'utf-8')\ntestLine = testFile.readline() # 测试文件读一行\ngoldWords = goldFile.readline().split('  ') # 结果文件读一行\nwhile len(testLine) and len(goldWords): # 没有读到文件尾\n\ttestWords = segmentor.parse(testLine) # 解析一行，得到结果词串\n\twordsSegmented += len(testWords) # 计数\n\tcorrectlySegmented += lcs.LCS_Length(testWords, goldWords) # 使用最长公共子序列计算正确分词个数，计数\n\tshouldBeSegmented += len(goldWords) # 计数\n\ttestLine = testFile.readline() # 继续循环\n\tgoldWords = goldFile.readline().split('  ') # 继续循环\ntestFile.close() # 关闭文件\ngoldFile.close()\n\n# 输出结果\nprint('wordsSegmented =', wordsSegmented)\nprint('correctlySegmented =', correctlySegmented)\nprint('shouldBeSegmented =', shouldBeSegmented)\np = correctlySegmented / wordsSegmented\nr = correctlySegmented / shouldBeSegmented\nprint('precision = ', p)\nprint('recall = ', r)\nprint('F-measure =', 2 * p * r / (p + r))\n```\n\n### 输入输出与运行截图\n\n以下测试均使用project-1/data/pku_training_words.utf8作为训练数据，向前窗口大小为5\n\n1.py不需要输入，输出如下：\n\n![1-1](img/1-1.png)\n\nplay.py可以自由输入，样例输入与输出如下：\n\n![1-2](img/1-2.png)\n\n如果把向前窗口设置为1，则不能够正确解析“北京大学”：\n\n![1-3](img/1-3.png)\n\n### 结果分析\n\n测试改变向前窗口大小对结果造成的影响。\n\n| 向前窗口大小 | 正确率 | 召回率 | F综合评估 |\n| --- | --- | --- | --- |\n| 1 | 80.49% | 70.08% | 0.85 |\n| 2 | 83.29% | 90.72% | 0.87 |\n| 3 | 84.56% | 91.00% | 0.88 |\n| 5 | 84.72% | 91.04% | 0.88 |\n| 8 | 84.74% | 91.05% | 0.88 |\n\n可以看出：\n- 增大向前窗口的大小可以提升系统性能。但是向前窗口为默认大小1的时候性能也不是很差。\n- 向前窗口大小由1提升为2的时候能够明显提升系统性能，但是向前窗口继续增加时性能提升并不明显。原因应该是没有很多很长的中文词。如果要继续提升性能，应该尝试优化算法而不是继续改变向前窗口大小。\n\n## Project-2 - N-gram Language Models.\n\n### 任务定义\n\nIn this assignent you will explore a simple, typical N-gram language model.\n\nThis model can be trained and tested on sentence-segented data of a Chinese text corpus. \"Word Perplexity\" is the most widely-used evaluation metric for language models.\n\nAdditional points:\n- Test how does the different \"Word Perplexity\" of the different \"N\" grams.\n- Test how does the different \"Word Perplexity\" of the different smoothing methods.\n\n### 文件目录\n\n- project-2/\n  - 2.py - 程序源代码\n  - data/\n    - TheThreeBodyProblem.txt - 小说《三体》，用作训练数据\n\n### 方法描述\n\n使用传统n gram模型\n\n定义一个class - `N_Gram`，其构造函数的参数包括了模型参数`n`。类内有不同的平滑方案，定义类的对象之后调用不同的平滑方案来得到不同的平滑结果。\n\n`N_Gram`类的结构如下：\n- `history`\n  - 历史的长度，即`n-1`\n- `data`\n  - 保存前`n-1`个词确定时最后一个词的出现次数。用来和`dataTimes`结合计算概率。\n- `dataTimes`\n  - 保存前`n-1`个词构成的词串的出现次数。用来和`data`结合计算概率。\n- `wordSet`\n  - 所有中文字的集合\n- `smoothing`\n  - 保存当前使用了哪种平滑方法\n- `__init__(self, n, fileNme = '')`\n  - 初始化成员变量。\n  - 根据参数`n`获得`history = n - 1`。\n  - 如果给出了`fileName`则调用`openFile`函数读取文件。\n- `openFile(self, fileName)`\n  - 根据指定文件填充`wordSet`。\n  - 以行为单位构造`data`和`dataTimes`\n  - 构造时会根据n gram的参数n在行首和行位添加适量的标记符\n- `historyGenerator(self, historyLength = -1)`\n  - 用于根据`self.history`长度和`self.wordSet`来构造历史词\n  - 为递归函数，使用了生成器技术，减少内存的使用\n- `additiveSmoothing(self, n = 1.0)`\n  - 使用加性平滑处理`data`和`dataTimes`，平滑系数n由参数给出\n  - 把`self.smoothing`设置为additive smoothing\n- `goodTuringSmoothing(self)`\n  - 使用Good-Turing平滑处理`data`和`dataTimes`\n  - 对于`n(r+1)`为0的情况，跳过不处理\n  - 把`self.smoothing`设置为Good-Turing smoothing\n- `parse(self, s)`\n  - 解析输入串`s`\n  - 如果输入串包含字库中没有的字，停止解析\n  - 如果输入串长度过短，停止解析\n  - 如果输入串中的字都存在在字库中，但是出现了`data`中没有的情况，则需要平滑处理，输出提示并停止解析\n  - 解析成功时，输出困惑度\n  - 如果使用了平滑方案，输出困惑度时会输出平滑方案\n\n`N_Gram.parse`代码逻辑如下：\n\n```python\ndef parse(self, s):\n# 判断是否存在不在词库中的字\nfor word in s:\n\tif not word in self.wordSet:\n\t\tprint('unknow word')\n\t\treturn\n\n# 判断输入串长度是否过短\ns = self.history * '\\n' + s + self.history * '\\n'\nif len(s) \u003c self.history + 1:\n\tprint('string too short')\nelse:\n\tresult = 1 # 结果困惑度\n\tfor i in range(self.history, len(s)):\n\t\tif s[i - self.history : i] in self.data and s[i] in self.data[s[i - self.history : i]]:\n\t\t\t# data中存在此情况\n\t\t\t# 计算困惑度\n\t\t\tresult *= (self.data[s[i - self.history : i]][s[i]] / \\\n\t\t\tself.dataTimes[s[i - self.history : i]]) ** -(1 / len(s)) # 乘方运算在此处进行以防止result太小变成0\n\t\telse:\n\t\t\t# 未知情况，应该平滑\n\t\t\tprint('error, unknow situation, please try to smooth data')\n\t\t\treturn\n\tif len(self.smoothing): # 如果使用了平滑策略，则输出平滑策略\n\t\tprint('After', self.smoothing, ':', end = '')\n\tprint('Word Perplexity:', result) # 输出困惑度\n```\n\n### 程序工作流程\n\n以使用bigram为例：\n\n```python\n# 定义5种bigram：不平滑、0.3-加性平滑、0.5-加性平滑、1-加性平滑和GT平滑\nb = N_Gram(2, 'data/TheThreeBodyProblem.txt') # bigram\nb_0_3 = N_Gram(2, 'data/TheThreeBodyProblem.txt') # bigram with add-0.3 smoothing\nb_0_3.additiveSmoothing(0.3)\nb_0_5 = N_Gram(2, 'data/TheThreeBodyProblem.txt') # bigram with add-0.5 smoothing\nb_0_5.additiveSmoothing(0.5)\nb_1 = N_Gram(2, 'data/TheThreeBodyProblem.txt') # bigram with add-1 smoothing\nb_1.additiveSmoothing(1)\nb_gt = N_Gram(2, 'data/TheThreeBodyProblem.txt') # bigram with good-turing smoothing\nb_gt.goodTuringSmoothing()\n\ns = input('input a line to parse, input a blank line to stop:')\nwhile len(s):\n\tb.parse(s)\n\tb_0_3.parse(s)\n\tb_0_5.parse(s)\n\tb_1.parse(s)\n\tb_gt.parse(s)\n\ts = input('input a line to parse, input a blank line to stop:')\n```\n\n### 输入输出与运行截图\n\n以bigram为例\n\n初始状态，系统读入文件，输出提示信息：\n\n![2-1](img/2-1.png)\n\n输入串，解析并获得困惑度：\n\n![2-2](img/2-2.png)\n\n输入一个不平滑时无法解析的串：\n\n![2-3](img/2-3.png)\n\n输入一个含有字库外的字的串：\n\n![2-4](img/2-4.png)\n\n### 结果分析\n\n输入串为“天气不错，要不要出去走走？”，对于不同的n gram参数n，以及不同的平滑方案，得到困惑度表格如下：\n\n| n | 不平滑 | 0.3加性平滑 | 0.5加性平滑 | 1加性平滑 | GT平滑 |\n| --- | --- | --- | --- | --- | --- |\n| 1 | 379.56 | 379.92 | 380.16 | 380.76 | 393.33 |\n| 2 | 118.14 | 178.71 | 207.71 | 266.81 | 129.30 |\n\nn=3时因内存不足(测试环境内存8G)，无法得到训练结果。\n\n结论：\n- 无论是否采取平滑或采取任何平滑，增加n都是减少困惑度最有效的方案\n- 模型所需内存随着n的增加而非常快速地增加，故高阶可计算性较差\n- 因为平滑把词库中存在的词的概率分配给了不存在的词，所以不平滑相比平滑困惑度较低，但是不平滑可能会无法识别一些词库外的情况。\n- 加性平滑的参数不建议使用1，使用较小的参数能够得到更低的困惑度\n- n=1时频率为0的情况较少，GT平滑效果不如加性平滑。但是n\u003e=2时概率为0的情况急剧增多，GT平滑效果明显优于加性平滑。\n\n## Project-3 - Part-of-speech tagging.\n\n### 任务定义\n\nThis data set contains one month of Chinese daily which are segmented and POS tagged under Peking Univ. standard.\n\nProject ideas:\n- Design a sequence learning methods to predicate a POS tags for each word in sentences.\n- Use 80% data for model training and other 20% for testing(or 5-fold cross validation to test learner's performance. So it could be interesting to seperate dataset).\n\n### 文件目录\n\n- project-3/\n  - 3.py - 程序源代码\n  - data/\n    - data.txt - 来自老师邮箱的数据集，已使用正则表达式去除了里面的中括号、大括号和连续换行等无关内容\n\n### 方法描述与程序执行流程\n\n使用80%数据作为训练集，20%数据作为测试集。使用隐马尔可夫模型，1-加性平滑策略。\n\n此次没有需要手动输入的模型参数，故没有像前两题一样设置一个class出来，完全使用面向过程编程。\n\n程序执行流程\n1. 加载文件，数据保存到`rawData`\n2. 从`rawData`中获取`tags`和`words`\n3. 把`rawData`按照比例划分为`trainingData`和`testData`\n4. 定义模型参数`tagTrans`为词性标签之间的状态转移概率，`emit`为某个标签发射某个词的概率\n5. 使用`trainingData`训练模型参数`tagTrans`和`emit`\n6. 使用Viterbi算法处理`testData`，得到模型输出结果\n7. 计数与输出\n\n处理`testData`的流程：\n\n```python\nprint('start parsing...') # 输出提示信息\nallCorrectCount = 0 # 整个测试数据中正确标签的词数。忽略段落起始符和结束符\nallWordsCount = 0 # 整个测试数据的词数，忽略段落起始符和结束符\n# 最终精度 = allCorrectCount / allWordsCount\nfor index in range(len(testData)):\n\tline = testData[index] # 每次从测试数据中提取一行，即line\n\tallWordsCount += len(line) # 计数，此时line不包括段落起始符和结束符\n\tline = ['$start$/$start$'] + line + ['$end$/$end$'] # 加上段落起始符和结束符\n\t# 从line中提取词和标签\n\ttestWords = []\n\ttestTags = []\n\tfor item in line:\n\t\ttestWords.append(item.split('/')[0])\n\t\ttestTags.append(item.split('/')[1])\n\n\t# 初始化Viterbi矩阵\n\tv = [[0 for x in range(len(tags))] for y in range(len(line))] # v[len(line)][len(tags)] Viterbi矩阵\n\tpath = [[0 for x in range(len(tags))] for y in range(len(line))] # path[len(line)][len(tags)] 记录路径\n\t# 初始化Viterbi矩阵\n\tfor j in range(len(tags)):\n\t\tv[1][j] = tagTrans[tags[0] + ' ' + tags[j]] * emit[tags[j] + ' ' + testWords[1]]\n\t\tpath[1][j] = 0 # path[1][j] = $start$\n\t# 前向递推\n\tfor t in range(2, len(testWords)):\n\t\tfor j in range(len(tags)):\n\t\t\tfor i in range(len(tags)):\n\t\t\t\tp = v[t - 1][i] * tagTrans[tags[i] + ' ' + tags[j]] * emit[tags[j] + ' ' + testWords[t]]\n\t\t\t\tif p \u003e v[t][j]:\n\t\t\t\t\tv[t][j] = p\n\t\t\t\t\tpath[t][j] = i\n\t# 结果概率为v[len(line) - 1][len(tags) - 1]\n\n\t# 构造结果list\n\tresultIndex = [0 for x in range(len(line))] # 记录结果的tag list在tags中的下标。包括段落起始符和结束符\n\tresultIndex[0] = 0 # $start$ 段落起始符\n\tresultIndex[-1] = len(tags) - 1 # $end$ 段落结束符\n\tfor i in reversed(range(len(line) - 1)): # 逆推path，即构造结果的tag路径\n\t\tresultIndex[i] = path[i + 1][resultIndex[i + 1]]\n\t# 根据resultIndex构造result\n\tresult = [] # 是tag的list\n\tfor i in resultIndex:\n\t\tresult.append(tags[i])\n\n\t# 输出结果\n\tcorrectCount = 0 # 单行正确标记个数\n\tfor i in range(len(result)):\n\t\tif result[i] == testTags[i]: # 标记正确\n\t\t\tcorrectCount += 1 # 单行正确标记个数+1\n\t\t\tallCorrectCount += 1 # 整个测试数据正确标记个数+1\n\tprint('progress:', (index + 1) * 100 // len(testData), '%', 'single line precision:', \\\n\t(correctCount - 2) * 100 // (len(result) - 2), '%') # 输出当前进度和单行结果。-2是为了去掉段落起始符和结束符\n\tallCorrectCount -= 2 # 段落起始符和结束符必定匹配正确。减去这两个符号\nprint('total precision:', allCorrectCount / allWordsCount) # 输出总正确率\n```\n\n### 输入输出与运行截图\n\n此程序不需输入。\n\n初始，输出如下提示信息：\n\n![3-2](img/3-2.png)\n\n开始解析测试集，输出当前进度和单行正确率：\n\n![3-3](img/3-3.png)\n\n解析结束，输出总正确率：\n\n![3-1](img/3-1.png)\n\n### 结果分析\n\n性能：总数据约20000行。其中测试数据约4000行(20%)，全部解析用时约30分钟。且需要在运行中申请大量存储空间，使用python性能明显不如其他同学使用c++编写的程序的性能。\n\n正确率：约70%。经过debug可以发现句首词性较易分析正确，一旦句子中出现一次分析错误，后面的词性可能都是错的，引起恶性循环。所以短句更容易匹配出100%，也可能出现0%。若要提高正确率，可以修改模型，尝试向前看更多的词。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiscretetom%2Fnlp-homework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdiscretetom%2Fnlp-homework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdiscretetom%2Fnlp-homework/lists"}