{"id":18458005,"url":"https://github.com/yongyehuang/zhihu-text-classification","last_synced_at":"2025-04-06T13:11:05.412Z","repository":{"id":54563012,"uuid":"101959872","full_name":"yongyehuang/zhihu-text-classification","owner":"yongyehuang","description":"[2017知乎看山杯 多标签 文本分类] ye组(第六名) 解题方案","archived":false,"fork":false,"pushed_at":"2018-04-07T03:26:39.000Z","size":21730,"stargazers_count":403,"open_issues_count":4,"forks_count":156,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-30T12:07:18.748Z","etag":null,"topics":["han","lstm","multi-label","tensorflow","text-classification","textcnn"],"latest_commit_sha":null,"homepage":"https://biendata.com/competition/zhihu/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yongyehuang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-31T04:40:42.000Z","updated_at":"2025-03-21T16:06:43.000Z","dependencies_parsed_at":"2022-08-13T19:50:23.025Z","dependency_job_id":null,"html_url":"https://github.com/yongyehuang/zhihu-text-classification","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongyehuang%2Fzhihu-text-classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongyehuang%2Fzhihu-text-classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongyehuang%2Fzhihu-text-classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yongyehuang%2Fzhihu-text-classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yongyehuang","download_url":"https://codeload.github.com/yongyehuang/zhihu-text-classification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247485287,"owners_count":20946398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["han","lstm","multi-label","tensorflow","text-classification","textcnn"],"created_at":"2024-11-06T08:16:22.585Z","updated_at":"2025-04-06T13:11:05.395Z","avatar_url":"https://github.com/yongyehuang.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 2017 知乎看山杯 多标签文本分类\r\n比赛总结： [2017知乎看山杯总结(多标签文本分类)](http://blog.csdn.net/Jerr__y/article/details/77751885)\r\n## 1.运行环境\r\n下面是我实验中的一些环境依赖，版本只提供参考。\r\n\r\n|环境/库|版本|\r\n|:---------:|----------|\r\n|Ubuntu|14.04.5 LTS|\r\n|python|2.7.12|\r\n|jupyter notebook|4.2.3|\r\n|tensorflow-gpu|1.2.1|\r\n|numpy|1.12.1|\r\n|pandas|0.19.2|\r\n|matplotlib|2.0.0|\r\n|word2vec|0.9.1|\r\n|tqdm|4.11.2|\r\n\r\n## 2.文件结构\r\n\r\n|- zhihu-text-classification\u003cbr/\u003e\r\n|　　|- raw_data　　　　　　　　　# 比赛提供的原始数据\u003cbr/\u003e\r\n|　　|- data　　　　　　　　　　　# 预处理得到的数据\u003cbr/\u003e\r\n|　　|- data_process　　　　　　　# 数据预处理代码\u003cbr/\u003e\r\n|　　|- models　　　　　　　　　　# 模型代码\u003cbr/\u003e\r\n|　　|　　|- wd-1-1-cnn-concat　　　　\u003cbr/\u003e\r\n|　　|　　|　　|- network.py　　　　　　# 定义网络结构\u003cbr/\u003e\r\n|　　|　　|　　|- train.py　　　　　　  # 模型训练\u003cbr/\u003e\r\n|　　|　　|　　|- predict.py　　　　　　# 验证集/测试集预测，生成概率矩阵\u003cbr/\u003e\r\n...\u003cbr/\u003e\r\n|　　|- ckpt　　　　　　　　　　　# 保存训练好的模型\u003cbr/\u003e\r\n|　　|- summary　　　　　　　　　　# tensorboard数据\u003cbr/\u003e\r\n|　　|- scores　　　　　　　　　　　# 测试集的预测概率矩阵\u003cbr/\u003e\r\n|　　|- local_scores　　　　　　　 # 验证集的预测概率矩阵\u003cbr/\u003e\r\n|　　|- doc　　　　　　　　　　    # 文档说明与相关论文\u003cbr/\u003e\r\n|　　|- notebook-old　　　　　　　# 比赛中未经过整理的代码\u003cbr/\u003e\r\n|　　|- local_ensemble.ipynb　　　# 验证集模型融合\u003cbr/\u003e\r\n|　　|- ensemble.py　　　　　　　　# 测试集模型融合\u003cbr/\u003e\r\n|　　|- data_helpers.py　　　　　　# 数据处理函数\u003cbr/\u003e\r\n|　　|- evaluator.py　　　　　　　 # 评价函数\u003cbr/\u003e\r\n\r\n\r\n## 3.数据预处理\r\n- 把比赛提供的所有数据解压到 raw_data/ 目录下。\r\n- 按照顺序依次执行各个 .py，不带任何参数。\u003cbr/\u003e\r\n  或者在当前目录下输入下面命令运行所有文件：\u003cbr/\u003e\r\n  dos2unix run_all_data_process.sh   # 使用cygwin工具dos2unix将script改为unix格式\u003cbr/\u003e\r\n  sh run_all_data_process.sh\r\n\r\n### 3.1 embed2ndarray.py\r\n赛方提供了txt格式的词向量和字向量，这里把embedding矩阵转成 np.ndarray 形式，分别保存为 data/word_embedding.npy 和 data/char_embedding.npy。\r\n用 pd.Series 保存词(字)对应 embedding 中的行号(id),存储在 data/sr_word2id.pkl 和 data/sr_char2id.pkl 中。\r\n\r\n### 3.2 question_and_topic_2id.py\r\n把问题和话题转为id形式，保存在 data/sr_question2id.pkl 和 data/sr_id2question.pkl 中。\r\n\r\n### 3.3 char2id.py\r\n利用上面得到的 sr_char2id，把所有问题的字转为对应的id, 存储为\u003cbr/\u003e\r\ndata/ch_train_title.npy\u003cbr/\u003e\r\ndata/ch_train_content.npy\u003cbr/\u003e\r\ndata/ch_eval_title.npy\u003cbr/\u003e\r\ndata/ch_eval_content.npy\u003cbr/\u003e\r\n\r\n### 3.4 word2id.py\r\n同 char2id.py\r\n\r\n### 3.5 creat_batch_data.py\r\n把所有的数据按照 batch_size(128) 进行打包，固定seed，随机取 10 万样本作为验证集。每个batch存储为一个 npz 文件，包括 X, y 两部分。\r\n这里所有的序列都进行了截断，长度不足的用0进行padding到固定长度。\u003cbr/\u003e\r\n保存位置：\u003cbr/\u003e\r\nwd_train_path = '../data/wd-data/data_train/'\u003cbr/\u003e\r\nwd_valid_path = '../data/wd-data/data_valid/'\u003cbr/\u003e\r\nwd_test_path = '../data/wd-data/data_test/'\u003cbr/\u003e\r\nch_train_path = '../data/ch-data/data_train/'\u003cbr/\u003e\r\nch_valid_path = '../data/ch-data/data_valid/'\u003cbr/\u003e\r\nch_test_path = '../data/ch-data/data_test/'\u003cbr/\u003e\r\n\r\n\r\n### 3.6 creat_batch_seg.py\r\n和 creat_batch_data.py 相同，只是对 content 部分进行句子划分。用于分层模型。\r\n划分句子长度：\u003cbr/\u003e\r\nwd_title_len = 30, wd_sent_len = 30, wd_doc_len = 10.(即content划分为10个句子，每个句子长度为30个词)\u003cbr/\u003e\r\nch_title_len = 52, ch_sent_len = 52, ch_doc_len = 10.\u003cbr/\u003e\r\n不划分句子：\u003cbr/\u003e\r\nwd_title_len = 30, wd_content_len = 150.\u003cbr/\u003e\r\nch_title_len = 52, ch_content_len = 300.\u003cbr/\u003e\r\n\r\n\r\n## 4.模型训练\r\n切换到模型所在位置，然后进行训练和预测。比如：\r\n```\r\ncd zhihu-text-classification/models/wd-1-1-cnn-concat/\r\n# 训练\r\npython train.py [--max_epoch 1 --max_max_epoch 6 --lr 1e-3 decay_rate 0.65 decay_step 15000 last_f1 0.4]\r\n# 预测\r\npython predict.py\r\n```\r\n这里只整理了部分模型，所有模型都用的词向量。如果想要使用字向量，只需要把模型中的输入和序列长度修改即可。\r\n\r\n## 5.模型融合\r\n线性加权融合，模拟梯度下降的策略进行权值搜索。见：local_ensemble.ipynb\r\n注意：\r\n- 此方法可能会对验证集过拟合，所以需要通过测试集进一步判断。在模型个数比较多时使用此方法效果更好。\r\n- 需要根据各个单模型的性能认为进行初始化。char 和 word 类型不能直接比较，char 的单模型的性能虽然较差，但是对融合提升非常明显。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyongyehuang%2Fzhihu-text-classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyongyehuang%2Fzhihu-text-classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyongyehuang%2Fzhihu-text-classification/lists"}