{"id":33150202,"url":"https://github.com/FudanNLP/nlpcc2017_news_headline_categorization","last_synced_at":"2025-11-20T11:01:44.396Z","repository":{"id":78772406,"uuid":"87178852","full_name":"FudanNLP/nlpcc2017_news_headline_categorization","owner":"FudanNLP","description":"NLPCC2017示例代码以及数据描述","archived":false,"fork":false,"pushed_at":"2017-06-12T02:56:57.000Z","size":20528,"stargazers_count":115,"open_issues_count":7,"forks_count":54,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-07-23T15:38:15.822Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FudanNLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-04-04T11:31:08.000Z","updated_at":"2025-01-23T06:07:33.000Z","dependencies_parsed_at":"2023-04-01T05:35:31.807Z","dependency_job_id":null,"html_url":"https://github.com/FudanNLP/nlpcc2017_news_headline_categorization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FudanNLP/nlpcc2017_news_headline_categorization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FudanNLP%2Fnlpcc2017_news_headline_categorization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FudanNLP%2Fnlpcc2017_news_headline_categorization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FudanNLP%2Fnlpcc2017_news_headline_categorization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FudanNLP%2Fnlpcc2017_news_headline_categorization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FudanNLP","download_url":"https://codeload.github.com/FudanNLP/nlpcc2017_news_headline_categorization/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FudanNLP%2Fnlpcc2017_news_headline_categorization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285422071,"owners_count":27168929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-20T02:00:05.334Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-15T18:00:21.544Z","updated_at":"2025-11-20T11:01:44.391Z","avatar_url":"https://github.com/FudanNLP.png","language":"Python","funding_links":[],"categories":["Contents 列表"],"sub_categories":["综合内容"],"readme":"# NLPCC 2017 新闻标题分类\n\n## 代码运行环境\npython2.7 (最好用anaconda2)\u003cbr\u003e\ntensorflow1.0.0 gpu版本或者cpu版本\u003cbr\u003e\n建议操作系统:Linux\u003cbr\u003e\n\n\u003e Linux 上的环境配置可以参考[Setup Deep Learning enviroment on linux](https://jerrikeph.github.io/setup-deep-learning-enviroment-on-linux.html)。注意要自己在tensorflow网站上找到自己要的版本\u003cbr\u003e\n\n\thttps://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.0.0-cp27-none-linux_x86_64.whl\n\thttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.0.0-cp27-none-linux_x86_64.whl\n\n## 快速上手\n\u003e 这里只提供在linux 上的上手攻略\n\n在满足上面运行环境之后，可以直接运行.\n\t\n\tbash ./quick_run_.sh\n\n就开始训练了。log保存在`./savings/save01/run.log` 里面。\n#### 里面做了这些事情：\n\n在命令行中`$tar zxvf nlpcc_data.tar.gz` 解压 nlpcc_data.tar.gz\u003cbr\u003e\n\n解压后的目录结构：\n\n\t.\n\t├── char\n\t│   ├── dev.txt\n\t│   ├── id2tag.txt\n\t│   ├── test.txt #测试文件在nlpcc_data中并没有给出，将dev.txt复制一份成test.txt\n\t│   ├── train.txt\n\t│   └── vocab.txt\n\t└── word\n\t    ├── dev.txt\n\t    ├── id2tag.txt\n\t    ├── test.txt #测试文件在nlpcc_data中并没有给出，将dev.txt复制一份成test.txt\n\t    ├── train.txt\n\t    ├── vocab.100k\n\t    └── vocab.all\n\n\n将`char`或者`word`中的copy到`all_data`中\u003cbr\u003e\n下面是运行命令，\n\n\t#在save01目录中生成一个config文件\n\tpython model.py --weight-path ./savings/save01 \n\t# 载入./savings/save01中的配置文件并且开始训练\n\tpython model.py --weight-path ./savings/save01 --load-config\n\t# 载入./savings/save01中的配置文件以及保存在改目录下的训练好的参数进行测试\n\tpython model.py --weight-path ./savings/save01 --load-config --train-test test\n\n`./savings/save01/config`文件可以修改，然后不用执行第一句命令生成配置文件。直接执行第二条命令载入修改好的配置文件。\u003cbr\u003e\n配置文件长这样：\n\n\t[General]\n\ttrain_data = ./all_data/train.txt\n\tval_data = ./all_data/dev.txt\n\ttest_data = ./all_data/test.txt\n\tvocab_path = ./all_data/vocab.txt\n\tid2tag_path = ./all_data/id2tag.txt\n\tembed_path = ./all_data/embed/embedding.\n\tneural_model = lstm_basic\n\tpre_trained = False\n\tvocab_size = 100000\n\tbatch_size = 64\n\tembed_size = 200\n\tmax_epochs = 50\n\tearly_stopping = 5\n\tdropout = 0.9\n\tlr = 0.001\n\tdecay_steps = 500\n\tdecay_rate = 0.9\n\tclass_num = 0\n\treg = 0.001\n\tnum_steps = 40\n\tfnn_numlayers = 1\n\n\t[lstm]\n\thidden_size = 300\n\trnn_numlayers = 1\n\n\t[cnn]\n\tnum_filters = 128\n\tfilter_sizes = [3, 4, 5]\n\tcnn_numlayers = 1\n\u003e 配置文件用来配置模型结结构\n\n\n## 数据描述\nchar目录中的数据是字符级别的新闻标题\u003cbr\u003e\nword目录中的数据是词级别的新闻标题 (分词工具为jieba，也可以用其他工具分)\u003cbr\u003e\n内容类似于：\n\u003e finance\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;建 行 按 揭 贷 余 额 超 3 万 亿 还 将 大 力 发 展\u003cbr\u003e\nsociety\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;头 号 老 赖 欠 款 2 亿 拆 东 墙 补 西 墙 终 欠 下 2 亿 元\u003cbr\u003e\nentertainment\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;对 卡 戴 珊 来 说 ， 每 一 次 换 装 都 是 一 次 宣 传\u003cbr\u003e\nentertainment\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;陈 妍 希 陈 晓 7 月 大 婚 ， 你 看 好 他 俩 吗 ？\u003cbr\u003e\ncar\t变 道 和 转 弯 ， 没 让 直 行 车 辆 后 果 可 不 轻\u003cbr\u003e\ngame\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;打 辅 助 位 的 正 统 T D ！ 坦 克 世 界 斯 太 尔 W T 的 战 场 理 解\u003cbr\u003e\ntech\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;大 数 据 人 才 炙 手 可 热 薪 酬 到 底 有 多 高 ？\u003cbr\u003e\ntravel\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;月 薪 3 0 0 0 元 的 常 州 人 ， 到 这 些 国 家 瞬 间 成 土 豪 ！\u003cbr\u003e\nhistory\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;清 朝 灭 亡 时 只 有 2 2 行 省 ， 现 在 却 有 3 4 个 ， 那 些 省 份 是 新 出 的 ？\u003cbr\u003e\n\n第一列是label后面是正文，中间用tab隔开。\ntrain.txt, test.txt, dev.txt分别是训练集，测试集以及开发集。id2tag.txt存的标签词典，vocab.txt为辞典。\n\u003e 所有的数据都是utf-8格式，以及如果要加入embedding的话，也应该是utf-8格式\u003cbr\u003e\n\u003e 注意test.txt并没有给出来，可以复制一份dev.txt成test.txt，或者直接改一下config文件\n\n给出的数据是总数据的30%，train跟dev比例为2:1. 其余的70%将会当作测试数据，会在截止日前一周发布。\n\n***预训的embedding 放在[百度云](https://pan.baidu.com/s/1mhPddpu)，可以自行下载。\u003cbr\u003e\n测试集已经放在/test目录下。也可以通过百度云下载[测试集](https://pan.baidu.com/s/1qXYzB5a)。\u003cbr\u003e\n测试结果应该**严格按照测试样本顺序**保存在result.txt文件中（一行一个标签，参考test/test.example文件格式），以附件的形式发送到邮箱 task2@nlpcc2017.info。\u003cbr\u003e\n\n## 代码描述\n\t├── Config.py\n\t├── helper.py\n\t├── model.py\n\t\n `Config.py`: 配置处理代码，用于生成或者加载配置文件。\u003cbr\u003e\n `helper.py`: 加载数据，加载辞典，计算准确度等等。\u003cbr\u003e\n### `model.py`: 模型代码。\n\u003e 如果需要增加或者修改模型，可以在Model类中的`add_model()`函数中添加一个新的模型模块，或者修改已经有的模型(lstm_basic, cnn_basic, cbow_basic)。\n\n## 引用\n\n```\n@InProceedings{qiu2017overview,\n  Title                    = {Overview of the NLPCC 2017 Shared Task: Chinese News Headline Categorization},\n  Author                   = {Xipeng Qiu and Jingjing Gong and Xuanjing Huang},\n  Booktitle                = {arXiv:1706.02883v1},\n  Year                     = {2017}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFudanNLP%2Fnlpcc2017_news_headline_categorization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFudanNLP%2Fnlpcc2017_news_headline_categorization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFudanNLP%2Fnlpcc2017_news_headline_categorization/lists"}