{"id":13496832,"url":"https://github.com/CALIL/citation","last_synced_at":"2025-03-28T21:31:25.031Z","repository":{"id":75813784,"uuid":"65821361","full_name":"CALIL/citation","owner":"CALIL","description":"Extract citation ISBNs from Wikipedia dump","archived":false,"fork":false,"pushed_at":"2024-12-09T03:10:10.000Z","size":54,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-12-09T04:19:52.013Z","etag":null,"topics":["code4lib-jp","wikipedia-dump"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CALIL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"citation.py","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-08-16T13:10:54.000Z","updated_at":"2024-12-09T03:10:14.000Z","dependencies_parsed_at":"2023-06-07T20:00:19.555Z","dependency_job_id":null,"html_url":"https://github.com/CALIL/citation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CALIL%2Fcitation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CALIL%2Fcitation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CALIL%2Fcitation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CALIL%2Fcitation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CALIL","download_url":"https://codeload.github.com/CALIL/citation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246104722,"owners_count":20724184,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code4lib-jp","wikipedia-dump"],"created_at":"2024-07-31T20:00:15.471Z","updated_at":"2025-03-28T21:31:25.018Z","avatar_url":"https://github.com/CALIL.png","language":"Python","funding_links":[],"categories":["オープンソース"],"sub_categories":[],"readme":"citation [![](https://img.shields.io/badge/python-3.12+-blue.svg)](https://docs.python.org/3.12/) [![Maintainability](https://api.codeclimate.com/v1/badges/d2ff9760abb138cd70bc/maintainability)](https://codeclimate.com/github/CALIL/citation/maintainability)\n=========================================================================================================================================================================================\nWikipediaのダンプファイルから出典ISBNを抽出するツール\n\n概要\n-----\n\n- 日本語版Wikipediaのダンプから出典ISBNを抽出\n- 抽出したデータはLine-delimited JSON形式で保存\n- ある程度の表記ゆれを吸収\n\n依存パッケージのインストール\n----\n\n```bash\npoetry install\n```\n\nコマンドライン\n----\n\n```bash\nUsage: citation.py [OPTIONS] INPUT_FILENAME EXPORT_FILENAME\n\nOptions:\n  --show-exclusion / --no-show-exclusion\n                                  除外した項目を表示する\n  --help                          Show this message and exit.\n```\n\n```bash\nwget https://dumps.wikimedia.org/jawiki/20190420/jawiki-20190420-pages-articles-multistream.xml.bz\npoetry run python citation.py jawiki-20190420-pages-articles-multistream.xml.bz2 citation-jawiki-20190420.jsonl\n```\n\n抽出されるデータ\n----\n\n```json\n{\n  \"isbn\": \"4772212272\",\n  \"raw\": \"4-7722-1227-2\",\n  \"title\": \"地理学\",\n  \"score\": 2.9,\n  \"h1\": \"参考文献\",\n  \"h2\": null,\n  \"is_ref\": true\n}\n```\n\n| 項目     | 型           | 概要                                               |\n|--------|-------------|--------------------------------------------------| \n| isbn   | String      | 正規化されたISBN（ISBN-10）                              |\n| raw    | String      | 解析される元のISBN表記                                    |\n| title  | String      | Wikipediaのページ名                                   |\n| score  | Number      | 独自指標により算出されたISBNの正確さ\u003cbr\u003e（スコアが低い場合は、誤って検出した場合がある） |\n| h1     | String/null | 見出し1                                             |\n| h2     | String/null | 見出し2                                             |\n| is_ref | Boolean     | 出典であることが明記されているか（作品リストなどではfalse）                 |\n\n処理済みデータのダウンロード\n----\n\n| ダンプ                                                                                                                                                  | 処理データ                                                                                                         |        件数 |\n|------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|----------:|\n| [jawiki-20190420-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20190420/jawiki-20190420-pages-articles-multistream.xml.bz2) | [citation-jawiki-20190420.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20190420.jsonl) |   672,155 |\n| [jawiki-20190601-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20190601/jawiki-20190601-pages-articles-multistream.xml.bz2) | [citation-jawiki-20190601.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20190601.jsonl) |   679,440 |\n| [jawiki-20190801-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20190801/jawiki-20190801-pages-articles-multistream.xml.bz2) | [citation-jawiki-20190801.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20190801.jsonl) |   688,393 |\n| [jawiki-20191220-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20191220/jawiki-20191220-pages-articles-multistream.xml.bz2) | [citation-jawiki-20191220.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20191220.jsonl) |   714,273 |\n| [jawiki-20200301-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20200301/jawiki-20200301-pages-articles-multistream.xml.bz2) | [citation-jawiki-20200301.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20200301.jsonl) |   728,278 |\n| [jawiki-20200801-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20200801/jawiki-20200801-pages-articles-multistream.xml.bz2) | [citation-jawiki-20200801.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20200801.jsonl) |   763,007 |\n| [jawiki-20201201-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20201201/jawiki-20201201-pages-articles-multistream.xml.bz2) | [citation-jawiki-20201201.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20201201.jsonl) |   788,068 |\n| [jawiki-20210620-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20201201/jawiki-20210620-pages-articles-multistream.xml.bz2) | [citation-jawiki-20210620.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20210620.jsonl) |   839,059 |\n| [jawiki-20210920-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20210920/jawiki-20210920-pages-articles-multistream.xml.bz2) | [citation-jawiki-20210920.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20210920.jsonl) |   864,341 |\n| [jawiki-20211120-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20211120/jawiki-20211120-pages-articles-multistream.xml.bz2) | [citation-jawiki-20211120.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20211120.jsonl) |   880,591 |\n| [enwiki-20211120-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/enwiki/20211120/enwiki-20211120-pages-articles-multistream.xml.bz2) | [citation-enwiki-20211120.jsonl](https://storage.googleapis.com/isbn-citation/citation-enwiki-20211120.jsonl) | 5,116,149 |\n| [jawiki-20221220-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20221220/jawiki-20221220-pages-articles-multistream.xml.bz2) | [citation-jawiki-20221220.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20221220.jsonl) |   970,869 |\n| [enwiki-20221220-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/enwiki/20221220/enwiki-20221220-pages-articles-multistream.xml.bz2) | [citation-enwiki-20221220.jsonl](https://storage.googleapis.com/isbn-citation/citation-enwiki-20221220.jsonl) | 6,064,901 |\n| [jawiki-20240401-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20240401/jawiki-20240401-pages-articles-multistream.xml.bz2) | [citation-jawiki-20240401.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20240401.jsonl) | 1,073,563 |\n| [enwiki-20240401-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/enwiki/20240401/jawiki-20240401-pages-articles-multistream.xml.bz2) | [citation-enwiki-20240401.jsonl](https://storage.googleapis.com/isbn-citation/citation-enwiki-20240401.jsonl) | 7,023,140 |\n| [jawiki-20241201-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/jawiki/20240401/jawiki-20240401-pages-articles-multistream.xml.bz2) | [citation-jawiki-20241201.jsonl](https://storage.googleapis.com/isbn-citation/citation-jawiki-20240401.jsonl) | 1,130,854 |\n| [enwiki-20241201-pages-articles-multistream.xml.bz2](https://dumps.wikimedia.org/enwiki/20240401/jawiki-20240401-pages-articles-multistream.xml.bz2) | [citation-enwiki-20241201.jsonl](https://storage.googleapis.com/isbn-citation/citation-enwiki-20240401.jsonl) | 8,669,996 |\n\n- [日本語版Wikipediaのダンプ](https://dumps.wikimedia.org/jawiki/)\n- [英語版Wikipediaのダンプ](https://dumps.wikimedia.org/enwiki/)\n- [保存場所の管理](https://console.cloud.google.com/storage/browser/isbn-citation) （管理者用）\n\n注意事項\n----\n\n- チェックデジットの一致により、ISBN以外を誤判定する場合があります。ただし、ISBNから参照記事を検索する目的では問題とならないため許容しています\n- チェックデジット間違いのISBNは抽出されません","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCALIL%2Fcitation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCALIL%2Fcitation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCALIL%2Fcitation/lists"}