{"id":20114823,"url":"https://github.com/gatenlp/corpusconversion-conll2003","last_synced_at":"2025-03-02T19:16:07.334Z","repository":{"id":145770852,"uuid":"102126628","full_name":"GateNLP/corpusconversion-conll2003","owner":"GateNLP","description":"Tool/scripts to help converting the CoNLL 2003 corpora to GATE format","archived":false,"fork":false,"pushed_at":"2018-05-04T14:32:08.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-01-13T06:11:01.865Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GateNLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-01T15:25:02.000Z","updated_at":"2019-07-29T18:56:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"59331316-3049-45fd-927c-0d6f0b690a8e","html_url":"https://github.com/GateNLP/corpusconversion-conll2003","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-conll2003","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-conll2003/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-conll2003/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-conll2003/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GateNLP","download_url":"https://codeload.github.com/GateNLP/corpusconversion-conll2003/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241557194,"owners_count":19981881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T18:32:14.033Z","updated_at":"2025-03-02T19:16:07.311Z","avatar_url":"https://github.com/GateNLP.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tools to convert the CoNLL2003 NER corpora to GATE format\n\nThis repository contains two sets of scripts for creating the Conll2003 NER corpora\nin GATE FastInfoset format: \n* prepare-deu.sh and prepare-eng.sh to create the conll-format files from\n  the text corpora (licensed, need to get obtained separately) and the annotation files \n  (available from https://www.clips.uantwerpen.be/conll2003/ner/)\n* convert-deu.sh and convert-eng.sh to convert the conll-format files to GATE format.\n\nNOTE: the prepare-deu.sh script does not work correctly at the moment and cannot be \nused, so the german conll-format files must get created by you separately and then\nbe put into directory ./conll2003-deu.\n\n## Preparing the CONLL-format files\n\nCreate the English conll2003-format files by running `./prepare-eng.sh \u003cconllloc\u003e \u003creutersloc\u003e`\n* `\u003cconllloc\u003e` is the directory that contains the downloaded ner.tgz file\n* `\u003cretersloc\u003e` is the directory that contains all the zip files of the Reuters corpus\n\nThis should place the three files eng.train, eng.testa and eng.tesb into the \n./conll2003-eng directory.\n\nNOTE: The preparation of the German conll2003-format files does not work properly right now.\nIf you find the problem, please let me know or provide a pull request!\n\n## Converting the CONLL-format files to GATE format\n\nRequirements:\n* Needs Java and Scala installed\n* Needs GATE 8.4.x installed and the environment variable `GATE_HOME` set to the installation directory\n* Only works on Linux, Mac and under Windows probably only in some form of Linux-compatibiity mode\n\nMake sure that the CONLL-format files are in conll2003-eng or conll2003-deu as needed!\n\nNow run `./convert-eng.sh` to convert English files and/or ./convert-deu.sh` to convert German files.\n\nEach of the result directories contains one GATE document in GATE XML format for each document identified in the corresponding input file. \n\n## Conversion Strategy\n\nThe following annotations are placed into the annotation set \"Original markups\":\n* LOC, MISC, ORG, PER: for the entity annotations from the input. These annotations have the single feature startLineNr which identifies the (1-based) number of the original CoNLL input file where this entity started\n* Token: for each input token one annotation is created. It contains the following features:\n  * chunkBIO: the original BIO value of the column for chunks\n  * lemma: the original value of the lemma columns (German only)\n  * lineNr: the line number (1-based) of that token in the original CoNLL input file as generated by the script described above\n  * neBIO: the original BIO value of the NE column\n  * pos: the original value of the POS column\n\nThe conversion algorithm separates all tokens by spaces except there are no spaces before punctuation caracters !,-.:;? and \nthere are no spaces after opening parentheses ({[ and before closing parentheses )}]\nThere is no space inserted before token 's but there is space inserted before a single ' because we cannot know if it is\nthe genetive of a plural or just used for quoting or something other.\nFor quote characters \" a space is inserted before but not after all odd occurrences in a document and after but not before\nall even occurrences.\n\nAll other tokens, including quote-like characters like '`, or characters like $£#~% if separate tokens in the input file are separated by \nspaces.\nNo space is added at the beginning or end of a sentence or beginning or end of a document.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fcorpusconversion-conll2003","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgatenlp%2Fcorpusconversion-conll2003","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fcorpusconversion-conll2003/lists"}