{"id":20114821,"url":"https://github.com/gatenlp/corpusconversion-bnc","last_synced_at":"2025-07-07T03:08:30.823Z","repository":{"id":145770853,"uuid":"80835894","full_name":"GateNLP/corpusconversion-bnc","owner":"GateNLP","description":"Tool to convert the British National Corpus to GATE format","archived":false,"fork":false,"pushed_at":"2017-02-03T14:27:06.000Z","size":7,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-03-09T19:55:27.453Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GateNLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-03T14:24:30.000Z","updated_at":"2021-06-18T20:44:42.000Z","dependencies_parsed_at":"2023-04-30T11:37:24.775Z","dependency_job_id":null,"html_url":"https://github.com/GateNLP/corpusconversion-bnc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/GateNLP/corpusconversion-bnc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-bnc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-bnc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-bnc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-bnc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GateNLP","download_url":"https://codeload.github.com/GateNLP/corpusconversion-bnc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fcorpusconversion-bnc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264005172,"owners_count":23542829,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T18:32:12.649Z","updated_at":"2025-07-07T03:08:30.802Z","avatar_url":"https://github.com/GateNLP.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conversion of the original British National Corpus documents, XML edition to GATE\n\nThe files in this repository can be used to convert the original XML fiels from the BNC corpus\n(see http://ota.ox.ac.uk/desc/2554) to usable GATE documents. \n\nNOTE: this depends on the following tools and software, not included here:\n* the runPipeline.sh command from https://github.com/johann-petrak/gatetool-runpipeline and the bin directory of\n  that tool must be on the binary path\n* the Java plugin is added as a submodule (https://github.com/johann-petrak/gateplugin-Java)\n* GATE (version 8.x)\n* JAVA SDK\n* Ant\n\n## Preparation\n\n* Make sure the Java submodule is actually fetched and compiled\n  * `git submodule init`\n  * `git submodule update`\n  * `git submodule foreach ant`\n* Make sure the British National Corpus is available in some directory in unzipped for, the directory is usually called \"2554\"\n* The conversion script will copy the BNC corpus into a local temporary directory, so make sure there is enough disk space\n  on the disk which contains the current directory (about 4.4G needed)\n* The GATE documents will require about 114G of disk space\n\n\n## Run the conversion\n\nJust run the convert.sh script and pass the location of the BNC corpus and the desired output directory as arguments:\n\n`./convert.sh bnccorpusdir outputdir`\n\n\n## Overview of how the conversion is done:\n* Load original files into GATE in XML format, but set the option\n  \"add space on markup unpack if needed\" to false\n* Now, in the \"Original markups\" set we get all the XML fields as annotations.\n* Remeber the following fields for document-features (if nothiing else specified, the document text for the field):\n  * availability\n  * bibl\n  * bncDoc.xml:id  gets converted to id\n  * catRef.targets\n  * change: doc text is documentation of change, features date amd who give additional info, should get \n    converted to single list? Should get converted to change.\\\u003cdate\\\u003e.\\\u003cwho\\\u003e and change.\\\u003cdate\\\u003e\n  * classCode: text and feature scheme\n  * creation.date\n  * date\n  * distributor\n  * edition\n  * extent\n  * imprint (text and feature n)\n  * keywords\n  * profileDesc\n  * pubPlace\n  * publicationStmt\n  * publisher\n  * respStmt\n  * sourceDesc\n  * tagUsage: empty span annots with features gi and occurs. Should get converted to features\n    tagUsage.\\\u003cgi\\\u003e=\\\u003coccurs\\\u003e \n  * titleStmt\n\nAnnotations relevant for the actual text:\n* wtext: covers the part we are interested in. Anything before should eventually get removed\n* w: words, features: c5, hw (lemma), pos\n* c: something about punctuation and quotes, needed in addition to \"w\". features: c5\n* mw: overlaps w annotations for multi-word stuff like \"up to\" and has feature c5=??\n* s: sentences, feature n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fcorpusconversion-bnc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgatenlp%2Fcorpusconversion-bnc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fcorpusconversion-bnc/lists"}