{"id":37711471,"url":"https://github.com/qurator-spk/sbb_ner","last_synced_at":"2026-01-16T13:19:43.451Z","repository":{"id":35234469,"uuid":"204686076","full_name":"qurator-spk/sbb_ner","owner":"qurator-spk","description":"Named Entity Recognition","archived":false,"fork":false,"pushed_at":"2025-04-04T13:32:05.000Z","size":331,"stargazers_count":17,"open_issues_count":3,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-04T13:35:31.517Z","etag":null,"topics":["bert-ner","named-entity-recognition","qurator"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qurator-spk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-27T11:11:58.000Z","updated_at":"2025-04-04T13:32:08.000Z","dependencies_parsed_at":"2025-04-04T13:29:46.952Z","dependency_job_id":"7abef3cd-6772-451e-be41-e9a7ed7945ea","html_url":"https://github.com/qurator-spk/sbb_ner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/qurator-spk/sbb_ner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qurator-spk%2Fsbb_ner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qurator-spk%2Fsbb_ner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qurator-spk%2Fsbb_ner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qurator-spk%2Fsbb_ner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qurator-spk","download_url":"https://codeload.github.com/qurator-spk/sbb_ner/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qurator-spk%2Fsbb_ner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert-ner","named-entity-recognition","qurator"],"created_at":"2026-01-16T13:19:43.390Z","updated_at":"2026-01-16T13:19:43.446Z","avatar_url":"https://github.com/qurator-spk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![sbb-ner-demo example](.screenshots/sbb_ner_demo.png?raw=true)\n\nHow the models have been obtained is described in our [paper](https://corpora.linguistik.uni-erlangen.de/data/konvens/proceedings/papers/KONVENS2019_paper_4.pdf).\n\n***\n\n# Installation:\n\nRecommended python version is 3.11. \nConsider use of [pyenv](https://github.com/pyenv/pyenv) if that python version is not available on your system. \n\nActivate virtual environment (virtualenv):\n```\nsource venv/bin/activate\n```\nor (pyenv):\n```\npyenv activate my-python-3.11-virtualenv\n```\n\nUpdate pip:\n```\npip install -U pip\n```\nInstall sbb_ner:\n```\npip install git+https://github.com/qurator-spk/sbb_ner.git\n```\nDownload required models: https://qurator-data.de/sbb_ner/models.tar.gz \n\nExtract model archive:\n```\ntar -xzf models.tar.gz\n```\n\nCopy [config file](qurator/sbb_ner/webapp/config.json) into working directory.\nSet USE_CUDA environment variable to True/False depending on GPU availability.\n\nRun webapp directly:\n\n```\nenv CONFIG=config.json env FLASK_APP=qurator/sbb_ner/webapp/app.py env FLASK_ENV=development env USE_CUDA=True/False flask run --host=0.0.0.0\n```\n\nFor production purposes rather use\n```\nenv CONFIG=config.json env USE_CUDA=True/False gunicorn --bind 0.0.0.0:5000 qurator.sbb_ner.webapp.wsgi:app\n```\n\n# Docker\n\n## CPU-only:\n\n```\ndocker build --build-arg http_proxy=$http_proxy  -t qurator/webapp-ner-cpu -f Dockerfile.cpu .\n```\n\n```\ndocker run -ti --rm=true --mount type=bind,source=data/konvens2019,target=/usr/src/qurator-sbb-ner/data/konvens2019 -p 5000:5000 qurator/webapp-ner-cpu\n```\n\n## GPU:\n\nMake sure that your GPU is correctly set up and that nvidia-docker has been installed.\n\n```\ndocker build --build-arg http_proxy=$http_proxy  -t qurator/webapp-ner-gpu -f Dockerfile .\n```\n\n```\ndocker run -ti --rm=true --mount type=bind,source=data/konvens2019,target=/usr/src/qurator-sbb-ner/data/konvens2019 -p 5000:5000 qurator/webapp-ner-gpu\n```\n\nNER web-interface is availabe at http://localhost:5000 . \n\n# REST - Interface\n\nGet available models:\n```\ncurl http://localhost:5000/models\n```\n\nOutput:\n\n```\n[\n  {\n    \"default\": true, \n    \"id\": 1, \n    \"model_dir\": \"data/konvens2019/build-wd_0.03/bert-all-german-de-finetuned\", \n    \"name\": \"DC-SBB + CONLL + GERMEVAL\"\n  }, \n  {\n    \"default\": false, \n    \"id\": 2, \n    \"model_dir\": \"data/konvens2019/build-on-all-german-de-finetuned/bert-sbb-de-finetuned\", \n    \"name\": \"DC-SBB + CONLL + GERMEVAL + SBB\"\n  }, \n  {\n    \"default\": false, \n    \"id\": 3, \n    \"model_dir\": \"data/konvens2019/build-wd_0.03/bert-sbb-de-finetuned\", \n    \"name\": \"DC-SBB + SBB\"\n  }, \n  {\n    \"default\": false, \n    \"id\": 4, \n    \"model_dir\": \"data/konvens2019/build-wd_0.03/bert-all-german-baseline\", \n    \"name\": \"CONLL + GERMEVAL\"\n  }\n]\n```\n\nPerform NER using model 1: \n\n```\ncurl -d '{ \"text\": \"Paris Hilton wohnt im Hilton Paris in Paris.\" }' -H \"Content-Type: application/json\" http://localhost:5000/ner/1\n```\n\nOutput:\n\n```\n[\n  [\n    {\n      \"prediction\": \"B-PER\", \n      \"word\": \"Paris\"\n    }, \n    {\n      \"prediction\": \"I-PER\", \n      \"word\": \"Hilton\"\n    }, \n    {\n      \"prediction\": \"O\", \n      \"word\": \"wohnt\"\n    }, \n    {\n      \"prediction\": \"O\", \n      \"word\": \"im\"\n    }, \n    {\n      \"prediction\": \"B-ORG\", \n      \"word\": \"Hilton\"\n    }, \n    {\n      \"prediction\": \"I-ORG\", \n      \"word\": \"Paris\"\n    }, \n    {\n      \"prediction\": \"O\", \n      \"word\": \"in\"\n    }, \n    {\n      \"prediction\": \"B-LOC\", \n      \"word\": \"Paris\"\n    }, \n    {\n      \"prediction\": \"O\", \n      \"word\": \".\"\n    }\n  ]\n]\n```\nThe JSON above is the expected input format of the \n[SBB named entity linking and disambiguation system](https://github.com/qurator-spk/sbb_ned).\n# Model-Training \n\n***\n## Preprocessing of NER ground-truth:\n\n\n### compile_conll\n\nRead CONLL 2003 ner ground truth files from directory and\nwrite the outcome of the data parsing to some pandas DataFrame that is\nstored as pickle.\n\n#### Usage\n\n```\ncompile_conll --help\n```\n\n### compile_germ_eval\n\nRead germ eval .tsv files from directory and write the\noutcome of the data parsing to some pandas DataFrame that is stored as\npickle.\n\n#### Usage\n\n```\ncompile_germ_eval --help\n```\n\n### compile_europeana_historic\n\nRead europeana historic ner ground truth .bio files from directory \nand write the outcome of the data parsing to some pandas\nDataFrame that is stored as pickle.\n\n#### Usage\n\n```\ncompile_europeana_historic --help\n```\n\n\n### compile_wikiner\n\nRead wikiner files from directory and write the outcome\nof the data parsing to some pandas DataFrame that is stored as pickle.\n\n#### Usage\n\n```\ncompile_wikiner --help\n```\n\n***\n## Train BERT - NER model:\n\n### bert-ner\n\nPerform BERT for NER supervised training and test/cross-validation.\n\n#### Usage\n\n```\nbert-ner --help\n```\n\n## BERT-Pre-training:\n\n### collectcorpus\n\n```\ncollectcorpus --help\n\nUsage: collectcorpus [OPTIONS] FULLTEXT_FILE SELECTION_FILE CORPUS_FILE\n\n  Reads the fulltext from a CSV or SQLITE3 file (see also altotool) and\n  write it to one big text file.\n\n  FULLTEXT_FILE: The CSV or SQLITE3 file to read from.\n\n  SELECTION_FILE: Consider only a subset of all pages that is defined by the\n  DataFrame that is stored in \u003cselection_file\u003e.\n\n  CORPUS_FILE: The output file that can be used by bert-pregenerate-trainingdata.\n\nOptions:\n  --chunksize INTEGER     Process the corpus in chunks of \u003cchunksize\u003e.\n                          default:10**4\n\n  --processes INTEGER     Number of parallel processes. default: 6\n  --min-line-len INTEGER  Lower bound of line length in output file.\n                          default:80\n\n  --help                  Show this message and exit.\n\n```\n\n### bert-pregenerate-trainingdata\n\nGenerate data for BERT pre-training from a corpus text file where \nthe documents are separated by an empty line (output of corpuscollect).\n\n#### Usage\n\n```\nbert-pregenerate-trainingdata --help\n```\n\n### bert-finetune\n\nPerform BERT pre-training on pre-generated data.\n\n#### Usage\n\n```\nbert-finetune --help\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqurator-spk%2Fsbb_ner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqurator-spk%2Fsbb_ner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqurator-spk%2Fsbb_ner/lists"}