{"id":13936357,"url":"https://github.com/openeventdata/mordecai","last_synced_at":"2025-05-16T10:07:47.861Z","repository":{"id":45659439,"uuid":"61832342","full_name":"openeventdata/mordecai","owner":"openeventdata","description":"Full text geoparsing as a Python library","archived":false,"fork":false,"pushed_at":"2021-09-17T11:54:38.000Z","size":2755,"stargazers_count":749,"open_issues_count":28,"forks_count":96,"subscribers_count":33,"default_branch":"master","last_synced_at":"2025-05-14T04:11:59.907Z","etag":null,"topics":["geocoding","geonames","geoparsing","nlp","spacy","toponym-resolution"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openeventdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-06-23T19:49:45.000Z","updated_at":"2025-05-06T14:19:29.000Z","dependencies_parsed_at":"2022-09-02T19:41:43.798Z","dependency_job_id":null,"html_url":"https://github.com/openeventdata/mordecai","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fmordecai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fmordecai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fmordecai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openeventdata%2Fmordecai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openeventdata","download_url":"https://codeload.github.com/openeventdata/mordecai/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509476,"owners_count":22082891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["geocoding","geonames","geoparsing","nlp","spacy","toponym-resolution"],"created_at":"2024-08-07T23:02:35.670Z","updated_at":"2025-05-16T10:07:42.854Z","avatar_url":"https://github.com/openeventdata.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"![](paper/mordecai_geoparsing.png)\n\nFull text geoparsing as a Python library. Extract the place names from a\npiece of English-language text, resolve them to the correct place, and return\ntheir coordinates and structured geographic information.\n\n**Mordecai is ready for an upgrade!** Please take the user survey [here](https://z0l4ihmu0ud.typeform.com/to/b8FmCfMt) \nto help shape what v3 will look like.\n\nExample usage\n-------------\n\n```\n\u003e\u003e\u003e from mordecai import Geoparser\n\u003e\u003e\u003e geo = Geoparser()\n\u003e\u003e\u003e geo.geoparse(\"I traveled from Oxford to Ottawa.\")\n\n[{'country_conf': 0.96474487,\n  'country_predicted': 'GBR',\n  'geo': {'admin1': 'England',\n   'country_code3': 'GBR',\n   'feature_class': 'P',\n   'feature_code': 'PPLA2',\n   'geonameid': '2640729',\n   'lat': '51.75222',\n   'lon': '-1.25596',\n   'place_name': 'Oxford'},\n  'spans': [{'end': 22, 'start': 16}],\n  'word': 'Oxford'},\n {'country_conf': 0.83302397,\n  'country_predicted': 'CAN',\n  'geo': {'admin1': 'Ontario',\n   'country_code3': 'CAN',\n   'feature_class': 'P',\n   'feature_code': 'PPLC',\n   'geonameid': '6094817',\n   'lat': '45.41117',\n   'lon': '-75.69812',\n   'place_name': 'Ottawa'},\n  'spans': [{'end': 32, 'start': 26}],\n  'word': 'Ottawa'}]\n```\n\nMordecai requires a running Elasticsearch service with Geonames in it. See\n\"Installation\" below for instructions.\n\n\nInstallation and Requirements\n--------------------\n\n1. Mordecai is on PyPI and can be installed for Python 3 with pip:\n\n```\npip install mordecai\n```\n\n**Note**: It's *strongly* recommended that you run Mordecai in a virtual\nenvironment. The libraries that Mordecai depends on are not always the most\nrecent versions and using a virtual environment prevents libraries from being\ndowngraded or running into other issues:\n\n```\npython -m venv mordecai-env\nsource mordecai-env/bin/activate\npip install mordecai\n```\n\n2. You should then download the required spaCy NLP model:\n\n```\npython -m spacy download en_core_web_lg\n```\n\n3. In order to work, Mordecai needs access to a Geonames gazetteer running in\nElasticsearch. The easiest way to set it up is by running the following\ncommands (you must have [Docker](https://docs.docker.com/engine/installation/)\ninstalled first).\n\n```\ndocker pull elasticsearch:5.5.2\nwget https://andrewhalterman.com/files/geonames_index.tar.gz --output-file=wget_log.txt\ntar -xzf geonames_index.tar.gz\ndocker run -d -p 127.0.0.1:9200:9200 -v $(pwd)/geonames_index/:/usr/share/elasticsearch/data elasticsearch:5.5.2\n```\n\nSee the [es-geonames](https://github.com/openeventdata/es-geonames) for the code used\nto produce this index.\n\nTo update the index, simply shut down the old container, re-download the index\nfrom s3, and restart the container with the new index.\n\nCiting\n------\n\nIf you use this software in academic work, please cite as \n\n```\n@article{halterman2017mordecai,\n  title={Mordecai: Full Text Geoparsing and Event Geocoding},\n  author={Halterman, Andrew},\n  journal={The Journal of Open Source Software},\n  volume={2},\n  number={9},\n  year={2017},\n  doi={10.21105/joss.00091}\n}\n```\n\nHow does it work?\n-----------------\n\nMordecai takes in unstructured text and returns structured geographic information extracted\nfrom it. \n\n- It uses [spaCy](https://github.com/explosion/spaCy/)'s named entity recognition to\n  extract placenames from the text.\n\n- It uses the [geonames](http://www.geonames.org/)\n  gazetteer in an [Elasticsearch](https://www.elastic.co/products/elasticsearch) index \n  (with some custom logic) to find the potential coordinates of\n  extracted place names.\n\n- It uses neural networks implemented in [Keras](https://keras.io/) and trained on new annotated\n  English-language data labeled with [Prodigy](https://prodi.gy/) to infer the correct country and correct gazetteer entries for each\n  placename. \n\nThe training data for the two models includes copyrighted text so cannot be\nshared freely. Applying Mordecai to non-English language text would require labeling data\nin the target language and retraining.\n\nAPI and Configuration\n---------------------\n\nWhen instantiating the `Geoparser()` module, the following options can be changed:\n\n- `es_hosts` : List of hosts where the Geonames Elasticsearch service is\n    running. Defaults to `['localhost']`, which is where it runs if you're using\n    the default Docker setup described above.\n- `es_port` : What port the Geonames Elasticsearch service is running on.\n    Defaults to `9200`, which is where the Docker setup has it\n- `es_ssl` : Whether Elasticsearch requires an SSL connection.\n    Defaults to `False`.\n- `es_auth` : Optional HTTP auth parameters to use with ES.\n    If provided, it should be a two-tuple of `(user, password)`.\n- `country_confidence` : Set the country model confidence below which no\n    geolocation will be returned. If it's really low, the model's probably\n    wrong and will return weird results. Defaults to `0.6`. \n- `verbose` : Return all the features used in the country picking model?\n    Defaults to `False`. \n- `threads`: whether to use threads to make parallel queries to the\n    Elasticsearch database. Defaults to `True`, which gives a ~6x speedup.\n\n`geoparse` is the primary endpoint and the only one that most users will need.\nOther, mostly internal, methods may be useful in some cases:\n\n- `lookup_city` takes a city name, country, and (optionally) ADM1/state/governorate and \n    does a rule-based lookup for the city.\n- `infer_country` take a document and attempts to infer the most probable\n    country for each.\n- `query_geonames` and `query_geonames_country` can be used for performing a\n    search over Geonames in Elasticsearch\n- methods with the `_feature` prefix are internal methods for\n    calculating country picking features from text.\n\n`batch_geoparse` takes in a list of documents and uses spaCy's `nlp.pipe`\nmethod to process them more efficiently in the NLP step. \n\nAdvanced users on large machines can increase the `lru_cache` parameter from 250\nto 1000. This will use more memory but will increase parsing speed.\n\nTests\n-----\n\nMordecai includes unit tests. To run the tests, `cd` into the\n`mordecai` directory and run:\n\n```\npytest\n```\n\nThe tests require access to a running Elastic/Geonames service to\ncomplete. The tests are currently failing on TravisCI with an unexplained\nsegfault but run fine locally. Mordecai has only been tested with Python 3.\n\n\nAcknowledgements\n----------------\n\nAn earlier verion of this software was donated to the Open Event Data Alliance\nby Caerus Associates.  See [Releases](https://github.com/openeventdata/mordecai/releases) \nor the [legacy-docker](https://github.com/openeventdata/mordecai/tree/legacy-docker) branch for the\n2015-2016 and the 2016-2017 production versions of Mordecai.\n\nThis work was funded in part by DARPA's XDATA program, the U.S. Army Research\nLaboratory and the U.S. Army Research Office through the Minerva Initiative\nunder grant number W911NF-13-0332, and the National Science Foundation under\naward number SBE-SMA-1539302. Any opinions, findings, and conclusions or\nrecommendations expressed in this material are those of the authors and do not\nnecessarily reflect the views of DARPA, ARO, Minerva, NSF, or the U.S.\ngovernment.\n\n\nContributing\n------------\n\nContributions via pull requests are welcome. Please make sure that changes\npass the unit tests. Any bugs and problems can be reported\non the repo's [issues page](https://github.com/openeventdata/mordecai/issues).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopeneventdata%2Fmordecai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopeneventdata%2Fmordecai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopeneventdata%2Fmordecai/lists"}