{"id":13557286,"url":"https://github.com/adbar/geokelone","last_synced_at":"2025-12-24T17:49:47.256Z","repository":{"id":147911789,"uuid":"104094242","full_name":"adbar/geokelone","owner":"adbar","description":"integrates spatial and textual data processing tools into a modular software package which features preprocessing, geocoding, disambiguation and visualization","archived":false,"fork":false,"pushed_at":"2019-06-11T16:56:27.000Z","size":266,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-04T07:35:53.176Z","etag":null,"topics":["digital-humanities","geocoder","mapping","text-visualization"],"latest_commit_sha":null,"homepage":"https://adrien.barbaresi.eu/blog/tag/digital-humanities.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adbar.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-09-19T15:38:35.000Z","updated_at":"2023-12-05T19:27:31.000Z","dependencies_parsed_at":"2023-05-27T22:00:16.793Z","dependency_job_id":null,"html_url":"https://github.com/adbar/geokelone","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adbar%2Fgeokelone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adbar%2Fgeokelone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adbar%2Fgeokelone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adbar%2Fgeokelone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adbar","download_url":"https://codeload.github.com/adbar/geokelone/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246993016,"owners_count":20865925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["digital-humanities","geocoder","mapping","text-visualization"],"created_at":"2024-08-01T12:04:15.626Z","updated_at":"2025-12-24T17:49:47.217Z","avatar_url":"https://github.com/adbar.png","language":"Python","funding_links":[],"categories":["Python","others"],"sub_categories":[],"readme":"``geokelone``\n=============\n\n.. image:: https://img.shields.io/pypi/v/geokelone.svg\n    :target: https://pypi.python.org/pypi/geokelone\n\n.. image:: https://img.shields.io/pypi/l/geokelone.svg\n    :target: https://pypi.python.org/pypi/geokelone\n\n.. image:: https://img.shields.io/pypi/pyversions/geokelone.svg\n    :target: https://pypi.python.org/pypi/geokelone\n\n.. image:: https://img.shields.io/travis/adbar/geokelone.svg\n    :target: https://travis-ci.org/adbar/geokelone\n\n.. image:: https://img.shields.io/codecov/c/github/adbar/geokelone.svg\n    :target: https://codecov.io/gh/adbar/geokelone\n\n\nWork towards the integration of spatial and textual data processing tools into a modular software package which features preprocessing, geocoding, disambiguation and visualization. Construction of gazzetteers and basic text processing functions are included. The installation works best with recent Linux and Mac systems (see below for more details).\n\nCurrent reference: Barbaresi, A. (2017). `Towards a toolbox to map historical text collections \u003chttps://hal.archives-ouvertes.fr/hal-01654526/document\u003e`_, *Proceedings of 11th Workshop on Geographic Information Retrieval*, ACM, Heidelberg.\n\n\n.. contents:: **Contents**\n    :backlinks: none\n\n\nUsage as a toolchain\n----------------------\n\n\nBootstrapped geographic databases\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nData helpers are included to derive geographic data from existing sources such as Geonames, Wikipedia or Wikidata (all under CC BY licenses), see for example Geonames with country codes:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from geokelone import data\n    # decide countries for which Geonames information is downloaded\n    \u003e\u003e\u003e countries = ['dk', 'fi'] # 2-letter tld-style country code\n    # go fetch the data\n    \u003e\u003e\u003e codesdict, metainfo = data.geonames.fetchdata(countries)\n    # write files for further use\n    \u003e\u003e\u003e data.geonames.writefile(codesdict, 'geonames-codes.dict')\n    \u003e\u003e\u003e data.geonames.writefile(metainfo, 'geonames-meta.dict')\n\n\nExtraction, disambiguation and mapping\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis tutorial uses a file provided in the ``tests`` folder and the information gathered above to go from a tagged sentence to a map:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from geokelone import data, geo, text\n    # read from a tagged text (one token per line)\n    \u003e\u003e\u003e splitted = text.readfile.readtagged('tests/data/fontane-stechlin.tagged')\n    # load default gazetteer info (Geonames, see above)\n    \u003e\u003e\u003e metainfo = data.load.geonames_meta('geonames-meta.dict')\n    \u003e\u003e\u003e codesdict = data.load.geonames_codes('geonames-codes.dict', metainfo)\n    # search for place names and store a list of resolved toponyms with metadata\n    \u003e\u003e\u003e results = geo.geocoding.search(splitted, codesdict, metainfo)\n    # write the results to a file\n    \u003e\u003e\u003e text.outputcontrol.writefile('results.tsv', results, dict())\n    # load results from a file\n    \u003e\u003e\u003e results = data.load.results_tsv('results.tsv')\n    # draw a map\n    \u003e\u003e\u003e geo.mapping.draw_map('testmap.png', results)\n\n\nUsage of single components\n--------------------------\n\nMapping\n~~~~~~~\n\nRequires a file containing results of a placename extraction. The minimal requirements are a toponym and coordinates, see the example file in the ``tests`` folder:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from geokelone import data, geo\n    \u003e\u003e\u003e results = data.load.results_tsv('tests/data/dummy-results.tsv')\n    \u003e\u003e\u003e geo.mapping.draw_map('testmap1.png', results)\n\nThe map window can be configured using the ``settings.py`` file.\n\n\nExtension and adaptation\n------------------------\n\n\nSpecial parameters\n~~~~~~~~~~~~~~~~~~\n\nDid you know there was a Jerusalem in Bavaria and a Leipzig in Ukraine?\n\nA series of parameters can be set to affect both search and visualization, see ``settings.py`` file.\n\nAllowed values for the filter level are ``MAXIMUM`` (conservative setting, recommended), ``MEDIUM`` and ``MINIMUM`` (better recall comes at a price).\n\n\nWhy curate special registers or gazetteers?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nEven with a touch of filtering, the token \"Berlin\" in Geonames resolves to a place north of Germany with a population of 0, see map below:\n\n.. image:: tests/example-wrong.png\n    :align: center\n    :alt: example\n\n\nCustom registers\n~~~~~~~~~~~~~~~~\n\nThe helper function in ``data.load.load_tsv()`` allow for additional registers to match particular needs, with particular levels (0 to 3), for example:\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from geokelone import data\n    # read from a TSV-file with three columns: name, latitude, longitude\n    \u003e\u003e\u003e customized = data.load.load_tsv('file-X.tsv')\n    # read from a CSV-file with optional level option (additional metadata)\n    # four columns expected: name, canonical name, latitude, longitude\n    \u003e\u003e\u003e customized = data.load.load_csv('file-Y.csv', level=1)\n    \u003e\u003e\u003e results = geo.geocoding.search(splitted, codesdict, metainfo, customized)\n\n\nUsing information from Wikipedia/Wikidata\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe module includes helpers to navigate categories, for example the `World Heritage Sites in England \u003chttps://en.wikipedia.org/wiki/Category:World_Heritage_Sites_in_England\u003e`_ or the `Cultural Landscapes of Japan \u003chttps://en.wikipedia.org/wiki/Category:Cultural_Landscapes_of_Japan\u003e`_ and to fetch coordinates for a given list by querying Wikipedia.\n\n.. code-block:: python\n\n    \u003e\u003e\u003e from geokelone.data import wikipedia\n    # chained operations for a list of categories\n    \u003e\u003e\u003e wikipedia.process_todolist('mytodolist.txt', outputfile='solved.tsv', categories=True)\n    # discover entries in a category\n    \u003e\u003e\u003e category_members = wikipedia.navigate_category('XYZ')\n    # process them one by one\n    \u003e\u003e\u003e for member in category_members:\n    \u003e\u003e\u003e     lat, lon = wikipedia.find_coordinates(member)\n    \u003e\u003e\u003e     print(member, lat, lon)\n    # change language code for search (default is 'en')\n    \u003e\u003e\u003e wikipedia.find_coordinates('Wien', language='de')\n    (48.208, 16.373)\n\nIntegration\n~~~~~~~~~~~\n\nFor language-independent solutions in the Python world, see `spacy \u003chttps://spacy.io/\u003e`_ or `polyglot \u003chttps://github.com/aboSamoor/polyglot\u003e`_.\n\nAPI-based geocoding solutions for Python: `geopy \u003chttps://github.com/geopy/geopy\u003e`_ and `geocoder \u003chttps://github.com/DenisCarriere/geocoder\u003e`_.\n\n\nInstallation\n------------\n\nThe instructions below have been tested on Linux with several system settings (see ``.travis.yml`` file). It works best with recent Linux and Mac systems and Python version \u003e= 3.5.\n\nThe cartographic components may need to be installed separately, for detailed instructions please refer to the Cartopy `documentation \u003chttp://scitools.org.uk/cartopy/docs/latest/installing.html#installing\u003e`_.\n\nUnofficial Windows binaries for Python packages are `available here \u003chttps://www.lfd.uci.edu/~gohlke/pythonlibs/\u003e`_.\n\n\nProj library\n~~~~~~~~~~~~\n\nThe `proj library \u003chttps://github.com/OSGeo/proj.4/\u003e`_ is needed. There are several ways to install it:\n\n- From a package repository (preferably posterior to 2016)\n\n  - there are several options (*libproj0* or *libproj9* or *libproj12*), to let the system decide:\n  - ``apt-get install libproj-dev proj-data proj-bin``\n\n- From source:\n\n  a. ``wget http://download.osgeo.org/proj/proj-5.2.0.tar.gz``\n  b. ``tar -xzvf proj-5.2.0.tar.gz``\n  c. ``cd proj-5.2.0 \u0026\u0026 ./configure --prefix=/usr \u0026\u0026 make \u0026\u0026 sudo make install``\n\nOther packages\n~~~~~~~~~~~~~~\n\n-  ``apt-get install libgeos-* libffi-dev libgdal-dev libxslt1-dev``\n\nPython packages\n~~~~~~~~~~~~~~~\n\nOnly Python3 (especially 3.4 onwards) is supported, although the scripts may work for Python 2.7.\n\nTwo options, from system repositories or through ``pip``:\n\n- *python3-dev python3-shapely python3-gdal python3-matplotlib python3-pyproj python3-shapely*\n- or simply ``pip3 install cairocffi GDAL matplotlib pyproj shapely``\n\nFor installation on Debian/Ubuntu simply follow the instructions (*before_install:*) in the ``travis.yml`` file\n\n*Additional note on GDAL in case problems occur during installation:*\n\n- ``gdal-config --version``\n- ``sudo pip3 install --global-option=build_ext --global-option=\"-I/usr/include/gdal\" GDAL==2.2.3``\n\n\nCartopy install notes\n~~~~~~~~~~~~~~~~~~~~~\n\nFinally, *cartopy* can be installed:\n\n- ``pip3 install Cython`` (if not installed already)\n- ``pip3 install cartopy``\n- or on newer systems: ``apt-get install python3-cartopy`` cf `\u003chttps://packages.ubuntu.com/source/zesty/python-cartopy\u003e`_\n- or see here: `\u003chttp://scitools.org.uk/cartopy/docs/latest/installing.html#installing\u003e`_\n\nInstall this package\n~~~~~~~~~~~~~~~~~~~~\n\nDirect installation of the latest version over pip is possible (see `build status \u003chttps://travis-ci.org/adbar/geokelone\u003e`_):\n\n-  ``pip3 install git+https://github.com/adbar/geokelone.git``\n\n\n\nAdditional info\n---------------\n\nWhy *geokelone*? `Because \u003chttps://en.wikipedia.org/wiki/Geochelone\u003e`_.\n\n**Work in progress**, see legacy page for more information: `\u003chttps://github.com/adbar/toponyms\u003e`_\n\nTODO\n~~~~\n\n- provide map configuration\n- integrate named entity recognition tool from Python repositories\n- add more import and export filters\n- write more tests\n- documentation\n\n\nReferences\n~~~~~~~~~~\n\nUses of the code base so far:\n\n- Barbaresi, A. (2018). `Borderlands of text mapping: Experiments on Fontane's Brandenburg \u003chttps://hal.archives-ouvertes.fr/hal-01951880/document\u003e`_. Proceedings of INF-DH-2018 workshop.\n- Barbaresi, A. (2018). `A constellation and a rhizome: two studies on toponyms in literary texts \u003chttps://hal.archives-ouvertes.fr/hal-01775127/document\u003e`_. In *Visual Linguistics*, Bubenhofer N. \u0026 Kupietz M. (Eds.), Heidelberg University Publishing, pp. 167-184.\n- Barbaresi, A. (2018). `Toponyms as Entry Points into a Digital Edition: Mapping Die Fackel \u003chttps://hal.archives-ouvertes.fr/hal-01775122/document\u003e`_. *Open Information Science*, 2(1), De Gruyter, pp.23-33.\n- Barbaresi, A. (2018). `Placenames analysis in historical texts: tools, risks and side effects \u003chttps://hal.archives-ouvertes.fr/hal-01775119/document\u003e`_. In *Proceedings of the Second Workshop on Corpus-Based Research in the Humanities (CRH-2)*, Dept. of Geoinformation, TU Vienna, pp. 25-34.\n- Barbaresi, A. (2017). `Towards a toolbox to map historical text collections \u003chttps://hal.archives-ouvertes.fr/hal-01654526/document\u003e`_, *Proceedings of 11th Workshop on Geographic Information Retrieval*, ACM, Heidelberg.\n- Barbaresi, A. and Biber, H. (2016). `Extraction and Visualization of Toponyms in Diachronic Text Corpora \u003chttps://hal.archives-ouvertes.fr/hal-01348696/document\u003e`_. In *Digital Humanities 2016: Book of Abstracts*, pp. 732-734.\n- Barbaresi, A. (2016). `Visualisierung von Ortsnamen im Deutschen Textarchiv \u003chttps://halshs.archives-ouvertes.fr/halshs-01287931/document\u003e`_. In *Proceedings of DHd 2016*, Digital Humanities im deutschprachigen Raum e.V. pp. 264-267.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadbar%2Fgeokelone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadbar%2Fgeokelone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadbar%2Fgeokelone/lists"}