{"id":13856693,"url":"https://github.com/eyeseast/geocode-sqlite","last_synced_at":"2025-04-06T01:07:23.351Z","repository":{"id":43153320,"uuid":"293361514","full_name":"eyeseast/geocode-sqlite","owner":"eyeseast","description":"Geocode rows in a SQLite database table","archived":false,"fork":false,"pushed_at":"2022-11-07T17:31:05.000Z","size":125,"stargazers_count":235,"open_issues_count":8,"forks_count":6,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-30T09:13:48.364Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eyeseast.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"eyeseast"}},"created_at":"2020-09-06T21:05:39.000Z","updated_at":"2025-03-03T17:13:51.000Z","dependencies_parsed_at":"2022-08-28T04:40:38.856Z","dependency_job_id":null,"html_url":"https://github.com/eyeseast/geocode-sqlite","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyeseast%2Fgeocode-sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyeseast%2Fgeocode-sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyeseast%2Fgeocode-sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eyeseast%2Fgeocode-sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eyeseast","download_url":"https://codeload.github.com/eyeseast/geocode-sqlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247419860,"owners_count":20936012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-05T03:01:09.187Z","updated_at":"2025-04-06T01:07:23.333Z","avatar_url":"https://github.com/eyeseast.png","language":"Python","readme":"# geocode-sqlite\n\n[![PyPI](https://img.shields.io/pypi/v/geocode-sqlite.svg)](https://pypi.org/project/geocode-sqlite/)\n[![Changelog](https://img.shields.io/github/v/release/eyeseast/geocode-sqlite?include_prereleases\u0026label=changelog)](https://github.com/eyeseast/geocode-sqlite/releases)\n[![Tests](https://github.com/eyeseast/geocode-sqlite/workflows/Test/badge.svg)](https://github.com/eyeseast/geocode-sqlite/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/geocode-sqlite/blob/master/LICENSE)\n\nGeocode rows from a SQLite table\n\n## Installation\n\nInstall this tool using `pip` or `pipx`:\n\n```sh\n# install inside a virtualenv\npip install geocode-sqlite\n\n# install globally\npipx install geocode-sqlite\n```\n\n## Usage\n\nLet's say you have a spreadsheet with addresses in it, and you'd like to map those locations.\nFirst, create a SQLite database and insert rows from that spreadsheet using `sqlite-utils`.\n\n```sh\nsqlite-utils insert data.db data data.csv --csv\n```\n\nNow, geocode it using OpenStreetMap's Nominatim geocoder.\n\n```sh\ngeocode-sqlite nominatim data.db data \\\n --location=\"{address}, {city}, {state} {zip}\" \\\n --delay=1 \\\n --user-agent=\"this-is-me\"\n```\n\nIn the command above, you're using Nominatim, which is free and only asks for a unique user agent (`--user-agent`).\n\nThis will connect to a database (`data.db`) and read all rows from the table `data` (skipping any that already\nhave both a `latitude` and `longitude` column filled).\n\nYou're also telling the geocoder how to extract a location query (`--location`) from a row of data, using Python's\nbuilt-in string formatting, and setting a rate limit (`--delay`) of one request per second.\n\nFor each row where geocoding succeeds, `latitude` and `longitude` will be populated. If you hit an error, or a rate limit,\nrun the same query and pick up where you left off.\n\nThe resulting table layout can be visualized with [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map).\n\nUnder the hood, this package uses the excellent [geopy](https://geopy.readthedocs.io/en/latest/) library, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consult [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders).\n\n### Supported Geocoders\n\nThe CLI currently supports these geocoders:\n\n- `bing`\n- `googlev3`\n- `mapquest` (and `open-mapquest`)\n- `mapbox`\n- `nominatim`\n- `opencage`\n\n#### Adding new geocoders\n\n1. Open an issue with the name of the geocoding service as the ticket title ([example](https://github.com/eyeseast/geocode-sqlite/issues/35)). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.\n2. Fork the repo and add a geocoder.\n3. Add an example to the `Makefile`. Add tests if there's new shared functionality.\n\n### Common arguments and options\n\nEach geocoder needs to know where to find the data it's working with. These are the first two arguments:\n\n- `database`: a path to a SQLite file, which must already exist\n- `table`: the name of a table, in that database, which exists and has data to geocode\n\nFrom there, we have a set of options passed to every geocoder:\n\n- `location`: a [string format](https://docs.python.org/3/library/stdtypes.html#str.format) that will be expanded with each row to build a full query, to be geocoded\n- `delay`: a delay between each call (some services require this)\n- `latitude`: latitude column name\n- `longitude`: longitude column name\n- `geojson`: store results as GeoJSON, instead of in latitude and longitude columns\n- `spatialite`: store results in a SpatiaLite geometry column, instead of in latitude and longitude columns\n- `raw`: store raw geocoding results in a JSON column\n\nEach geocoder takes additional, specific arguments beyond these, such as API keys. Again, [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) is an excellent resource.\n\n## Using SpatiaLite\n\nThe `--spatialite` flag will store results in a [geometry column](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/cookbook_topics.adminstration.html#topic_TABLE_to_SpatialTable), instead of `latitude` and `longitude` columns. This is useful if you're doing other GIS operations, such as using a [spatial index](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=SpatialIndex). See the [SpatiaLite cookbook](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/index.html) and [functions list](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html) for more of what's possible.\n\n## Capturing additional geocoding data\n\nGeocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the `--raw` flag. By default, this will add a `raw` column and store the full geocoding response as JSON. If you want to rename that column, pass a value, like `--raw custom_raw`.\n\nThe shape of this response object will vary between services. You can query specific values using [SQLite's built-in JSON functions](https://www.sqlite.org/json1.html). For example, this will work with Google's geocoder:\n\n```sql\nselect\n  json_extract(raw, '$.formatted_address') as address,\n  json_extract(raw, '$.geometry.location_type') as location_type\nfrom\n  innout_test\n```\n\nCheck each geocoding service's documentation for what's included in the response.\n\n## Python API\n\nThe command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.\n\nAs with the CLI, this assumes you already have a SQLite database and a table of location data.\n\n```python\nfrom geocode_sqlite import geocode_table\nfrom geopy.geocoders import Nominatim\n\n# create a geocoder instance, with some extra options\nnominatim = Nominatim(user_agent=\"this-is-me\", domain=\"nominatim.local.dev\", scheme=\"http\")\n\n# assuming our database is in the same directory\ncount = geocode_table(\"data.db\", \"data\", query_template=\"{address}, {city}, {state} {zip}\")\n\n# when it's done\nprint(f\"Geocoded {count} rows\")\n```\n\nAny [geopy geocoder](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) can be used with the Python API.\n\n## Development\n\nTo contribute to this tool, first checkout the code. Then create a new virtual environment:\n\n```sh\ncd geocode-sqlite\npython -m venv .venv\nsource .venv/bin/activate\n```\n\nOr if you are using `pipenv`:\n\n```sh\npipenv shell\n```\n\nNow install the dependencies and tests:\n\n```sh\npip install -e '.[test]'\n```\n\nTo run the tests:\n\n```sh\npytest\n```\n\nPlease remember that this library is mainly glue code between other well-tested projects, specifically: [click](https://click.palletsprojects.com/), [geopy](https://geopy.readthedocs.io/en/stable/) and [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/). Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.\n\nTo that end, there is a test geocoder included: `geocode_sqlite.testing.DummyGeocoder`. That geocoder works with an included dataset of In-N-Out Burger locations provided by [AllThePlaces](https://www.alltheplaces.xyz/). It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.\n","funding_links":["https://github.com/sponsors/eyeseast"],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feyeseast%2Fgeocode-sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feyeseast%2Fgeocode-sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feyeseast%2Fgeocode-sqlite/lists"}