{"id":28579699,"url":"https://github.com/mkai/dictionarydb","last_synced_at":"2025-06-11T02:09:56.868Z","repository":{"id":38797583,"uuid":"296440466","full_name":"mkai/dictionarydb","owner":"mkai","description":"Python-based CLI tool to set up the database for a dictionary app","archived":false,"fork":false,"pushed_at":"2023-01-18T15:03:57.000Z","size":676,"stargazers_count":0,"open_issues_count":14,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-03-12T03:36:36.472Z","etag":null,"topics":["database","dictionary","education","importer","language","translations"],"latest_commit_sha":null,"homepage":"https://dictionarydb.herokuapp.com/docs#/default/lookup_lookup_get","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mkai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-17T20:55:46.000Z","updated_at":"2021-03-24T21:20:13.000Z","dependencies_parsed_at":"2023-02-10T15:45:14.835Z","dependency_job_id":null,"html_url":"https://github.com/mkai/dictionarydb","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkai%2Fdictionarydb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkai%2Fdictionarydb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkai%2Fdictionarydb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkai%2Fdictionarydb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mkai","download_url":"https://codeload.github.com/mkai/dictionarydb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkai%2Fdictionarydb/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259184763,"owners_count":22818271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","dictionary","education","importer","language","translations"],"created_at":"2025-06-11T02:09:45.990Z","updated_at":"2025-06-11T02:09:55.411Z","avatar_url":"https://github.com/mkai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dictionarydb\n\nA Python-based CLI tool to set up the database for a dictionary app.\n\nThe data file included with the freely licensed [Ding dictionary lookup program](https://www-user.tu-chemnitz.de/~fri/ding/) can be used to populate the database with English ↔ German translations.\n\n[![Build][github-actions-image]][github-actions-url]\n[![Coverage][codecov-image]][codecov-url]\n\n[github-actions-image]: https://github.com/mkai/dictionarydb/workflows/Test/badge.svg?branch=main\u0026event=push\n[github-actions-url]: https://github.com/mkai/dictionarydb/actions?query=branch%3Amain+event%3Apush\n[codecov-image]: https://codecov.io/gh/mkai/dictionarydb/branch/main/graph/badge.svg\n[codecov-url]: https://codecov.io/gh/mkai/dictionarydb\n\n## Features\n\n- Imports data quickly using bulk insertion and configurable chunking. It takes about 40 seconds to import 367 000 translations (753 000 words) into a new SQLite database.\n- Uses Python generators to process large amounts of data efficiently, i.e. without consuming too much memory.\n- The import process is atomic: if an error occurs midway through the task, it is aborted and the database remains in its original state (uses SQL transactions).\n- Abstraction of SQL/DDL specifics (via [SQLAlchemy](https://www.sqlalchemy.org)) allows for easily adding support for new databases (tested with [PostgreSQL](https://www.postgresql.org) and [SQLite](https://www.sqlite.org)).\n- Flexible command line interface (CLI) which supports reading from standard input (via [Click](https://click.palletsprojects.com)).\n- Reasonably scalable, OpenAPI-compliant web API (via [FastAPI](https://fastapi.tiangolo.com)).\n\n## The CLI tool\n\nType `dictionarydb` to start the CLI tool:\n\n```shell\n$ dictionarydb\n```\n\nSince no [command](#commands) was specified, it will print a block of usage information:\n\n```\nUsage: dictionarydb [OPTIONS] COMMAND [ARGS]...\n\n  Set up and populate a translation dictionary database.\n\nOptions:\n  --version  Show the version and exit.\n  --help     Show this message and exit.\n\nCommands:\n  api     Start the API server.\n  import  Import new entries into the dictionary database.\n  init    Create the database schema for the dictionary database.\n```\n\n### \u003ca name=\"commands\"\u003e\u003c/a\u003eCLI commands\n\nThree commands are available:\n\n* `dictionarydb init` to initialise a new database (see [Initialising the database](#init)).\n* `dictionarydb import` to import translations into the database (see [Importing translations](#import)).\n* `dictionarydb api` to run the lookup API server (see [Starting the API server](#api)).\n\nYou can run each command with the `--help` argument to show the available options. For example, to show usage information for the `init` command:\n\n```shell\n$ dictionarydb init --help\n```\n\nThis would give you the following output:\n\n```\nUsage: dictionarydb init [OPTIONS]\n\n  Create the database schema for the dictionary database.\n\nOptions:\n  -u, --database-url TEXT   URL of the database to initialize.\n  --confirm / --no-confirm  Whether or not to ask for confirmation before\n                            proceeding.\n\n  --help                    Show this message and exit.\n```\n\n## \u003ca name=\"init\"\u003e\u003c/a\u003eInitialising the database\n\nUse the `init` command to create the database schema:\n\n```shell\n$ dictionarydb init\n```\n\nBy default, this will create a new SQLite database in the `data/` directory. If you want to use PostgreSQL instead, use the `--database-url` option:\n\n```shell\n$ dictionarydb init --database-url='postgresql://localhost:5432/dictionary'\n```\n\n**Note:** you will need to create the database (`dictionary` in this example) manually before running the command.\n\nWhen all is done, the following schema will have been created in your database:\n\n![Image showing the schema of the dictionary database](./docs/images/database_schema.png?raw=true \"Dictionary database schema\")\n\n## \u003ca name=\"import\"\u003e\u003c/a\u003eImporting translations\n\n### Downloading the data file\n\nNow you can populate the database with data. First, download the [Ding dictionary](https://www-user.tu-chemnitz.de/~fri/ding/) data file and unpack it:\n\n```shell\n$ curl --output de-en.txt.xz https://ftp.tu-chemnitz.de/pub/Local/urz/ding/de-en-devel/de-en.txt.xz\n$ xz --decompress de-en.txt.xz\n```\n\nThis should give you a file named `de-en.txt` in the current directory.\n\n### Running the importer\n\nYou can now use the `dictionarydb import` command and point it at your `de-en.txt` file in order to run an import:\n\n```shell\n$ dictionarydb import ./de-en.txt --source-language=\"deu\" --target-language=\"eng\"\n```\n\nIt will give you the following output:\n\n```\nStarting dictionary import from file \"./de-en.txt\"…\nThis will remove all existing entries. Continue? [y/N]: y\nRemoving existing dictionary entries…\nCreating languages…\nStoring new dictionary entries…\nCommitting transaction…\nSuccessfully completed dictionary import (0 deleted, 376541 added, 39.67 seconds elapsed).\n```\n\nOnce the import is successful, your database should contain about 750 000 words:\n\n![Image showing the contents of the dictionary database after import](./docs/images/database_contents.png?raw=true \"Dictionary database contents\")\n\n**Note:** if you want to use PostgreSQL instead, use the `--database-url` option again as described above. Set the `DICTIONARYDB_DATABASE_URL` environment variable to the same value to make it persistent (see also: [Configuration](#configuration)).\n\n#### Using standard input\n\nThe CLI tool also supports reading data from another shell command. Pass `-` as the input filename (`stdin`) in this scenario:\n\n```shell\n$ xzcat ./de-en.txt.xz | dictionarydb import - --source-language=\"deu\" --target-language=\"eng\" --no-confirm\n```\n\nYou could even do a streaming import directly over HTTP:\n\n```shell\n$ curl --silent https://ftp.tu-chemnitz.de/pub/Local/urz/ding/de-en-devel/de-en.txt.xz | xzcat | dictionarydb import - --source-language=\"deu\" --target-language=\"eng\" --no-confirm\n```\n\n### Automating the import\n\nYou could use a cron job to populate the database automatically and keep it up to date in an unattended fashion.\n\nFor example, the following command will download the latest source file and then decompress and import it into the database:\n\n```shell\n$ curl --output de-en.txt.xz https://ftp.tu-chemnitz.de/pub/Local/urz/ding/de-en-devel/de-en.txt.xz \u0026\u0026 xzcat de-en.txt.xz | dictionarydb import - --source-language=\"deu\" --target-language=\"eng\" --min-entries=370000 --no-confirm\n```\n\n**Note:** this uses the `--min-entries` option to enforce that a minimum number of valid entries is imported successfully. If that is not the case, then the import will fail cleanly, meaning the database transaction will be rolled back and the database will remain in its original state.\n\nThe `--no-confirm` option is used to prevent the shell from waiting for the user's confirmation indefinitely.\n\n## \u003ca name=\"configuration\"\u003e\u003c/a\u003eConfiguration\n\nWhile most of the options can be passed to `dictionarydb` using command line flags, you might want to make some settings persistent. You can do this by setting one or more of the following environment variables:\n\n* [`DICTIONARYDB_LOG_LEVEL`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L15): The log level (verbosity) to use. Defaults to \"INFO\".\n* [`DICTIONARYDB_LOG_COLORS`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L25): Whether or not to color the log output. Defaults to true.\n* [`DICTIONARYDB_DATABASE_URL`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L69): A connection URL to use for connecting to the database. The default is to create a new SQLite database file in the `data/` directory.\n* [`DICTIONARYDB_IMPORT_CHUNK_SIZE`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L84): Maximum number of entries to hold in memory at once during the import. Data will be sent to the database (and freed from memory) once _n_ entries have been read. Defaults to _10 000_.\n* [`DICTIONARYDB_API_HOST`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L96): Network address on which the API server should listen. Defaults to _localhost_.\n* [`DICTIONARYDB_API_PORT`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L106): TCP port number on which the API server should run. Defaults to _8080_.\n* [`DICTIONARYDB_API_TRUST_PROXY_IPS`](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/config.py#L127): Proxy IP addresses to trust when determining the client's IP, port and protocol. By default, only _127.0.0.1_ (i.e. a proxy running locally) is trusted.\n\n**Hint:** clicking the name of a setting will take you to a more detailed description of the respective setting along with configuration examples.\n\n### Example: setting the database URL\n\nTo make the `dictionarydb` tool always use a local PostgreSQL database, you could set the `DICTIONARYDB_DATABASE_URL` environment variable as follows:\n\n```shell\nexport DICTIONARYDB_DATABASE_URL=\"postgresql://localhost:5432/dictionary\"\n```\n\n## Querying the database\n\nSee [this query](https://github.com/mkai/dictionarydb/blob/756acaa4c4deefde296b392e67cbca12d2a180f4/dictionarydb/api.py#L31) for an example of how you could look up a word and its available translations using SQL.\n\n## \u003ca name=\"api\"\u003e\u003c/a\u003eStarting the API server\n\nThe `dictionarydb api` command lets you start the API server:\n\n```shell\n$ dictionarydb api\n```\n\nIf all goes well, the API is now running:\n\n```\nStarting API server on http://localhost:8080…\n```\n\nHit Ctrl+C if you need to shut it down again.\n\n## Consuming the API\n\n### Using `curl`\n\nPoint `curl` to the `/lookup` endpoint as follows to translate the word _conscientious_):\n\n```shell\n$ curl --request GET \"http://localhost:8080/lookup?source_language=eng\u0026target_language=deu\u0026search_string=conscientious\u0026max_results=3\" --header  \"Accept: application/json\"\n```\n\nNote that you need to pass three `GET` parameters:\n\n* `source_language`: [ISO 639-3](https://iso639-3.sil.org/code_tables/639/data) code of the language you want to translate from.\n* `target_language`: ISO 639-3 code of the language into which you want to translate.\n* `search_string`: the word (or a substring of it) you would like to look up.\n\nThe `max_results` query parameter is optional. It can be used to limit the number of results that are returned.\n\n`curl` should give you a response as follows:\n\n```json\n{\n  \"results\": [\n    {\n      \"word\": \"conscientious\",\n      \"language\": \"eng\",\n      \"translation\": \"gewissenhaft {adj}\",\n      \"translation_language\": \"deu\",\n      \"relevance\": 1.0\n    },\n    {\n      \"word\": \"conscientiously\",\n      \"language\": \"eng\",\n      \"translation\": \"gewissenhaft {adv}\",\n      \"translation_language\": \"deu\",\n      \"relevance\": 0.7647058963775635\n    },\n    {\n      \"word\": \"conscientiousness\",\n      \"language\": \"eng\",\n      \"translation\": \"Gewissenhaftigkeit {f} [psych.]\",\n      \"translation_language\": \"deu\",\n      \"relevance\": 0.6842105388641357\n    }\n  ]\n}\n```\n\n### Using a REST client\n\nConsider using a graphical API client like [Insomnia](https://insomnia.rest) for a more comfortable experience:\n\n![Image showing how to use the lookup API using a graphical API client](./docs/images/api_lookup.png?raw=true \"Querying the API using Insomnia\")\n\n### API documentation\n\nFinally, there is an OpenAPI documentation site available at [http://localhost:8080/docs](http://localhost:8080/docs). Use the \"Try it out\" button on the _/lookup_ resource to perform dictionary lookups.\n\n## Demo\n\nA demo deployment of the dictionary lookup API is available at [https://dictionarydb.herokuapp.com](https://dictionarydb.herokuapp.com). It uses the [Heroku](https://www.heroku.com) scheduler add-on to keep the database up to date with the latest translations automatically.\n\n## Contributing\n\nContributions welcome! See the [CONTRIBUTING.md](./CONTRIBUTING.md) document for an overview of how to set up the project for development.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkai%2Fdictionarydb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmkai%2Fdictionarydb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkai%2Fdictionarydb/lists"}