{"id":15722187,"url":"https://github.com/next-li/datasette-csv-importer","last_synced_at":"2026-01-29T23:31:40.103Z","repository":{"id":72006100,"uuid":"355625718","full_name":"next-LI/datasette-csv-importer","owner":"next-LI","description":"A Datasette plugin for live-importing CSV files with a user-friendly configuration UI, without the need to restart your application","archived":false,"fork":false,"pushed_at":"2021-09-15T15:54:35.000Z","size":896,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-07T07:49:43.526Z","etag":null,"topics":["csv-import","datasette","datasette-live","datasette-plugin","dynamic"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/next-LI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-07T17:15:19.000Z","updated_at":"2023-08-31T08:03:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"ab516d5f-693b-4857-856f-ad59e5dc9bba","html_url":"https://github.com/next-LI/datasette-csv-importer","commit_stats":{"total_commits":52,"total_committers":2,"mean_commits":26.0,"dds":"0.17307692307692313","last_synced_commit":"8551990cd9debc7b8aa0870e72c9d82820468a17"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/next-LI%2Fdatasette-csv-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/next-LI%2Fdatasette-csv-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/next-LI%2Fdatasette-csv-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/next-LI%2Fdatasette-csv-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/next-LI","download_url":"https://codeload.github.com/next-LI/datasette-csv-importer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247615481,"owners_count":20967182,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv-import","datasette","datasette-live","datasette-plugin","dynamic"],"created_at":"2024-10-03T22:05:03.371Z","updated_at":"2026-01-29T23:31:40.047Z","avatar_url":"https://github.com/next-LI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# datasette-csv-importer\n\nA Datasette plugin for live-uploading CSV files with a user-friendly configuration UI that doesn't require the application to be restarted to work.\n\nThis plugin was inspired by the [datasette-upload-csvs](https://github.com/simonw/datasette-upload-csvs) plugin and uses [CSVs-to-SQlite](https://github.com/simonw/csvs-to-sqlite) to actually perform the import. Configuration comes from a parsed version of the CLI tool's `--help` output.\n\n## Installation\n\nVia Datasette:\n\n    datasette install git+https://github.com/next-LI/datasette-csv-importer.git\n\nOr a specific version:\n\n    datasette install git+https://github.com/next-LI/datasette-csv-importer.git@v1.0.4\n\n## Development Installation\n\nThis repository uses submodules:\n\n    git clone --recursive https://github.com/next-LI/datasette-csv-importer\n\nIf you've already cloned non-recursively you can run this to get the submodules:\n\n    git submodule update --init\n\nThen just install like any normal Datasette plugin:\n\n    python setup.py install\n\nIt accepts the following configuration in your `metadata.json` file:\n\n    {\n      ... metadata.json ...\n      \"plugins\": {\n        \"datasette-csv-importer\": {\n          \"status_table\": \"_csv_importer_progress_\",\n          \"status_database\": \"_internal\",\n          \"database_path\": \"/data\",\n          \"csvs_path\": \"/data/csvs\"\n        },\n        ... the rest of your plugins configuration ...\n      }\n    }\n\nDetails on the configuration:\n\n- `status_database` - name of the database that we're going to use to store import status. It defaults to the internal DB, `_internal`.\n- `status_table` - name of the table that we'll write import status rows to.\n- `database_path` - path to the directory that we'll use to write SQlite databases to. By default, the plugin will use the current working directory of the process running it.\n- `csvs_path` - path to the directory where raw uploaded CSVs will be saved, along with their `csvs-to-sqlite` import settings. If this isn't set, then this plugin won't do CSV/config saving.\n- `use_db_metadata` - (boolean)  Whether or not to build the `__metadata` table as supported by the [next-LI/datasette_live_config](https://github.com/next-LI/datasette_live_config) plugin, giving default access to the current user.\n- `use_live_permissions` - (boolean) Whether or not to integrate with the [next-LI/datasette-live-permissions](https://github/com/next-LI/datasette-live-permissions) and grant the uploading user access upon upload.\n\n\n## Usage\n\n**tl;dr**: Go to `/-/csv-importer`, select a CSV, choose import options, click submit 💥 and your data is live on the dashboard!\n\nThe plugin adds an interface at `/-/csv-importer` for importing a CSV file. Once you drag and drop a CSV file, you'll be shown a set of options for importing your data into Datasette using the [CSVs-to-SQlite](https://github.com/simonw/csvs-to-sqlite) tool. Clicking \"Submit\" below will start the import process and you'll be given information about success or failure.\n\n## Permissions\n\nThis plugin uses Datasette's permissioning system. Granting actors the `csv-importer` permission will allow access to CSV importing. The `root` actor is granted access by default.\n\n## Development\n\nThere's two parts to this plugin: converting the CSV importer CLI tool's arguments list to a JSON schema which renders a form (this is mostly done by hand and ends up in `templates/schema.json` and the webapp that does the actual import.\n\nForm configuration can be done via the form schema in `templates/form.json` and documentation for supported options [are here](https://github.com/jsonform/jsonform/wiki)\n\n### KNOWN ISSUES\n\nThere's an async race condition that can sometimes be triggered when uploading and inserting a new database. You'll get a 500 error and a console traceback like the following:\n\n```\nINFO:     127.0.0.1:52790 - \"POST /-/csv-importer HTTP/1.1\" 200 OK\n/Users/brandon/.pyenv/versions/nextli/lib/python3.8/site-packages/pandas/core/generic.py:2779: UserWarning: The spaces in these column names will not be changed. In pandas versions \u003c 0.14, spaces were converted to underscores.\n  sql.to_sql(\nThere is no current event loop in thread 'Thread-2'.\nTraceback (most recent call last):\n  File \"/Users/brandon/.pyenv/versions/nextli/lib/python3.8/site-packages/datasette-0.56-py3.8.egg/datasette/app.py\", line 1144, in route_path\n    response = await view(request, send)\n  File \"/Users/brandon/.pyenv/versions/nextli/lib/python3.8/site-packages/datasette-0.56-py3.8.egg/datasette/views/base.py\", line 146, in view\n    return await self.dispatch_request(\n  File \"/Users/brandon/.pyenv/versions/nextli/lib/python3.8/site-packages/datasette-0.56-py3.8.egg/datasette/views/base.py\", line 119, in dispatch_request\n    await self.ds.refresh_schemas()\n  File \"/Users/brandon/.pyenv/versions/nextli/lib/python3.8/site-packages/datasette-0.56-py3.8.egg/datasette/app.py\", line 343, in refresh_schemas\n    for database_name, db in self.databases.items():\nRuntimeError: OrderedDict mutated during iteration\n```\n\nAs far as I can tell, you can ignore this error. If your import got to the point where this bug can be triggered, your data was successfully imported.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnext-li%2Fdatasette-csv-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnext-li%2Fdatasette-csv-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnext-li%2Fdatasette-csv-importer/lists"}