{"id":14155116,"url":"https://github.com/c-3lab/dim","last_synced_at":"2026-01-17T06:23:45.648Z","repository":{"id":41367256,"uuid":"398181253","full_name":"c-3lab/dim","owner":"c-3lab","description":"📦 dim: Manage the open data in your project like a package manager.","archived":false,"fork":false,"pushed_at":"2024-01-21T12:09:03.000Z","size":308747,"stargazers_count":131,"open_issues_count":31,"forks_count":22,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-01-15T14:41:55.675Z","etag":null,"topics":["cli","commads","command-line-tool","data","dataops","dim","gpt","gpt-3","llm","opendata","package-manager","public-data","public-dataset"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c-3lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-20T06:42:41.000Z","updated_at":"2025-06-10T15:29:27.000Z","dependencies_parsed_at":"2024-01-21T12:47:28.785Z","dependency_job_id":null,"html_url":"https://github.com/c-3lab/dim","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/c-3lab/dim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-3lab%2Fdim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-3lab%2Fdim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-3lab%2Fdim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-3lab%2Fdim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c-3lab","download_url":"https://codeload.github.com/c-3lab/dim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-3lab%2Fdim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28502214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T04:31:57.058Z","status":"ssl_error","status_checked_at":"2026-01-17T04:31:45.816Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","commads","command-line-tool","data","dataops","dim","gpt","gpt-3","llm","opendata","package-manager","public-data","public-dataset"],"created_at":"2024-08-17T08:01:57.217Z","updated_at":"2026-01-17T06:23:45.608Z","avatar_url":"https://github.com/c-3lab.png","language":"TypeScript","readme":"# dim\n\n[![codecov](https://codecov.io/gh/c-3lab/dim/branch/main/graph/badge.svg?token=2PXVJGUNWP)](https://codecov.io/gh/c-3lab/dim)\n[![Github All Releases](https://img.shields.io/github/downloads/c-3lab/dim/total.svg)](https://github.com/c-3lab/dim/releases/latest)\n[![Github All Releases](https://img.shields.io/github/downloads/c-3lab/dim/latest/total.svg)](https://github.com/c-3lab/dim/releases/latest)\n\nData Installation Manager: Manage the open data in your project like a package manager.\n\n![8bket-vzuiv](https://user-images.githubusercontent.com/6661165/148486923-a29f6ea5-ecbc-4d69-9f40-66bed34e3f99.gif)\n\n## Join community\n\nWe are looking for members to develop together as an open source community.\n\n[Slack](https://join.slack.com/t/c3lab-hq/shared_invite/zt-v6zz66n9-1VYkVXC4zoQViWSMdzMTLg)\n\n# Features\n\n- [📀 Record the source url and post-processing, etc., of downloaded open-data](#install)\n- [🔧 Prepare all open data needed for the project in one command by using the `dim.json` recorded by someone else](#install-the-all-data)\n- [🚀 General post-processing, such as unzip, encoding, etc., is available from the start](#specify-the-installation-post-process)\n- [🔍 Search open-data from CKAN](#search)\n- [🧠 Generate code to process data using **GPT-3**](#generate)\n\n# Document\n\nFor more information about how to use it, please refer to [this document](/doc).\n\n# Quick Start\n\n## Install the dim\n\n[Install the dim from binary files](#install-the-dim-from-binary-files) or\n[Run the dim using Deno](#run-the-dim-using-deno)\n\n### Install the dim from binary files\n\nDownload the dim from binary files.\n\n[aarch64-apple-darwin](https://github.com/c-3lab/dim/releases/latest/download/aarch64-apple-darwin-dim)\n\n```\ncurl -L https://github.com/c-3lab/dim/releases/latest/download/aarch64-apple-darwin-dim -o /usr/local/bin/dim\n```\n\n[x86_64-apple-darwin](https://github.com/c-3lab/dim/releases/latest/download/x86_64-apple-darwin-dim)\n\n```\ncurl -L https://github.com/c-3lab/dim/releases/latest/download/x86_64-apple-darwin-dim -o /usr/local/bin/dim\n```\n\n[x86_64-pc-windows-msvc](https://github.com/c-3lab/dim/releases/latest/download/x86_64-pc-windows-msvc-dim.exe)\n\n```\ncurl https://github.com/c-3lab/dim/releases/latest/download/x86_64-pc-windows-msvc-dim.exe -o C:\\Users\\user-name\\dim.exe\n```\n\n[x86_64-unknown-linux-gnu](https://github.com/c-3lab/dim/releases/latest/download/x86_64-unknown-linux-gnu-dim)\n\n```\ncurl -L https://github.com/c-3lab/dim/releases/latest/download/x86_64-unknown-linux-gnu-dim -o /usr/local/bin/dim\n```\n\n#### Grant user execution permission\n\n```\nchmod u+x /usr/local/bin/dim\n```\n\n## New Project\n\n1. init the project\n\nGenerate `dim.json`, `dim-lock.json` and `data_files/` by init command.\n\n```\n$ dim init\n```\n\n2. Install a data\n\nThis command stores information about installed data in `dim.json` and `dim-lock.json`.\n\n```\n$ dim install https://example.com -n \"example\"\n```\n\n3. Installed data is saved in `data_files/`.\n\n```\n$ ls ./data_files\n```\n\n## Install all data written to dim.json shared by members\n\nInstall all data written to `dim.json` shared by members.\n\n![r30y7-wcbx7](https://user-images.githubusercontent.com/6661165/148490980-c1ae8195-a3fd-430f-aa10-c11c7cf1fd64.gif)\n\n1. Make sure existing the dim.json in current directory\n\n```\n$ ls ./\n\ndim.json  ....\n```\n\n2. Install all data written in the dim.json\n\n```\n$ dim install\n```\n\n3. Installed data is saved in `data_files/`.\n\n```\n$ ls ./data_files\n```\n\n# Command Usage\n\n## [Init](doc/en/init.md)\n\nGenerate `dim.json`, `dim-lock.json` and `data_files/`.\n\n```\n$ dim init\n```\n\n## [Install](doc/en/install.md)\n\n### Install the all data.\n\nInstall all data written to `dim.json`.\n\n```\n$ dim install\n```\n\nInstall from a specified local `dim.json`.\n\n```\n$ dim install -f ./path/dim.json\n```\n\nInstall from a specified remote `dim.json` in internet.\n\n```\n$ dim install -f https://raw.githubusercontent.com/xxxx/xxxx/main/dim.json\n```\n\n### Install the specified data.\n\n```\n$ dim install https://example.com -n \"example\"\n```\n\n#### Specify headers.\n\n```\n$ dim install https://example.com -n \"example\" -H \"Authorization: 1234567890abc\" -H \"Fiware-Service: example\"\n```\n\n#### Specify the installation post-process\n\nPostprocess unzip\n\n```\n$ dim install https://example.com -n \"example\" -p unzip\n```\n\nPostprocess encoding\n\n```\n$ dim install https://example.com -n \"example\" -p \"encode utf-8\"\n```\n\nPostprocess xlsx-to-csv\n\n```\n$ dim install https://example.com -n \"example\" -p xlsx-to-csv\n```\n\nPostprocess csv-to-json\n\n```\n$ dim install https://example.com -n \"example\" -p csv-to-json\n```\n\nPostprocess custom command\n\nYou can specify a custom command after **\"cmd\"**.\n\n```\n$ dim install https://example.com -n \"example\" -p \"cmd ******\"\n```\n\nThe file path will be passed as an argument at the end of the specified command.\n\n```\n$ dim install https://example.com -n \"example\" -p \"cmd python ./tests/test_custom_command.py\"\n```\n\nCommand to be executed during postprocessing.\n\n```\n$ python ./tests/test_custom_command.py ./data_files/***/***.xx\n```\n\n#### Install by specifying the HTML page\n\nInstall data by specifying regular expressions for links within a specified page.\n\n```\n$ dim install -P https://example.com -e \".pdf\" -n \"example\"\n```\n\n#### Forced execution\n\nForced install. Overwrite already exist data file.\n\n```\n$ dim install https://example.com -n \"example\" -F\n```\n\n## [Uninstall](doc/en/uninstall.md)\n\nDelete data with the specified name from `dim.json`, `dim-lock.json` and `data_files/`.\n\n```\n$ dim uninstall [name]\n```\n\n## [List](doc/en/list.md)\n\nDisplay the information described in `dim-lock.json`.\n\n```\n$ dim list\n```\n\n### Simple List\n\n```\n$ dim list -s\n```\n\n## [Verify](doc/en/verify.md)\n\nVerify the data\n\n```\n$ dim verify\n```\n\n## [Update](doc/en/update.md)\n\nUpdate the all data.\n\n```\n$ dim update\n```\n\nUpdate the specified data.\n\n```\n$ dim update [name]\n```\n\n## Clean\n\nClean the project. Delete the data_files and init the project.\n\n```\n$ dim clean\n```\n\n## [Search](doc/en/search.md)\n\nSearch data from package_search CKAN API.\n\nUse [データカタログ横断検索システム](https://search.ckan.jp/) by default to do the search.\n\n```\n$ dim search 避難所\n```\n\nSpecify the number of data to get by option -n (default 10).\n\n```\n$ dim search 避難所 -n 3\n```\n\n### Interactive installation\n\nWrite data information to `dim.json` from ckan.\n\nStore the data to `data_files`.\n\n```\n$ dim search -i \"東京 避難所\"\n\n131105_東京都_目黒区_大地震時における地域避難所\n  - Catalog URL        : https://www.geospatial.jp/ckan/dataset/131105-002\n  - Catalog Description: ####大地震時における地域避難所のデータです。\n####東京都目黒区のオープンデータです。【リソース】大地震時における地域避難所 / ####大地震時における地域避難所のXLSXです。\n【キーワード】東京都 / 目黒区 / 避難所\n  - Catalog License    : クリエイティブ・コモンズ 表示\n    1. 大地震時における地域避難所\n      * Resource URL        : https://www.geospatial.jp/ckan/dataset/1e07b569-80a5-4c31-8a7b-be88d1e8f327/resource/8d8de117-2342-4c61-a98d-8f7a9c5b71a2/download/131105evacuationspace.xlsx\n      * Resource Description: ####大地震時における地域避難所のXLSXです。\n      * Created             : 2018-10-30T02:55:40.179726\n      * Format              : XLSX\n\n131059_東京都_文京区_緊急避難場所・避難所\n  - Catalog URL        : https://www.geospatial.jp/ckan/dataset/131059-025\n  - Catalog Description: ####緊急避難場所・避難所のデータです。\n####東京都文京区のオープンデータです。【リソース】緊急避難場所・避難所 / ####文京区の避難所・緊急避難場所の一覧データのCSVです。####更新日:2018年10月23日 / ####文京区の避難所・緊急避難場所の一覧データのXLSXです。\n####更新日:2018年10月23日【キーワード】文京区 / 東京都 / 避難場所 / 避難所\n  - Catalog License    : CC-BY2.1\n    2. 緊急避難場所・避難所\n      * Resource URL        : https://www.geospatial.jp/ckan/dataset/b17c1f51-ce1c-4e6a-8ff9-5ff0203b1e43/resource/008d34ad-61a5-4dbd-8996-fa6d647c2986/download/kinkyuhinanbasyo-hinanjo.csv\n      * Resource Description: ####文京区の避難所・緊急避難場所の一覧データのCSVです。\n####更新日:2018年10月23日\n      * Created             : 2018-10-30T05:44:44.623645\n      * Format              : CSV\n    3. 緊急避難場所・避難所\n      * Resource URL        : https://www.geospatial.jp/ckan/dataset/b17c1f51-ce1c-4e6a-8ff9-5ff0203b1e43/resource/0c4942d4-a149-4091-a52f-69b7da8fa143/download/kinkyuhinanbasyo-hinanjo.xlsx\n      * Resource Description: ####文京区の避難所・緊急避難場所の一覧データのXLSXです。\n####更新日:2018年10月23日\n      * Created             : 2018-10-30T05:44:46.127915\n      * Format              : XLSX\n...\n? Enter the number of data to install \u003e 1\n? Enter the name. Enter blank if want to use CKAN resource name. \u003e \n? Enter the post-processing you wish to add. Enter blank if not required. \u003e xlsx-to-csv\n? Is there a post-processing you would like to add next? (Y/n) \u003e No\nConvert xlsx to csv.\nInstalled to ./data_files/131105_東京都_目黒区_大地震時における地域避難所_大地震時における地域避難所/131105evacuationspace.xlsx\n```\n\n## [Generate](doc/en/generate.md)\n\nAuto-generate code about target data using **GPT-3**. For example, conversion processing, visualization processing,\netc..\n\u003cimg src=\"https://user-images.githubusercontent.com/6661165/216978343-0c1cbedb-e709-4501-8068-deca12e805b0.gif\" width=800 /\u003e\n\nExport APIKey of OpenAI to `OPENAI_API_KEY`.\n\n```\n$ export OPENAI_API_KEY=xxxxxxxxxxxxxxxxxxxxx\n```\n\nYou can get APIKey URL: https://platform.openai.com/account/api-keys\n\nCombine the specified target data and prompt, send it to GPT-3 API, output the code, and save it.\n\n```\n$ dim generate -t \"./data.csv\" \"Python code that converts this csv data to geojson\"\n```\n\nSpecify the data name managed by dim using `-t`\n\n```\n$ dim generate -t \"shelter\" \"Python code that converts this csv data to geojson\"\n```\n\n### Example prompt List\n\n```\nPython code that converts this csv data to geojson\n```\n\n```\nPython code that remove id column from this csv data\n```\n\n```\nPython code that visualizes this csv data as a map\n```\n\n```\nPython code that visualizes this csv data as a map\n```\n\n```\nPython code that visualizes this csv data as HTML page\n```\n\n```\nPython code that saves this csv data to PostgreSQL\n```\n\n```\nPython code that converts full-width numbers in this csv file to half-width numbers\n```\n\n## [Help](doc/en/help.md)\n\n```\n$ dim help\n```\n\n# Run the dim using Deno\n\n1. Install Deno\n\n- Deno == 1.28.2\n\n```\n$ curl -fsSL https://deno.land/install.sh | sh\n$ echo 'export DENO_INSTALL=~/.deno' \u003e\u003e ~/.bashrc\n$ echo 'export PATH=\"$DENO_INSTALL/bin:$PATH\"' \u003e\u003e ~/.bashrc\n$ source ~/.bashrc\n```\n\n2. Clone the repository\n\n```\n$ git clone https://github.com/c-3lab/dim.git\n```\n\n```\n$ cd dim\n```\n\n3. Run the dim commands\n\n```\n$ deno run -A dim.ts init\n$ deno run -A dim.ts install https://xxxxxx/data.json -n 'data_name'\n```\n\n4. Install dim\n\n```\n$ deno install --unstable --allow-read --allow-write --allow-run --allow-net --allow-env dim.ts\n```\n\n# Run test and display coverage\n\n1. Run test\n\n```\n$ deno test -A --coverage=tests/coverage\n```\n\n2. Display coverage\n\n```\n$ deno coverage ./tests/coverage\n```\n\n# Upgrade the dim version\n\nYou need to be able to run Deno in a local environment.\\\nIf you don't have an execution environment of Deno, please re-install.\\\n[Install the dim from binary files](#install-the-dim-from-binary-files)\n\n```\n$ dim upgrade\n```\n\n# Contributers\n\n\u003ca href=\"https://github.com/c-3lab/dim/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contributors-img.web.app/image?repo=c-3lab/dim\" /\u003e\n\u003c/a\u003e\n\nMade with [contributors-img](https://contributors-img.web.app).\n\n# LICENSE\n\n[MIT LICENSE](./LICENSE)\n","funding_links":[],"categories":["cli"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-3lab%2Fdim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc-3lab%2Fdim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-3lab%2Fdim/lists"}