{"id":22919637,"url":"https://github.com/couchbase-ecosystem/hf-to-cb-dataset-migrator","last_synced_at":"2025-07-12T07:02:25.572Z","repository":{"id":263725067,"uuid":"865784059","full_name":"Couchbase-Ecosystem/hf-to-cb-dataset-migrator","owner":"Couchbase-Ecosystem","description":"Hugging face to Couchbase dataset migrator","archived":false,"fork":false,"pushed_at":"2025-04-11T11:07:29.000Z","size":13657,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T12:53:08.621Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Couchbase-Ecosystem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-01T06:05:48.000Z","updated_at":"2025-04-11T11:07:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"133d28aa-9156-4c54-bbe6-7effa54de32d","html_url":"https://github.com/Couchbase-Ecosystem/hf-to-cb-dataset-migrator","commit_stats":null,"previous_names":["couchbase-ecosystem/hf-to-cb-dataset-migrator"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Couchbase-Ecosystem/hf-to-cb-dataset-migrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fhf-to-cb-dataset-migrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fhf-to-cb-dataset-migrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fhf-to-cb-dataset-migrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fhf-to-cb-dataset-migrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Couchbase-Ecosystem","download_url":"https://codeload.github.com/Couchbase-Ecosystem/hf-to-cb-dataset-migrator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fhf-to-cb-dataset-migrator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264951976,"owners_count":23687993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T07:12:38.955Z","updated_at":"2025-07-12T07:02:25.562Z","avatar_url":"https://github.com/Couchbase-Ecosystem.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hugging Face Dataset to Couchbase Migrator CLI\n\nA command-line tool to interact with Hugging Face datasets and migrate them to Couchbase, with support for streaming data.\n\n## Installation\n\n```bash\npip install -r requirements.txt\npython setup.py install\n```\n\n## Commands\n\nThe CLI provides the following commands:\n\n### 1. List Configurations\n\nLists all available configurations for a dataset.\n\n```bash\nhf_to_cb_dataset_migrator list-configs --path dataset\n```\n\nFlags:\n\n- `--path`: Path or name of the dataset (required)\n- `--revision`: Version of the dataset script to load\n- `--download-config`: Specific download configuration parameters\n- `--download-mode`: Download mode (reuse_dataset_if_exists or force_redownload)\n- `--dynamic-modules-path`: Path to dynamic modules\n- `--data-files`: Path(s) to source data file(s)\n- `--token`: Authentication token for private datasets\n- `--json-output`: Output the configurations in JSON format\n- `--debug`: Enable debug output\n- `--trust-remote-code`: Allow loading arbitrary code from the dataset repository\n\n### 2. List Splits\n\nLists all available splits for a dataset.\n\n```bash\nhf_to_cb_dataset_migrator list-splits --path dataset\n```\n\nFlags:\n\n- `--path`: Path or name of the dataset (required)\n- `--name`: Configuration name of the dataset\n- `--data-files`: Path(s) to source data file(s)\n- `--download-config`: Specific download configuration parameters\n- `--download-mode`: Download mode (reuse_dataset_if_exists or force_redownload)\n- `--revision`: Version of the dataset script to load\n- `--token`: Authentication token for private datasets\n- `--json-output`: Output the splits in JSON format\n- `--debug`: Enable debug output\n- `--trust-remote-code`: Allow loading arbitrary code from the dataset repository\n\n### 3. List Fields\n\nLists all fields (columns) in a dataset.\n\n```bash\nhf_to_cb_dataset_migrator list-fields --path dataset\n```\n\nFlags:\n\n- `--path`: Path or name of the dataset (required)\n- `--name`: Name of the dataset configuration\n- `--data-files`: Paths to source data files\n- `--download-config`: Specific download configuration parameters\n- `--revision`: Version of the dataset script to load\n- `--token`: Hugging Face token for private datasets\n- `--split`: Which split of the data to load\n- `--json-output`: Output the fields in JSON format\n- `--debug`: Enable debug output\n- `--trust-remote-code`: Allow loading arbitrary code from the dataset repository\n\n### 4. Migrate Dataset\n\nMigrates data from Hugging Face to Couchbase.\n\n```bash\nhf_to_cb_dataset_migrator migrate \\\n    --path dataset \\\n    --id-fields id_field \\\n    --cb-url couchbase://localhost \\\n    --cb-username user \\\n    --cb-password pass \\\n    --cb-bucket my_bucket \\\n    --cb-scope my_scope \\\n    --cb-collection my_collection\n```\n\nFlags:\n\n- `--path`: Path or name of the dataset (required)\n- `--id-fields`: Comma-separated list of field names to use as document ID (required)\n- `--cb-url`: Couchbase cluster URL (required)\n- `--cb-username`: Couchbase username (required)\n- `--cb-password`: Couchbase password (required)\n- `--cb-bucket`: Couchbase bucket name (required)\n- `--cb-scope`: Couchbase scope name (required)\n- `--cb-collection`: Couchbase collection name\n- `--name`: Configuration name of the dataset\n- `--data-files`: Path(s) to source data file(s)\n- `--split`: Which split of the data to load\n- `--cache-dir`: Cache directory for datasets\n- `--download-config`: Specific download configuration parameters\n- `--download-mode`: Download mode (reuse_dataset_if_exists or force_redownload)\n- `--verification-mode`: Verification mode (no_checks, basic_checks, or all_checks)\n- `--keep-in-memory`: Keep dataset in memory\n- `--save-infos`: Save dataset information\n- `--revision`: Version of the dataset script to load\n- `--token`: Authentication token for private datasets\n- `--no-streaming`: Disable streaming mode\n- `--num-proc`: Number of processes to use\n- `--storage-options`: Storage options for remote filesystems\n- `--trust-remote-code`: Allow loading arbitrary code from the dataset repository\n- `--cb-batch-size`: Number of documents to insert per batch (default: 1000)\n- `--debug`: Enable debug output\n\n## Examples\n\n- List configurations for a public dataset:\n\n```bash\nhf_to_cb_dataset_migrator list-configs --path dataset\n```\n\n- List configurations for a private dataset:\n\n```bash\nhf_to_cb_dataset_migrator list-configs --path my-dataset --token YOUR_HF_TOKEN\n```\n\n- List splits for a dataset with specific configuration:\n\n```bash\nhf_to_cb_dataset_migrator list-splits --path dataset --name config-name\n```\n\n- List fields in JSON format:\n\n```bash\nhf_to_cb_dataset_migrator list-fields --path dataset --json-output\n```\n\n- List fields for a specific split:\n\n```bash\nhf_to_cb_dataset_migrator list-fields --path dataset --split train\n```\n\n- List fields with download configuration:\n\n```bash\nhf_to_cb_dataset_migrator list-fields \\\n    --path dataset \\\n    --download-config '{\"force_download\": true}' \\\n    --trust-remote-code\n```\n\n- Migrate a dataset with multiple ID fields:\n\n```bash\nhf_to_cb_dataset_migrator migrate \\\n    --path dataset \\\n    --id-fields field1,field2 \\\n    --cb-url couchbase://localhost \\\n    --cb-username user \\\n    --cb-password pass \\\n    --cb-bucket my_bucket \\\n    --cb-scope my_scope \\\n    --cb-collection my_collection\n```\n\n- Migrate a specific split with streaming enabled:\n\n```bash\nhf_to_cb_dataset_migrator migrate \\\n    --path dataset \\\n    --split train \\\n    --id-fields id_field \\\n    --cb-url couchbase://localhost \\\n    --cb-username user \\\n    --cb-password pass \\\n    --cb-bucket my_bucket \\\n    --cb-scope my_scope\n```\n\n## Error Handling\n\nThe CLI will exit with a non-zero status code if an error occurs during execution. Error messages will be displayed on stderr.\n\n## Logging\n\n- Use `--debug` flag with any command to enable debug-level logging\n- JSON output options are available for machine-readable output\n- Progress information is displayed during migration\n\n## Authentication\n\n- For private Hugging Face datasets, use the `--token` option\n- Couchbase credentials are required for migration operations\n- Credentials can be provided via command-line options\n\n---\n\n## 📢 Support Policy\n\nWe truly appreciate your interest in this project!  \nThis project is **community-maintained**, which means it's **not officially supported** by our support team.\n\nIf you need help, have found a bug, or want to contribute improvements, the best place to do that is right here — by [opening a GitHub issue](https://github.com/Couchbase-Ecosystem/hf-to-cb-dataset-migrator/issues).  \nOur support portal is unable to assist with requests related to this project, so we kindly ask that all inquiries stay within GitHub.\n\nYour collaboration helps us all move forward together — thank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcouchbase-ecosystem%2Fhf-to-cb-dataset-migrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcouchbase-ecosystem%2Fhf-to-cb-dataset-migrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcouchbase-ecosystem%2Fhf-to-cb-dataset-migrator/lists"}