{"id":18421033,"url":"https://github.com/thunderbug1/firebatch","last_synced_at":"2026-02-03T08:31:56.453Z","repository":{"id":228670585,"uuid":"774481957","full_name":"thunderbug1/firebatch","owner":"thunderbug1","description":"A cli tool for batch CRUD operations on firestore. Allows for easy import/export/update of data based on json files","archived":false,"fork":false,"pushed_at":"2024-04-15T19:12:56.000Z","size":83,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-13T02:06:27.398Z","etag":null,"topics":["cli","firebase-firestore","firestore"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thunderbug1.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-19T16:15:26.000Z","updated_at":"2024-06-04T14:09:29.000Z","dependencies_parsed_at":"2024-04-15T20:31:10.990Z","dependency_job_id":"2ecaf089-50bb-454d-ad31-ce760129dfe7","html_url":"https://github.com/thunderbug1/firebatch","commit_stats":null,"previous_names":["thunderbug1/firebatch"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/thunderbug1/firebatch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbug1%2Ffirebatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbug1%2Ffirebatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbug1%2Ffirebatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbug1%2Ffirebatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thunderbug1","download_url":"https://codeload.github.com/thunderbug1/firebatch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunderbug1%2Ffirebatch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259704393,"owners_count":22898857,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","firebase-firestore","firestore"],"created_at":"2024-11-06T04:24:01.642Z","updated_at":"2026-02-03T08:31:56.420Z","avatar_url":"https://github.com/thunderbug1.png","language":"Python","funding_links":["https://www.buymeacoffee.com/thinx"],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://www.buymeacoffee.com/thinx\" target=\"_blank\"\u003e\u003cimg src=\"https://cdn.buymeacoffee.com/buttons/default-orange.png\" alt=\"Buy Me A Coffee\" style=\"height: 41px !important;width: 174px !important;\" \u003e\u003c/a\u003e\r\n\r\n\u003e :coffee: **Open-source tools thrive on caffeine. If you like this project, please consider supporting it.**\r\n\r\n# Firebatch: Advanced Firestore CLI Tool\r\n\r\nFirebatch streamlines batch operations on Firestore, offering developers a robust CLI tool for managing Google Firestore databases. It simplifies CRUD operations, supports advanced data type conversions, and facilitates efficient data manipulation directly from your command line.\r\n\r\n\u003e :warning: **The usage of Firebatch counts towards your read and write quota, use cautiously with large databases.**\r\n\r\n## Key Features\r\n\r\n### Read/Download\r\nFetch documents with customizable query conditions, ordering, and limits. Supports raw mode and Firestore type conversions.\r\n\r\n### Write\r\nBatch upload documents with server timestamp support and automatic format detection.\r\n\r\n### Update\r\nPerform batch updates with upsert functionality and optional data validation.\r\n\r\n### Delete\r\nBulk delete documents, with support for recursive subcollection deletion.\r\n\r\n### Validation\r\nValidate documents against custom Pydantic models before writing or updating.\r\n\r\n### Verbose and Dry Run Modes\r\nEnable detailed operation logs and simulate write operations without database changes.\r\n\r\n### Flexible Input Formats\r\nSupports JSON, JSONL, and auto-detects input data formats.\r\n\r\n### Timestamp and Geopoint Conversion\r\nAutomatically converts Python datetime and geopoint data to Firestore's Timestamp and GeoPoint types.\r\n\r\n### Collection Group Queries\r\nQuery across all collections with the same name, regardless of their database location.\r\n\r\n### List Collections\r\nQuickly list all top-level collections in your Firestore database.\r\n\r\n---\r\n\r\n## Installation\r\n\r\n\u003e :warning: **Ensure you have Python 3.6 or newer installed before proceeding.**\r\n\r\nInstall Firebatch using pip:\r\n\r\n```sh\r\npip install firebatch\r\n# For additional validation support:\r\npip install firebatch[validation]\r\n```\r\n## Examples\r\n\r\n### Workflow 1: Geotagging User Posts\r\n\r\nSuppose you want to add location data to user posts that lack this information. You can use Firebatch to read, update, and validate geopoint data in bulk.\r\n\r\n1. **Export posts lacking geotags** to a JSONL file for review:\r\n   ```sh\r\n   firebatch read --collection posts --where \"location == null\" --format jsonl \u003e posts_without_geotags.jsonl\r\n   ```\r\n2. **Manually add geotags** to the posts in `posts_without_geotags.jsonl` using your preferred text editor or a script.\r\n3. **Validate and update posts** with geopoint conversion enabled to ensure data integrity:\r\n   ```sh\r\n   firebatch update --collection posts --geopoint-convert --validator my_validators:PostValidator --verbose updates_with_geotags.jsonl\r\n   ```\r\n\r\n### Workflow 2: Archiving Old Orders\r\n\r\nFor orders older than a year, you might want to move them to an archive collection to keep your active orders collection lean and performant.\r\n\r\n1. **Identify and export old orders** using timestamp conversion to detect dates properly:\r\n   ```sh\r\n   firebatch read --collection orders --where \"date \u003c 2023-01-01\" --timestamp-convert --format jsonl --verbose \u003e old_orders.jsonl\r\n   ```\r\n2. **Review the exported data** to ensure accuracy.\r\n3. **Import old orders into the archive** with automatic timestamp updates:\r\n   ```sh\r\n   firebatch write --collection archived_orders --timestamp-field archived_at --timestamp-convert --format jsonl --verbose \u003c old_orders.jsonl\r\n   ```\r\n4. **Delete the original old orders** after confirming the archive's integrity (use dry-run mode first for safety):\r\n   ```sh\r\n   firebatch delete --collection orders --verbose --dry-run old_orders.jsonl\r\n   ```\r\n   After verification, remove `--dry-run` to proceed with deletion.\r\n\r\n### Workflow 3: Consolidating User Feedback\r\n\r\nImagine you have feedback stored in multiple collections (e.g., `feedback_2023`, `feedback_2024`) and you want to consolidate all feedback into a single collection for easier analysis.\r\n\r\n1. **Perform collection group queries** to fetch all feedback documents:\r\n   ```sh\r\n   firebatch read --collection feedback --collection-group --format jsonl --verbose \u003e all_feedback.jsonl\r\n   ```\r\n2. **Optionally process the feedback data** to fit the new unified format.\r\n3. **Batch upload the consolidated feedback** to a new `unified_feedback` collection:\r\n   ```sh\r\n   firebatch write --collection unified_feedback --format jsonl --verbose all_feedback.jsonl\r\n   ```\r\n\r\n---\r\n\r\n## Contributing\r\n\r\n\u003e :heart: **Your contributions make Firebatch better.**\r\n\r\nReport bugs, suggest enhancements, or submit pull requests on our GitHub repository. Join our community to make Firestore more accessible and efficient for developers worldwide.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunderbug1%2Ffirebatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunderbug1%2Ffirebatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunderbug1%2Ffirebatch/lists"}