{"id":22039317,"url":"https://github.com/mdornseif/datastore-backup","last_synced_at":"2026-05-08T13:48:05.418Z","repository":{"id":45314690,"uuid":"440244693","full_name":"mdornseif/datastore-backup","owner":"mdornseif","description":"Programatic Backup of Google Cloud Datastore","archived":false,"fork":false,"pushed_at":"2022-01-20T09:03:14.000Z","size":350,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-28T19:17:19.918Z","etag":null,"topics":["google-cloud","google-cloud-platform"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/datastore-backup","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mdornseif.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-20T16:56:37.000Z","updated_at":"2021-12-23T23:55:20.000Z","dependencies_parsed_at":"2022-09-21T10:13:35.888Z","dependency_job_id":null,"html_url":"https://github.com/mdornseif/datastore-backup","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdornseif%2Fdatastore-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdornseif%2Fdatastore-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdornseif%2Fdatastore-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdornseif%2Fdatastore-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mdornseif","download_url":"https://codeload.github.com/mdornseif/datastore-backup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245104528,"owners_count":20561380,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-cloud","google-cloud-platform"],"created_at":"2024-11-30T11:10:19.293Z","updated_at":"2026-05-08T13:48:00.398Z","avatar_url":"https://github.com/mdornseif.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://img.shields.io/travis/mdornseif/datastore-backup.svg?style=flat-square\u0026color=%23007a1f)](https://travis-ci.org/mdornseif/datastore-backup)\n[![version](https://img.shields.io/npm/v/datastore-backup.svg?style=flat-square)](https://npmjs.org/datastore-backup)\n[![package size](https://packagephobia.com/badge?p=datastore-backup)](https://packagephobia.com/result?p=datastore-backup)\n[![license](https://img.shields.io/npm/l/datastore-backup?color=%23007a1f\u0026style=flat-square)](https://github.com/mdornseif/datastore-backup/blob/master/LICENSE)\n[![dependencies](https://img.shields.io/librariesio/release/npm/datastore-backup?color=%23007a1f\u0026style=flat-square)](https://libraries.io/npm/datastore-backup)\n[![downloads](https://img.shields.io/npm/dm/datastore-backup?style=flat-square\u0026color=%23007a1f)](https://npmcharts.com/compare/datastore-backup)\n\n# datastore-backup\n\n[Exporting Data from Google Cloud Datastore](https://cloud.google.com/datastore/docs/export-import-entities#rest_) seems easy:\n\n[From the command line](https://cloud.google.com/sdk/gcloud/reference/datastore/export):\n\n```\ngcloud datastore export gs://exampleBucket/backupdir\n```\n\nOr in node.js:\n\n```js\nconst datastore = new Datastore();\ndatastore.export({ outputUrlPrefix: 'gs://exampleBucket/backupdir' });\n```\n\nSo why do you need a library for that? Because it is in the details!\n\nFor one thing the code above contains an export containing all kinds [can not be loaded into BigQuery](https://cloud.google.com/bigquery/docs/loading-data-cloud-datastore): Data exported without specifying an entity filter cannot be loaded into BigQuery.\"\n\nSo you need a list of all Kinds in the Datastore. But there are [Limits on entity filters](https://cloud.google.com/datastore/docs/export-import-entities#entity_filter): \"Each request is limited to 100 entity filter combinations, where each combination of filtered kind and namespace counts as a separate filter towards this limit.\" (It was 50 e few years before.)\n\nAnd there is [an other limit](https://cloud.google.com/datastore/docs/export-import-entities): \"The managed export and import service limits the number of concurrent exports and imports to 50 and allows a maximum of 20 export and import requests per minute for a project.\"\n\nSo if you have a lot of entities you have to do some slicing and dicing.\n\n## CLI Usage\n\nCommand line usage is simple. You have to provide the projectId of the datastore and a bucket name. Both have to be in the same region.\n\n```\n% yarn ts-node src/bin/backup.ts --help\nusage: backup.ts [-h] [-v] [-d BACKUPDIR] [-n BACKUPNAME] [-s NAMESPACE] projectId bucket\n\nBackup Datastore.\n\npositional arguments:\n  projectId             Datastore project ID\n  bucket                GCS bucket to store backup\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -d BACKUPDIR, --backupDir BACKUPDIR\n                        prefix/dir within bucket\n  -n BACKUPNAME, --backupName BACKUPNAME\n                        name of backup (default: auto-generated)\n  -s NAMESPACE, --namespace NAMESPACE\n                        datastore namespace\n\nPlease provide `GOOGLE_APPLICATION_CREDENTIALS` via the Environment!\n```\n\n```\n% export GOOGLE_APPLICATION_CREDENTIALS=~/sampleproj-b0a74af0545e.json\n% yarn ts-node src/bin/backup.ts sampleproj sampleproj-tmp\nℹ backup to gs://sampleproj-tmp/bak/20211221T212651-sampleproj\nℹ Dumping datastore sampleproj\n✔ 2 Kinds\n2 kinds  ℹ Dumping NumberingAncestor, NumberingItem\n2 kinds  ℹ Dumping to gs://sampleproj-tmp/bak/20211221T212651-sampleproj-0\n2 kinds  ✔ Dumping finished 27150 records (4.04 MB) in 51s\n2 kinds  ✔ written gs://sampleproj-tmp/bak/20211221T212651-sampleproj-0/20211221T212651-sampleproj-0.overall_export_metadata\n✨  Done in 54.16s.\n```\n\nThis generates a single folder structure under `gs://sampleproj-tmp/bak/20211221T212651-sampleproj-0`. For projects with more than 100 entities more folders will be generated.\n\nThere is one subfolder per kind containing the Entities encoded in [LevelDB Format](https://github.com/google/leveldb).\n\n## Programmatic Usage\n\n```\nimport { Datastore } from '@google-cloud/datastore';\nimport { dumpAllKinds } from '../lib/datastore-backup';\n\nawait dumpAllKinds(new Datastore({ projectId: 'sampleproj' }), 'sampleproj-tmp')\n```\n\n# See also\n\n- [datastore-to-bigquery](https://www.npmjs.com/package/datastore-to-bigquery) to load the data produced by this module into BigQuery.\n- [Google DatastoreImport/Export Documentation](https://cloud.google.com/datastore/docs/export-import-entities#rest)\n- [node.js low-level Datastore export API](https://googleapis.dev/nodejs/datastore/latest/google.datastore.admin.v1.DatastoreAdmin.html#exportEntities2)\n- [firestore-to-bigquery-export npm module](https://www.npmjs.com/package/@pokutuna/firestore-to-bigquery) for Cloud Functions\n- [firestore-to-bigquery-export npm module](https://www.npmjs.com/package/firestore-to-bigquery-export) an other one wth some documentation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdornseif%2Fdatastore-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmdornseif%2Fdatastore-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdornseif%2Fdatastore-backup/lists"}