{"id":23722666,"url":"https://github.com/remko/dsbackups","last_synced_at":"2025-06-29T22:35:33.933Z","repository":{"id":57620102,"uuid":"390748943","full_name":"remko/dsbackups","owner":"remko","description":"Package to work with Google Cloud Datastore exports","archived":false,"fork":false,"pushed_at":"2021-07-30T09:00:25.000Z","size":67,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-30T23:55:06.628Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/remko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-29T14:21:51.000Z","updated_at":"2024-09-08T16:34:07.000Z","dependencies_parsed_at":"2022-09-16T19:30:20.377Z","dependency_job_id":null,"html_url":"https://github.com/remko/dsbackups","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remko%2Fdsbackups","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remko%2Fdsbackups/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remko%2Fdsbackups/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remko%2Fdsbackups/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/remko","download_url":"https://codeload.github.com/remko/dsbackups/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239818071,"owners_count":19702064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-30T23:55:22.166Z","updated_at":"2025-02-20T10:24:20.128Z","avatar_url":"https://github.com/remko.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `dsbackups` [![Go Reference](https://pkg.go.dev/badge/github.com/remko/dsbackups.svg)](https://pkg.go.dev/github.com/remko/dsbackups)\n\nThis package contains code to work with [Google Cloud Datastore exports](https://cloud.google.com/datastore/docs/export-import-entities).\n(e.g. to restore a subset of entities from a backup, which is not possible using 'official' interfaces)\n\n**DISCLAIMERS**: \n- The format of Datastore exports is undocumented. This package is based on reverse engineering and the little\ninformation known about the datastore export format. None of this is officially supported by Google.\n  Since all this is unofficial and undocumented, this may break at some point (although I doubt it will any time soon, as it hasn't for years).\n- This package exists mostly as proof-of-concept and documentation, based on my own usage of datastore. \n  Don't assume this code works and doesn't delete all your data.  \n  The safest method of using this in production is probably to use this code to import data into a fresh local datastore emulator, verify the imported data\n  (e.g. using [DSAdmin](https://github.com/remko/dsadmin)), [export the data from your emulator](https://cloud.google.com/datastore/docs/tools/emulator-export-import),\n  and import your export [using the official methods](https://cloud.google.com/datastore/docs/export-import-entities).\n\n\n## Documentation\n\n- [Go reference](https://pkg.go.dev/github.com/remko/dsbackups).\n- See the commands in `cmd/dsbackuputil/dsbackuputil.go` for examples using this package.\n- [Google Cloud Datastore Export Format description](https://github.com/remko/dsbackups/blob/master/doc/DatastoreExportFormat.md)\n\n## `dsbackuputil`\n\nThe `dsbackuputil` command contains utility commands to inspect or import backups.\nYou can use it as is for quick backup inspections, or as a basis for more advanced commands.\n\nSee `dsbackuputil help` for a full list of commands.\n\n\nExamples: \n\n- List all entities in an export (on GCS):\n\n      export GOOGLE_APPLICATION_CREDENTIALS=credentials.json \n      dsbackuputil list \\\n        gs://my-bucket/mybackup/mybackup.overall_export_metadata\n\n- List all sub-exports in an export (on GCS):\n\n      export GOOGLE_APPLICATION_CREDENTIALS=credentials.json \n      dsbackuputil describe-exports \\\n        gs://my-bucket/mybackup/mybackup.overall_export_metadata\n\n  Output:\n\n      MyKind: default_namespace/kind_MyKind/default_namespace_kind_MyKind.export_metadata (3308 items, 808448b)\n      MyOtherKind: default_namespace/kind_MyOtherKind/default_namespace_kind_MyOtherKind.export_metadata (360 items, 5944254b)\n\n- List all output files of a single sub-export in an export:\n\n      export GOOGLE_APPLICATION_CREDENTIALS=credentials.json \n      dsbackuputil describe-export \\\n        gs://my-bucket/mybackup/default_namespace/kind_MyKind/default_namespace_kind_MyKind.export_metadata \n\n  Output: \n  \n      Start: 2021-07-29 15:00:03.000612651 +0200 CEST\n      End: 2021-07-29 15:02:55.000135493 +0200 CEST\n      Outputs:\n        output-0\n        output-1\n        output-2\n        output-3\n\n- Import a kind from an export on GCS into a local datastore emulator:\n\n      $ export GOOGLE_APPLICATION_CREDENTIALS=credentials.json \n      $ dsbackuputil import --project=my-project --kind=MyKind \\\n        gs://my-bucket/mybackup/mybackup.overall_export_metadata\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fremko%2Fdsbackups","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fremko%2Fdsbackups","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fremko%2Fdsbackups/lists"}