https://github.com/rustyx/dsutil
Google Cloud DataStore utilities: import, export, delete
https://github.com/rustyx/dsutil
datastore utilities
Last synced: 8 months ago
JSON representation
Google Cloud DataStore utilities: import, export, delete
- Host: GitHub
- URL: https://github.com/rustyx/dsutil
- Owner: rustyx
- License: mit
- Created: 2020-04-25T16:34:41.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2025-09-09T12:22:24.000Z (9 months ago)
- Last Synced: 2025-09-09T15:38:37.692Z (9 months ago)
- Topics: datastore, utilities
- Language: Go
- Size: 5.43 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## dsutil - Google Cloud DataStore import/export utilities
[](https://app.travis-ci.com/rustyx/dsutil)
### Installing
```
go get github.com/rustyx/dsutil
```
### Command Line Usage
```
dsutil [options] command
command:
export - export records from DataStore
import ... - import records into DataStore
delete - delete records from DataStore
set - update records in DataStore (type is: string, int, double)
convert - convert exported records from JSON to Go object notation
Note: ending with ".gz" will be automatically g(un)zipped
-project string
Google Cloud project name (deduced if not provided)
-kind string
DataStore table name (required for export)
-filter string
Filter field name (optional)
-from string
Filter >= value (optional)
-to string
Filter < value (optional)
-eq string
Filter = value (optional)
```
### API Usage
It is possible to read an export file and process each entity programmatically.
There are two interfaces: [`ImportFile`](https://pkg.go.dev/github.com/rustyx/dsutil/dsio#ImportFile), based on key-value pairs, and [`ImportFileReflect`](https://pkg.go.dev/github.com/rustyx/dsutil/dsio#ImportFileReflect), which is useful for ORM. Here's an example of how to load an export file into a PostgreSQL database using `go-pg` ORM API:
```
type MyEntity struct {
Id int `datastore:"-"`
SomeColumn string `pg:"type:varchar(40)"`
SomeColumn2 string `pg:"type:varchar(40)"`
// . . .
}
inputFile := "my-export.ds"
log.Printf("Importing %v", inputFile)
insertFunc := func(kind string, rows []interface{}) error {
log.Printf("Inserting %v %s(s)", len(rows), kind)
_, err := pgdb.Model(rows...).Insert()
return err
}
modelMap := []dsio.ModelMapping{
{Kind: "MyEntity", TypePtr: &MyEntity{}, ImportFunc: insertFunc, BatchSize: 200},
}
if err := dsio.ImportFileReflect(inputFile, modelMap); err != nil {
log.Fatalf("import %v failed: %v", inputFile, err)
}
```
For more documentation refer to [API Docs](https://pkg.go.dev/github.com/rustyx/dsutil/dsio).