https://github.com/txn2/dmk
Declarative data migration.
https://github.com/txn2/dmk
Last synced: 8 months ago
JSON representation
Declarative data migration.
- Host: GitHub
- URL: https://github.com/txn2/dmk
- Owner: txn2
- License: mit
- Created: 2018-04-27T06:26:59.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2018-10-31T19:21:27.000Z (over 7 years ago)
- Last Synced: 2025-06-29T07:41:47.620Z (12 months ago)
- Language: Go
- Homepage:
- Size: 11.7 MB
- Stars: 5
- Watchers: 4
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/txn2/dmk/releases)
[](https://travis-ci.org/txn2/dmk)
[](https://goreportcard.com/report/github.com/txn2/dmk)
# DMK - Data Migration Kit
## Work in Progress
This project is under development.
-------
## Alpha Preview
On MacOS with [brew]() installed.
**Run:**
```bash
brew install txn2/tap/dmk
```
**Upgrade:**
```bash
brew upgrade dmk
```
## Testing
Use `docker-compose` to bring up Cassandra and MySql test databases.
```bash
$ docker-compose up
```
Create example Keyspace and table in Cassandra:
```bash
$ docker run -it --rm -v $(pwd)/dev/cassandra.cql:/setup.cql --net host \
cassandra cqlsh localhost 39042 -f /setup.cql
```
Use the official Cassandra image to open a `cqlsh` session to
the local Cassandra running from the `docker-compose` above.
```bash
$ docker run -it --rm --net host cassandra cqlsh localhost 39042
```
Run the following example migrations in order:
```bash
go run ./dmk.go -d examples -p example run -v example_csv_to_cassandra
# check: select * from example.migration_data;
go run ./dmk.go -d examples -p example run -v cassandra_to_cassandra_by_name example
# check: select * from example.migration_data_name;
go run ./dmk.go -d examples -p example run -v cassandra_to_cassandra_name_lookup example
# check: select * from example.migration_name;
go run ./dmk.go -d examples -p example run -v cassandra_to_cassandra_using_collector example
# check: select * from example.migration_sets;
```
## Todo
- Reuse DB connection for script run sub-migrations.
- Better error messaging (location of error)
- General Performance improvements.
## Development
Run `go run ./dmk.go -d ./examples/`
## Vendor Package Management
see: https://github.com/Masterminds/glide
## Docker Build
```bash
docker build --build-arg VERSION=1.1.0 -t txn2/dmk:1.1.0-2 -f ./dockerfiles/cmd/Dockerfile .
```
Container with no entry point:
```bash
docker build --build-arg VERSION=1.1.0-noep -t txn2/dmk:1.1.0-2-noep -f ./dockerfiles/noep/Dockerfile .
```
## Release Development
Uses [goreleaser](https://goreleaser.com):
Install goreleaser with brew (mac):
```
brew install goreleaser/tap/goreleaser`
Build without releasing:
`goreleaser --skip-publish --rm-dist --skip-validate`
Release:
```bash
GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist
```
#### Containers
Containers for testing MySql and Cassandra databases.