Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/copybird/copybird
Cloud-native universal backup tool for databases and files
https://github.com/copybird/copybird
backup clickhouse cloudnative database devops elasticsearch etcd kubernetes kubernetes-operator mariadb mongodb mysql postgresql
Last synced: about 1 month ago
JSON representation
Cloud-native universal backup tool for databases and files
- Host: GitHub
- URL: https://github.com/copybird/copybird
- Owner: copybird
- License: apache-2.0
- Created: 2019-06-20T12:07:25.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2020-02-20T06:48:09.000Z (almost 5 years ago)
- Last Synced: 2024-08-03T18:20:51.725Z (4 months ago)
- Topics: backup, clickhouse, cloudnative, database, devops, elasticsearch, etcd, kubernetes, kubernetes-operator, mariadb, mongodb, mysql, postgresql
- Language: Go
- Homepage: https://copybird.org
- Size: 2.68 MB
- Stars: 65
- Watchers: 10
- Forks: 6
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-clickhouse - copybird/copybird - Copybird is a cloud-native universal backup tool for databases and files. (Integrations / Data Transfer and Synchronization)
README
# Copybird
[](https://maddevs.io/)
[](https://www.repostatus.org/#wip)
[](https://microbadger.com/images/copybird/copybird)
[](https://microbadger.com/images/copybird/copybird)
[](http://godoc.org/github.com/copybird/copybird)
[](https://github.com/copybird/copybird/releases)
[](https://coveralls.io/github/copybird/copybird)
[](https://goreportcard.com/report/github.com/copybird/copybird)
[](https://opensource.org/licenses/Apache-2.0)
## About
Copybird is open-source **cloud-native** universal backup tool for databases and files.
It allows you to:
1. Create database backup
2. Compress backup stream
3. Encrypt backup stream
4. Send it to various destinations fast and secure
5. Get notification about backup status in messagers and notification services
6. Enjoy simple backup as a service with k8s backup controller
Backup process not using local storage for temp files.
learn more at [copybird.org](https://copybird.org). Note that this repository is in Work in Progress status. Feel free to contribure. Read more about contributing below.
## Databases
Currently Copybird supports the following databases:
- MySQL
- Postgres
- MongoDB
- Etcd (v2 and v3 API)
## Compression
Copybird compresses with the following tools:
- gzip
- lz4
## Encryption
Copybird uses AES-GCM for Efficient Authenticated Encryption
## Output
Copybird can deliver encrypted compressed backup to the following destinations:
- store the file locally
- save it on [GCP](https://cloud.google.com/)
- save it on [S3](https://aws.amazon.com/s3/)
- send over HTTP
- send over SCP
## Notification services
Copybird currently supports the following notification services:
- Slack
- Telegram
- AWS SES
- AWS SQS
- get notificatoin on email
- Kafka
- Nats
- Create issue in PagerDuty
- Pushbullet
- RabbitMQ
- Twilio
- Webcallback
If you would like to add additional service, please submit an issue with feature request or add it yourself and send a Pull Request.
## How to Run the tool
There are different ways you can use this tool:
### Run locally
First get the source code on your machine
```
go get -u github.com/copybird/copybird
```
Then run it with `go run main.go` to see helpers for various optional parameters
Example creating MySQL dump:
```
go run -v main.go backup -i 'mysql::dsn=root:root@tcp(localhost:3306)/test' -o local::file=dump.sql
```
### Run with Docker
Run `docker run copybird/copybird` to see the available optional parameters
### Use Backup Custom Controller/Operator for k8s
First create custom resource definition in your cluster:
```
kubectl apply -f operator/crd/crd.yaml
```
To run the controller:
```
go run main.go operator
```
And then in a separate shell, create custom resource:
```
kubectl create -f operator/example/backup-example.yaml
```
As output you get the following logs when creating, updating or deleting custom resource:
```
INFO[0000] Successfully constructed k8s client
INFO[0000] Starting Foo controller
INFO[0000] Waiting for informer caches to sync
INFO[0001] Starting workers
INFO[0001] Started workers
```
You can modify example file as you wish to get proper configuration for your jobs
## Tests
To run tests against MySQL module proceed with the following commands:
```
docker run --name test_mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=test -d percona:latest
docker exec -i test_mysql mysql -uroot -proot test < samples/mysql.sql
cd modules/backup/input/mysql/
go test -v -cover
```
To clean up after you finish with tests:
```
docker kill test_mysql
docker rm test_mysql
```
To run tests against MySQLDump module, first make sure that you have `mysqldump` binary
available in `$PATH` and then proceed with the following commands:
```
docker run --name test_mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=test -d percona:latest
docker exec -i test_mysql mysql -hlocalhost -uroot -proot test < samples/mysql.sql
cd modules/backup/input/mysqldump/
go test -v -cover
```
To clean up after you finish with tests:
```
docker kill test_mysql
docker rm test_mysql
```
To run tests against Postgres module proceed with the following commands:
```
docker run --name test_postgres -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=test -d postgres:latest
docker exec -i test_postgres psql -U postgres test < samples/postgres.sql
cd modules/backup/input/postgresql/
go test -v -cover
```
To clean up after you finish with tests:
```
docker kill test_postgres
docker rm test_postgres
```
## Contributing
Pull requests are more than welcomed. For major changes, please open an issue first to discuss what you would like to change.
Before submission of pull request make sure you pulled recent updates, included tests for your code that covers at least the core functionality and you submitted a desciptive issue that will be fixed with your pull request. Do not forget to mention the issue in the pull request.
Project started by Artem Andreenko, Andrew Minkin, and Mad Devs.
Built with Mad Devs support for the community
P.S. Love distributed and mesh? Meshbird.