https://github.com/ironcore864/mydumpergo
https://github.com/ironcore864/mydumpergo
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/ironcore864/mydumpergo
- Owner: IronCore864
- Created: 2021-03-22T14:29:49.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-03-22T14:30:32.000Z (over 5 years ago)
- Last Synced: 2025-09-04T07:29:08.898Z (11 months ago)
- Language: Go
- Size: 3.91 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Mydumper in Go
A wrapper for [mydumper](https://github.com/maxbube/mydumper) written in Golang with limited available storage in mind.
## Usage
Example:
```bash
go get ./...
go build
BUCKET=mybucket-123123123 ./mydumpergo
```
## Configuration
Using ENV variables. For more details, see file [config/config.go](config/config.go).
Explanation:
- AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY: used to access AWS.
- REGION: the S3 bucket region, defaults to "eu-central-1".
- BUCKET: the name of the S3 bucket for uploading. Mandatory.
- CHUNK_FILE_SIZE_MB: Split tables into chunks of this output file size. This value is in MB. Defaults to 1 for local testing purpose.
- MAX_FILE_COUNT: maximum files allowed before pausing and uploading to S3. Defaults to 10 for local testing purpose.
- HOST: mysql host, defaults to "localhost".
- PORT: mysql port, defaults to 3306.
- USERNAME: mysql username, defaults to "root" for local testing purpose.
- PASSWORD: mysql password, defaults to empty string "" for local testing purpose.
- OUTPUTDIR: defaults to "output", used as the `-o` parameter for mydumper.
**NOTE:**
`CHUNK_FILE_SIZE_MB` * `MAX_FILE_COUNT` decides maximum storage usage. For example, if `CHUNK_FILE_SIZE_MB=50` and `MAX_FILE_COUNT=20`, total max disk usage is 1GB. This is an MVP implementation, so counting file number is used (which is simpler) rather than calculating output dir size.
## Main Logic
- start the mydumper process
- start a goroutine that checks disk usage, which loops:
- if disk usage is higher than pre-configured threshold, send SIGTSTP to the mydumper process to pause it, then:
- for each existing file in the output folder, upload the file (using the s3manager package's Uploader, which provides concurrent upload by taking advantage of S3's Multipart API)
- after all existing files are uploaded, delete them to release disk usage
- send SIGCONT to the mydumper process to continue
- after the mydumper process ends, clean up the output folder.
## TODOs
To make it production-ready, there are a few improvements to be done:
- Change counting files to counting disk usage. Counting file is only meant as a simple implementation for local testing purpose where you don't have large tables and output files.
- Add robust error handling, like retry for S3 upload, only delete successfully uploaded files, etc.
- Add unit test and functional test.
- Running in K8s: docker [multistage build](https://docs.docker.com/develop/develop-images/multistage-build/), adding mydumper into the same image; run as a K8s [job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) wth persistent volume, using the mountPath as the output dir.