https://github.com/islishude/gotgz
Similar to tar + gzip, but supports S3
https://github.com/islishude/gotgz
go gzip lz4 s3 tar zstd
Last synced: 7 months ago
JSON representation
Similar to tar + gzip, but supports S3
- Host: GitHub
- URL: https://github.com/islishude/gotgz
- Owner: islishude
- Created: 2023-02-15T14:35:54.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-03-11T00:23:33.000Z (7 months ago)
- Last Synced: 2025-03-11T01:24:22.247Z (7 months ago)
- Topics: go, gzip, lz4, s3, tar, zstd
- Language: Go
- Homepage:
- Size: 70.3 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# gotgz
Similar to `tar + gzip`, but supports S3
**differences from tar**
1. gzip is enabled by default, but you can also use zstd or lz4
2. `-xvf` short form is not supported, you should use `-x -v -f`
3. Only regular files, directories and symbol links are supported
## Compress
```console
$ gotgz -c -f s3://test/testdata.tar.gz -e 'parent/.exclude/**' -relative -suffix date testdata
2025/01/30 19:19:36 INFO append target=testdata
2025/01/30 19:19:36 INFO append target=testdata/parent
2025/01/30 19:19:36 INFO append target=testdata/parent/README
2025/01/30 19:19:36 INFO append target=testdata/parent/README.md
2025/01/30 19:19:36 INFO append target=testdata/parent/css
2025/01/30 19:19:36 INFO append target=testdata/parent/css/index.css
2025/01/30 19:19:36 INFO append target=testdata/parent/favicon-32x32.png
2025/01/30 19:19:36 INFO append target=testdata/parent/index.html
2025/01/30 19:19:36 INFO append target=testdata/parent/index.json
2025/01/30 19:19:36 INFO append target=testdata/parent/javascript
2025/01/30 19:19:36 INFO append target=testdata/parent/js
2025/01/30 19:19:36 INFO append target=testdata/parent/js/index.js
2025/01/30 19:19:36 INFO Time cost: period=15.129041ms
$ aws s3 ls s3://test
2025-01-30 11:19:36 2332 testdata-20250130.tar.gz
```
`-c` is used to compress files.
`-f` is used to specify the target file, it supports local path and S3 path.
`-e` is used to exclude files or directories, it's a shell glob pattern.
`-relative` is used to keep the relative path in tar ball, if the source directory is `/data` and the file path is `/data/file.txt`, the relative path in tar ball is `file.txt`.
`-suffix` option is used to add a suffix to the file name, date is a built-in suffix.
the last argument is the source directory, it supports multiple directories.
You can use `s3://your-s3-bucket/path.tgz?key=value` to add metadata to the object.
The default compression method is gzip.
To use zstd or lz4, you need use `--algo` with options:
```
gotgz -c -algo 'zstd?level=1' -f s3://your-s3-bucket/path.tgz /data
gotgz -c -algo 'lz4?level=1' -f s3://your-s3-bucket/path.tgz /data
```
## Decompress
```console
$ gotgz -x -f s3://test/testdata.tar.gz -suffix=date -strip-components=1 tmp
2025/01/30 19:21:09 INFO extract target=.
2025/01/30 19:21:09 INFO extract target=parent
2025/01/30 19:21:09 INFO extract target=parent/README
2025/01/30 19:21:09 INFO extract target=parent/README.md
2025/01/30 19:21:09 INFO extract target=parent/css
2025/01/30 19:21:09 INFO extract target=parent/css/index.css
2025/01/30 19:21:09 INFO extract target=parent/favicon-32x32.png
2025/01/30 19:21:09 INFO extract target=parent/index.html
2025/01/30 19:21:09 INFO extract target=parent/index.json
2025/01/30 19:21:09 INFO extract target=parent/javascript
2025/01/30 19:21:09 INFO extract target=parent/js
2025/01/30 19:21:09 INFO extract target=parent/js/index.js
2025/01/30 19:21:09 INFO Time cost: period=4.628542ms
$ tree tmp
tmp
├── README -> README.md
├── README.md
├── css
│ └── index.css
├── favicon-32x32.png
├── index.html
├── index.json
├── javascript -> js
└── js
└── index.js
4 directories, 7 files
```
`-f` also supports a local path.
The `-strip-components=N` to remove the leading N directories from the file names.
By default, the gotgz will overwrite the existing files, you can use `-no-overwrite=true` to prevent it.
If you want to keep the file permission and user infomation, you can use `-no-same-permissions=false -no-same-owner=false`.
Don't forget to add `-algo` if the file is compressed by zstd or lz4.