https://github.com/ihoro/tar-stream-chunker
Splits stdin onto chunk files and writes resulting TAR archive to stdout
https://github.com/ihoro/tar-stream-chunker
backup shell tar tarsnap unix-way utility
Last synced: 2 days ago
JSON representation
Splits stdin onto chunk files and writes resulting TAR archive to stdout
- Host: GitHub
- URL: https://github.com/ihoro/tar-stream-chunker
- Owner: ihoro
- License: bsd-2-clause
- Created: 2020-01-11T17:27:39.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-09-03T08:12:55.000Z (almost 6 years ago)
- Last Synced: 2025-09-11T17:57:52.785Z (10 months ago)
- Topics: backup, shell, tar, tarsnap, unix-way, utility
- Language: JavaScript
- Homepage:
- Size: 78.1 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.adoc
- License: LICENSE
Awesome Lists containing this project
README
= tar-stream-chunker
image:https://github.com/ihoro/tar-stream-chunker.c/workflows/linux/badge.svg[link="https://github.com/ihoro/tar-stream-chunker.c/actions?query=workflow%3Alinux"]
image:https://github.com/ihoro/tar-stream-chunker.c/workflows/macos/badge.svg[link="https://github.com/ihoro/tar-stream-chunker.c/actions?query=workflow%3Amacos"]
image:https://img.shields.io/lgtm/alerts/g/ihoro/tar-stream-chunker.c.svg?logo=lgtm&logoWidth=18[link="https://lgtm.com/projects/g/ihoro/tar-stream-chunker.c/alerts/"]
image:https://img.shields.io/lgtm/grade/cpp/g/ihoro/tar-stream-chunker.c.svg?logo=lgtm&logoWidth=18[link="https://lgtm.com/projects/g/ihoro/tar-stream-chunker.c/context:cpp"]
image:https://img.shields.io/lgtm/grade/javascript/g/ihoro/tar-stream-chunker.c.svg?logo=lgtm&logoWidth=18[link="https://lgtm.com/projects/g/ihoro/tar-stream-chunker.c/context:javascript"]
image:https://img.shields.io/badge/FreeBSD-port-informational?logo=freebsd[link="https://www.freshports.org/archivers/tar-stream-chunker"]
Splits stdin onto chunk files and writes resulting TAR archive to stdout.
It can help to handle data stream of unknown size without hitting a disk. As long as creating TAR archive as a stream needs to know file size in advance, that is why the idea of the chunker is to split input stream onto chunks with known size.
Initial motivation was to use it for backup streaming to tarsnap.com.
== Build from sources
Expecting you have a POSIX compliant system:
`$ make`
== Test suite
End-to-end test suite requires node.js:
`$ make e2e`
== FreeBSD port/package
- `pkg install tar-stream-chunker`
- or `cd /usr/ports/archivers/tar-stream-chunker && make install clean`
== Usage
`$ tar_stream_chunker { --file-name | -f } { --chunk-size | -s } `
And redirect its stdout to your target and that's it.
== Usage examples
PostgreSQL dump what is streamed to tarsnap:
`$ pg_dump | tar_stream_chunker -f dump.sql -s 500000000 | tarsnap -cvf dump.daily.$(date +%Y%m%d.%H%M%S) @-`
Let's see what's inside a resulting TAR:
```
$ cat 3MB-file | tar_stream_chunker --file-name file --chunk-size 1000000 > result.tar
$ tar -tf result.tar
file.00001
file.00002
file.00003
```
And original input can be re-assembled as follows:
```
$ tar -xf result.tar; cat file.* > original-3MB-file
$ tar -xOf result.tar > original-3MB-file
```