https://github.com/aurora-is-near/tarserv

Tarserv serves streaming tar files from filesystem snapshots.
https://github.com/aurora-is-near/tarserv

Last synced: 10 days ago
JSON representation

Tarserv serves streaming tar files from filesystem snapshots.

• Host: GitHub
• URL: https://github.com/aurora-is-near/tarserv
• Owner: aurora-is-near
• License: cc0-1.0
• Created: 2021-11-29T18:15:53.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2021-11-30T03:34:44.000Z (about 3 years ago)
• Last Synced: 2024-06-21T18:01:00.250Z (7 months ago)
• Language: Go
• Size: 24.4 KB
• Stars: 1
• Watchers: 5
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
tarserv

=======

A collection of tools that allow serving large datasets from local filesystem snapshots.

It is meant for serving big amounts of data to shell scripted receivers.

Process:

 

  1. Take snapshot of directory containing data "/var/datadir/.snapshot12345".

  2. Create build index of the snapshot and store it into index directory:\

     `$ createindex /var/index/snapshot12345.taridx /var/datadir/.snapshot12345`

  3. Serve snapshots via HTTP: `$ tarserv -l 127.0.0.1:8080 -i /var/index/`

  4. Download snapshots via HTTP: `$ curl http://127.0.0.1:8080/snapshot12345/data.tar`

Features:

  - The tar files are created on the fly from the index file and filesystem snapshot/directory.

  - Downloading is resilient, it allows range requests and so can be efficiently restarted.

  - Furthermore the tar files produced contain a file ".version" that contains the source snapshot name.

  - If given the "lastfile" GET parameter, the served snapshot will start with the file named by the parameter.

  - All contained paths are normalized to current directory "./".