https://github.com/anistark/azazel

File transporter
https://github.com/anistark/azazel

anistark aws aws-s3 azazel checksum downlaod file files filesystem s3 s3-bucket s3-storage sha256 transporter upload validate

Last synced: 21 days ago
JSON representation

File transporter

• Host: GitHub
• URL: https://github.com/anistark/azazel
• Owner: anistark
• License: apache-2.0
• Created: 2017-09-18T07:51:15.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2018-09-01T06:36:35.000Z (over 6 years ago)
• Last Synced: 2024-10-25T15:39:48.792Z (2 months ago)
• Topics: anistark, aws, aws-s3, azazel, checksum, downlaod, file, files, filesystem, s3, s3-bucket, s3-storage, sha256, transporter, upload, validate
• Language: JavaScript
• Homepage: https://anistark.github.io/azazel
• Size: 26.4 KB
• Stars: 0
• Watchers: 4
• Forks: 1
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Azazel

[![Coverage Status](https://coveralls.io/repos/github/anistark/azazel/badge.svg)](https://coveralls.io/github/anistark/azazel)

[![Build Status](https://travis-ci.org/anistark/azazel.svg?branch=master)](https://travis-ci.org/anistark/azazel)

File transporter.

## Description

Transports file from any place(downloadable) to s3(or another storage). 2 queues are used to operate. Any node trying to use Azazel must run a task to push the files to be uploaded and write the files to their corresponding database after reading from result queue.

Azazel can run concurrently across multiple instances in case of scalability requirements.

## How does it work?

Steps:

 1. Upload the data to a queue(currently works with AWS SQS).

 2. Azazel will download the file from the queue to the temp folder in this project. (Can be replaced with any other directory location).

 3. Azazel will then upload the file to s3 and provide with the sqs url.

 4. The sqs url is now pushed to another queue(AWS SQS).

## Setup

* Setup node app: `npm install`

* Setup sqs queue

* Create config file: `cp config_example.js config.js`

* Update the config file according to your requirements.

## Run

`node app.js`

# Todo

- [x] Add Documentation/Wiki.

- [ ] Add a landing gh-pages to be seen at https://anistark.github.io/azazel.

- [ ] Run SQS queue across multiple instances at the same time.

- [ ] Add Test-Cases.

- [ ] Add Coveralls.

- [ ] Support more queues.