https://github.com/nitor-infotech-oss/go-filewatcher
FileWatcher is GoLang based AWS lambda function is looking for the file arrival on S3 bucket and move the files to target s3 bucket with decompression.
https://github.com/nitor-infotech-oss/go-filewatcher
Last synced: 3 months ago
JSON representation
FileWatcher is GoLang based AWS lambda function is looking for the file arrival on S3 bucket and move the files to target s3 bucket with decompression.
- Host: GitHub
- URL: https://github.com/nitor-infotech-oss/go-filewatcher
- Owner: nitor-infotech-oss
- Created: 2020-08-07T11:35:05.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2020-08-07T11:41:13.000Z (about 5 years ago)
- Last Synced: 2025-02-01T09:22:16.181Z (8 months ago)
- Language: Go
- Size: 8.79 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# go-filewatcher
[](https://travis-ci.org/joemccann/dillinger)
# Objective
- Develop a daemon based technology solution that will given a list of source watch directories and REGEX patterns for filenames. Move files to a corresponding list of target directories with decompression, if the source file is compressed.
- This solution would in addition send alerts via slack to inform a group of recipients that a move event was triggered and the details of the event.
- The list of source, target directories and rules should be configurable and stored in json file.
# Solution Approach
AWS Lambda is a serverless offering by AWS cloud and identified as s best fit for File Watcher components.
Below are the important components of the solution –
- This lambda function is the fileWatcher program which can be invoked using the api endpoint from external program.
- It reads the configurations from json file and perform actions on the file like unzip, move it to target bucket by creating the require directory structure on the target bucket.
- Logs the execution of the process in the and send an alert with execution details to Slack
### Dependencies
GoLang packages dependencies:
* [aws-sdk-go](https://github.com/aws/aws-sdk-go)
* [golang-x](https://golang.org/x/sync)
# Configurations
- Settings.json - This will work as configuration file for golang package and contains below information.
Credentials
{
"naming": {
"prepend": "autogen"
},
"key": "XXXXXXXXXXXXXXXXXXXXXXXXXXX",
"secret": "XXXXXXXXXXXXXXXXXXXXXXXX",
"region": "us-east-2",
"sourcebucket": "bucket-001",
"sourcepath": "inputpath",
"archivalpath": "archival",
"destbucket": "bucket-002",
"destpath": "outputpath",
"webhookurl": "https://hooks.slack.com/services/TKDC1KVV1/XXXXXX/XXXXXXXXXXXXXX",
"regexrules": ["^20200801"]
}}
### Imprting Package
```
import (
"github.com/nitor-infotech-oss/go-filewatcher/filewatcher"
)
```
Package can be imported by simply using the path of repo where it resides.
### Steps to execute
- Setup $GOPATH
- Install required dependencies
```
go run filewatcher.go
```
### Deploy
- Deployed over Amazon ec2 instance
- Can be deployed lambda function to make the executions faster