Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/scribd/objinsync
Continuously synchronize directories from remote object store to local filesystem
https://github.com/scribd/objinsync
airflow cplat s3
Last synced: 3 days ago
JSON representation
Continuously synchronize directories from remote object store to local filesystem
- Host: GitHub
- URL: https://github.com/scribd/objinsync
- Owner: scribd
- License: mit
- Created: 2020-02-25T23:25:33.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-09-07T20:35:09.000Z (2 months ago)
- Last Synced: 2024-09-07T21:40:04.001Z (2 months ago)
- Topics: airflow, cplat, s3
- Language: Go
- Homepage: https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html
- Size: 121 KB
- Stars: 94
- Watchers: 14
- Forks: 21
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
ObjInSync
=========
Daemon to continuously and incrementally synchronize a directory from remote
object store to a local directory.
Usage
-----
```bash
objinsync pull --exclude '**/__pycache__/**' s3://bucket/keyprefix ./localdir
```
When running in daemon mode (without `--once` flag), a health check endpoint is
served at `:8087/health` and a prometheus metrics endpoint is served at
`:8087/metrics`. You can use `--status-addr` to override the binding address.
Objinsync also comes with builtin Sentry integration. To enable it, set the
`SENTRY_DSN` environment variable.
You can also run objinsync in pull once mode, which behaves just like `aws s3 sync`:
```bash
objinsync pull --once s3://bucket/keyprefix ./localdir
```
To use with [Minio](https://docs.min.io/) instead of S3, you can set
`--s3-endpoint` and `--disable-ssl` flags for `pull` command as you see fit.
The `-i` or `--interval` flags allows to configure the pull time interval, which is 5 seconds by default:
```bash
objinsync pull --interval 20s s3://bucket/keyprefix ./localdir
```
---
Enable debug logs by setting the `DEBUG` environment variable `DEBUG=1 objinsync pull ...`
Installation
------------
Simply download the prebuilt single binary from [release page](https://github.com/scribd/objinsync/releases) or use `go get` command:
```bash
go get github.com/scribd/objinsync
```
Pre-built docker images are available at https://github.com/orgs/scribd/packages/container/package/objinsync.
Development
------------
Run tests
```bash
make test
```
Run from source
```bash
AWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags
```
To cut a release, push tag to remote in the format of `vx.x.x`.