{"id":13621988,"url":"https://github.com/scribd/objinsync","last_synced_at":"2025-04-05T11:07:45.030Z","repository":{"id":44593451,"uuid":"243124415","full_name":"scribd/objinsync","owner":"scribd","description":"Continuously synchronize directories from remote object store to local filesystem","archived":false,"fork":false,"pushed_at":"2025-02-21T22:59:04.000Z","size":126,"stargazers_count":104,"open_issues_count":16,"forks_count":21,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-29T10:05:29.454Z","etag":null,"topics":["airflow","cplat","s3"],"latest_commit_sha":null,"homepage":"https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scribd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-25T23:25:33.000Z","updated_at":"2025-03-26T11:01:42.000Z","dependencies_parsed_at":"2024-01-27T05:24:00.143Z","dependency_job_id":"54d74685-993c-470f-8461-e372d1f6920f","html_url":"https://github.com/scribd/objinsync","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fobjinsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fobjinsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fobjinsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scribd%2Fobjinsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scribd","download_url":"https://codeload.github.com/scribd/objinsync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325693,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","cplat","s3"],"created_at":"2024-08-01T21:01:12.692Z","updated_at":"2025-04-05T11:07:44.986Z","avatar_url":"https://github.com/scribd.png","language":"Go","readme":"ObjInSync\n=========\n\n![CI/CD](https://github.com/scribd/objinsync/workflows/CI/CD/badge.svg)\n\nDaemon to continuously and incrementally synchronize a directory from remote\nobject store to a local directory.\n\n\nUsage\n-----\n\n```bash\nobjinsync pull --exclude '**/__pycache__/**' s3://bucket/keyprefix ./localdir\n```\n\nWhen running in daemon mode (without `--once` flag), a health check endpoint is\nserved at `:8087/health` and a prometheus metrics endpoint is served at\n`:8087/metrics`. You can use `--status-addr` to override the binding address.\n\nObjinsync also comes with builtin Sentry integration. To enable it, set the\n`SENTRY_DSN` environment variable.\n\nYou can also run objinsync in pull once mode, which behaves just like `aws s3 sync`:\n\n```bash\nobjinsync pull --once s3://bucket/keyprefix ./localdir\n```\n\nTo use with [Minio](https://docs.min.io/) instead of S3, you can set\n`--s3-endpoint` and `--disable-ssl` flags for `pull` command as you see fit.\n\nThe `-i` or `--interval` flags allows to configure the pull time interval, which is 5 seconds by default:\n\n```bash\nobjinsync pull --interval 20s s3://bucket/keyprefix ./localdir\n```\n\n---\n\nEnable debug logs by setting the `DEBUG` environment variable `DEBUG=1 objinsync pull ...`\n\n\nInstallation\n------------\n\nSimply download the prebuilt single binary from [release page](https://github.com/scribd/objinsync/releases) or use `go get` command:\n\n```bash\ngo get github.com/scribd/objinsync\n```\n\nPre-built docker images are available at https://github.com/orgs/scribd/packages/container/package/objinsync.\n\n\nDevelopment\n------------\n\nRun tests\n\n```bash\nmake test\n```\n\nRun from source\n\n```bash\nAWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags\n```\n\nTo cut a release, push tag to remote in the format of `vx.x.x`.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscribd%2Fobjinsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscribd%2Fobjinsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscribd%2Fobjinsync/lists"}