{"id":36496588,"url":"https://github.com/oneconcern/datamon","last_synced_at":"2026-01-12T02:04:41.648Z","repository":{"id":37797187,"uuid":"138348481","full_name":"oneconcern/datamon","owner":"oneconcern","description":"Datamon manages infinite reflections of data","archived":false,"fork":false,"pushed_at":"2023-04-07T06:00:18.000Z","size":10330,"stargazers_count":15,"open_issues_count":58,"forks_count":6,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-06-19T01:55:54.556Z","etag":null,"topics":["data-management-platform"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oneconcern.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-22T21:09:55.000Z","updated_at":"2023-09-19T08:27:12.000Z","dependencies_parsed_at":"2024-06-19T01:31:48.122Z","dependency_job_id":"1d690f7b-6fdd-4272-90a5-aca8b2caa783","html_url":"https://github.com/oneconcern/datamon","commit_stats":null,"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"purl":"pkg:github/oneconcern/datamon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneconcern%2Fdatamon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneconcern%2Fdatamon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneconcern%2Fdatamon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneconcern%2Fdatamon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oneconcern","download_url":"https://codeload.github.com/oneconcern/datamon/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oneconcern%2Fdatamon/sbom","scorecard":{"id":706936,"data":{"date":"2025-08-11","repo":{"name":"github.com/oneconcern/datamon","commit":"e32a50eee3333304c6af8a1a57a0a4aa4fb97b72"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.4,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/24 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v2.6.9 not signed: https://api.github.com/repos/oneconcern/datamon/releases/90388227","Warn: release artifact v2.6.8 not signed: https://api.github.com/repos/oneconcern/datamon/releases/90334057","Warn: release artifact v2.6.7 not signed: https://api.github.com/repos/oneconcern/datamon/releases/88623749","Warn: release artifact v2.6.6 not signed: https://api.github.com/repos/oneconcern/datamon/releases/88546365","Warn: release artifact v2.6.5 not signed: https://api.github.com/repos/oneconcern/datamon/releases/88490704","Warn: release artifact v2.6.9 does not have provenance: https://api.github.com/repos/oneconcern/datamon/releases/90388227","Warn: release artifact v2.6.8 does not have provenance: https://api.github.com/repos/oneconcern/datamon/releases/90334057","Warn: release artifact v2.6.7 does not have provenance: https://api.github.com/repos/oneconcern/datamon/releases/88623749","Warn: release artifact v2.6.6 does not have provenance: https://api.github.com/repos/oneconcern/datamon/releases/88546365","Warn: release artifact v2.6.5 does not have provenance: https://api.github.com/repos/oneconcern/datamon/releases/88490704"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 18 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: dockerfiles/alpine-base.Dockerfile:4","Warn: containerImage not pinned by hash: dockerfiles/binaries.Dockerfile:3","Warn: containerImage not pinned by hash: dockerfiles/builder.Dockerfile:4: pin your Docker image by updating golang:1.19 to golang:1.19@sha256:3025bf670b8363ec9f1b4c4f27348e6d9b7fec607c47e401e40df816853e743a","Warn: containerImage not pinned by hash: dockerfiles/datamon-util.Dockerfile:3","Warn: containerImage not pinned by hash: dockerfiles/datamon-util.Dockerfile:5","Warn: containerImage not pinned by hash: dockerfiles/datamon-util.Dockerfile:10: pin your Docker image by updating alpine:3.14 to alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed","Warn: containerImage not pinned by hash: dockerfiles/datamon.Dockerfile:1","Warn: containerImage not pinned by hash: dockerfiles/migrate.Dockerfile:2","Warn: containerImage not pinned by hash: dockerfiles/pgsidecar-base.Dockerfile:5","Warn: containerImage not pinned by hash: dockerfiles/sidecar-base.Dockerfile:4: pin your Docker image by updating debian to debian@sha256:6d87375016340817ac2391e670971725a9981cfc24e221c47734681ed0f6c0f5","Warn: containerImage not pinned by hash: dockerfiles/sidecar-pg.Dockerfile:4","Warn: containerImage not pinned by hash: dockerfiles/sidecar-pg.Dockerfile:6","Warn: containerImage not pinned by hash: dockerfiles/sidecar.Dockerfile:3","Warn: containerImage not pinned by hash: dockerfiles/sidecar.Dockerfile:5","Warn: containerImage not pinned by hash: dockerfiles/wrapper.Dockerfile:4: pin your Docker image by updating alpine to alpine@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1","Warn: containerImage not pinned by hash: hack/fuse-demo/coord-app-pg.Dockerfile:7","Warn: containerImage not pinned by hash: hack/fuse-demo/coord-app.Dockerfile:2","Warn: containerImage not pinned by hash: hack/fuse-demo/shell.Dockerfile:2","Warn: containerImage not pinned by hash: k8s/migratev2/migrator.Dockerfile:1","Warn: containerImage not pinned by hash: k8s/migratev2/migrator.Dockerfile:7","Warn: containerImage not pinned by hash: k8s/migratev2/migrator.Dockerfile:28","Warn: goCommand not pinned by hash: dockerfiles/builder.Dockerfile:8-19","Warn: goCommand not pinned by hash: dockerfiles/builder.Dockerfile:8-19","Warn: goCommand not pinned by hash: dockerfiles/builder.Dockerfile:8-19","Warn: goCommand not pinned by hash: dockerfiles/datamon-util.Dockerfile:8","Info:   0 out of  21 containerImage dependencies pinned","Info:   1 out of   5 goCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":7,"reason":"3 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0635","Warn: Project is vulnerable to: GO-2022-0646","Warn: Project is vulnerable to: GO-2025-3770"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T06:54:01.386Z","repository_id":37797187,"created_at":"2025-08-22T06:54:01.386Z","updated_at":"2025-08-22T06:54:01.386Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28331523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T00:36:25.062Z","status":"online","status_checked_at":"2026-01-12T02:00:08.677Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-management-platform"],"created_at":"2026-01-12T02:02:16.671Z","updated_at":"2026-01-12T02:04:41.635Z","avatar_url":"https://github.com/oneconcern.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/oneconcern/datamon/tree/master.svg?style=svg\u0026circle-token=e827ee1509892d8ba85db2a819b692ca451a7a97)](https://circleci.com/gh/oneconcern/datamon/tree/master)\n[![Coverage Status](https://coveralls.io/repos/github/oneconcern/datamon/badge.svg?t=VQE2s3)](https://coveralls.io/github/oneconcern/datamon)\n[![GitHub release](https://img.shields.io/github/v/release/oneconcern/datamon)](https://github.com/oneconcern/datamon/releases/latest)\n[![license](https://img.shields.io/badge/license-MIT-green)](https://raw.githubusercontent.com/oneconcern/datamon/master/LICENSE)\n[![GoDoc](https://godoc.org/github.com/oneconcern/datamon?status.svg)](http://godoc.org/github.com/oneconcern/datamon)\n\n# Datamon\n\nDatamon is a data science tool sponsored by [OneConcern](https://www.oneconcern.com) that helps manage data at scale.\n\n## Primer\n\n### Goals\n\nThe primary goal of datamon is to manage versioned data at rest, providing CLI tools for creation, access and tracking\nin an environment where data repositories and their lifecycles are linked.\n\nDatamon links the various sources of data, how they are processed and tracks the\noutput/new data that is generated from the existing data.\n\n[More on design and architecture](docs/design.md).\n\n### Features\n\n* Manage data sets as versioned repositories stored on a cloud storage backend\n* Manage metadata for these data sets (versions, labels, file sets...)\n* Multi-tenancy using contexts\n* Lineage tracking backed by cloud authentication\n* Store data sets as fixed size deduplicated blobs, using blake hashing\n* Versions (\"bundles\") may be uploaded then downloaded on local storage\n* Versions may be accessed directly on a mounted file system (fuse)\n* CLI management tool\n* Metrics collection\n\n### Added value\n\n* Leverages low-cost frozen storage (e.g. S3, GCS)\n* Optimized billed operations for storage: no fancy billable backend store options are used (like concurrency control, etc)\n* Optimized for speed: parallel I/Os together with deduplication vastly outperform usual tools like `gsutil`\n* A well-defined and tested immutable metadata model ensures that no data is ever lost or unrecoverable. Datamon is an effective substitute to many bespoke gsutil scripting utilities.\n* Versioning \u0026 tagging occur on whole data sets and not individual files. This makes it easy to restore consistent inputs to some reproducible computation\n* Less storage bucket administration: datamon uses only a few buckets, defined according to IAM policies (i.e. a datamon context)\n* Repositories make up a convenient abstration for datasets, and share the same underlying cloud storage bucket configuration (abstracted as a \"context\")\n\n#### Extra tools\n\n* Scripted interface to use as a sidecar container (e.g. for ARGO workflows)\n* [Cleaning-up unused storage](docs/purge.md)\n\n#### Experimental\n\n* Mutable fuse mount, to commit versioned data sets directly from a mounted file system\n\n#### Coming soon...\n\n* [X] Diamond workflow: several collaborating nodes produce a versioned dataset in parallel\n* [ ] Python bindings\n* [ ] Write Ahead / Read Ahead logs\n\n### Environment\n\nAlthough flexible in its concepts and architecture, the current version of datamon is primarily developed and tested \nagainst the Google Cloud environment. Note that AWS S3 storage buckets are supported (see [datamover tool](docs/datamover.md)).\n\n#### Storage backends\n\nDatamon supports the following cloud storage backends:\n* Google Cloud Storage \n* AWS S3\n\n### [Concepts](docs/concepts.md)\n\n- [**Repo**](docs/concepts.md#repo): analogous to a git repo. A repo in datamon is a dataset that has a unified lifecycle.\n- [**Bundle**](docs/concepts.md#bundle): a bundle is a point in time read-only view of a rep:branch and is composed of individual files. Analogous to a commit in git.\n- [**Label**](docs/concepts.md#label): a name given to a bundle, analogous to tags in git. Examples: Latest, production.\n- [**Context**](docs/concepts.md#context): a context provides a way to define multiple instances of datamon.\n- [**Write Ahead Log**](docs/concepts.md#write-ahead-log): a WAL tracks data updates and their ordering.\n- [**Read Log**](docs/concepts.md#read-log): logs all read operations, with their originator.\n- [**Authentication**](docs/auth.md): datamon keeps track of who contributed what, when and in which order (WAL) and who accessed what (Read Log).\n\n## Installation\n\nPlease follow the [installation instructions](docs/install.md).\n\n## Migrating from v1 to v2\n\nv2 comes with breaking changes. The migration process replaces older repos by new ones.\n\nSee the [migration guide](https://github.com/oneconcern/datamon/tree/master/k8s/migratev2).\n\n## CLI guide\n\nDatamon comes as a CLI tool: see [usage](docs/usage/datamon.md).\n\n## Use cases\n\n* ARGO ML pipeline\n* [Datamover container guide](docs/datamover.md)\n* [Datamon as sidecar](docs/sidecar.md)\n* [Kubernetes integration](docs/kubernetes.md)\n\n## Feature requests and bugs\n\nPlease file [GitHub issues](https://github.com/oneconcern/datamon/issues) for feature requests or bug reports.\n\n## Contributing\n\nPlease read our [contributing guidelines](CONTRIBUTING.md)\n\n## License\n\nDatamon is developed by [OneConcern Inc.](https://wwww.oneconcern.com) under the [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneconcern%2Fdatamon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foneconcern%2Fdatamon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneconcern%2Fdatamon/lists"}