{"id":18733920,"url":"https://github.com/mgoltzsche/k8storagex","last_synced_at":"2026-07-15T01:37:37.454Z","repository":{"id":52535197,"uuid":"291839016","full_name":"mgoltzsche/k8storagex","owner":"mgoltzsche","description":"host path and layered cache file system provisioner for distributed (CI) jobs on Kubernetes (alpha)","archived":false,"fork":false,"pushed_at":"2021-05-30T23:15:01.000Z","size":407,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-08-02T19:44:48.670Z","etag":null,"topics":["cache","cache-provisioner","cache-storage","cache-synchronizations","copy-on-write","cow","k8s-sig-storage","storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mgoltzsche.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-31T22:35:11.000Z","updated_at":"2022-01-24T11:46:50.000Z","dependencies_parsed_at":"2022-09-06T14:50:52.025Z","dependency_job_id":null,"html_url":"https://github.com/mgoltzsche/k8storagex","commit_stats":null,"previous_names":["mgoltzsche/cache-provisioner"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/mgoltzsche/k8storagex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgoltzsche%2Fk8storagex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgoltzsche%2Fk8storagex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgoltzsche%2Fk8storagex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgoltzsche%2Fk8storagex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mgoltzsche","download_url":"https://codeload.github.com/mgoltzsche/k8storagex/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgoltzsche%2Fk8storagex/sbom","scorecard":{"id":639459,"data":{"date":"2025-08-11","repo":{"name":"github.com/mgoltzsche/k8storagex","commit":"6b75e780adaae234a22280f69ce8f73ffb7636f6"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.8,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/27 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/pr.yml:1","Warn: no topLevel permission defined: .github/workflows/release.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pr.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/mgoltzsche/k8storagex/pr.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pr.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/mgoltzsche/k8storagex/pr.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/mgoltzsche/k8storagex/release.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/mgoltzsche/k8storagex/release.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/mgoltzsche/k8storagex/release.yml/master?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:2","Warn: containerImage not pinned by hash: Dockerfile:22: pin your Docker image by updating gcr.io/distroless/static:nonroot to gcr.io/distroless/static:nonroot@sha256:cdf4daaf154e3e27cfffc799c16f343a384228f38646928a1513d925f473cb46","Warn: containerImage not pinned by hash: Dockerfile-layerfs:2","Warn: containerImage not pinned by hash: Dockerfile-layerfs:22: pin your Docker image by updating alpine:3.12 to alpine:3.12@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69","Warn: npmCommand not pinned by hash: .github/workflows/release.yml:32","Info:   0 out of   5 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 npmCommand dependencies pinned","Info:   0 out of   4 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 4 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"13 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0345 / GHSA-7638-r9r3-rmjj","Warn: Project is vulnerable to: GO-2022-0417 / GHSA-c3g4-w6cv-6v7h","Warn: Project is vulnerable to: GO-2022-1008 / GHSA-fjm8-m7m6-2fjp","Warn: Project is vulnerable to: GO-2025-3414 / GHSA-5vpc-35f4-r8w6","Warn: Project is vulnerable to: GO-2024-2658","Warn: Project is vulnerable to: GO-2024-3186 / GHSA-586p-749j-fhwp","Warn: Project is vulnerable to: GO-2024-3169 / GHSA-fhqq-8f65-5xfc","Warn: Project is vulnerable to: GO-2024-3171 / GHSA-mc76-5925-c5p6","Warn: Project is vulnerable to: GO-2024-2842 / GHSA-6wvf-f2vw-3425","Warn: Project is vulnerable to: GO-2021-0100 / GHSA-7qw8-847f-pggm","Warn: Project is vulnerable to: GHSA-77vh-xpmg-72qh","Warn: Project is vulnerable to: GO-2022-0493 / GHSA-p782-xgp4-8hr8","Warn: Project is vulnerable to: GO-2021-0064 / GHSA-8cfg-vx93-jvxw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-21T10:17:01.992Z","repository_id":52535197,"created_at":"2025-08-21T10:17:01.993Z","updated_at":"2025-08-21T10:17:01.993Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279015284,"owners_count":26085683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","cache-provisioner","cache-storage","cache-synchronizations","copy-on-write","cow","k8s-sig-storage","storage"],"created_at":"2024-11-07T15:11:48.066Z","updated_at":"2025-10-13T13:08:59.015Z","avatar_url":"https://github.com/mgoltzsche.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Extensible Kubernetes storage provisioners\n\nA host directory and cache provisioner for Kubernetes.  \n\nIt provides a simple, fast and scalable caching solution for distributed (build) jobs using containers as layered, Copy-on-Write cache file systems that can be provisioned as `PersistentVolumes`.  \n\nFormerly known as \"cache-provisioner\".\n\n## Motivation\n\nBuild tools that require a lot of dynamic dependencies like Docker/Podman or\nMaven don't perform well without its cache backed by persistent storage.\nWhen distributed builds are set up without such a persistent cache\nthey are often slower than locally run builds which can impact development\nvelocity.  \n\nTools like Kaniko or Makisu that already support distributed caching are still\nslower than local builds since they need to synchronize/transfer the cache\nduring the build which increases the build duration while still not guaranteeing\nfull cache consistency since the last cache writer wins (which is acceptable though).  \n\nAs a simple solution a single PVC per build could be used but this reduces build concurrency and availability.\nAlternatively a `hostpath` Volume could be configured per project but that requires privileges,\nis not synchronized between nodes, cannot be size limited and may cause problems\nwhen written by multiple jobs concurrently.  \n\nSince job cache storage is a general problem that applies to most build\npipelines in general it should be solved as an infrastructure problem rather\nthan that of a particular application or pipeline.\nDoing so also allows to reduce the time cache synchronizations are blocking builds.\n\n## Idea\n\nA `PersistentVolume` should be requested for a particular cache name with accessMode `ReadWriteOnce`.\nA cache name is unique per Kubernetes `Namespace`.\nWhen a new `PersistentVolume` is created it is initialized with the contents\nof the cache identified by the given name.\nWhen a `PersistentVolume` is deleted (and it is the master for its cache name on that node)\nits contents are written back to the cache name from where the next `PersistentVolume`\nthat is requested afterwards for the same cache name is initialized.\nHowever during that process each `PersistentVolume` remains both writeable\nand isolated with the help of overlayfs using the shared cache as lower layer\nand a separate dir per `PersistentVolume` as upper layer (like docker's image layer fs).  \n\nThe cache name as well as eligibility to write the master cache could be\nspecified as PVC annotations. Distinction between builds that can read and\nbuilds that can write the shared cache is necessary at least in OpenSource\nprojects where a PR build could inject malware into regular releases\nby manipulating the shared cache.  \n\nThough the shared cache is only committed when a `PersistentVolume` is deleted\nwhich therefore must happen directly after each build but PVC/PV deletion is\nblocked by the `kubernetes.io/pvc-protection` finalizer as long as the build\npod refers to it which is the case in e.g. a\n[Tekton Pipeline](https://github.com/tektoncd/pipeline/blob/v0.15.1/docs/pipelineruns.md#specifying-resources).\nHowever an additional controller could be written that watches Pods and,\non Pod termination (if Pod's `restartPolicy: Never`), deletes the associated PVC,\nwaits for it to be in `Terminating` state and remove the finalizer\nso that the corresponding PV gets committed/deleted.\n\n\n## Development\n\n### Generate code/manifests\n\n```sh\nmake generate manifests static-manifests\n```\n\n### Build\nBuild binaries:\n```sh\nmake layerfs manager\n```\n\n### Test\nTest layerfs binary:\n```sh\nmake test-layerfs\n```\n\n### Deploy\n\nDeploy to a local Kubernetes cluster (using [kpt](https://github.com/GoogleContainerTools/kpt): `kpt live apply config/static/*`):\n```sh\nmake deploy-$TARGET\n```\nUndeploy:\n```sh\nmake undeploy-$TARGET\n```\nwhere `$TARGET` can be one of:\n* `default` - manager and provisioners without registry, pulling public images\n* `registry` - manager, provisioners and registry, pulling public images\n* `minikube` - like `registry` but builds and deploys the local changes to minikube\n* `kind` - like `registry` but builds and deploys the local changes to kind\n\n### Run the run the manager binary directly\n\nAfter deploying the manifests scale down the manager Deployment to 0:\n```sh\nkubectl scale -n k8storagex --replicas=0 deployment/k8storagex-controller-manager\n```\n\nBuild and run the binary:\n```sh\nmake run\n```\n\n## Example\n\nDeploy a Pod with a PersistentVolumeClaim that points to a cache named `example-project`:\n```sh\nkubectl apply -f e2e/test-pod.yaml\n```\nWatch the Pod being created and how it fetches and runs a podman image:\n```sh\n$ kubectl logs -f cached-build\nTrying to pull docker.io/library/alpine:3.12...\nGetting image source signatures\nCopying blob sha256:801bfaa63ef2094d770c809815b9e2b9c1194728e5e754ef7bc764030e140cea\nCopying config sha256:389fef7118515c70fd6c0e0d50bb75669942ea722ccb976507d7b087e54d5a23\nWriting manifest to image destination\nStoring signatures\nhello from nested container\n```\n\nAfter the Pod terminated the PVC is removed and a `Cache` resource is created that refers to the node the Pod ran on:\n```sh\n$ kubectl get cache example-project\nNAME              AGE\nexample-project   7s\n```\n\nWhen another Pod is deployed that points to the same cache its volume has the same contents as it had when the last Pod that was using the cache terminated.\nFor the sake of the example let's delete and recreate the previously applied Pod and PersistentVolumeClaim:\n```sh\n$ kubectl delete -f e2e/test-pod.yaml\n$ kubectl apply -f e2e/test-pod.yaml\n```\n\nAlthough the new Pod has a new PersistentVolumeClaim and PersistentVolume the image is cached and doesn't need to be pulled again:\n```sh\n$ kubectl logs -f cached-build\nhello from nested container\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgoltzsche%2Fk8storagex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmgoltzsche%2Fk8storagex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgoltzsche%2Fk8storagex/lists"}