{"id":13671043,"url":"https://github.com/datashim-io/datashim","last_synced_at":"2025-12-15T03:04:52.373Z","repository":{"id":37091772,"uuid":"211152022","full_name":"datashim-io/datashim","owner":"datashim-io","description":"A kubernetes based framework for hassle free handling of datasets","archived":false,"fork":false,"pushed_at":"2025-03-21T00:11:40.000Z","size":12123,"stargazers_count":497,"open_issues_count":56,"forks_count":68,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-09T10:01:56.416Z","etag":null,"topics":["csi","dataset-lifecycle-framework","kubernetes","nfs","noobaa","s3"],"latest_commit_sha":null,"homepage":"http://datashim-io.github.io/datashim","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datashim-io.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/Roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-26T18:06:14.000Z","updated_at":"2025-04-08T23:08:42.000Z","dependencies_parsed_at":"2024-05-20T14:45:43.399Z","dependency_job_id":"92c4c924-f391-4207-a702-124702e9c03f","html_url":"https://github.com/datashim-io/datashim","commit_stats":null,"previous_names":["ibm/dataset-lifecycle-framework"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datashim-io%2Fdatashim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datashim-io%2Fdatashim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datashim-io%2Fdatashim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datashim-io%2Fdatashim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datashim-io","download_url":"https://codeload.github.com/datashim-io/datashim/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251154297,"owners_count":21544476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csi","dataset-lifecycle-framework","kubernetes","nfs","noobaa","s3"],"created_at":"2024-08-02T09:00:56.781Z","updated_at":"2025-12-15T03:04:47.332Z","avatar_url":"https://github.com/datashim-io.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![Go Report Card](https://goreportcard.com/report/github.com/datashim-io/datashim)](https://goreportcard.com/report/github.com/datashim-io/datashim)\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4821/badge)](https://bestpractices.coreinfrastructure.org/projects/4821)\n# Datashim\n\n\u003e [!NOTE]  \n\u003e **_Help us shape the future of Datashim - fill in our short anonymous user survey [here](https://forms.gle/wMSkDGfeGzw5XyCM7)_** \n\n\u003cimg src=\"./docs/pictures/lfaidata-project-badge-incubation-color.png\" alt=\"drawing\" width=\"200\"/\u003e\n\n\u003eOur Framework introduces the **Dataset** CRD which is a pointer to existing S3 and NFS data sources. It includes the\n\u003enecessary logic to map these Datasets into Persistent Volume Claims and ConfigMaps which users can reference in their\n\u003epods, letting them focus on the workload development and not on configuring/mounting/tuning the data access. Thanks to\n\u003e[Container Storage Interface](https://kubernetes-csi.github.io/docs/) it is extensible to support additional data sources in the future.\n\n![DLF](./docs/pictures/dlf.png)\n\nA Kubernetes Framework to provide easy access to S3 and NFS **Datasets** within pods. Orchestrates the provisioning of\n**Persistent Volume Claims** and **ConfigMaps** needed for each **Dataset**. Find more details in our [FAQ](https://datashim-io.github.io/datashim/FAQ/)\n\n\u003e [!WARNING]\n\u003e 🚨 (23 Jan 2024) - Group Name Change\n\u003e \n\u003e __If you have an existing installation of Datashim, please DO NOT follow the instructions below to upgrade it to version `0.4.0` or latest__. The group name of the Dataset and DatasetInternal CRDs (objects) is changing from `com.ie.ibm.hpsys` to `datashim.io`. An upgrade in place will invalidate your Dataset definitions and will cause problems in your installation. You can upgrade up to version `0.3.2` without any problems. \n\u003e\n\u003eTo upgrade to `0.4.0` and beyond, please a) delete all datasets safely; b) uninstall Datashim; and c) reinstall Datashim either through Helm or using the manifest file as follows.\n\n## Quickstart\n\nFirst, create the namespace for installing Datashim, if not present\n\n```bash\nkubectl create ns dlf\n```\n\nIn order to quickly deploy Datashim, based on your environment execute **one** of the following commands:\n\n- **Kubernetes/Minikube/kind**\n```bash\nkubectl apply -f https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf.yaml\n```\n- **Kubernetes on IBM Cloud**\n```bash\nkubectl apply -f https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf-ibm-k8s.yaml\n```\n- **Openshift**\n```bash\nkubectl apply -f https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf-oc.yaml\n```\n- **Openshift on IBM Cloud**\n```bash\nkubectl apply -f https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf-ibm-oc.yaml\n```\n\nWait for all the pods to be ready :)\n```bash\nkubectl wait --for=condition=ready pods -l app.kubernetes.io/name=datashim -n dlf\n```\n\nAs an **optional** step, label the namespace(or namespaces) you want in order have the pods labelling functionality (see below for an example with default namespace).\n```bash\nkubectl label namespace default monitor-pods-datasets=enabled\n```\n\n\u003e [!TIP]\n\u003e In case don't have an existing S3 Bucket follow our wiki to [deploy an Object Store](https://github.com/datashim-io/datashim/wiki/Deployment-and-Usage-of-S3-Object-Stores)\nand populate it with data.\n\nWe will create now a Dataset named `example-dataset` pointing to your S3 bucket.\n```yaml\ncat \u003c\u003cEOF | kubectl apply -f -\napiVersion: datashim.io/v1alpha1\nkind: Dataset\nmetadata:\n  name: example-dataset\nspec:\n  local:\n    type: \"COS\"\n    accessKeyID: \"{AWS_ACCESS_KEY_ID}\"\n    secretAccessKey: \"{AWS_SECRET_ACCESS_KEY}\"\n    endpoint: \"{S3_SERVICE_URL}\"\n    bucket: \"{BUCKET_NAME}\"\n    readonly: \"true\" #OPTIONAL, default is false  \n    region: \"\" #OPTIONAL\nEOF\n```\n\nIf everything worked okay, you should see a PVC and a ConfigMap named `example-dataset` which you can mount in your pods.\nAs an easier way to use the Dataset in your pod, you can instead label the pod as follows:\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: nginx\n  labels:\n    dataset.0.id: \"example-dataset\"\n    dataset.0.useas: \"mount\"\nspec:\n  containers:\n    - name: nginx\n      image: nginx\n```\n\nAs a convention the Dataset will be mounted in `/mnt/datasets/example-dataset`. If instead you wish to pass the connection\ndetails as environment variables, change the `useas` line to `dataset.0.useas: \"configmap\"`\n\nFeel free to explore our [other examples](./examples)\n\n\u003e [!IMPORTANT]\n\u003e We recommend using secrets to pass your S3/Object Storage Service credentials to Datashim, as shown in [this example](./examples/templates/example-dataset-s3-provision.yaml).\n\n\u003e [!NOTE]\n\u003e Datashim uses a Mutating Webhook which uses a self-signed certificate. We recommend the use of `cert-manager` to manage this certificate. Please take a look at [this note](docs/cert-manager.md) for instructions to do this. \n\n## Helm Installation\n\nHosted Helm charts have been made available for installing Datashim. This is how you can do a Helm install:\n\n```bash\nhelm repo add datashim https://datashim-io.github.io/datashim/\n```\n```bash\nhelm repo update\n```\nThis should produce an output of `...Successfully got an update from the \"datashim\" chart repository` in addition to the other Helm repositories you may have.\n\nTo install, search for the latest stable release\n\n```bash\nhelm search repo datashim --versions\n```\nwhich will result in:\n```\nNAME                    \tCHART VERSION\tAPP VERSION\tDESCRIPTION\ndatashim/datashim-charts\t0.4.0        \t0.4.0      \tDatashim chart\ndatashim/datashim-charts\t0.3.2        \t0.3.2      \tDatashim chart\n```\n\n\u003e [!CAUTION]\n\u003e Version `0.3.2` still has `com.ie.ibm.hpsys` as the apiGroup name. So, please proceed with caution. It is fine for upgrading from an existing Datashim installation but going forward the apiGroup will be `datashim.io` \n\n\nPass the option to create namespace, if you are installing Datashim for the first time:\n```bash\nhelm install --namespace=dlf --create-namespace datashim datashim/datashim-charts --version \u003cversion_string\u003e\n```\nDo not forget to label the target namespace to support pod labels, as shown in the previous section\n\n### Uninstalling through Helm\n\nTo uninstall, use `helm uninstall` like so:\n```bash\nhelm uninstall -n dlf datashim\n```\n\n### Installing intermediate releases\n\nYou can query the Helm repo for intermediate releases (`.alpha`, `.beta`, etc). To do this, you need to pass `--devel` flag to Helm repo search, like so:\n\n```bash\nhelm search repo datashim --devel\n```\n\nTo install an intermediate version, \n```bash\nhelm install --namespace=dlf --create-namespace datashim datashim/datashim-charts --devel --version \u003cversion_name\u003e\n```\n\n## Questions\n\nThe wiki and [Frequently Asked Questions](https://datashim-io.github.io/datashim/FAQ) documents are a bit out of date. We recommend browsing [the issues](https://github.com/datashim-io/datashim/issues?q=is%3Aissue+label%3Aquestion) for previously answered questions. Please open an issue if you are not able to find the answers to your questions, or if you have discovered a bug. \n\n## Contributing\n\nWe welcome all contributions to Datashim. Please read [this document](./docs/GitWorkflow.md) for setting up a Git workflow for contributing to Datashim. This project uses [DCO (Developer Certificate of Origin)](https://github.com/apps/dco) to certify code ownership and contribution rights. \n\nIf you use VSCode, then we have [recommendations for setting it up for development](./docs/GolangVSCodeGit.md). \n\nIf you have an idea for a feature request, please open an issue. Let us know in the issue description the problem or the pain point, and how the proposed feature would help solve it. If you are looking to contribute but you don't know where to start, we recommend looking at the open issues first. Thanks!\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatashim-io%2Fdatashim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatashim-io%2Fdatashim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatashim-io%2Fdatashim/lists"}