{"id":17030291,"url":"https://github.com/vsoch/cdb","last_synced_at":"2025-04-12T12:33:16.978Z","repository":{"id":62561211,"uuid":"271890376","full_name":"vsoch/cdb","owner":"vsoch","description":"Container database metadata extraction and data-container builder","archived":false,"fork":false,"pushed_at":"2024-03-07T05:54:12.000Z","size":16358,"stargazers_count":4,"open_issues_count":5,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-26T07:11:17.354Z","etag":null,"topics":["containerdb","data-container","docker"],"latest_commit_sha":null,"homepage":"https://vsoch.github.io/cdb/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vsoch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"vsoch"}},"created_at":"2020-06-12T21:07:52.000Z","updated_at":"2025-01-09T16:45:23.000Z","dependencies_parsed_at":"2024-06-12T20:46:49.259Z","dependency_job_id":null,"html_url":"https://github.com/vsoch/cdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fcdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fcdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fcdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsoch%2Fcdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vsoch","download_url":"https://codeload.github.com/vsoch/cdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248566565,"owners_count":21125686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containerdb","data-container","docker"],"created_at":"2024-10-14T08:05:47.076Z","updated_at":"2025-04-12T12:33:16.955Z","avatar_url":"https://github.com/vsoch.png","language":"Python","funding_links":["https://github.com/sponsors/vsoch"],"categories":[],"sub_categories":[],"readme":"# Container Database (cdb)\n\nThis is the Python support tool for [containerdb](https://github.com/vsoch/containerdb)\nto support generation of [data containers](https://github.com/singularityhub/data-container).\nPython is more friendly to generating arbitrary data structures, and is popular among the\ndata science community, so I chose it for metadata generation instead of using GoLang.\n\n[![PyPI version](https://badge.fury.io/py/cdb.svg)](https://badge.fury.io/py/cdb)\n\n\u003e Have your data and use it too!\n\n![docs/assets/img/logo/logo.png](docs/assets/img/logo/logo.png)\n\nFor documentation and full examples see [vsoch.github.io/cdb](https://vsoch.github.io/cdb). These\nexamples are also available in the [examples](examples) folder.\n\n## Getting Started\n\n### What is a Data Container?\n\nA data container is generally an operating-system-less container that is optimized\nto provide data, either for query/search, or binding for analysis. The qualities of\nthe data container should be:\n\n 1. It can be mounted to containers with operating systems to run analysis\n 2. It can be interacted with on it's own to search metadata about the data\n 3. It should not have an operating system.\n\n### How do we generate one?\n\nThe generation is fairly simple! It comes down to a three step multistage build:\n\n 1. **Step 1** We install [cdb](https://github.com/vsoch/cdb) to generate a GoLang template for an [in-memory database](https://github.com/vsoch/containerdb) for our data) \n 2. **Step 2** We compile the binary into an entrypoint\n 3. **Step 3** We add the data and the binary entrypoint to a scratch container (no operating system).\n\nAnd then we interact with it! This tutorial will show you the basic steps to\nperform the multistage-build using a simple [Dockerfile](https://github.com/vsoch/cdb/tree/master/examples/docker-simple/Dockerfile) along with the data folder. The [Dockerfile](Dockerfile) in the base of the repository also is a good example.\n\n## Usage\n\n### Docker Usage\n\nThe intended usage is via Docker, so you don't need to worry about installation of\nPython, GoLang, and multistage builds to basically:\n\n 1. Generate a db.go template\n 2. Compile it\n 3. Add to scratch with data as data container entrypoint.\n\nThus, to run the dummy example here using the [Dockerfile](Dockerfile):\n\n```bash\n$ docker build -t data-container .\n```\n\nWe then have a simple way to do the following:\n\n**metadata**\n\nIf we just run the container, we get a listing of all metadata alongside the key.\n\n```bash\n$ docker run entrypoint \n/data/avocado.txt {\"size\": 9, \"sha256\": \"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4\"}\n/data/tomato.txt {\"size\": 8, \"sha256\": \"3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816\"}\n```\n\n**list** \n\nWe can also just list data files with `-ls`\n\n```bash\n$ docker run entrypoint -ls\n/data/avocado.txt\n/data/tomato.txt\n```\n\n**orderby**\n\nOr we can list ordered by one of the metadata items:\n\n```bash\n$ docker run entrypoint -metric size\nOrder by size\n/data/tomato.txt: {\"size\": 8, \"sha256\": \"3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816\"}\n/data/avocado.txt: {\"size\": 9, \"sha256\": \"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4\"}\n```\n\n**search**\n\nOr search for a specific metric based on value.\n\n```bash\n$ docker run entrypoint -metric size -search 8\n/data/tomato.txt 8\n\n$ docker run entrypoint -metric sha256 -search 8\n/data/avocado.txt 327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4\n/data/tomato.txt 3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816\n```\n\n**get**\n\nOr we can get a particular file metadata by it's name:\n\n```bash\n$ docker run entrypoint -get /data/avocado.txt\n/data/avocado.txt {\"size\": 9, \"sha256\": \"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4\"}\n```\n\nor a partial match:\n\n```bash\n$ docker run entrypoint -get /data/\n/data/avocado.txt {\"size\": 9, \"sha256\": \"327bf8231c9572ecdfdc53473319699e7b8e6a98adf0f383ff6be5b46094aba4\"}\n/data/tomato.txt {\"size\": 8, \"sha256\": \"3b7721618a86990a3a90f9fa5744d15812954fba6bb21ebf5b5b66ad78cf5816\"}\n```\n\n**start**\n\nThe start command is intended to keep the container running, if we are using\nit with an orchestrator.\n\n```bash\n$ docker run data-container -start\n```\n\n### Orchestration\n\nIt's more likely that you'll want to interact with files in the container via\nsome analysis, or more generally, another container. Let's put together\na quick `docker-compose.yml` to do exactly that.\n\n```\nversion: \"3\"\nservices:\n  base:\n    restart: always\n    image: busybox\n    entrypoint: [\"tail\", \"-f\", \"/dev/null\"]\n    volumes:\n      - data-volume:/data\n\n  data:\n    restart: always\n    image: data-container\n    command: [\"-start\"]\n    volumes:\n      - data-volume:/data\n\nvolumes:\n  data-volume:\n```\n\nNotice that the command for the data-container to start is `-start`, which\nis important to keep it running. After building our `data-container`, we can then bring these containers up:\n\n\n```bash\n$ docker-compose up -d\nStarting docker-simple_base_1   ... done\nRecreating docker-simple_data_1 ... done\n```\n```bash\n$ docker-compose ps\n        Name                Command         State   Ports\n---------------------------------------------------------\ndocker-simple_base_1   tail -f /dev/null    Up           \ndocker-simple_data_1   /entrypoint -start   Up           \n```\n\nWe can then shell inside and see our data!\n\n```bash\n$ docker exec -it docker-simple_base_1 sh\n/ # ls /data/\navocado.txt  tomato.txt\n```\n\nThe metadata is still available for query by interacting with the data-container\nentrypoint:\n\n```bash\n$ docker exec docker-simple_data_1 /entrypoint -ls\n/data/avocado.txt\n/data/tomato.txt\n```\n\nDepending on your use case, you could easily make this available inside the\nother container. This is very simple usage, but the idea is powerful! We can interact with the dataset\nand search it without needing an operating system. It follows that we can develop\ncustomized data-containers based on the format / organization of the data inside \n(e.g., a data-container that knows how to expose inputs, outputs, etc.)\n\n\n## Python Usage\n\nThe above doesn't require you to install the Container Database (cdb) metadata\ngenerator, however if you want to (to develop or otherwise interact) you\ncan do the following. First, install cdb from pypi or a local repository:\n\n```bash\n$ pip install cdb\n```\nor\n\n```bash\ngit clone git@github.com:vsoch/cdb\ncd cdb\npip install -e .\n```\n\n### Command Line \n\nThe next step is to generate the goLang file to compile.\nYou'll next want to change directory to somewhere you have a dataset folder.\nFor example, in [tests](tests) we have a dummy \"data\" folder.\n\n```bash\ncd tests/\n```\n\nWe might then run `cdb generate` to create a binary for our container, targeting\nthe [tests/data](tests/data) folder:\n\n```bash\n$ cdb generate data --out db.go\n```\n\nThe db.go file is then in the present working directory. You can either\nbuild it during a multistage build as is done in the [Dockerfile](Dockerfile),\nor do it locally with your own GoLang install and then add to the container.\nFor example, to compile:\n\n```bash\ngo get github.com/vsoch/containerdb \u0026\u0026 \\\nGOOS=linux GOARCH=amd64 go build -ldflags=\"-w -s\" -o /db -i /db.go\n```\n\nAnd then a very basic Dockerfile would need to add the data at the path specified,\nand the compiled entrypoint.\n\n```bash\nFROM scratch\nWORKDIR /data\nCOPY data/ .\nCOPY db /db\nCMD [\"/db\"]\n```\n\nA more useful entrypoint will be developed soon! This is just a very\nbasic start to the library.\n\n### Python\n\nYou can run the same generation functions interactively with Python.\n\n```python\nfrom cdb.main import ContainerDatabase\ndb = ContainerDatabase(path=\"data\")\n# \u003ccdb.main.ContainerDatabase at 0x7fcaa9cb8950\u003e\n```\n\nView that there is a files generator at `db.files`\n\n```python\ndb.files\n\u003cgenerator object recursive_find at 0x7fcaaa4ae950\u003e\n```\n\n\nAnd then generate! If you don't provide an output file, a string will be returned.\nOtherwise, the output file name is returned.\n\n```python\noutput = db.generate(output=\"db.go\", force=True)\n```\n\nCurrently, functions for parsing metadata are named in [cdb/functions.py](cdb/functions.py),\nhowever you can also define a custom import path. This has not yet been tested \nand will be soon. We will also be added more real world examples soon.\n\n## License\n\n * Free software: MPL 2.0 License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsoch%2Fcdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvsoch%2Fcdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsoch%2Fcdb/lists"}