{"id":21438154,"url":"https://github.com/bsc-dom/optanedc-miniapps","last_synced_at":"2025-08-12T00:36:41.500Z","repository":{"id":70631994,"uuid":"251566231","full_name":"bsc-dom/optanedc-miniapps","owner":"bsc-dom","description":"Applications for Optane DC evaluation with dataClay","archived":false,"fork":false,"pushed_at":"2020-12-02T09:46:26.000Z","size":223,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-23T09:44:58.969Z","etag":null,"topics":["nvm","optane-dc"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bsc-dom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-31T10:08:44.000Z","updated_at":"2020-12-02T09:46:29.000Z","dependencies_parsed_at":"2023-02-21T14:45:13.877Z","dependency_job_id":null,"html_url":"https://github.com/bsc-dom/optanedc-miniapps","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsc-dom%2Foptanedc-miniapps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsc-dom%2Foptanedc-miniapps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsc-dom%2Foptanedc-miniapps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bsc-dom%2Foptanedc-miniapps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bsc-dom","download_url":"https://codeload.github.com/bsc-dom/optanedc-miniapps/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243950809,"owners_count":20373664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nvm","optane-dc"],"created_at":"2024-11-23T00:33:24.019Z","updated_at":"2025-03-16T23:43:03.478Z","avatar_url":"https://github.com/bsc-dom.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# optanedc-miniapps\n\nApplications for Optane DC evaluation with [dataClay](https://dataclay.bsc.es/) active object store.\n\nThe following applications can be found in this repository:\n\n - `histogram` a histogram application\n - `kmeans` a _k_-means implementation using the standard algorithm\n - `matsum` a matrix addition\n - `matmul` a matrix multiplication\n\nNote that all aplications are self-contained, meaning that the data is randomly\ngenerated. The timing is based on calls to `time.time()` (wall clock time).\n\nThe NVM placement hints for the applications are done through the usage of the\n[`npp2nvm` python library](https://github.com/bsc-dom/npp2nvm/).\n\n## Registration of the data model\n\nAfter initializing the dataClay stack, you need to register the data model. Each\n`cfgfiles` folder contains a default configuration for the client which assumes\nthat dataClay is started at localhost.\n\nExecution of the `register.sh` will register the data model in dataClay. Each application\nhas its own script. You will need to have the dataClay client libraries installed.\n\n## Execution of the application\n\nEach application folder has its own python main script. After completing the registration\nof the data model you can execute the application by doing:\n\n    $ python histogram.py\n\nChange `histogram.py` depending on the application.\n\nThe application will generate the random data and execute the application. When the application\nfinishes, it will print in `stdout` the execution time.\n\nIf you need to change the dataset size, edit the python script and change the constants value (near\nthe beginning).\n\n## Kernel evaluation\n\nEach application has a `kernel_eval.py` script that can be used in order to evaluate the isolated performance of the kernel routines in different scenarios.\n\nYou will need to set up the appropriate environment variables per each application and per each scenario evaluation.\n\n**Caution!** Memory Mode may be tricky to evaluate and probably you may want to consider two different scenarios: fits-in-DRAM and does-not-fit-in-DRAM. You can force each of those two scenarios by changing the `NUMBER_OF_ITERATIONS` environment variable.\n\n## Analysis\n\nThere is a folder `analysis` where there are some Jupyter notebooks used for analysis and\nfigures generation.\n\nYou will need to install install `jupyter` and `seaborn` in order to use it.\n\nThe notebooks expect to find a csv file with the results, in that same folder. The column names \nfor the csv will depend for each application but in all of them you will need to include the \ndataset size and the _mode_ (e.g. `ad-wram` or `mm`).\n\n# Usage of DAOS -- Optane DC\n\n## Prerequisites\n\nInstall DAOS and its Python bindings. You may follow the \n[official documentation](https://daos-stack.github.io/admin/installation/)\nand be sure that at the end you have the Python module installed. You may need\nto update your `PYTHONPATH` environment variable (YMMV).\n\n## Preparing the configuration\n\nWe will be using a DCPM-only configuration of DAOS --as this will\nresult in a meaningful and fair comparison. You should tweak the\nconfiguration files for DAOS into something like the following:\n\n```yaml\n# daos_server.yaml\n\n# (...)\n\ntransport_config:\n   allow_insecure: true\n\nsocket_dir: # \u003c-- default /var/run only while using root!\n\nservers:\n  - #(...)\n    scm_class: dcpm\n    # Update according to your hardware/OS configuration:\n    scm_mount: /mnt/daos0 # map to -s /mnt/daos\n    scm_list: [/dev/pmem0.2]\n```\n\n```yaml\n# daos_agent.yaml\nruntime_dir: # \u003c-- default /var/run only while using root!\n             # (match the one in daos_server.yml)\n```\n\n## Starting the DAOS stack\n\nDo the following:\n\n```\n$ daos_server start -o install/etc/daos_server.yml\n$ dmg -i storage prepare\n$ dmg -i storage format\n$ daos_agent -i -o install/etc/daos_agent.yml\n```\n\nIf you are restarting the stack, you may need to skip the `prepare` step and instead change the format into `dmg storage -i format --reformat`. Also you may need to clean up files from `/mnt/daos0`. Keep all that in mind. \n\nI recommend using tmux/screen/byobu or similar to keep both `daos_server` and `daos_agent` commands running in foreground/persistent --you don't want them to be killed due to ssh/network hiccups on your side.\n\n## Preparing the _pool_ and the _container_\n\nBefore starting the Python applications, you need to define pool and container. I will be using the daos management tool. First, the _pool_:\n\n```\n$ dmg pool create -i --scm-size=800G --nvme-size=0G\n```\n\nAs stated above, it will be a DCPM-only setup, thus the `nvme-size=0` parameter.\n\nThis, if succeeds, will spit a UUID for the created pool. Example output:\n\n```\nPool-create command SUCCEEDED: UUID: d8c2786c-75e4-4fd3-914c-a76c49d10676, Service replicas: 0\n```\n\n**The scripts manage the container by themselves, the following steps are not required\nfor typical application benchmarking**\n\nThen use it to create the _container_:\n\n```\n$ daos cont create --pool=d8c2786c-75e4-4fd3-914c-a76c49d10676 --svc=0\nSuccessfully created container 886a015d-aa9b-4f77-b94b-ea750b6fbaa1\n```\n\nExport the variables and you are ready to start the applications:\n\n```\n$ export DAOS_POOL=d8c2786c-75e4-4fd3-914c-a76c49d10676\n# DAOS_CONT is not required, current scripts manage those\n$ export DAOS_CONT=886a015d-aa9b-4f77-b94b-ea750b6fbaa1\n```\n\nOf course, you will need to change all the UUID to use the ones that you get.\n\n## Start the applications\n\nEnsure that you have `DAOS_POOL` environment defined (check `env`) and ensure that `daos` python module is available (`import pydaos` in a Python interpreter should not give an `ImportError`).\n\nEach application has a `\u003capp_name\u003e_daos.py` Python application that runs the application. E.g.:\n\n```\n$ cd apps/matsum\n$ python matsum_daos.py\nStarting experiment with the following:\n(...)\n-----------------------------------------\n-------------- RESULTS ------------------\n-----------------------------------------\nInitialization time: ###.######\nMatsum time: ###.######\n-----------------------------------------\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbsc-dom%2Foptanedc-miniapps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbsc-dom%2Foptanedc-miniapps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbsc-dom%2Foptanedc-miniapps/lists"}