{"id":23278547,"url":"https://github.com/hdfgroup/vol-daos","last_synced_at":"2025-08-21T11:31:38.065Z","repository":{"id":38330462,"uuid":"258642383","full_name":"HDFGroup/vol-daos","owner":"HDFGroup","description":"HDF5 VOL connector for DAOS","archived":false,"fork":false,"pushed_at":"2023-09-28T21:52:42.000Z","size":22281,"stargazers_count":5,"open_issues_count":12,"forks_count":11,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-05-09T11:13:02.747Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HDFGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2020-04-24T22:58:50.000Z","updated_at":"2022-11-03T04:02:34.000Z","dependencies_parsed_at":"2023-02-18T03:15:55.732Z","dependency_job_id":null,"html_url":"https://github.com/HDFGroup/vol-daos","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-daos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-daos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-daos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-daos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HDFGroup","download_url":"https://codeload.github.com/HDFGroup/vol-daos/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230508023,"owners_count":18237108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-19T22:32:17.078Z","updated_at":"2024-12-19T22:32:17.724Z","avatar_url":"https://github.com/HDFGroup.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HDF5 DAOS VOL connector\n\n[![Latest version][vol-daos-release-svg]][vol-daos-release-link]\n\n## Table of Contents\n1. [Description](#1-Description)\n2. [Installation](#2-Installation)\n    * [Prerequisites](#Prerequisites)\n    * [Build instructions](#Build-instructions)\n    * [CMake options](#CMake-options)\n    * [Connector options](#Connector-options)\n3. [Testing and Usage](#3-Testing-And-Usage)\n4. [More information](#4-More-Information)\n    * [DAOS VOL](#DAOS-VOL)\n    * [DAOS](#DAOS)\n\n## 1. Description\n\nThe HDF5 DAOS VOL connector is a Virtual Object Layer (VOL) connector for HDF5\nthat allows for direct interfacing with the Distributed Asynchronous Object\nStorage (DAOS) system, bypassing both MPI I/O and POSIX for efficient and\nscalable I/O, removing the limitations of the native HDF5 file format and\nenabling new features such as independent creation of objects in parallel,\nkey-value store objects, data recovery, asynchronous I/O, etc.\n\nApplications already using HDF5 can, using this VOL connector and a DAOS-enabled\nsystem, benefit of some of these features with minimal code changes.\nThe connector is built as a plugin library that is external to HDF5, meaning\nthat it must be dynamically-loaded in the same fashion as HDF5 filter plugins.\n\n## 2. Installation\n\nBelow is set a of instructions that is compiled to provide a minimal\ninstallation of the DAOS VOL connector on a DAOS-enabled system.\n\n### Prerequisites\n\nTo build the DAOS VOL connector, the following libraries are required:\n\n+ `libhdf5` - The [HDF5](https://www.hdfgroup.org/downloads/hdf5/) library.\n            Minimum version required is 1.14.0, compiled with\n            support for both parallel I/O and map objects \n            (i.e., `-DHDF5_ENABLE_MAP_API=ON` CMake option).\n\n+ `libdaos` - The [DAOS](https://github.com/daos-stack/daos) library.\n            Minimum version required is 1.3.106-tb.\n\n+ `libuuid` - UUID support.\n\nCompiled libraries must either exist in the system's library paths or must be\npointed to during the DAOS VOL connector build process.\n\n### Build instructions\n\nThe HDF5 DAOS VOL connector is built using CMake. CMake version 2.8.12.2 or\ngreater is required for building the connector itself, but version 3.1 or\ngreater is required to build the connector's tests.\n\nIf you install the full sources, put the tarball in a directory where you have permissions (e.g., your home directory) and unpack it:\n\n    gzip -cd hdf5_vol_daos-X.tar.gz | tar xvf -\n\nor\n\n    bzip2 -dc hdf5_vol_daos-X.tar.bz2 | tar xvf -\n\nReplace 'X' with the version number of the package.\n\nAfter obtaining the connector's source code, you can create a build directory\nwithin the source tree and run the `ccmake` or `cmake` command from it:\n\n    cd hdf5_vol_daos-X\n    mkdir build\n    cd build\n    ccmake ..\n\nIf using `ccmake`, type `'c'` multiple times and choose suitable options or if\nusing `cmake`, pass these options with `-D`. Some of these options may be needed\nif, for example, the required components mentioned previously are not located in\ndefault paths.\n\nSetting include directory and library paths may require you to toggle to\nthe advanced mode by typing `'t'`. Once you are done and do not see any\nerrors, type `'g'` to generate makefiles. Once you exit the CMake\nconfiguration screen and are ready to build the targets, do:\n\n    make\n\nVerbose build output can be generated by appending `VERBOSE=1` to the\n`make` command.\n\nAssuming that the `CMAKE_INSTALL_PREFIX` has been set\nand that you have write permissions to the destination directory, you can\ninstall the connector by simply doing:\n\n     make install\n\n### CMake options\n\n  * `CMAKE_INSTALL_PREFIX` - This option controls the install directory that the resulting output files are written to. The default value is `/usr/local`. \n  * `CMAKE_BUILD_TYPE` - This option controls the type of build used for the VOL connector. Valid values are `Release`, `Debug`, `RelWithDebInfo`,\n  `MinSizeRel`, `Ubsan`, `Asan`; the default build type is `RelWithDebInfo`.\n\n### Connector options\n\n  * `BUILD_TESTING` - This option is used to enable/disable building of the\n  DAOS VOL connector's tests. The default value is `ON`.\n  * `BUILD_EXAMPLES` - This option is used to enable/disable building of the\n  DAOS VOL connector's HDF5 examples. The default value is `OFF`.\n  * `HDF5_C_COMPILER_EXECUTABLE` - This option controls the HDF5 compiler\n  wrapper script used by the VOL connector build process. It should be set to\n  the full path to the HDF5 compiler wrapper (usually `bin/h5cc`), including\n  the name of the wrapper script. The following two options may also need to be\n  set.\n  * `HDF5_C_LIBRARY_hdf5` - This option controls the HDF5 library used by the\n  VOL connector build process. It should be set to the full path to the HDF5\n  library, including the library's name (e.g., `/path/libhdf5.so`). Used in\n  conjunction with the `HDF5_C_INCLUDE_DIR` option.\n  * `HDF5_C_INCLUDE_DIR` - This option controls the HDF5 include directory used\n  by the VOL connector build process. Used in conjunction with the\n  `HDF5_C_LIBRARY_hdf5` variable.\n  * `DAOS_LIBRARY` - This option controls the DAOS library used by the VOL\n  connector build process. It should be set to the full path to the DAOS\n  library, including the library's name (e.g., `/path/libdaos.so`). Used in\n  conjunction with the `DAOS_UNS_LIBRARY` and `DAOS_INCLUDE_DIR` options.\n  * `DAOS_UNS_LIBRARY` - This option controls the DAOS unified namespace library\n  used by the VOL connector build process. It should be set to the full path to\n  the DAOS `libduns` library, including the library's name\n  (e.g., `/path/libduns.so`). Used in conjunction with the `DAOS_LIBRARY` and `DAOS_INCLUDE_DIR` options.\n  * `DAOS_INCLUDE_DIR` - This option controls the DAOS include directory used by\n  the VOL connector build process. Used in conjunction with the `DAOS_LIBRARY`\n  and `DAOS_UNS_LIBRARY` options.\n  * `MPI_C_COMPILER` - This option controls the MPI C compiler used by the VOL\n  connector build process. It should be set to the full path to the MPI C\n  compiler, including the name of the executable.\n\n## 3. Testing and Usage\nIn the connector, each chunk is stored in a different DAOS dkey, and data in a single dkey is stored in a single DAOS storage target. Therefore, splitting the data into different chunks stripes the data across different dkeys and different storage targets. This improves I/O performance by allowing DAOS to read from or write to multiple storage targets at once.\n\n\nThe bandwidth improvement from using different storage targets is so vital that, if *h5pset_chunk*() is not used, i.e., contiguous datasets, the connector will automatically set a chunk size. The connector, by default, tries to size these chunks to approximately 1 MiB. The environment variable **HDF5_DAOS_CHUNK_TARGET_SIZE** (in bytes) sets the chunk target size. Setting this variable to 0 disables automatic chunking, and contiguous datasets will stay contiguous (and will therefore only be stored on a single storage target). Better performance may be obtained by choosing a larger chunk target size, such as 4-8 MiB.\n\nFor further information on how to use the DAOS VOL connector with an HDF5 application,\nas well as how to test that the VOL connector is functioning properly, please\nrefer to the DAOS VOL User's Guide under _docs/users_guide.pdf_.\n\n## 4. More Information\n\n### DAOS VOL\nDesign documentation for the DAOS VOL can be found under _docs/design_doc.pdf_.\n\nJournal paper:\n* J. Soumagne, J. Henderson, M. Chaarawi, N. Fortner, S. Breitenfeld, S. Lu, D. Robinson, E. Pourmal, J. Lombardi, \"__Accelerating HDF5 I/O for Exascale Using DAOS__,\" in _IEEE Transactions on Parallel and Distributed Systems_, vol. 33, no. 4, pp. 903-914, April 2022. | [paper][doi_paper] |\n\n### DAOS\nDAOS installation and usage instructions can\nbe found on the DAOS website: https://docs.daos.io/\n\n[vol-daos-release-svg]: https://img.shields.io/github/release/HDFGroup/vol-daos/all.svg\n[vol-daos-release-link]: https://github.com/HDFGroup/vol-daos/releases\n[doi_paper]: https://dx.doi.org/10.1109/TPDS.2021.3097884\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhdfgroup%2Fvol-daos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhdfgroup%2Fvol-daos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhdfgroup%2Fvol-daos/lists"}