{"id":23278474,"url":"https://github.com/hdfgroup/vol-rest","last_synced_at":"2025-08-21T11:31:28.648Z","repository":{"id":56367698,"uuid":"263087727","full_name":"HDFGroup/vol-rest","owner":"HDFGroup","description":"HDF5 REST VOL Connector","archived":false,"fork":false,"pushed_at":"2025-01-10T22:26:18.000Z","size":39472,"stargazers_count":5,"open_issues_count":8,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-08-12T21:11:38.105Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HDFGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-11T15:49:04.000Z","updated_at":"2025-01-10T22:26:23.000Z","dependencies_parsed_at":"2022-08-15T17:30:38.857Z","dependency_job_id":"f12efa93-c3b2-4aa5-9177-5ec62ef15671","html_url":"https://github.com/HDFGroup/vol-rest","commit_stats":{"total_commits":417,"total_committers":8,"mean_commits":52.125,"dds":0.3932853717026379,"last_synced_commit":"82f89c3efe2807354ab59af00d6739aca328ed51"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HDFGroup/vol-rest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-rest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-rest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-rest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-rest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HDFGroup","download_url":"https://codeload.github.com/HDFGroup/vol-rest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fvol-rest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271470103,"owners_count":24765315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-19T22:31:45.501Z","updated_at":"2025-08-21T11:31:28.636Z","avatar_url":"https://github.com/HDFGroup.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HDF5 REST VOL connector \u003c!-- omit in toc --\u003e\r\n\r\n[![build status](https://img.shields.io/github/actions/workflow/status/HDFGroup/vol-rest/main.yml?branch=master\u0026label=build%20and%20test)](https://github.com/HDFGroup/vol-rest/actions?query=branch%3Amaster)\r\n\r\n### Table of Contents: \u003c!-- omit in toc --\u003e\r\n\r\n- [I. Introduction](#i-introduction)\r\n- [II. Installation](#ii-installation)\r\n  - [II.A. Prerequisites](#iia-prerequisites)\r\n    - [II.A.i. External Libraries](#iiai-external-libraries)\r\n    - [II.A.ii. HDF5 REST API server access](#iiaii-hdf5-rest-api-server-access)\r\n  - [II.B. Building the REST VOL connector](#iib-building-the-rest-vol-connector)\r\n    - [II.B.i. Obtaining the Source](#iibi-obtaining-the-source)\r\n    - [II.B.ii. One-step Build](#iibii-one-step-build)\r\n      - [II.B.ii.a. Build Script Options](#iibiia-build-script-options)\r\n    - [II.B.iii. Manual Build](#iibiii-manual-build)\r\n      - [II.B.iii.a Manual Build with Autotools](#iibiiia-manual-build-with-autotools)\r\n      - [II.B.iii.b Autotools options](#iibiiib-autotools-options)\r\n      - [II.B.iii.c Manual Build with CMake](#iibiiic-manual-build-with-cmake)\r\n      - [II.B.iii.d CMake options](#iibiiid-cmake-options)\r\n    - [II.B.iv. Building at HDF5 Build Time](#iibiv-building-at-hdf5-build-time)\r\n    - [II.B.v. Build Results](#iibv-build-results)\r\n- [III. Using/Testing the REST VOL connector](#iii-usingtesting-the-rest-vol-connector)\r\n- [IV. More Information](#iv-more-information)\r\n\r\n# I. Introduction\r\n\r\nThe HDF5 REST VOL connector is a plugin for HDF5 designed with the goal of\r\nallowing HDF5 applications to utilize web-based storage systems by translating\r\nHDF5 API calls into HTTP-based REST calls, as defined by the HDF5 REST API\r\n(See section V. for more information on RESTful HDF5).\r\n\r\nUsing a VOL connector allows an existing HDF5 application to interface with\r\ndifferent storage systems with minimal changes necessary. The connector accomplishes\r\nthis by utilizing the HDF5 Virtual Object Layer in order to re-route HDF5's\r\npublic API calls to specific callbacks in the connector which handle all of the\r\nusual HDF5 operations. The HDF5 Virtual Object Layer is an abstraction layer\r\nthat sits directly between HDF5's public API and the underlying storage system.\r\nIn this manner of operation, the mental data model of an HDF5 application can\r\nbe preserved and transparently mapped onto storage systems that differ from a\r\nnative filesystem, such as Amazon's S3.\r\n\r\nThe REST VOL connector is under development, and details given here may change.\r\n\r\n--------------------------------------------------------------------------------\r\n\r\n# II. Installation\r\n\r\nNotes and instructions related to obtaining, building and installing the REST VOL\r\nconnector.\r\n\r\n## II.A. Prerequisites\r\n\r\nBefore building and using the HDF5 REST VOL connector, a few requirements must be met.\r\n\r\n### II.A.i. External Libraries\r\n\r\nTo build the REST VOL connector, the following libraries are required:\r\n\r\n+ libhdf5 - The [HDF5](https://www.hdfgroup.org/downloads/hdf5/) library. The HDF5 library\r\n            used must be at least version 1.14.0. Only the shared library should be built for\r\n            maximal compatibility with the REST VOL connector. Using statically-built HDF5 \r\n            libraries can cause issues with the REST VOL connector under certain circumstances. \r\n            Additionally, the HDF5 library must have its high-level interface enabled at build time\r\n            unless the REST VOL examples are disabled.\r\n\r\n+ libcurl (ver. 7.61.0 or greater) - networking support\r\n    + https://curl.haxx.se/\r\n\r\n+ libyajl (ver. 2.0.4 or greater) - JSON parsing and construction\r\n    + https://lloyd.github.io/yajl/\r\n\r\nCompiled libraries must either exist in the system's library paths or must be\r\npointed to during the REST VOL connector build process. Refer to section II.B.ii.\r\nbelow for more information.\r\n\r\n\r\n### II.A.ii. HDF5 REST API server access\r\n\r\nThe HDF5 REST VOL connector requires access to a server which implements\r\nthe HDF5 REST API. The Highly Scalable Data Service (HSDS) is one such server.\r\n\r\nFor more information on The HDF Group's officially supported service, please see\r\nhttps://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/.\r\n\r\n\r\n## II.B. Building the REST VOL connector\r\n\r\n### II.B.i. Obtaining the Source\r\n\r\nThe latest and most up-to-date REST VOL connector code can be viewed at:\r\n\r\nhttps://github.com/HDFGroup/vol-rest\r\n\r\nand can directly be obtained from:\r\n\r\n`git clone https://github.com/HDFGroup/vol-rest`\r\n\r\nA source distribution of the HDF5 library has been included in the REST VOL connector\r\nsource in the `/src/hdf5` directory.\r\n\r\n### II.B.ii. One-step Build\r\n\r\nUse one of the supplied Autotools or CMake build scripts, depending on preference or\r\nsystem support.\r\n\r\n+ Autotools\r\n\r\n    Run `build_vol_autotools.sh`. See section II.B.ii.a for configuration options.\r\n\r\n\r\n+ CMake\r\n\r\n    Run `build_vol_cmake.sh` (Linux or OS X) or `build_vol_cmake.bat` (Windows).\r\n    See section II.B.ii.a for configuration options.\r\n\r\n\r\nBy default, these build scripts will compile and link with the provided HDF5 source\r\ndistribution. However, if you wish to use a manually built version of the HDF5 library,\r\ninclude the flag `-H \u003cdir\u003e` where `dir` is the path to the HDF5 install prefix.\r\n\r\nNOTE: For those who are capable of using both build systems, the autotools build currently\r\ndoes not support out-of-tree builds. If the REST VOL source directory is used for an autotools\r\nbuild, it is important not to reuse the source directory for a later build using CMake.\r\nThis can cause build conflicts and result in strange and unexpected behavior.\r\n\r\n\r\n#### II.B.ii.a. Build Script Options\r\n\r\nThe following configuration options are available to all of the build scripts:\r\n\r\n    -h      Prints out a help message indicating script usage and available options.\r\n\r\n    -d      Enables debugging information printouts within the REST VOL connector.\r\n\r\n    -c      Enables debugging information printouts from cURL within the REST VOL connector.\r\n\r\n    -m      Enables memory usage tracking within the REST VOL connector. This option is\r\n            mostly useful in helping to diagnose any possible memory leaks or other\r\n            memory errors within the connector.\r\n\r\n    -P DIR  Specifies where the REST VOL connector should be installed. The default\r\n            installation prefix is `rest_vol_build` inside the REST VOL connector source\r\n            root directory.\r\n\r\n    -H DIR  Prevents building of the provided HDF5 source. Instead, uses the compiled\r\n            library found at directory `DIR`, where `DIR` is the path used as the\r\n            installation prefix when building HDF5 manually.\r\n\r\n    -C DIR  Specifies the top-level directory where cURL is installed. Used if cURL is\r\n            not installed to a system path or used to override\r\n\r\n    -Y DIR  Specifies the top-level directory where YAJL is installed. Used if YAJL is\r\n            not installed to a system path or used to override\r\n\r\nThe following configuration options are specific to `build_vol_autotools.sh`:\r\n\r\n    -g      Enables symbolic debugging of the REST VOL code.\r\n\r\nThe following configuration options are specific to the CMake build scripts:\r\n\r\n    -u      Specifies that a static cURL library should be used as a dependency.\r\n\r\n    -t      Specifies that a static YAJL library should be used as a dependency.\r\n\r\n    -B DIR  Specifies the directory that CMake should use as the build tree location.\r\n            The default build tree location is `rest_vol_cmake_build_files` inside the\r\n            REST VOL connector source root directory. Note that the REST VOL does not\r\n            support in-source CMake builds.\r\n\r\n    -G DIR  Specifies the CMake Generator to use when generating the build files\r\n            for the project. On Unix systems, the default is \"Unix Makefiles\" and if\r\n            this is not changed, the build script will automatically attempt to build\r\n            the project after generating the Makefiles. If the generator is changed, the\r\n            build script will only generate the build files and the build command to\r\n            build the project will have to be run manually.\r\n\r\n### II.B.iii. Manual Build\r\n\r\nIn general, the process for building the REST VOL connector involves either obtaining a VOL-enabled\r\nHDF5 distribution or building one from source. Then, the REST VOL connector is built using that\r\nHDF5 distribution by including the appropriate header files and linking against the HDF5 library.\r\n\r\nOnce you have a VOL-enabled HDF5 distribution available, follow the instructions below for your\r\nrespective build system in order to build the REST VOL connector against the HDF5 distribution.\r\n\r\n#### II.B.iii.a Manual Build with Autotools\r\n\r\nTo perform a manual build of the REST VOL using autotools:\r\n\r\n```bash\r\n$ cd rest-vol\r\n$ ./autogen.sh\r\n$ ./configure --prefix=INSTALL_DIR --with-hdf5=HDF5_DIR [options]\r\n$ make\r\n$ make check (requires HDF5 REST API server access -- see section II.A.ii.)\r\n$ make install\r\n```\r\n\r\n#### II.B.iii.b Autotools Options\r\n\r\nWhen building the REST VOL connector manually using Autotools, the following options are\r\navailable to `configure`:\r\n\r\n    -h, --help      Prints out a help message indicating script usage and available\r\n                    options.\r\n\r\n    --prefix=DIR    Specifies the location for the resulting files. The default location\r\n                    is `rest_vol_build` in the same directory as configure.\r\n\r\n    --enable-build-mode=(production|debug)\r\n                    Sets the build mode to be used.\r\n                    Debug - enable debugging printouts within the REST VOL connector.\r\n                    Production - Focus more on optimization.\r\n\r\n    --enable-curl-debug\r\n                    Enables debugging information printouts from cURL within the\r\n                    REST VOL connector.\r\n\r\n    --enable-mem-tracking\r\n                    Enables memory tracking within the REST VOL connector. This option is\r\n                    mostly useful in helping to diagnose any possible memory leaks or\r\n                    other memory errors within the connector.\r\n\r\n    --enable-tests\r\n                    Enables/Disables building of the REST VOL connector tests.\r\n\r\n    --enable-examples\r\n                    Enables/Disables building of the REST VOL HDF5 examples.\r\n\r\n    --with-hdf5=DIR Used to specify the directory where an HDF5 distribution that uses\r\n                    the VOL layer has already been built. This is to help the REST VOL\r\n                    connector locate the HDF5 header files that it needs to include.\r\n\r\n    --with-curl=DIR Used to specify the top-level directory where cURL is installed, if\r\n                    cURL is not installed to a system path.\r\n\r\n    --with-yajl=DIR Used to specify the top-level directory where YAJL is installed, if\r\n                    YAJL is not installed to a system path.\r\n\r\n\r\n#### II.B.iii.c Manual Build with CMake\r\n\r\nFirst, create a build directory within the source tree:\r\n\r\n```bash\r\n$ cd rest-vol\r\n$ mkdir build\r\n$ cd build\r\n```\r\n\r\nThen, if all of the required components (HDF5, cURL and YAJL) are located within the system path,\r\nbuilding the connector should be as simple as running the following two commands to first have CMake\r\ngenerate the build files to use and then to build the connector. If the required components are\r\nlocated somewhere other than the system path, refer to section II.B.iii.b. for information on how to\r\npoint to their locations.\r\n\r\n```bash\r\n$ cmake -DPREBUILT_HDF5_DIR=HDF5_DIR [options] ..\r\n$ make \u0026\u0026 make install (command may differ depending on platform and cmake generator used)\r\n```\r\n\r\nand, optionally, run the following to generate a system package for the REST VOL connector:\r\n\r\n```bash\r\n$ cpack\r\n```\r\n\r\nThe options that can be specified to control the build process are covered in section II.B.iii.d\r\nNote that by default CMake will generate Unix Makefiles for the build, but other build files can\r\nbe generated by specifying the `-G` option for the `cmake` command; \r\nsee [CMake Generators](https://cmake.org/cmake/help/v3.16/manual/cmake-generators.7.html) for more\r\ninformation.\r\n\r\n#### II.B.iii.d CMake Options\r\n\r\nWhen building the REST VOL connector manually using CMake, the following CMake variables are\r\navailable for controlling the build process. These can be supplied to the `cmake` command by\r\nprepending them with `-D`. Some of these options may be needed if, for example, the required\r\ncomponents mentioned previously cannot be found within the system path.\r\n\r\n  * `CMAKE_INSTALL_PREFIX` - This option controls the install directory that the resulting output files are written to. The default value is `/usr/local`.\r\n  * `CMAKE_BUILD_TYPE` - This option controls the type of build used for the VOL connector. Valid values are Release, Debug, RelWithDebInfo and MinSizeRel; the default build type is RelWithDebInfo.\r\n  * `PREBUILT_HDF5_DIR` - Specifies a directory which contains a pre-built HDF5 distribution which uses the VOL abstraction layer. By default, the REST VOL connector's CMake build will attempt to build the included HDF5 source distribution, then use that to build the connector itself. However, if a VOL-enabled HDF5 distribution is already available, this option can be set to point to the directory of the HDF5 distribution. In this case, CMake will use that HDF5 distribution to build the REST VOL connector and will not attempt to build HDF5 again.\r\n  * `BUILD_TESTING` - This option is used to enable/disable building of the REST VOL connector's tests. The default value is `ON`.\r\n  * `BUILD_EXAMPLES` - This option is used to enable/disable building of the REST VOL connector's HDF5 examples. The default value is `ON`.\r\n  * `BUILD_SHARED_LIBS` - This option is used to enable/disable building the REST VOL connector's shared library. The default value is `ON`.\r\n  * `BUILD_STATIC_LIBS` - This option is used to enable/disable building the REST VOL connector's static library. The default value is `OFF`.\r\n  * `BUILD_STATIC_EXECS` - This option is used to enable/disable building the REST VOL connector's static executables. The default value is `OFF`.\r\n  * `HDF5_VOL_REST_ENABLE_COVERAGE` - Enables/Disables code coverage for HDF5 REST VOL connector libraries and programs. The default value is `OFF`.\r\n  * `HDF5_VOL_REST_ENABLE_DEBUG` - Enables/Disables debugging printouts within the REST VOL connector. The default value is `OFF`.\r\n  * `HDF5_VOL_REST_ENABLE_EXAMPLES` - Indicate that building of the examples should be enabled. The default value is `ON`.\r\n  * `HDF5_VOL_REST_ENABLE_CURL_DEBUG` - Enables/Disables debugging information printouts from cURL within the REST VOL connector. The default value is `OFF`.\r\n  * `HDF5_VOL_REST_ENABLE_MEM_TRACKING` - Enables/Disables memory tracking within the REST VOL connector. This option is mostly useful in helping to diagnose any possible memory leaks or other memory errors within the connector. The default value is `OFF`.\r\n  * `HDF5_VOL_REST_THREAD_SAFE` - Enables/Disables linking to HDF5 statically compiled with thread safe option. The default value is `OFF`.\r\n  * `YAJL_USE_STATIC_LIBRARIES` - Indicate if the static YAJL libraries should be used for linking. The default value is `OFF`.\r\n  * `CURL_USE_STATIC_LIBRARIES` - Indicate if the static CURL libraries should be used for linking. The default value is `OFF`.\r\n\r\nNote, when setting BUILD_SHARED_LIBS=ON and YAJL_USE_STATIC_LIBRARIES=ON, the static YAJL libraries have be build with the position independent code (PIC) option enabled. In the static YAJL build,\r\nthis PIC option has been turned off by default.\r\n\r\n### II.B.iv. Building at HDF5 Build Time\r\n\r\nIt is also possible to build the REST VOL as part of the build process for the HDF5 library, using CMake's FetchContent module. This can be done using a local copy of the REST VOL's source code, or by providing the information for the repository to be automatically cloned from a branch of a Github repository. For full instructions on this process, see [Building and testing HDF5 VOL connectors with CMake FetchContent](https://github.com/HDFGroup/hdf5/blob/develop/doc/cmake-vols-fetchcontent.md).\r\n\r\n### II.B.v. Build Results\r\n\r\nIf the build is successful, the following files will be written into the installation directory on a Linux or OSX machine:\r\n\r\n```\r\nbin/\r\n\r\ninclude/\r\n     rest_vol_config.h - The header file containing the configuration options for the built REST VOL connector\r\n     rest_vol_public.h - The REST VOL connector's public header file to include in HDF5 applications\r\n\r\nlib/\r\n    pkgconfig/\r\n        hdf5_vol_rest-\u003cversion\u003e.pc - The REST VOL connector pkgconfig file\r\n\r\n    libhdf5_vol_rest.a - The REST VOL connector static library\r\n    libhdf5_vol_rest.settings - The REST VOL connector build settings\r\n    libhdf5_vol_rest.so - The REST VOL connector shared library\r\n\r\nshare/\r\n    cmake/\r\n        hdf5_vol_rest/\r\n            hdf5_vol_rest-config.cmake\r\n            hdf5_vol_rest-config-version.cmake\r\n            hdf5_vol_rest-targets.cmake\r\n            hdf5_vol_rest-targets-\u003cbuild mode\u003e.cmake\r\n```\r\n\r\nOn a Windows system, the installation directory will have a few differences. If shared libraries are built, the resulting `hdf5_vol_rest.dll` will be found in the `bin` directory. If tests are enabled, `test_rest_vol(-shared).exe` will also be in the `bin` directory. If static libraries are built, `lib` will contain `libhdf5_vol_rest.lib`.\r\n\r\nIf the REST VOL connector was built using one of the included build scripts, all of the usual files\r\nfrom an HDF5 source build should appear in the respective `bin`, `include`, `lib` and `share`\r\ndirectories in the install directory. Notable among these is `bin/h5cc`, a special-purpose compiler wrapper script that streamlines the process of building HDF5 applications.\r\n\r\n\r\n--------------------------------------------------------------------------------\r\n\r\n# III. Using/Testing the REST VOL connector\r\n\r\nFor information on how to use the REST VOL connector with an HDF5 application,\r\nas well as how to test that the connector is functioning properly, please refer\r\nto the REST VOL User's Guide under `docs/users_guide.pdf`.\r\n\r\n\r\n--------------------------------------------------------------------------------\r\n\r\n# IV. More Information\r\n\r\n+ Highly Scalable Data Service (HSDS) - A python-based implementation of the HDF5 REST API which\r\n  can send and receive HDF5 data through the use of HTTP requests\r\n    + https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/\r\n    + https://github.com/HDFGroup/hsds\r\n    + https://www.youtube.com/watch?v=9b5TO7drqqE\r\n\r\n+ RESTful HDF5 - A description of the HDF5 REST API\r\n    + https://support.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf\r\n    + http://hdf-rest-api.readthedocs.io/en/latest/\r\n\r\n+ HDF5-JSON - A specification of and tools for representing HDF5 in JSON\r\n    + http://hdf5-json.readthedocs.io/en/latest/\r\n\r\n+ HDF in the Cloud\r\n  + https://www.slideshare.net/HDFEOS/hdf-cloud-services\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhdfgroup%2Fvol-rest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhdfgroup%2Fvol-rest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhdfgroup%2Fvol-rest/lists"}