{"id":23278492,"url":"https://github.com/hdfgroup/hcl","last_synced_at":"2025-10-04T19:46:15.088Z","repository":{"id":51142769,"uuid":"214226252","full_name":"HDFGroup/hcl","owner":"HDFGroup","description":"Hermes Container Library","archived":false,"fork":false,"pushed_at":"2021-05-21T15:39:31.000Z","size":152,"stargazers_count":5,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"dev","last_synced_at":"2025-09-08T20:58:17.606Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HDFGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-10T15:59:27.000Z","updated_at":"2022-05-16T04:05:20.000Z","dependencies_parsed_at":"2022-09-19T05:10:55.115Z","dependency_job_id":null,"html_url":"https://github.com/HDFGroup/hcl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HDFGroup/hcl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fhcl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fhcl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fhcl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fhcl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HDFGroup","download_url":"https://codeload.github.com/HDFGroup/hcl/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HDFGroup%2Fhcl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278366608,"owners_count":25975090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-19T22:31:57.424Z","updated_at":"2025-10-04T19:46:15.070Z","avatar_url":"https://github.com/HDFGroup.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HCL: Hermes Container Library\n\n[![cmake](https://github.com/HDFGroup/hcl/actions/workflows/main.yml/badge.svg)](https://github.com/HDFGroup/hcl/actions/workflows/main.yml)\n\nIn order to keep up with the demand for high performance at extreme-scale,\napplications have become very highly distributed. Distributed applications\ntypically require global coordination for distributed algorithms such as stencil\ncomputations, collective I/O, tridiagonal systems, etc. This coordination is\nachieved using mechanisms such as inter-process communication, global\ndistributed coordinators, or leader election. These coordination mechanisms\nrequire storing state information for consistent global access by multiple\nprocesses via distributed data structure platforms. In order to facilitate the\ncoordination of applications at extreme-scale, we propose Hermes Container\nLibrary (HCL), a user-space platform for distributing data structures. HCL\nprovides wrappers for C++ Standard Library (STL) containers, which it\ndistributes and manages transparently across nodes. HCL has been designed to be\neasy-to-use, highly programmable, and portable. Data access is optimized via a\nhybrid data model of shared memory and RPC. It supports decoupled and ephemeral\ndeployment models, with deployment configured based on application requirements.\nIt's primary goals are to provide\n\n* a familiar STL-like interface\n* a flexible programming paradigm\n* a hybrid data access model optimized for High-Performance Computing (HPC)\n* a high-performance data container infrastructure that leverages new hardware and software innovations (e.g., RDMA, RoCE, one-sided communications)\n\nHCL consists of the following templated data structures:\n\n * global_clock\n * map\n * multimap\n * priority_queue\n * queue\n * global_sequence (sequencer)\n * set\n * unordered_map\n\n## Compilation\n\nThe HCL Library compiles with cmake, so the general procedure is\n\n```bash\ncd hcl\nmkdir build\ncmake -HCL_ENABLE_RPCLIB=true ..\nmake\nsudo make install\n```\nIf you want to install somewhere besides `/usr/local`, then use\n\n```bash\ncd hcl\nmkdir build\ncmake -DCMAKE_INSTALL_PREFIX:PATH=/wherever -DHCL_ENABLE_RPCLIB=true ..\nmake\nmake install\n```\n\nA flag should be added to cmake to indicate the preferred RPC library, otherwise\ncompilation will fail. If compiling with RPCLib, use `-DHCL_ENABLE_RPCLIB`. If\ncompiling with Thallium, use either `-DHCL_ENABLE_THALLIUM_TCP` or\n`-DHCL_ENABLE_THALLIUM_ROCE`\n\n### Dependencies\n- MPI\n- Boost (interprocess module)\n- RPC layer (pick one and compile appropriately)\n  - rpclib\n  - Thallium (wrapper over Mercury)\n- glibc (for librt and posix threads)\n\n#### Recommended Versions\n\nHCL has been tested with mpich 3.3.1, boost 1.69.0, rpclib 2.2.1, mercury 1.0.1,\nmargo 0.5, and thallium 0.4.0. Please consider patching mercury using the patch\nspecified below, especially if you're using the Thallium RoCE transport. Also of\nnote is that we uses libfabric (ofi) as the default Mercury transport (for TCP\nand RoCE (verbs)). We used libfabric version 1.8.x. If you would rather use a\ndifferent Mercury transport, please change the configuration via\ninclude/hcl/common/configuration_manager.h. The TCP_CONF string is used for tcp\nvia Mercury, and the VERBS_CONF string is for verbs via Mercury. The\nVERBS_DOMAIN is the domain used for RoCE.\n\n## Usage\n\nSince libhcl uses MPI, data structures have to be declared on the server and\nclients. In the `test` directory, you will find examples of how to use each data\nstructure, and also for the clock and sequencer. Data structures typically\nassume that we are running the server and clients on the same node, but this\nneed not be the case. You can easily configure clients to work with servers that\nare not on their node.\n\n### Structure Initialization\n\nStructure configuration is managed via the following macros:\n\n``` c++\nHCL_CONF-\u003eIS_SERVER = is_server;\nHCL_CONF-\u003eMY_SERVER = my_server;\nHCL_CONF-\u003eNUM_SERVERS = num_servers;\nHCL_CONF-\u003eSERVER_ON_NODE = server_on_node || is_server;\nHCL_CONF-\u003eSERVER_LIST_PATH = \"./server_list\";\n```\n\n * `IS_SERVER`: `true` when the data structure in the current process will act as a\n   server (store the internal data structure), `false` otherwise.\n\n * `MY_SERVER`: The relative rank of your server (if servers are on ranks 3 and\n   5, server relative ranks are still 0 and 1)\n\n * `NUM_SERVERS`: The total number of servers.\n\n * `SERVER_ON_NODE`: `true` when `MY_SERVER` is on the same node as the current process.\n\n * `SERVER_LIST_PATH`: An absolute or relative path to a file that lists\n   `NUM_SERVERS` hostnames or addresses, one per line, of the server processes.\n\nConstructor example:\n\n``` c++\nhcl::unordered_map(std::string name);\n```\n\n * `name`: A unique name used to identify the shared memory.\n\nSee the [wiki](https://github.com/HDFGroup/hcl/wiki) for more information.\n\n\n## Refer to this work using this citation\n\n### Citation\n```\nH. Devarajan, A. Kougkas, K. Bateman, and X. Sun. \"HCL: Distributing Parallel Data Structures in Extreme Scales.\" In 2020 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 2020.\n```\n\n### Bibtex\n\n```\n@inproceedings{devarajan2020hcl,\n  title={HCL: Distributing Parallel Data Structures in Extreme Scales},\n  author={Devarajan, Hariharan and Kougkas, Anthony and Bateman, Keith and Sun, Xian-He},\n  booktitle={2020 IEEE International Conference on Cluster Computing (CLUSTER)},\n  year={2020},\n  organization={IEEE}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhdfgroup%2Fhcl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhdfgroup%2Fhcl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhdfgroup%2Fhcl/lists"}