{"id":19281727,"url":"https://github.com/scality/hdclient","last_synced_at":"2025-04-22T01:31:03.491Z","repository":{"id":37740027,"uuid":"136958561","full_name":"scality/hdclient","owner":"scality","description":"HyperDrive Client Library","archived":false,"fork":false,"pushed_at":"2025-04-16T07:31:38.000Z","size":4475,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":56,"default_branch":"development/1","last_synced_at":"2025-04-16T10:00:03.960Z","etag":null,"topics":["artesca","zenko"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scality.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-11T17:23:37.000Z","updated_at":"2025-04-16T07:31:40.000Z","dependencies_parsed_at":"2024-05-02T14:45:49.672Z","dependency_job_id":"03adea28-baca-4ddb-9d36-14d4326e84f2","html_url":"https://github.com/scality/hdclient","commit_stats":{"total_commits":321,"total_committers":9,"mean_commits":"35.666666666666664","dds":"0.36760124610591904","last_synced_commit":"bf7b9b56ea008fb0babd816ef9e51384ddd03f2f"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fhdclient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fhdclient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fhdclient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Fhdclient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scality","download_url":"https://codeload.github.com/scality/hdclient/tar.gz/refs/heads/development/1","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250161964,"owners_count":21385014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artesca","zenko"],"created_at":"2024-11-09T21:23:56.621Z","updated_at":"2025-04-22T01:31:03.192Z","avatar_url":"https://github.com/scality.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NodeJS hyperdrive client API\n\n## Contributing\n\nIn order to contribute, please follow the\n[Contributing Guidelines](\nhttps://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md).\n\n## Overview\n\nThe Hyperdrive client is managing the Hyperdrive Controller endpoint for CloudServer.\nIt exposes a simple API for a group of HDController.\n\nIt does not change the data, it is merely a proxy to HDController.\n\n### HDController protocol\n\nBasically a subset of HTTP 1.1 using POST, GET and DELETE verbs to create, to get and to delete an object.\n\nOn POST, the HDController will create a key that can be used in GET / DELETE to access the data stored.\n\n## Usage\n\n### Installation\n\n```shell\nnpm install --save scality/hdclient\n# Check for dependencies vulnerabilities\nnpm audit\n```\n\n### Running tests\n\nTo use the linter and run the tests:\n\n```shell\n\n# All tests\nnpm test tests/\n\n# Other options and Mocha help\nnpm test -- -h\n\n# Code coverage\nnpm run coverage\n```\n\n### Generating documentation\n\n```shell\n# Generating JSDoc, to start browsing open docs/jsdoc/index.html\nnpm run jsdoc\n```\n\n### Performance diagnostic tool\n\n[Node-clinic](https://github.com/nearform/node-clinic) is installed by default as dev dependency. It can be used to diagnose general performance, I/O specific, event-loop issues, etc. It can also be used to generate flame graphs. Below are some usage examples usage. Note that data acquisition and visualization can be sepearated, they are not in the examples.\n\n```shell\n# Node clinic diagnosis tool - help\nnpm run clinic\n\n# Clinic doctor help\nnpm run clinic doctor\n\n# Diagnosing on the same machine - use --on-port to kickstart load generator\nNODE_ENV=production npm run clinic doctor -- \\\n                   --on-port='for i in {..10}; do curl -XPUT --data-binary @/etc/hosts \"http://localhost:6767/bucket/testobj$i\" ; done' \\\n                   -- node scripts/server.js 6767 scripts/example_hdclient_proxy.conf.json\n\nNODE_ENV=production npm run clinic doctor -- \\\n                   -- node scripts/server.js 6767 scripts/example_hdclient_proxy.conf.json \u0026\npid=$!\n# Start and wait for load generator to finish from somewhere else...\nkill -SIGINT $pid # Or keep process in foreground and Ctrl-C when done\nwait $pid\n\n# Flame graph help\nnpm run clinic flame\n\n# Flame graph - load-generator on different machine, to be started whenever the server is up\nNODE_ENV=production npm run clinic flame -- node scripts/server.js \u003cport\u003e \u003cconfig file\u003e\n...\nCtrl-C\nAnalysing data\n...\n```\n\n### Running as standalone\n\nBecause deploying the full S3 server might be too much of a hassle for your specific need, a HTTP server using HdClient is provided. The mapping between object key and internal keys (the ones actually stored on the hyperdrive) is stored in-memory only. The internal keys are not accessible on the outside, mirrorring behavior of Zenko-like deployment.\n\n ```shell\n# Start Hyperdrive 'proxy'\n# example conf assumes 1 hdcontroller listening on localhost:18888\nNODE_ENV=production node scripts/server.js 8888 scripts/example_hdclient_proxy.conf.json \u0026\n\n# Have fun\ncurl -XPUT --data-binary @/etc/hosts -v http://localhost:8888/mybucket/testobj\n*   Trying 127.0.0.1...\n* Connected to localhost (127.0.0.1) port 8888 (#0)\n\u003e PUT /mybucket/testobj HTTP/1.1\n\u003e Host: localhost:8888\n\u003e User-Agent: curl/7.47.0\n\u003e Accept: */*\n\u003e Content-Length: 267\n\u003e Content-Type: application/x-www-form-urlencoded\n\u003e\n* upload completely sent off: 267 out of 267 bytes\n\u003c HTTP/1.1 200 OK\n\u003c Date: Wed, 27 Jun 2018 10:44:32 GMT\n\u003c Connection: keep-alive\n\u003c Transfer-Encoding: chunked\n\u003c\n* Connection #0 to host localhost left intact\n\ncurl -v http://localhost:8888/mybucket/testobj\n*   Trying 127.0.0.1...\n* Connected to localhost (127.0.0.1) port 8888 (#0)\n\u003e GET /mybucket/testobj HTTP/1.1\n\u003e Host: localhost:8888\n\u003e User-Agent: curl/7.47.0\n\u003e Accept: */*\n\u003e\n\u003c HTTP/1.1 200 OK\n\u003c Content-Length: 267\n\u003c Date: Wed, 27 Jun 2018 10:44:39 GMT\n\u003c Connection: keep-alive\n\u003c\n* Connection #0 to host localhost left intact\n\u003cpayload...\u003e\n\n# \u003curl\u003e/\u003cbucket\u003e/\u003cobject\u003e/\u003cversion\u003e\ncurl -XDELETE -v http://localhost:8888/mybucket/testobj/64\n*   Trying 127.0.0.1...\n* Connected to localhost (127.0.0.1) port 8888 (#0)\n\u003e DELETE /mybucket/testobj HTTP/1.1\n\u003e Host: localhost:8888\n\u003e User-Agent: curl/7.47.0\n\u003e Accept: */*\n\u003e\n\u003c HTTP/1.1 200 OK\n\u003c Date: Wed, 27 Jun 2018 10:44:58 GMT\n\u003c Connection: keep-alive\n\u003c Transfer-Encoding: chunked\n\u003c\n* Connection #0 to host localhost left intact\n```\n\n### Running as a CloudServer data backend\n\nHow to run integrated hyperdrive client inside S3 or Zenko deployment? This section is only a work in progress\nsince actual S3 integration code is not yet merged.\n\n```shell\n# Checkout S3 repository and checkout proper hdclient integration branch\ngit clone https://github.com/scality/CloudServer.git\ncd CloudServer/\ngit checkout feature/RING-28500-add-hyperdrive-client-data-backend-real\n\n# Modify package.json to use the version of hdclient you want in case latest development/1.0 is not good\n# To use a local repository\n# sed s%scality/hdclient%file:\u003cpath to hdclient repository% package.json\n# To use a tag or commit\n# sed -i s%scality/hdclient%scality/hdclient#\u003ctag/commit\u003e% package.json\n\n# Add new locationConstraints\n# Region us-east-1 is mandatory, since the default config still references it\ncat \u003c\u003cEOF \u003e hdclient_locationConfig.json\n{\n    \"us-east-1\": {\n        \"type\": \"file\",\n        \"objectId\": \"iod1\",\n        \"legacyAwsBehavior\": true,\n        \"details\": {}\n    },\n    \"hyperdrive-cluster-1\": {\n        \"type\": \"scality\",\n        \"objectId\": \"oid2\",\n        \"legacyAwsBehavior\": false,\n        \"details\": {\n            \"connector\": {\n                \"hdclient\" : {\n                  \"bootstraps\": \"localhost:18888\",\n                  \"path\": \"/store/\",\n                }\n            }\n        }\n    }\n}\nEOF\n\n# Pattern match restEndpoints - haven't found a better way yet...\n# Edit config.json restEndpoints section to use hyperdrive-cluster-1\n# e.g. to map localhost onto hdclient: sed -i %\"localhost\": \"us-east-1\"%\"localhost\": \"hyperdrive-cluster-1\"\n# e.g. to map 127.0.0.1 onto hdclient: sed -i %\"127.0.0.1\": \"us-east-1\"%\"127.0.0.1\": \"hyperdrive-cluster-1\"\n\n# Install dependencies\nnpm install\n\n# Start a Kafka instance on passed kafkaBrokers parameters (in the example 127.0.0.1:6666)\n\n# Start CloudServer (memory backend ie metadata in-memory)\n# More informations inside S3 repository documentation\nNODE_ENV=production S3DATA=multiple S3_LOCATION_FILE=hdclient_locationConfig.json npm run mem_backend\n```\n\nIn a separate tab, have fun with AWS CLI\n\n```shell\n# Running S3 server uses default accessKey and secretKey\nexport AWS_ACCESS_KEY_ID=accessKey1\nexport AWS_SECRET_ACCESS_KEY=verySecretKey1\n\n# Create a bucket\naws  --endpoint-url=http://localhost:8000 s3 mb s3://brandnewbucket\n\n# List buckets\naws  --endpoint-url=http://localhost:8000 s3 ls\n\n# Put data\naws  --endpoint-url=http://localhost:8000 s3 cp /etc/hosts s3://brandnewbucket/shiny_new_object\n\n# List bucket content\naws  --endpoint-url=http://localhost:8000 s3 ls s3://brandnewbucket\n\n# Get data\naws  --endpoint-url=http://localhost:8000 s3 cp s3://brandnewbucket/shiny_new_object /tmp/retrieved\n\n# Delete data\naws  --endpoint-url=http://localhost:8000 s3 rm s3://brandnewbucket/shiny_new_object\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fhdclient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscality%2Fhdclient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Fhdclient/lists"}