{"id":22035525,"url":"https://github.com/amperity/blocks-adl","last_synced_at":"2025-07-31T22:39:14.846Z","repository":{"id":57713113,"uuid":"105822099","full_name":"amperity/blocks-adl","owner":"amperity","description":"Content-addressable Azure Data Lake block store","archived":false,"fork":false,"pushed_at":"2020-04-15T16:59:08.000Z","size":37,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":62,"default_branch":"develop","last_synced_at":"2025-03-31T13:27:57.005Z","etag":null,"topics":["azure-data-lake","clojure","content-addressable-storage"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amperity.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-04T21:50:13.000Z","updated_at":"2020-04-15T16:59:08.000Z","dependencies_parsed_at":"2022-09-06T02:10:41.206Z","dependency_job_id":null,"html_url":"https://github.com/amperity/blocks-adl","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amperity%2Fblocks-adl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amperity%2Fblocks-adl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amperity%2Fblocks-adl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amperity%2Fblocks-adl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amperity","download_url":"https://codeload.github.com/amperity/blocks-adl/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252945810,"owners_count":21829662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure-data-lake","clojure","content-addressable-storage"],"created_at":"2024-11-30T10:25:13.111Z","updated_at":"2025-05-07T19:42:56.895Z","avatar_url":"https://github.com/amperity.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"blocks-adl\n==========\n\n[![CircleCI](https://circleci.com/gh/amperity/blocks-adl.svg?style=svg)](https://circleci.com/gh/amperity/blocks-adl)\n\nThis library implements a [content-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage)\n[block store](https://github.com/greglook/blocks) backed by an Azure Data Lake\nstorage account.\n\n\n## Installation\n\nLibrary releases are published on Clojars. To use the latest version with\nLeiningen, add the following dependency to your project definition:\n\n[![Clojars Project](http://clojars.org/amperity/blocks-adl/latest-version.svg)](http://clojars.org/amperity/blocks-adl)\n\n\n## Usage\n\nThe `blocks.store.adl` namespaces provides the `adl-block-store` constructor. This accepts\nthe fully-qualified hostname of the datalake store and an optional root path for the block\nstore.\n\nThe record created eventually expects a `token-provider` which is a AccessTokenProvider from\nthe datalake oauth2 library. The example uses client credentials from an AD service principal,\nbut other methods are possible, like using a refresh token or user password.\n\nThe principal needs full authorization for the datalake store's path starting from the given root.\n\nSee the [Datalake ACL Docs][1] for more information about how this authorization works.\n\n\n```clojure\n=\u003e (require '[blocks.core :as block]\n            '[blocks.store.adl :refer [adl-block-store]])\n=\u003e (import (com.microsoft.azure.datalake.store.oauth2\n             ClientCredsTokenProvider))\ncom.microsoft.azure.datalake.store.oauth2.ClientCredsTokenProvider\n\n; Create the token provider. This example uses client credentials via the oauth2 token endpoint.\n; This token provider will fetch access tokens as needed for the store. If a token has expired\n; it will fetch a new one with the given credentials.\n=\u003e (def token-endpoint \"https://login.microsoftonline.com/c6b6362b-b1d3-4bbe-bda9-2be62711f5e4/oauth2/token\")\n=\u003e (def token-provider (ClientCredsTokenProvider. token-endpoint \"theclientid\" \"theclientsecret\"))\n#'token-provider\n\n; Create the block store backed by the  datalake (ADL) store \"mystore\" under the root \"/example/blocks\".\n=\u003e (def store' (adl-block-store \"mystore.azuredatalakestore.net\"\n                 :root \"/example/blocks\"\n                 :token-provider token-provider))\n=\u003e (require '[com.stuartsierra.component :as component])\n; Start the store, which will fetch an access token and check the store status.\n=\u003e (def store (component/start store'))\nINFO blocks.store.adl - Connecting Azure Data Lake client to mystore.azuredatalakestore.net\nDEBUG com.microsoft.azure.datalake.store.oauth2.ClientCredsTokenProvider - AADToken: refreshing client-credential based token\nDEBUG com.microsoft.azure.datalake.store.oauth2.AzureADAuthenticator - AADToken: fetched token with expiry Fri Nov 17 12:10:01 PST 2017\nINFO blocks.store.adl - Store contains 0.0 MB in 0 blocks\n#'store\n\n=\u003e (block/list store {})\n()\n\n=\u003e (def b1 (block/read! \"abcd\" :sha2-512))\n=\u003e (def b2 (block/read! \"1234\" :sha1))\n=\u003e (count (block/put-batch! store [b1 b2]))\n2\n=\u003e (def b2-id (-\u003e (block/list store {:algorithm :sha1}) first :id))\n=\u003e (= (:id b2) b2-id)\ntrue\n\n=\u003e (block/get store b2-id)\nBlock[hash:sha1:7110eda4d09e062aa5e4a390b0a572ac0d2c0220 4 ~]\n\n; Like other persistent remote block stores, the block returned is lazy\n; it must be opened to read content which will stream from ADL.\n=\u003e (def b2' *1)\n=\u003e (block/lazy? b2')\ntrue\n=\u003e (= b2 b2')\ntrue\n=\u003e (slurp (block/open b2'))\n\"1234\"\n```\n\n\n## License\n\nLicensed under the Apache License, Version 2.0. See the [LICENSE](LICENSE) file\nfor rights and restrictions.\n\n[1]:https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-access-control\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famperity%2Fblocks-adl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famperity%2Fblocks-adl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famperity%2Fblocks-adl/lists"}