{"id":13591656,"url":"https://github.com/mpenet/spandex","last_synced_at":"2025-04-04T21:09:21.158Z","repository":{"id":14442612,"uuid":"76551430","full_name":"mpenet/spandex","owner":"mpenet","description":"Elasticsearch client for Clojure (built on new ES 8.x java client)","archived":false,"fork":false,"pushed_at":"2024-05-29T11:02:13.000Z","size":264,"stargazers_count":262,"open_issues_count":2,"forks_count":31,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-28T20:08:07.537Z","etag":null,"topics":["clojure","elasticsearch","elasticsearch-client"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpenet.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":"FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"mpenet"}},"created_at":"2016-12-15T10:50:11.000Z","updated_at":"2024-12-14T12:32:15.000Z","dependencies_parsed_at":"2024-11-13T16:41:31.613Z","dependency_job_id":"ef4dc4f7-227a-4023-916a-9eefebb34109","html_url":"https://github.com/mpenet/spandex","commit_stats":{"total_commits":287,"total_committers":17,"mean_commits":16.88235294117647,"dds":"0.27177700348432055","last_synced_commit":"4420eb6163c14df4ea0545aaab369d2f53ab824f"},"previous_names":[],"tags_count":64,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpenet%2Fspandex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpenet%2Fspandex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpenet%2Fspandex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpenet%2Fspandex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpenet","download_url":"https://codeload.github.com/mpenet/spandex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247249530,"owners_count":20908212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","elasticsearch","elasticsearch-client"],"created_at":"2024-08-01T16:01:00.290Z","updated_at":"2025-04-04T21:09:21.138Z","avatar_url":"https://github.com/mpenet.png","language":"Clojure","funding_links":["https://github.com/sponsors/mpenet"],"categories":["Clojure","Database"],"sub_categories":[],"readme":"# spandex\n\u003c!-- [![Build Status](https://travis-ci.org/mpenet/spandex.svg?branch=master)](https://travis-ci.org/mpenet/spandex) --\u003e\n\n[![Clojars Project](https://img.shields.io/clojars/v/cc.qbits/spandex.svg)](https://clojars.org/cc.qbits/spandex)\n\nElasticsearch new low level [rest-client](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html) wrapper\n\n## Why?\n\nTo quote [\"State of the official Elasticsearch Java clients\"](https://www.elastic.co/blog/state-of-the-official-elasticsearch-java-clients)\n\n\u003e The Java REST client is the future for Java users of Elasticsearch.\n\nBecause the legacy native client is a bit of a nightmare to deal with\n(for many reasons) and the new REST client is quite capable and fast\ntoo, see [\"Benchmarking REST client and transport client\"](https://www.elastic.co/blog/benchmarking-rest-client-transport-client)\n\nNot to mention it supports some interesting features:\n\n* compatibility with any Elasticsearch version\n\n* load balancing across all available nodes\n\n* failover in case of node failures and upon specific response codes\n\n* failed connection penalization\n\n* persistent connections\n\n* trace logging of requests and responses\n\n* optional automatic discovery of cluster nodes (also known as sniffing)\n\n## Goals\n\n* Be minimal \u0026 performant\n\n* RING inspired\n\n* All \"exotic\" features should be optional\n\n* Not a giant DSL over another DSL, just maps everywhere.\n  Read ElasticSearch doc -\u003e done, not another layer of indirection\n\n* Provide minimal (and optional) utils to do the boring stuff\n  (bulk, scroll queries, compose urls)\n\n* Can do async via simple callbacks based api or `core.async`\n\n* Provide [specs](https://github.com/mpenet/spandex/blob/master/src/clj/qbits/spandex/spec.clj)\n\n### Setup\n\n```clojure\n(require '[qbits.spandex :as s])\n\n(def c (s/client {:hosts [\"http://127.0.0.1:9200\" \"https://foo2:3838\"]}))\n\n;; add optional sniffer\n(def s (s/sniffer c {... options ...}))\n```\n\n#### Work with `https` via ssh tunneling\n\nFirst setup and make sure that you have appropriate access to the host via tunneling.\n\ne.g. Add/edit your `~/.ssh/config` to look something like\n\n```\n# Example of tunneling in ~/.ssh/config\n# .. more config\nHost my-aws-elasticsearch-host \n  HostName 10.123.345.456\n  User ec2-user\n  IdentitiesOnly yes\n  IdentityFile ~/.ssh/my-aws-elasticsearch.pem\n  LocalForward 9200 vpc-my-aws-elasticsearch-host-lb43i.us-east-1.es.amazonaws.com:443\n  ServerAliveInterval 240\n# .. more config\n```\n\nYou can then start ssh tunneling with \n\n``` sh\n# see manpage of `ssh` for more details\nssh -oStrictHostKeyChecking=no my-aws-elasticsearch-host -N \n```\n\nThen you can create your client using the following `:http-client` options like\n\n```clojure\n;; if you are using tunnelling to host in AWS e.g.\n(def client (s/client {:hosts [\"https://localhost:9200\"]\n                       :http-client {:ssl-context (client/ssl-context-trust-all)\n                                     :ssl-noop-hostname-verifier? true}}))\n```\n\n#### Constructing URLs\n\nMost of spandex request functions take a request map as parameter. The\n`:url` key differs a bit from the original RING spec, it allows to\npass a raw string but also a sequence (potentially 2d) of encodable\nthings, keywords, .toString'able objects that make sense or nil (which\ncould be caused by a missing :url key).\n\n``` clojure\n(s/request c {:url [:foo :bar :_search] ...})\n(s/request c {:url [:foo [:bar :something \"more\"] :_search] ...})\n(s/request c {:url :_search ...})\n(s/request c {:url \"/index/_search\" ...})\n(s/request c {:url (java.util.UUID/randomUUID) ...})\n(s/request c {...}) ;; defaults to \"/\"\n```\n### Blocking requests\n\n```clojure\n(s/request c {:url [:entries :entry :_search]\n              :method :get\n              :body {:query {:match_all {}}}})\n\n\u003e\u003e {:body {:_index \"entries\", :_type \"entry\", :_id \"AVkDDJvdkd2OsNWu4oYk\", :_version 1, :_shards {:total 2, :successful 1, :failed 0}, :created true}, :status 201, :headers {\"Content-Type\" \"application/json; charset=UTF-8\", \"Content-Length\" \"141\"}, :host #object[org.apache.http.HttpHost 0x62b90fad \"http://127.0.0.1:9200\"]}\n```\n\n### Async requests (callbacks)\n\n```clojure\n(s/request-async c {:url \"/urls/url/\"\n                    :method :get\n                    :body {:query {:match {:message \"this is a test\"}}}\n                    :success (fn [response-as-clj] ... )\n                    :error (fn [ex] :boom)})\n```\n\n### Async requests: `core.async/promise-chan`\n\n``` clojure\n(async/\u003c!! (s/request-chan c {:url \"/urls/url/\"\n                              :method :get\n                              :body {:query {:match {:message \"this is a test\"}}}}))\n```\n\n### Scrolling\n\nScrolling via core.async (fully NIO internally), interruptible if you\nasync/close! the returned chan.\n\n``` clojure\n(async/go\n  (let [ch (s/scroll-chan client {:url \"/foo/_search\" :body {:query {:match_all {}}}})]\n    (loop []\n      (when-let [page (async/\u003c! ch)]\n        (do-something-with-page page)\n        (recur)))))\n```\n\n### Bulk requests scheduling\n\n\"Faux streaming\" of _bulk requests (flushes bulk request after\nconfigurable interval or threshold. Uses request-chan internally, so\nit's quite cheap.\n\n```clojure\n(let [{:keys [input-ch output-ch]} (bulk-chan client {:flush-threshold 100\n                                                      :flush-interval 5000\n                                                      :max-concurrent-requests 3})]\n  ;; happily takes a sequence of actions or single fragments\n  (async/put! input-ch [{:delete {:_index \"foo\" :_id \"1234\"}} {:_index :bar} {:create {...}}])\n  (async/put! input-ch {\"delete\" {\"_index\" \"website\" \"_type\" \"blog\" \"_id\" \"123\"}}))\n\n;; setup an response consumer (we just want to make sure we don't clog this channel)\n(future\n  (loop []\n    (when (async/\u003c!! (:output-ch c))\n      (recur))))\n```\n\n## Installation\n\n[![Clojars Project](https://img.shields.io/clojars/v/cc.qbits/spandex.svg)](https://clojars.org/cc.qbits/spandex)\n\n## API Docs\n\n[QuickDoc](API.md)\n\nOr the [clj.specs](https://github.com/mpenet/spandex/blob/master/src/clj/qbits/spandex/spec.clj) if that's your thing:\n\n## Running the tests\n\n``` clojure\nclj -X:test qbits.spandex.test-runner/run\n```\n\nclj -X:test qbits.spandex-test-runner/run\n\n## License\n\nCopyright © 2018 [Max Penet](http://twitter.com/mpenet)\n\nDistributed under the\n[Eclipse Public License](http://www.eclipse.org/legal/epl-v10.html),\nthe same as Clojure.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpenet%2Fspandex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpenet%2Fspandex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpenet%2Fspandex/lists"}