Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dakrone/clj-http
An idiomatic clojure http client wrapping the apache client. Officially supported version.
https://github.com/dakrone/clj-http
Last synced: 4 days ago
JSON representation
An idiomatic clojure http client wrapping the apache client. Officially supported version.
- Host: GitHub
- URL: https://github.com/dakrone/clj-http
- Owner: dakrone
- License: mit
- Created: 2011-09-07T21:37:58.000Z (over 13 years ago)
- Default Branch: 3.x
- Last Pushed: 2024-08-19T19:50:50.000Z (4 months ago)
- Last Synced: 2024-11-25T15:55:00.118Z (18 days ago)
- Language: Clojure
- Homepage: http://clojars.org/clj-http
- Size: 1.98 MB
- Stars: 1,784
- Watchers: 37
- Forks: 410
- Open Issues: 66
-
Metadata Files:
- Readme: README.org
- Changelog: changelog.org
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-clojure - clj-http
- stars - dakrone/clj-http - An idiomatic clojure http client wrapping the apache client. Officially supported version. \[*MIT License*\] (⭐️1787) (Clojure)
- stars - dakrone/clj-http - An idiomatic clojure http client wrapping the apache client. Officially supported version. \[*MIT License*\] (⭐️1782) (Clojure)
README
#+TITLE: clj-http documentation
#+AUTHOR: Lee Hinman
#+STARTUP: align fold nodlcheck lognotestate showall
#+OPTIONS: H:4 num:nil toc:t \n:nil @:t ::t |:t ^:{} -:t f:t *:t
#+OPTIONS: skip:nil d:(HIDE) tags:not-in-toc auto-id:t
#+PROPERTY: header-args :results code :exports both :noweb yes
#+HTML_HEAD: body {margin-right:15%; margin-left:15%;}
#+LANGUAGE: en
[[https://clojars.org/clj-http][file:https://img.shields.io/clojars/v/clj-http.svg]] [[https://github.com/dakrone/clj-http/actions?query=workflow%3A%22Clojure+CI%22][file:https://github.com/dakrone/clj-http/workflows/Clojure%20CI/badge.svg]] [[https://gitter.im/clj-http/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge][file:https://badges.gitter.im/clj-http/Lobby.svg]]
* Table of Contents :TOC_3:
:PROPERTIES:
:CUSTOM_ID: h-aaf075ea-2f0e-4a45-871a-0f89c838fb4b
:END:
- [[#branches][Branches]]
- [[#introduction][Introduction]]
- [[#overview][Overview]]
- [[#philosophy][Philosophy]]
- [[#installation][Installation]]
- [[#quickstart][Quickstart]]
- [[#head][HEAD]]
- [[#get][GET]]
- [[#put][PUT]]
- [[#post][POST]]
- [[#delete][DELETE]]
- [[#async-http-request][Async HTTP Request]]
- [[#cancelling-requests][Cancelling Requests]]
- [[#coercions][Coercions]]
- [[#input-coercion][Input coercion]]
- [[#output-coercion][Output coercion]]
- [[#headers][Headers]]
- [[#query-string-parameters][Query-string parameters]]
- [[#meta-tag-headers][Meta Tag Headers]]
- [[#link-headers][Link Headers]]
- [[#redirects][Redirects]]
- [[#how-to-create-a-custom-redirectstrategy][How to create a custom RedirectStrategy]]
- [[#cookies][Cookies]]
- [[#cookiestores][Cookiestores]]
- [[#keystores-trust-stores][Keystores, Trust-stores]]
- [[#exceptions][Exceptions]]
- [[#decompression][Decompression]]
- [[#debugging][Debugging]]
- [[#logging][Logging]]
- [[#caching][Caching]]
- [[#authentication][Authentication]]
- [[#basic-auth][Basic Auth]]
- [[#digest-auth][Digest Auth]]
- [[#ntlm-auth][NTLM Auth]]
- [[#oauth2][oAuth2]]
- [[#advanced-usage][Advanced Usage]]
- [[#raw-request][Raw Request]]
- [[#boolean-options][Boolean options]]
- [[#persistent-connections][Persistent Connections]]
- [[#re-using-httpclient-between-requests][Re-using =HttpClient= between requests]]
- [[#proxies][Proxies]]
- [[#custom-middleware][Custom Middleware]]
- [[#modifying-apache-specific-features-of-the-httpclientbuilder-and-httpasyncclientbuilder][Modifying Apache-specific features of the =HttpClientBuilder= and =HttpAsyncClientBuilder=]]
- [[#incrementally-json-parsing][Incrementally JSON Parsing]]
- [[#dns-resolution][DNS Resolution]]
- [[#development][Development]]
- [[#faking-responses][Faking Responses]]
- [[#optional-dependencies][Optional Dependencies]]
- [[#clj-http-lite][clj-http-lite]]
- [[#troubleshooting][Troubleshooting]]
- [[#verifyerror-class-orgcodehausjacksonsmilesmileparser-overrides-final-method-getbinaryvalue][VerifyError class org.codehaus.jackson.smile.SmileParser overrides final method getBinaryValue...]]
- [[#nohttpresponseexception--due-to-stale-connections][NoHttpResponseException ... due to stale connections**]]
- [[#tests][Tests]]
- [[#testimonials][Testimonials]]
- [[#other-libraries-providing-middleware][Other Libraries Providing Middleware]]
- [[#license][License]]
* Branches
:PROPERTIES:
:CUSTOM_ID: h-e390585c-cbd8-4e94-b36b-4e9c27c16720
:END:
There are branches for the major version numbers:
- 2.x (no longer maintained except for security issues)
- 3.x (current stable releases and the main Github branch)
- master (which is 4.x, unreleased, based on version 5 of the Apache HTTP Client)
* Introduction
:PROPERTIES:
:CUSTOM_ID: h-d893078a-b20b-4086-9272-3d9c28c86846
:END:
** Overview
:PROPERTIES:
:CUSTOM_ID: h-d8b17d06-124e-44fd-9c86-0399f39b0254
:END:
clj-http is an HTTP library wrapping the [[http://hc.apache.org/][Apache HttpComponents]] client. This
library has taken over from mmcgrana's clj-http.
** Philosophy
:PROPERTIES:
:CUSTOM_ID: h-aa21d07d-333b-4ff2-93a9-ffdca31d8949
:END:
The design of =clj-http= is inspired by the [[https://github.com/ring-clojure/ring][Ring]] protocol for Clojure HTTP
server applications.
The client in =clj-http.core= makes HTTP requests according to a given Ring
request map and returns [[https://github.com/ring-clojure/ring/blob/master/SPEC.md][Ring response maps]] corresponding to the resulting HTTP
response. The function =clj-http.client/request= uses Ring-style middleware to
layer functionality over the core HTTP request/response implementation. Methods
like =clj-http.client/get= are sugar over this =clj-http.client/request=
function.
* Installation
:PROPERTIES:
:CUSTOM_ID: h-ddfce0e2-6797-4774-add5-d5cf5bfaaa17
:END:
=clj-http= is available as a Maven artifact from [[http://clojars.org/clj-http][Clojars]].
With Leiningen/Boot:
#+BEGIN_SRC clojure
[clj-http "3.13.0"]
#+END_SRC
If you need an older version, a 2.x release is also available.
#+BEGIN_SRC clojure
[clj-http "2.3.0"]
#+END_SRC
clj-http 3.12.x supports clojure 1.6.0 and higher.
clj-http 3.13.x supports clojure 1.8.0 and higher.
clj-http 4.x will support clojure 1.7.0 and higher.
* Quickstart
:PROPERTIES:
:CUSTOM_ID: h-65f0132e-1f96-4711-a84e-973817f37dd3
:END:
The main HTTP client functionality is provided by the =clj-http.client= namespace.
First, require it in the REPL:
#+BEGIN_SRC clojure
(require '[clj-http.client :as client])
#+END_SRC
Or in your application:
#+BEGIN_SRC clojure
(ns my-app.core
(:require [clj-http.client :as client]))
#+END_SRC
The client supports simple =get=, =head=, =put=, =post=, =delete=, =copy=,
=move=, =patch=, and =options= requests. Response are returned as [[https://github.com/ring-clojure/ring/blob/master/SPEC][Ring-style
response maps]]:
** HEAD
:PROPERTIES:
:CUSTOM_ID: h-79d1bb5f-c695-46a6-af4e-a64ca599c978
:END:
#+BEGIN_SRC clojure
(client/head "http://example.com/resource")
(client/head "http://example.com/resource" {:accept :json})
#+END_SRC
** GET
:PROPERTIES:
:CUSTOM_ID: h-89c164fb-85c2-4953-a8c4-a50867adf42a
:END:
Example requests:
#+BEGIN_SRC clojure
(client/get "http://example.com/resources/id")
;; Setting options
(client/get "http://example.com/resources/3" {:accept :json})
(client/get "http://example.com/resources/3" {:accept :json :query-params {"q" "foo, bar"}})
;; Specifying headers as either a string or collection:
(client/get "http://example.com"
{:headers {"foo" ["bar" "baz"], "eggplant" "quux"}})
;; Using either string or keyword header names:
(client/get "http://example.com"
{:headers {:foo ["bar" "baz"], :eggplant "quux"}})
;; Completely ignore cookies:
(client/post "http://example.com" {:cookie-policy :none})
;; There are also multiple ways to handle cookies
(client/post "http://example.com" {:cookie-policy :default})
(client/post "http://example.com" {:cookie-policy :netscape})
(client/post "http://example.com" {:cookie-policy :standard})
(client/post "http://example.com" {:cookie-policy :standard-strict})
;; Cookies can be completely configurable with a custom spec by adding a
;; function to return a cookie spec for parsing the cookie. For example, if you
;; wanted to configure a spec provider to have a certain compatibility level:
(client/post "http://example.com"
{:cookie-spec
(fn [http-context]
(println "generating a new cookie spec")
(.create
(org.apache.http.impl.cookie.RFC6265CookieSpecProvider.
org.apache.http.impl.cookie.RFC6265CookieSpecProvider$CompatibilityLevel/IE_MEDIUM_SECURITY
(PublicSuffixMatcherLoader/getDefault))
http-context))})
;; Or a version with relaxed compatibility
(client/post "http://example.com"
{:cookie-spec
(fn [http-context]
(println "generating a new cookie spec")
(.create
(org.apache.http.impl.cookie.RFC6265CookieSpecProvider.
org.apache.http.impl.cookie.RFC6265CookieSpecProvider$CompatibilityLevel/RELAXED
(PublicSuffixMatcherLoader/getDefault))
http-context))})
;; Sometimes you want to do your own validation or something, which you can do
;; by proxying the CookieSpecBase. Note that this doesn't actually return the
;; cookies, because clj-http does its own cookie parsing. If you want to store
;; the cookies from these methods you'll need to use a cookie store or put it in
;; some datastructure yourself.
(client/post "http://example.com"
{:cookie-spec
(fn [http-context]
(proxy [org.apache.http.impl.cookie.CookieSpecBase] []
;; Version and version header
(getVersion [] 0)
(getVersionHeader [] nil)
;; parse headers into cookie objects
(parse [header cookie-origin] (java.util.ArrayList.))
;; Validate a cookie, throwing MalformedCookieException if the
;; cookies isn't valid
(validate [cookie cookie-origin]
(println "validating:" cookie))
;; Determine if a cookie matches the target location
(match [cookie cookie-origin] true)
;; Format a list of cookies into a list of headers
(formatCookies [cookies] (java.util.ArrayList.))))})
;; If you have created your own registry for cookie policies, you can provide
;; :cookie-policy-registry to use it. See
;; clj-http.core/create-custom-cookie-policy-registry for an example of a custom
;; registry
(client/post "http://example.com"
{:cookie-policy-registry my-custom-policy-registry
:cookie-policy "my-policy"})
;; Need to contact a server with an untrusted SSL cert?
(client/get "https://alioth.debian.org" {:insecure? true})
;; If you don't want to follow-redirects automatically:
(client/get "http://example.com/redirects-somewhere" {:redirect-strategy :none})
;; Only follow a certain number of redirects:
(client/get "http://example.com/redirects-somewhere" {:max-redirects 5})
;; Avoid throwing exceptions if redirected too many times:
(client/get "http://example.com/redirects-somewhere" {:max-redirects 5 :redirect-strategy :graceful})
;; Throw an exception if the get takes too long. Timeouts in milliseconds.
(client/get "http://example.com/redirects-somewhere" {:socket-timeout 1000 :connection-timeout 1000})
;; Query parameters
(client/get "http://example.com/search" {:query-params {"q" "foo, bar"}})
;; "Nested" query parameters
;; (this yields a query string of `a[e][f]=6&a[b][c]=5`)
(client/get "http://example.com/search" {:query-params {:a {:b {:c 5} :e {:f 6}}}})
;; Provide cookies — uses same schema as :cookies returned in responses
;; (see the cookie store option for easy cross-request maintenance of cookies)
(client/get "http://example.com"
{:cookies {"ring-session" {:discard true, :path "/", :value "", :version 0}}})
;; Tell clj-http not to decode cookies from the response header
(client/get "http://example.com" {:decode-cookies false})
;; Support for IPv6!
(client/get "http://[2001:62f5:9006:e472:cabd:c8ff:fee3:8ddf]")
;; Super advanced, your own http-client-context and request-config
(client/get "http://example.com/get"
{:http-client-context my-http-client-context
:http-request-config my-request-config})
#+END_SRC
The client will also follow redirects on the appropriate =30*= status codes.
The client transparently accepts and decompresses the =gzip= and =deflate=
content encodings.
=:trace-redirects= will contain the chain of the redirections followed.
** PUT
:PROPERTIES:
:CUSTOM_ID: h-1582cd6e-a6e8-49c8-96e3-28eee6128c31
:END:
#+BEGIN_SRC clojure
(client/put "http://example.com/api" {:body "my PUT body"})
#+END_SRC
** POST
:PROPERTIES:
:CUSTOM_ID: h-32c8ca7a-0ef2-41b8-8158-20b0e2945e5d
:END:
#+BEGIN_SRC clojure
;; Various options:
(client/post "http://example.com/api"
{:basic-auth ["user" "pass"]
:body "{\"json\": \"input\"}"
:headers {"X-Api-Version" "2"}
:content-type :json
:socket-timeout 1000 ;; in milliseconds
:connection-timeout 1000 ;; in milliseconds
:accept :json})
;; Send form params as a urlencoded body (POST or PUT)
(client/post "http://example.com" {:form-params {:foo "bar"}})
;; Send form params as a json encoded body (POST or PUT)
(client/post "http://example.com" {:form-params {:foo "bar"} :content-type :json})
;; Send form params as a json encoded body (POST or PUT) with options
(client/post "http://example.com" {:form-params {:foo "bar"}
:content-type :json
:json-opts {:date-format "yyyy-MM-dd"}})
;; You can also specify the encoding of form parameters
(client/post "http://example.com" {:form-params {:foo "bar"}
:form-param-encoding "ISO-8859-1"})
;; Send form params as a Transit encoded JSON body (POST or PUT) with options
(client/post "http://example.com" {:form-params {:foo "bar"}
:content-type :transit+json
:transit-opts
{:encode {:handlers {}}
:decode {:handlers {}}}})
;; Send form params as a Transit encoded MessagePack body (POST or PUT) with options
(client/post "http://example.com" {:form-params {:foo "bar"}
:content-type :transit+msgpack
:transit-opts
{:encode {:handlers {}}
:decode {:handlers {}}}})
;; Multipart form uploads/posts
;; takes a vector of maps, to preserve the order of entities, :name
;; will be used as the part name unless :part-name is specified
(client/post "http://example.org" {:multipart [{:name "title" :content "My Awesome Picture"}
{:name "Content/type" :content "image/jpeg"}
{:name "foo.txt" :part-name "eggplant" :content "Eggplants"}
{:name "file" :content (clojure.java.io/file "pic.jpg")}]
;; You can also optionally pass a :mime-subtype
:mime-subtype "foo"})
;; Multipart :content values can be one of the following:
;; String, InputStream, File, a byte-array, or an instance of org.apache.http.entity.mime.content.ContentBody
;; Some Multipart bodies can also support more keys (like :encoding
;; and :mime-type), check src/clj-http/multipart.clj to see all flags
;; Apache's http client automatically retries on IOExceptions, if you
;; would like to handle these retries yourself, you can specify a
;; :retry-handler. Return true to retry, false to stop trying:
(client/post "http://example.org" {:multipart [["title" "Foo"]
["Content/type" "text/plain"]
["file" (clojure.java.io/file "/tmp/missing-file")]]
:retry-handler (fn [ex try-count http-context]
(println "Got:" ex)
(if (> try-count 4) false true))})
;; to handle a file with non-ascii filename, try :multipart-charset "UTF-8" and :multipart-mode BROWSER_COMPATIBLE
;; see also: https://stackoverflow.com/questions/3393445/international-characters-in-filename-in-mutipart-formdata
(import (org.apache.http.entity.mime HttpMultipartMode))
(client/post "http://example.org" {:multipart [{:content (clojure.java.io/file "日本語.txt")}]
:multipart-mode HttpMultipartMode/BROWSER_COMPATIBLE
:multipart-charset "UTF-8"} )
#+END_SRC
A word about flattening nested =:query-params= and =:form-params= maps. There are essentially three
different ways to handle flattening them:
- =:ignore-nested-query-string= :: Do not handle nested query parameters specially, treat them as
the exact text they come in as. Defaults to *false*.
- =:flatten-nested-form-params= :: Flatten nested (map within a map) =:form-params= before encoding
it as the body. Defaults to *false*, meaning form params are encoded only
=x-www-form-urlencoded=.
- =:flatten-nested-keys= :: An advanced way of specifying which keys having nested maps should be
flattened. A middleware function checks the previous two options
(=:ignore-nested-query-string= and =:flatten-nested-form-params=) and modifies this to be the
list that will be flattened.
** DELETE
:PROPERTIES:
:CUSTOM_ID: h-c7165d6b-232a-439d-9390-8c05e6ef1e6f
:END:
#+BEGIN_SRC clojure
(client/delete "http://example.com/resource")
#+END_SRC
** Async HTTP Request
:PROPERTIES:
:CUSTOM_ID: h-0e3eb987-5b2b-4874-97ef-b834394d083d
:END:
The new async HTTP request API is a Ring-style async API.
All options for synchronous requests can be used in asynchronous requests.
Starting an async request is easy, for example:
#+BEGIN_SRC clojure
;; :async? in options map need to be true
(client/get "http://example.com"
{:async? true}
;; respond callback
(fn [response] (println "response is:" response))
;; raise callback
(fn [exception] (println "exception message is: " (.getMessage exception))))
#+END_SRC
All exceptions thrown during the request will be passed to the raise callback.
*** Cancelling Requests
:PROPERTIES:
:CUSTOM_ID: cancelling-requests
:END:
Calls to the http methods with =:async true= return an Apache [[https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/concurrent/BasicFuture.html][BasicFuture]] that you can call =.get=
or =.cancel= on. See the Javadocs for =BasicFuture= [[https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/concurrent/BasicFuture.html][here]]. For instance:
#+BEGIN_SRC clojure
(import '(java.util.concurrent TimeoutException TimeUnit))
(let [future (client/get "http://example.com/slow-url"
{:async true :oncancel #(println "request was cancelled")}
#(println :got %) #(println :err %))]
(try
(.get future 1 TimeUnit/SECONDS)
(catch TimeoutException e
;; Cancel the request, it's taken too long
(.cancel future true))))
#+END_SRC
** Coercions
:PROPERTIES:
:CUSTOM_ID: h-8902cd95-e01e-4d9b-9dc8-5f5c8f04504b
:END:
clj-http allows coercing the body of the request either before it is sent (input coercion), or after
it's received (output coercion) from the server.
*** Input coercion
:PROPERTIES:
:CUSTOM_ID: h-bed01743-2209-473d-ae86-bd187f059e0c
:END:
#+BEGIN_SRC clojure
;; body as a byte-array
(client/post "http://example.com/resources" {:body my-byte-array})
;; body as a string
(client/post "http://example.com/resources" {:body "string"})
;; :body-encoding is optional and defaults to "UTF-8"
(client/post "http://example.com/resources"
{:body "string" :body-encoding "UTF-8"})
;; body as a file
(client/post "http://example.com/resources"
{:body (clojure.java.io/file "/tmp/foo") :body-encoding "UTF-8"})
;; :length is optional for passing in an InputStream; if not
;; supplied it will default to -1 to signal to HttpClient to use
;; chunked encoding
(client/post "http://example.com/resources"
{:body (clojure.java.io/input-stream "/tmp/foo")})
(client/post "http://example.com/resources"
{:body (clojure.java.io/input-stream "/tmp/foo") :length 1000})
#+END_SRC
*** Output coercion
:PROPERTIES:
:CUSTOM_ID: h-0c8966a6-f220-4f1e-a79e-a520fb313f9e
:END:
#+BEGIN_SRC clojure
;; The default output is a string body
(client/get "http://example.com/foo.txt")
;; Coerce as a byte-array
(client/get "http://example.com/favicon.ico" {:as :byte-array})
;; Coerce as something other than UTF-8 string
(client/get "http://example.com/string.txt" {:as "UTF-16"})
;; Coerce as json
(client/get "http://example.com/foo.json" {:as :json})
(client/get "http://example.com/foo.json" {:as :json-string-keys})
;; Coerce as Transit encoded JSON or MessagePack
(client/get "http://example.com/foo" {:as :transit+json})
(client/get "http://example.com/foo" {:as :transit+msgpack})
;; Coerce as a clojure datastructure
(client/get "http://example.com/foo.clj" {:as :clojure})
;; Coerce as x-www-form-urlencoded
(client/post "http://example.com/foo" {:as :x-www-form-urlencoded})
;; Try to automatically coerce the output based on the content-type
;; header (this is currently a BETA feature!). Currently supports
;; text, json and clojure (with automatic charset detection)
;; clojure coercion requires "application/clojure" or
;; "application/edn" in the content-type header
(client/get "http://example.com/foo.json" {:as :auto})
;; Return the body as a stream
(client/get "http://example.com/bigrequest.html" {:as :stream})
;; Note that the connection to the server will NOT be closed until the
;; stream has been read
;; Return the body as a java.io.BufferedReader
(client/get "http://example.com/bigrequest.html" {:as :reader})
;; As above, the connection will remain open until the stream has been
;; read. The reader will attempt to respect the server-specified charset,
;; if any, defaulting to UTF-8.
#+END_SRC
Output coercion with =:as :json=, =:as :json-string-keys= or =:as :x-www-form-urlencoded=, will only work with an optional dependency, see [[#optional-dependencies][Optional Dependencies]].
By default, JSON coercion is only applied when the response's status
is considered "unexceptional". If the =:unexceptional-status= option
is provided, then its value is a function which specifies what status
codes are unexceptional. =:unexceptional-status= defaults to
=clj-http.client/unexceptional-status?=.
If you would like to change under what conditions coercion is applied,
you can send the =:coerce= option, which can be set to:
#+BEGIN_SRC clojure
:always ;; always json decode the body
:unexceptional ;; json decode when an HTTP response is considered unexceptional
:exceptional ;; json decode when an HTTP response is considered exceptional
#+END_SRC
The =:coerce= setting defaults to =:unexceptional=.
** Headers
:PROPERTIES:
:CUSTOM_ID: h-ef64574f-f9dc-4356-95b7-d55cc6737b44
:END:
clj-http's treatment of headers is a little more permissive than the [[https://github.com/ring-clojure/ring/blob/master/SPEC][ring spec]]
specifies.
Rather than forcing all request headers to be lowercase strings,
clj-http allows strings or keywords of any case. Keywords will be
transformed into their canonical representation, so the :content-md5
header will be sent to the server as "Content-MD5", for instance.
String keys in request headers, however, will be sent to the server
with their casing unchanged.
Response headers can be read as keywords or strings of any case. If
the server responds with a "Date" header, you could access the value
of that header as :date, "date", "Date", etc.
If for some reason you require access to the original header name that
the server specified, it is available by invoking (keys ...) on the
header map.
This special treatment of headers is implemented in the
wrap-header-map middleware, which (like any middleware) can be
disabled by using with-middleware to specify different behavior.
** Query-string parameters
:PROPERTIES:
:CUSTOM_ID: h-dd49992c-a516-4af0-9735-4f4340773361
:END:
There are four different ways that query string parameters for array values can
be generated, depending on what the resulting query string should look like,
they are:
- A repeating parameter (default)
- Array style
- Indexed array style
- Comma separated style
Here is an example of the input and output for the ~:query-params~ parameter,
controlled by the ~:multi-param-style~ option:
#+BEGIN_SRC clojure
;; default style, with :multi-param-style unset
:a [1 2 3] => "a=1&a=2&a=3"
;; with :multi-param-style :array, a repeating param with array suffix
;; (PHP-style):
:a [1 2 3] => "a[]=1&a[]=2&a[]=3"
;; with :multi-param-style :indexed, a repeating param with array suffix and
;; index (Rails-style):
:a [1 2 3] => "a[0]=1&a[1]=2&a[2]=3"
;; with :multi-param-style :comma-separated, a param with comma-separated values
:a [1 2 3] => "a=1,2,3"
#+END_SRC
** Meta Tag Headers
:PROPERTIES:
:CUSTOM_ID: h-01663a63-8bc8-45da-8a3d-341402f3f3fa
:END:
HTML 4.01 allows using the tag ~~ and HTML 5 allows
using the tag ~~ to specify a header that should be
treated as an HTTP response header. By default, clj-http will ignore the body of
the response (other than the regular output coercion), but if you need clj-http
to parse the headers out of the body, you can use the =:decode-body-headers=
option:
#+BEGIN_SRC clojure
;; without decoding body headers (defaults to off):
(:headers (client/get "http://www.yomiuri.co.jp/"))
=> {"server" "Apache",
"content-encoding" "gzip",
"content-type" "text/html",
"date" "Tue, 09 Oct 2012 18:02:41 GMT",
"cache-control" "max-age=0, no-cache",
"expires" "Tue, 09 Oct 2012 18:02:41 GMT",
"etag" "\"1dfb-2686-4cba2686fb8b1\"",
"pragma" "no-cache",
"connection" "close"}
;; with decoding body headers, notice the content-type,
;; content-style-type and content-script-type headers:
(:headers (client/get "http://www.yomiuri.co.jp/" {:decode-body-headers true}))
=> {"server" "Apache",
"content-encoding" "gzip",
"content-script-type" "text/javascript",
"content-style-type" "text/css",
"content-type" "text/html; charset=Shift_JIS",
"date" "Tue, 09 Oct 2012 18:02:59 GMT",
"cache-control" "max-age=0, no-cache",
"expires" "Tue, 09 Oct 2012 18:02:59 GMT",
"etag" "\"1dfb-2686-4cba2686fb8b1\"",
"pragma" "no-cache",
"connection" "close"}
#+END_SRC
This can be used to have clj-http correctly interpret the body's charset by
using:
#+BEGIN_SRC clojure
(client/get "http://www.yomiuri.co.jp/" {:decode-body-headers true :as :auto})
=> ;; correctly formatted :body (Shift_JIS charset instead of UTF-8)
#+END_SRC
Note that this feature is currently beta and uses [[https://github.com/weavejester/crouton][Crouton]] to parse the body of
the request. If you want to use this feature, you can include Crouton in
addition to clj-http as a dependency like so:
#+BEGIN_SRC clojure
(defproject foo "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.3.0"]
[clj-http "0.6.0"]
[crouton "1.0.0"]])
#+END_SRC
Note also that HEAD requests will not return a body, in which case this setting will have no effect.
clj-http will automatically disable the =:decode-body-headers= option.
** Link Headers
:PROPERTIES:
:CUSTOM_ID: h-f7464c54-4928-474f-9132-08e6b6f3c19d
:END:
clj-http parses any [[http://tools.ietf.org/html/rfc5988][link headers]] returned in the response, and adds them to the
=:links= key on the response map. This is particularly useful for paging RESTful
APIs:
#+BEGIN_SRC clojure
(:links (client/get "https://api.github.com/gists"))
=> {:next {:href "https://api.github.com/gists?page=2"}
:last {:href "https://api.github.com/gists?page=22884"}}
#+END_SRC
** Redirects
:PROPERTIES:
:CUSTOM_ID: h-71c966ae-f764-4bd7-801c-0f3c8413c502
:END:
clj-http conforms its behaviour regarding automatic redirects to the [[https://tools.ietf.org/html/rfc2616#section-10.3][RFC]].
It means that redirects on status =301=, =302=, =307= and =308= are not redirected on
methods other than =GET= and =HEAD=. If you want a behaviour closer to what most
browser have, you can set =:redirect-strategy= to =:lax= in your request to have
automatic redirection work on all methods by transforming the method of the
request to =GET=.
Redirect Options:
- =:trace-redirects= :: If true, clj-http will enhance the response object with a
list of redirected URLs with key: =:trace-redirects=.
- =:redirect-strategy= :: Sets the redirect strategy for clj-http. Accepts the following:
- =:none= - Perform no redirects
- =:default= - See https://hc.apache.org/httpcomponents-client-4.5.x/current/httpclient/apidocs/org/apache/http/impl/client/DefaultRedirectStrategy.html
- =:lax= - See https://hc.apache.org/httpcomponents-client-4.5.x/current/httpclient/apidocs/org/apache/http/impl/client/LaxRedirectStrategy.html
- =:graceful= - Similar to =:default=, but does not throw exceptions when max redirects is reached. This is the redirects behaviour in 2.x
- =nil= - When nil, assumes =:default=
You may also pass in an instance of RedirectStrategy (in the =:redirect-strategy= key) if you want a
behavior that's not implemented.
Additionally, clj-http will attempt to validate that a redirect host is not invalid, you can disable
this by setting =:validate-redirects false= in the request (the default is true)
NOTE: The options =:force-redirects= and =:follow-redirects= (present in clj-http 2.x are no longer
used). You can use =:graceful= to mostly emulate the old redirect behaviour.
*** How to create a custom RedirectStrategy
:PROPERTIES:
:CUSTOM_ID: h:a3b8b124-411f-4c4c-ac4b-777624e76bf1
:END:
As mentioned earlier, it's possible to pass a custom instance of RedirectStrategy. The snippet below shows how to create a custom =RedirectStrategy= by wrapping the default strategy.
#+begin_src clojure
(def default-strategy org.apache.http.impl.client.DefaultRedirectStrategy/INSTANCE)
(def logging-redirect-strategy
(reify org.apache.http.client.RedirectStrategy
(getRedirect [this request response context]
(println "attempting redirect...")
(.getRedirect default-strategy request response context))
(isRedirected [this request response context]
(println "checking isRedirected")
(.isRedirected default-strategy request response context))))
(client/get "https://httpbin.org/absolute-redirect/3" {:redirect-strategy logging-redirect-strategy})
;; this will output the following:
;;
;; checking isRedirected
;; attempting redirect...
;; checking isRedirected
;; attempting redirect...
;; checking isRedirected
;; attempting redirect...
;; checking isRedirected
#+end_src
** Cookies
:PROPERTIES:
:CUSTOM_ID: h-3bb89b16-4be3-455e-98ec-c5ca5830ddb9
:END:
*** Cookiestores
:PROPERTIES:
:CUSTOM_ID: h-1d86fe30-f690-4c2a-9a1c-231669f4591a
:END:
clj-http can simplify the maintenance of cookies across requests if it is
provided with a _cookie store_.
#+BEGIN_SRC clojure
(binding [clj-http.core/*cookie-store* (clj-http.cookies/cookie-store)]
(client/post "http://example.com/login" {:form-params {:username "..."
:password "..."}})
(client/get "http://example.com/secured-page")
...)
#+END_SRC
(The =clj-http.cookies/cookie-store= function returns a new empty instance of a
default implementation of =org.apache.http.client.CookieStore=.)
This will allow cookies to only be _written_ to the cookie store. Cookies from
the cookie-store will not automatically be sent with future requests.
If you would like cookies from the cookie-store to automatically be sent with
each request, specify the cookie-store with the =:cookie-store= option:
#+BEGIN_SRC clojure
(let [my-cs (clj-http.cookies/cookie-store)]
(client/post "http://example.com/login" {:form-params {:username "..."
:password "..."}
:cookie-store my-cs})
(client/post "http://example.com/update" {:body my-data
:cookie-store my-cs}))
#+END_SRC
You can also use the =get-cookies= function to retrieve the cookies
from a cookie store:
#+BEGIN_SRC clojure
(def cs (clj-http.cookies/cookie-store))
(client/get "http://google.com" {:cookie-store cs})
(clojure.pprint/pprint (clj-http.cookies/get-cookies cs))
{"NID"
{:domain ".google.com",
:expires #,
:path "/",
:value
"58=c387....",
:version 0},
"PREF"
{:domain ".google.com",
:expires #,
:path "/",
:value
"ID=3ba...:FF=0:TM=133...:LM=133...:S=_iRM...",
:version 0}}
#+END_SRC
*** Keystores, Trust-stores
:PROPERTIES:
:CUSTOM_ID: h-7968467a-1441-4a73-9307-9a7a5fd8e733
:END:
You can also specify your own keystore/trust-store to be used:
#+BEGIN_SRC clojure
(client/get "https://example.com" {:keystore "/path/to/keystore.ks"
:keystore-type "jks" ; default: jks
:keystore-pass "secretpass"
:trust-store "/path/to/trust-store.ks"
:trust-store-type "jks" ; default jks
:trust-store-pass "trustpass"})
#+END_SRC
The =:keystore/:trust-store= values may be either paths to keystore
files or =KeyStore= instances.
** Exceptions
:PROPERTIES:
:CUSTOM_ID: h-ed9e04f1-1c7b-4c2e-9259-94d2a3e65a89
:END:
The client will throw exceptions on, well, exceptional status codes, meaning all
HTTP responses other than =#{200 201 202 203 204 205 206 207 300 301 302 303 304
307}=. clj-http will throw a [[http://github.com/scgilardi/slingshot][Slingshot]] Stone that can be caught by a regular
=(catch Exception e ...)= or in Slingshot's =try+= block:
#+BEGIN_SRC clojure
(client/get "http://example.com/broken")
=> ExceptionInfo clj-http: status 404 clj-http.client/wrap-exceptions/fn--583 (client.clj:41)
;; Or, if you would like the Exception message to contain the entire response:
(client/get "http://example.com/broken" {:throw-entire-message? true})
=> ExceptionInfo clj-http: status 404 {:status 404,
:headers {"server" "nginx/1.0.4",
"x-runtime" "12ms",
"content-encoding" "gzip",
"content-type" "text/html; charset=utf-8",
"date" "Mon, 17 Oct 2011 23:15 :36 GMT",
"cache-control" "no-cache",
"status" "404 Not Found",
"transfer-encoding" "chunked",
"connection" "close"},
:body "...body here..."}
clj-http.client/wrap-exceptions/fn--584 (client.clj:42
;; You can also ignore HTTP-status-code exceptions and handle them yourself:
(client/get "http://example.com/broken" {:throw-exceptions false})
;; Or ignore an unknown host (methods return 'nil' if this is set to
;; true and the host does not exist:
(client/get "http://example.invalid" {:ignore-unknown-host? true})
;; Or customize the http statuses that will not throw:
(client/get "http://example.com/broken" {:unexceptional-status #(<= 200 % 299)})
#+END_SRC
(spacing added by me to be human readable)
How to use with Slingshot:
#+BEGIN_SRC clojure
; Response map is thrown as exception obj.
; We filter out by status codes
(try+
(client/get "http://example.com/broken")
(catch [:status 403] {:keys [request-time headers body]}
(log/warn "403" request-time headers))
(catch [:status 404] {:keys [request-time headers body]}
(log/warn "NOT Found 404" request-time headers body))
(catch Object _
(log/error (:throwable &throw-context) "unexpected error")
(throw+)))
#+END_SRC
** Decompression
:PROPERTIES:
:CUSTOM_ID: h-f780c96c-90be-4d83-9b53-227a9e5942ab
:END:
By default, clj-http will add the ={"Accept-Encoding" "gzip, deflate"}= header
to requests, and automatically decompress the resulting gzip or deflate stream
if the =Content-Encoding= header is found on the response. If this is undesired,
the ={:decompress-body false}= option can be specified:
#+BEGIN_SRC clojure
;; Auto-decompression used: (google requires a user-agent to send gzip data)
(def h {"User-Agent" "Mozilla/5.0 (Windows NT 6.1;) Gecko/20100101 Firefox/13.0.1"})
(def resp (client/get "http://google.com" {:headers h}))
(:orig-content-encoding resp)
=> "gzip" ;; <= google sent response gzipped
;; and without decompression:
(def resp2 (client/get "http://google.com" {:headers h :decompress-body false})
(:orig-content-encoding resp2)
=> nil
#+END_SRC
If clj-http decompresses something, the "content-encoding" header is removed
from the headers (because the encoding is no longer true). This allows clj-http
to be used as a pass-through proxy with ring. The original content-encoding is
available as =:orig-content-encoding= in the response map if auto-decompression
is enabled.
** Debugging
:PROPERTIES:
:CUSTOM_ID: debugging
:END:
There are four debugging methods you can use:
#+BEGIN_SRC clojure
;; print request info to *out*:
(client/get "http://example.org" {:debug true})
;; print request info to *out*, including request body:
(client/post "http://example.org" {:debug true :debug-body true :body "..."})
;; save the request that was sent in a :request key in the response:
(client/get "http://example.org" {:save-request? true})
;; save the request that was sent in a :request key in the response,
;; including the body content:
(client/get "http://example.org" {:save-request? true :debug-body true})
;; add an HttpResponseInterceptor to the request. This callback
;; is called for each redirects with the following args:
;; ^HttpResponse resp, HttpContext^ ctx
;; this allows low level debugging + access to socket.
;; see http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpResponseInterceptor.html
(client/get "http://example.org" {:response-interceptor (fn [resp ctx] (println ctx))})
#+END_SRC
*** Logging
:PROPERTIES:
:CUSTOM_ID: h-0d505652-d453-48a2-a868-46aef2b8af66
:END:
Finally, if you want to access the logging that the Apache client does internally, you can set up
your dependencies to add the [[https://logging.apache.org/log4j/2.x/][log4j2]] libraries and configure the logging for clj-http. In order to do
this, you'll need to add
#+BEGIN_SRC clojure
[org.apache.logging.log4j/log4j-api "2.11.0"]
[org.apache.logging.log4j/log4j-core "2.11.0"]
[org.apache.logging.log4j/log4j-1.2-api "2.11.0"]
#+END_SRC
To your =project.clj= and have a usable log4j2.properties. I have provided one in
=resources/log4j2.properties=. Make sure to set:
#+BEGIN_SRC fundamental
rootLogger.level = debug
#+END_SRC
If you want to see debug information (or "trace" for trace logging). When you perform a request you
should see something akin to this in the logs:
#+BEGIN_SRC fundamental
[2018-03-20T20:36:34,635][DEBUG][o.a.h.c.p.RequestAddCookies] CookieSpec selected: default
[2018-03-20T20:36:34,635][DEBUG][o.a.h.c.p.RequestAuthCache] Auth cache not set in the context
[2018-03-20T20:36:34,635][DEBUG][o.a.h.i.c.BasicHttpClientConnectionManager] Get connection for route {s}->https://example.com:443
[2018-03-20T20:36:34,636][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] http-outgoing-1: set socket timeout to 0
[2018-03-20T20:36:34,636][DEBUG][o.a.h.i.e.MainClientExec ] Opening connection {s}->https://example.com:443
[2018-03-20T20:36:34,644][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] Connecting to example.com/10.0.0.1:443
[2018-03-20T20:36:34,644][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Connecting socket to example.com/10.0.0.1:443 with timeout 0
[2018-03-20T20:36:34,692][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
[2018-03-20T20:36:34,693][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ... etc ...]
[2018-03-20T20:36:34,693][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Starting handshake
[2018-03-20T20:36:34,841][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Secure session established
[2018-03-20T20:36:34,842][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] negotiated protocol: TLSv1.2
[2018-03-20T20:36:34,842][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] peer principal: CN=example.com
[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] peer alternative names: [example.com, www.example.com]
[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] issuer principal: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] Connection established 192.168.0.29:36792<->10.0.0.1:443
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Executing request POST /post HTTP/1.1
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Target auth state: UNCHALLENGED
[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Proxy auth state: UNCHALLENGED
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> POST /post HTTP/1.1
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Connection: close
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> accept-encoding: gzip, deflate
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Content-Length: 14
[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Content-Type: text/plain; charset=UTF-8
[2018-03-20T20:36:34,846][DEBUG][o.a.h.headers ] http-outgoing-1 >> Host: example.com
[2018-03-20T20:36:34,846][DEBUG][o.a.h.headers ] http-outgoing-1 >> User-Agent: Apache-HttpClient/4.5.5 (Java/9.0.1)
[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "POST /post HTTP/1.1[\r][\n]"
[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Connection: close[\r][\n]"
[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "accept-encoding: gzip, deflate[\r][\n]"
[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Content-Length: 14[\r][\n]"
[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Host: example.com[\r][\n]"
etc etc it will go on forever and be very verbose
#+END_SRC
This provides both the data sent and received on the wire for debugging purposes.
I've also provided an example for changing the log level from clojure in
=examples/logging-apache-requests.clj=.
* Caching
:PROPERTIES:
:CUSTOM_ID: h-2c4ee611-ca22-432e-9c33-18040566661e
:END:
clj-http supports Apache's caching client, essentially it "provides an HTTP/1.1-compliant caching
layer to be used with HttpClient--the Java equivalent of a browser cache." (see [[https://hc.apache.org/httpcomponents-client-ga/tutorial/html/caching.html][the explanation in
the apache docs]]). In order to use the cache, a reusable connection manager *and* http-client must be
used.
An example of basic usage with the default options:
#+BEGIN_SRC clojure
(let [cm (conn/make-reusable-conn-manager {})
client (:http-client (http/get "http://example.com"
{:connection-manager cm :cache true}))]
(http/get "http://example.com"
{:connection-manager cm :http-client client :cache true})
(http/get "http://example.com"
{:connection-manager cm :http-client client :cache true})
(http/get "http://example.com"
{:connection-manager cm :http-client client :cache true}))
#+END_SRC
You can build your own cache config by providing either a map of caching configuration options, or
by providing a =CacheConfig= object, as seen below:
#+BEGIN_SRC clojure
(let [cm (conn/make-reusable-conn-manager {})
cache-config (core/build-cache-config
{:cache-config {:max-object-size 4096}})
client (:http-client (http/get "http://example.com"
{:connection-manager cm :cache true}))]
(http/get "http://example.com"
;; Use the default cache config settings
{:connection-manager cm :http-client client :cache true})
(http/get "http://example.com"
{:connection-manager cm :http-client client :cache true
;; Provide cache configuration options as a map
:cache-config {:max-object-size 9152
:max-cache-entries 100}})
(http/get "http://example.com"
{:connection-manager cm :http-client client :cache true
;; Provide the cache configuration as a CacheConfig object
:cache-config cache-config}))
#+END_SRC
In the response, clj-http provides the =:cached= key to indicate whether the response was cached,
missed, etc:
- nil :: Caching was not used for this request
- =:CACHE_HIT= :: A response was generated from the cache with no requests sent upstream.
- =:CACHE_MISS= :: The response came from an upstream server.
- =:CACHE_MODULE_RESPONSE= :: The response was generated directly by the caching module.
- =:VALIDATED= :: The response was generated from the cache after validating the entry with the origin server.
* Authentication
:PROPERTIES:
:CUSTOM_ID: h-87f38469-36b4-44c6-ae74-0d8f5e80c2ed
:END:
** Basic Auth
:PROPERTIES:
:CUSTOM_ID: h-d3ea348f-88ed-4193-bb16-d8d5accdc2aa
:END:
#+BEGIN_SRC clojure
(client/get "http://example.com/protected" {:basic-auth ["user" "pass"]})
(client/get "http://example.com/protected" {:basic-auth "user:pass"})
#+END_SRC
** Digest Auth
:PROPERTIES:
:CUSTOM_ID: h-d1904589-e71e-43db-8b93-0f94ccecaabe
:END:
#+BEGIN_SRC clojure
(client/get "http://example.com/protected" {:digest-auth ["user" "pass"]})
#+END_SRC
** NTLM Auth
:PROPERTIES:
:CUSTOM_ID: h-AE80FFDC-2016-4883-9512-2BE16640339D
:END:
#+BEGIN_SRC clojure
(client/get "http://example.com/protected" {:ntlm-auth ["user" "pass" "host" "domain"]})
#+END_SRC
** oAuth2
:PROPERTIES:
:CUSTOM_ID: h-dd077440-a1de-437e-b34e-5d6d0d1da4bd
:END:
#+BEGIN_SRC clojure
(client/get "http://example.com/protected" {:oauth-token "secret-token"})
#+END_SRC
* Advanced Usage
:PROPERTIES:
:CUSTOM_ID: h-d52ca837-a575-402f-81fe-53241d85f2db
:END:
** Raw Request
:PROPERTIES:
:CUSTOM_ID: h-0d2eadbf-c1ad-4514-a932-9d173582a790
:END:
A more general =request= function is also available, which is useful as a
primitive for building higher-level interfaces:
#+BEGIN_SRC clojure
(defn api-action [method path & [opts]]
(client/request
(merge {:method method :url (str "http://example.com/" path)} opts)))
#+END_SRC
*** Boolean options
:PROPERTIES:
:CUSTOM_ID: h-a37c718c-43bb-43ce-936a-21ef65147295
:END:
Since 0.9.0, all boolean options can be expressed as either ={:debug true}= or
={:debug? true}=, with or without the question mark.
** Persistent Connections
:PROPERTIES:
:CUSTOM_ID: h-4e9f116d-c293-4a0c-8e11-435c440bfe97
:END:
clj-http can use persistent connections to speed up connections if multiple
connections are being used:
#+BEGIN_SRC clojure
(with-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
(get "http://example.org/1")
(post "http://example.org/2")
(get "http://example.org/3")
...
(get "http://example.org/999"))
#+END_SRC
For async request, you can use =with-async-connection-pool=
#+BEGIN_SRC clojure
(with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
(get "http://example.org/1" {:async? true} resp1 exce1)
(post "http://example.org/2" {:async? true} resp2 exce2)
(get "http://example.org/3" {:async? true} resp3 exce3)
...
(get "http://example.org/999" {:async? true} resp999 exce999))
#+END_SRC
This is MUCH faster than sequentially performing all requests, because a
persistent connection can be used instead creating a new connection for each
request.
If you want to start an async request in the =respond= callback of an async request and
reuse the pool context, just use =reuse-pool=.
#+BEGIN_SRC clojure
(with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
(get "http://example.org/1" {:async? true} resp1 exce1)
(post "http://example.org/2"
{:async? true}
(fn [resp] (get "http://example.org/3"
(reuse-pool {:async? true} resp)
resp3 exce3))
exce2))
#+END_SRC
There are many advanced options available when creating asynchronous connection pools that can be
configured by passing an =:io-config= map in the connection manager parameters. It supports:
- =:connect-timeout=
- =:interest-op-queued=
- =:io-thread-count=
- =:rcv-buf-size=
- =:select-interval=
- =:shutdown-grace-period=
- =:snd-buf-size=
- =:so-keep-alive=
- =:so-linger=
- =:so-timeout=
- =:tcp-no-delay=
See the docstring on =with-async-connection-pool= for more information about these options.
If you would prefer to handle managing the connection manager yourself, you can
create a connection manager and specify it for each request:
#+BEGIN_SRC clojure
(def cm (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 2 :threads 3}))
(def cm2 (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 10 :threads 1}))
(get "http://example.org/1" {:connection-manager cm2})
(post "http://example.org/2" {:connection-manager cm})
(get "http://example.org/3" {:connection-manager cm2})
;; Don't forget to shut it down when you're done!
(clj-http.conn-mgr/shutdown-manager cm)
(clj-http.conn-mgr/shutdown-manager cm2)
#+END_SRC
See the docstring on =make-reusable-conn-manager= for options and default
values.
In the current version, pooled async request CANNOT specify connection manager.
** Re-using =HttpClient= between requests
:PROPERTIES:
:CUSTOM_ID: h-b79b07fb-d024-49a2-a7f7-53863d1b8d6d
:END:
In some cases, you may want to re-use the same =HttpClient= object between requests, either so you
don't have to build it every time, or because you make some configuration change to the request.
clj-http will return the built HTTP client in =:http-client= which you can then specify in
subsequent requests (with =:http-client=). Note that in order to reuse the client a connection
manager must be used.
#+BEGIN_SRC clojure
;; Re-use the HttpClient clj-http builds for you:
(let [cm (conn/make-reusable-conn-manager {})
resp (client/get "http://example.com" {:connection-manager cm})
hclient (:http-client resp)]
(client/get "http://example.com/1"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/2"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/3"
{:connection-manager cm :http-client hclient}))
;; You can also build your own, using clj-http's helper or manually building it:
(let [cm (conn/make-reusable-conn-manager {})
hclient (core/build-http-client {} false cm)]
(client/get "http://example.com/1"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/2"
{:connection-manager cm :http-client hclient})
(client/get "http://example.com/3"
{:connection-manager cm :http-client hclient}))
;; Async http clients may also be created and re-used:
(let [acm (conn/make-reuseable-async-conn-manager {})
ahclient (core/build-async-http-client {} acm)]
(client/get "http://example.com/1"
{:connection-manager cm :http-client ahclient}
handle-response handle-failure)
(client/get "http://example.com/2"
{:connection-manager cm :http-client ahclient}
handle-response handle-failure)
(client/get "http://example.com/3"
{:connection-manager cm :http-client ahclient}
handle-response handle-failure))
#+END_SRC
** Proxies
:PROPERTIES:
:CUSTOM_ID: h-49f9ca81-0bad-4cd8-87ac-c09a85fa5500
:END:
A proxy can be specified by setting the Java properties: =.proxyHost=
and =.proxyPort= where == is the client scheme used (normally
'http' or 'https'). =http.nonProxyHosts= allows you to specify a pattern for
hostnames which do not require proxy routing - this is shared for all schemes.
Additionally, per-request proxies can be specified with the =proxy-host= and
=proxy-port= options (this overrides =http.nonProxyHosts= too):
#+BEGIN_SRC clojure
(client/get "http://example.com" {:proxy-host "127.0.0.1" :proxy-port 8118})
#+END_SRC
Proxy credentials can also be explicitly set as
#+BEGIN_SRC clojure
(client/get "http://example.com" {:proxy-host "127.0.0.1" :proxy-port 8118 :proxy-user "proxy-user" :proxy-pass "superSecurePassword"})
#+END_SRC
You can also specify the =proxy-ignore-hosts= parameter with a list of
hosts where the proxy should be ignored. By default this list is
=#{"localhost" "127.0.0.1"}=.
A SOCKS proxy can be used by creating a proxied connection manager with
=clj-http.conn-mgr/make-socks-proxied-conn-manager=. Then using that connection
manager in the request.
For example if you wanted to connect to a local socks proxy on port =8081= you
would:
#+BEGIN_SRC clojure
(ns foo.bar
(:require [clj-http.client :as client]
[clj-http.conn-mgr :as conn-mgr]))
(client/get "https://google.com"
{:connection-manager
(conn-mgr/make-socks-proxied-conn-manager "localhost" 8081)})
#+END_SRC
If your SOCKS connection requires a keystore / trust-store, you can specify that too:
#+BEGIN_SRC clojure
(ns foo.bar
(:require [clj-http.client :as client]
[clj-http.conn-mgr :as conn-mgr]))
(client/get "https://google.com"
{:connection-manager
(conn-mgr/make-socks-proxied-conn-manager "localhost" 8081
{:keystore "/path/to/keystore.ks"
:keystore-type "jks" ; default: jks
:keystore-pass "secretpass"
:trust-store "/path/to/trust-store.ks"
:trust-store-type "jks" ; default jks
:trust-store-pass "trustpass"})})
#+END_SRC
You can also store the proxied connection manager and reuse it later.
** Custom Middleware
:PROPERTIES:
:CUSTOM_ID: h-c51cba6c-5c1b-4941-93c3-f769bb533562
:END:
Sometime it is desirable to run a request with some middleware enabled and some
left out, the =with-middleware= method provides this functionality:
#+BEGIN_SRC clojure
(with-middleware [#'clj-http.client/wrap-method
#'clj-http.client/wrap-url
#'clj-http.client/wrap-exceptions]
(get "http://example.com")
(post "http://example.com/foo" {:body (.getBytes "foo")}))
#+END_SRC
To see available middleware, check the =clj-http.client/default-middleware= var,
which is a vector of the default middleware that clj-http uses.
=clj-http.client/*current-middleware*= is bound to the current list of
middleware during request time.
** Modifying Apache-specific features of the =HttpClientBuilder= and =HttpAsyncClientBuilder=
:PROPERTIES:
:CUSTOM_ID: h:844f078c-531e-445e-b7ce-76092bcc9928
:END:
While clj-http tries to provide the features needed, there are times when it does not provide access
to a parameter that you need. In these cases, you can use a couple of advanced parameters to provide
arbitrary configuration functions to be run on the =HttpClientBuilder= by specifying
=:http-builder-fns= and =:async-http-builder-fns=.
Each of these variables is a sequence of functions of two arguments, the http builder
(=HttpClientBuilder= for =:http-builder-fns= and =HttpAsyncClientBuilder= for
=:async-http-builder-fns=) and the request map.
#+BEGIN_SRC clojure
;; A function that takes a builder and disables Apache's cookie management
(defun my-cookie-disabler [^HttpClientBuilder builder
request]
(when (:disable-cookies request)
(.disableCookieManagement builder)))
;; The functions to modify the builder are passed in
(http/post "http://www.example.org" {:http-builder-fns [my-cookie-disabler]
:disable-cookies true})
#+END_SRC
The functions are run in the order they are passed in (inside a =doseq=).
By specifying =:http-client-builder=, your own instance of
=HttpClientBuilder= will be used. A supplied =HttpClientBuilder= which
sets the connection manager, redirect strategy, retry handler, route
planner, cache, or cookie spec registry may find these overridden by
clj-http's =:connection-manager=, =:redirect-strategy=,
=:retry-handler=, =:cache=, or =:cookie-policy-registry= or
=:cookie-spec=, respectively.
** Incrementally JSON Parsing
:PROPERTIES:
:CUSTOM_ID: h:b01c16e8-7179-468e-8890-316939ec0e38
:END:
[[https://github.com/dakrone/cheshire][cheshire]] supports incrementally parsing JSON using lazy sequences. This approach can useful for
processing large top-level JSON arrays because it doesn't require upfront work consuming the entire stream.
#+begin_src clojure
(require '[cheshire.core :as json])
(defn print-all-pokemon-names [pokemons]
(for [pokemon pokemons]
(println (get-in pokemon [:name :english]))))
(let [url "https://raw.githubusercontent.com/fanzeyi/pokemon.json/master/pokedex.json"
response (get url {:as :reader})]
(with-open [reader (:body response)] ; closes the underlying connection when we're done
(let [pokemons (json/parse-stream reader true)]
; You must perform all reads from the stream inside `with-open`,
; any , any lazy
(doall (print-all-pokemon-names pokemons)))))
#+end_src
Keep in mind that the =reader= object wraps a HTTP connection. The user needs to be aware of two
things:
1. The user should close the reader after processing the stream, otherwise the underlying HTTP
Connection may leak and create subtle bugs. Clojure's [[https://clojuredocs.org/clojure.core/with-open][with-open]] is useful here.
2. You should realize any lazy sequences before closing the connection. Use [[https://clojuredocs.org/clojure.core/doall][doall]] or [[https://clojure.org/reference/transducers][transducers]] to
prevent bugs from lazy IO. See [[https://stuartsierra.com/2015/08/25/clojure-donts-lazy-effects][Clojure Don'ts: Lazy Effects]].
In previous versions of =clj-http= (<= 3.10.0), =clj-http= defaulted to lazily parsing JSON, but this
was slow and also confused users who didn't expect laziness.
** DNS Resolution
:PROPERTIES:
:CUSTOM_ID: h:52CC15DF-57A5-425E-9AFC-10C9B4C4FA83
:END:
Users may add their own DNS resolver function to override the default DNS Resolver. This is useful in situations where you are unable to change the name to IP Address mapping. It is analogous to the =--resolve= flag present in =curl=. This example uses =org.apache.http.impl.conn.InMemoryDnsResolver= to resolve =example.com= to IP Address =127.0.0.1=.
#+BEGIN_SRC clojure
(client/get "https://example.com" {:dns-resolver (doto (InMemoryDnsResolver.)
(.add "example.com" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))})
#+END_SRC
This option is supported for all of the connection managers.
The =dns-resolver= can be any instance of =DnsResolver=. Here is an example of a custom implementation that attempts to look up the hostname in the supplied map and falls back to the default SystemDnsResolver if not found. Note how IPV6 addresses are specified.
#+BEGIN_SRC clojure
(defn custom-dns-resolver
[host-map]
(let [system-dns-resolver (org.apache.http.impl.conn.SystemDefaultDnsResolver.)]
(reify
org.apache.http.conn.DnsResolver
(^"[Ljava.net.InetAddress;" resolve [this ^String host]
(if-let [address (get host-map host)]
(into-array [(java.net.InetAddress/getByAddress host (byte-array address))])
(.resolve system-dns-resolver host))))))
(client/get "https://example.com" {:dns-resolver (custom-dns-resolver {"example.com" [127 0 0 1]
"www.google.com" [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]})})
#+END_SRC
* Development
:PROPERTIES:
:CUSTOM_ID: h-65bbf017-2e8b-4c43-824b-24b89cc27a70
:END:
Please send a pull request or open an issue if you have any problems. See =CONTRIBUTING.md= for more
information.
** Faking Responses
:PROPERTIES:
:CUSTOM_ID: h-c3d9c7e0-cc3f-47bf-91e3-b12567b08eb6
:END:
If you need to fake clj-http responses (for things like testing and such), check
out the [[https://github.com/myfreeweb/clj-http-fake][clj-http-fake]] library.
** Optional Dependencies
:PROPERTIES:
:CUSTOM_ID: h-f1fbdad3-cf40-41e0-8ae0-8716419be228
:END:
In 2.0.0+ clj-http's optional dependencies at excluded by default, in order to
use the features you will need to add them to your =project.clj= file.
clj-http currently has four optional dependencies, =cheshire=, =crouton=,
=tools.reader= and =ring/ring-codec=. Any number of them may be included by
adding them with the clj-http dependency in your project.clj:
#+BEGIN_SRC clojure
;; optional dependencies
[cheshire] ;; for :as :json
[crouton] ;; for :decode-body-headers
[org.clojure/tools.reader] ;; for :as :clojure
[ring/ring-codec] ;; for :as :x-www-form-urlencoded
[com.cognitect/transit-clj] ;; for transit support
#+END_SRC
Prior to 2.0.0, you can /exclude/ the dependencies and clj-http will work
without them.
** clj-http-lite
:PROPERTIES:
:CUSTOM_ID: h-ba6b263b-74a5-40f3-afc1-b0d785554c2b
:END:
Like clj-http but need something more lightweight without as any external
dependencies? Check out [[https://github.com/clj-commons/clj-http-lite][clj-http-lite]] for a project that can be used as a
drop-in replacement for clj-http.
** Troubleshooting
:PROPERTIES:
:CUSTOM_ID: h-c543201e-a0e5-4e84-8eb2-6bf3e21a3140
:END:
*** VerifyError class org.codehaus.jackson.smile.SmileParser overrides final method getBinaryValue...
:PROPERTIES:
:CUSTOM_ID: h-c3a8ebc3-a247-4327-8b71-0097d1380873
:END:
This is actually caused by your project attempting to use [[https://github.com/mmcgrana/clj-json/][clj-json]] and [[https://github.com/dakrone/cheshire][cheshire]]
in the same classloader. You can fix the issue by either not using clj-json (and
thus choosing cheshire), or specifying an exclusion for clj-http in your project
like this:
#+BEGIN_SRC clojure
(defproject foo "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.3.0"]
[clj-http "0.3.4" :exclusions [cheshire]]])
#+END_SRC
Note that if you exclude cheshire, json decoding of response bodies
and json encoding of form-params cannot happen, you are responsible
for your own encoding/decoding.
As of clj-http 0.3.5, you should no longer see this, as Cheshire 3.1.0
and clj-json can now live together without causing problems.
*** NoHttpResponseException ... due to stale connections**
:PROPERTIES:
:CUSTOM_ID: h-9d7cf050-ed5b-4d23-8b02-97a9b9c94737
:END:
Persistent connections kept alive by the connection manager become stale: the
target server shuts down the connection on its end without HttpClient being able
to react to that event, while the connection is being idle, thus rendering the
connection half-closed or 'stale'.
This can be solved by using (with-connection-pool) as described in the
'Using Persistent Connection' section above.
* Tests
:PROPERTIES:
:CUSTOM_ID: h-a52feb3d-d966-4287-a07e-ad7aa7918fd5
:END:
To run the tests:
#+BEGIN_SRC
$ lein deps
$ lein test
Run all tests (including integration):
$ lein test :all
Run tests against all clojure versions
$ lein all test
$ lein all test :all
#+END_SRC
* Testimonials
:PROPERTIES:
:CUSTOM_ID: h-3044d1f7-6772-43c2-9ded-8c71c7f9ada2
:END:
With over [[https://clojars.org/clj-http][three million]] downloads, clj-http is a widely used, battle-tested clojure library. It is
also included in other libraries (like database clients) as a low-level http wrapper.
Libraries using clj-http:
- [[https://github.com/mattrepl/clj-oauth][clj-oauth]]
- [[https://github.com/clojurewerkz/elastisch][elasticsearch]]
- [[https://github.com/olauzon/capacitor][influxdb]]
Libraries inspired by clj-http:
- [[https://github.com/mpenet/jet][jet]]
- [[https://github.com/clj-commons/clj-http-lite][clj-http-lite]]
* Other Libraries Providing Middleware
:PROPERTIES:
:CUSTOM_ID: other-middleware
:END:
- [[https://github.com/sharetribe/aws-sig4][aws-sig4]] :: a pure clojure implementation of AWS v4 signature request signing as middleware
(feel free to open a PR or issue if you'd like to add middleware here)
* License
:PROPERTIES:
:CUSTOM_ID: h-2de3db75-7a1b-42b8-ad3b-6ef27fc2a5ea
:END:
Released under the MIT License:
# Local Variables:
# fill-column: 100
# End: