Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mirage/ocaml-cohttp

An OCaml library for HTTP clients and servers using Lwt or Async
https://github.com/mirage/ocaml-cohttp

http http-client lwt ocaml unix

Last synced: 2 days ago
JSON representation

An OCaml library for HTTP clients and servers using Lwt or Async

Awesome Lists containing this project

README 

        
## ocaml-cohttp -- an OCaml library for HTTP clients and servers [![Main workflow](https://github.com/mirage/ocaml-cohttp/actions/workflows/workflow.yml/badge.svg)](https://github.com/mirage/ocaml-cohttp/actions/workflows/workflow.yml)



Cohttp is an OCaml library for creating HTTP daemons. It has a portable

HTTP parser, and implementations using various asynchronous programming

libraries:



* `Http` provides essential type definitions used in Cohttp and an extremely

  fast http parser. It is designed to have no dependencies and make it easy

  for other packages to easily interoperate with Cohttp.

* `Cohttp_lwt_unix` uses the [Lwt](https://ocsigen.org/lwt/) library, and

  specifically the UNIX bindings. It uses [ocaml-tls](https://github.com/mirleft/ocaml-tls)

  as the TLS implementation to handle HTTPS connections.

* `Cohttp_async` uses the [Async](https://realworldocaml.org/v1/en/html/concurrent-programming-with-async.html)

  library and `async_ssl` to handle HTTPS connections.

* `Cohttp_lwt` exposes an OS-independent Lwt interface, which is used

  by the [Mirage](https://mirage.io/) interface to generate standalone

  microkernels (use the cohttp-mirage subpackage).

* `Cohttp_lwt_jsoo` compiles to a JavaScript module that maps the Cohttp

  calls to XMLHTTPRequests.  This is used to compile OCaml libraries like

  the GitHub bindings to JavaScript and still run efficiently.

* `Cohttp_curl` uses `libcurl`, via `ocurl`, as backend. It also comes

  with lwt (`Cohttp_curl_lwt`) and async backends (`Cohttp_curl_async`).

* `Cohttp_eio` uses `eio` to leverage new features from multicore ocaml 5.0.

* `Cohttp_server_lwt_unix` uses lwt to implement a more efficient web server

  with a minimal interface.



You can implement other targets using the parser very easily. Look at the `IO`

signature in `lib/s.mli` and implement that in the desired backend.



You can find help from cohttp users and maintainers at the

[discuss.ocaml.org](https://discuss.ocaml.org) forum or on the

[OCaml discord server](https://discord.gg/cCYQbqN).



## Table of contents



- [Installation](#installation)

- [Client Tutorial](#client-tutorial)

  * [Compile and execute with dune](#compile-and-execute-with-dune)

- [Dealing with timeouts](#dealing-with-timeouts)

- [Managing sessions](#managing-sessions)

- [Multipart form data](#multipart-form-data)

- [Creating custom resolver: a Docker Socket Client example](#creating-custom-resolver--a-docker-socket-client-example)

- [Dealing with redirects](#dealing-with-redirects)

- [Basic Server Tutorial](#basic-server-tutorial)

  * [Compile and execute with dune](#compile-and-execute-with-dune-1)

- [Installed Binaries](#installed-binaries)

- [Debugging](#debugging)

- [Important Links](#important-links)



## Installation



Latest stable version should be obtained from `opam`. Make sure to install the

specific backends you want as well. E.g.



```

$ opam install cohttp-lwt-unix cohttp-async

```



You can also obtain the development release:



```

$ opam pin add cohttp --dev-repo

```



## Client Tutorial



Cohttp provides clients for Async, Lwt, and Js_of_ocaml (Lwt based). In this tutorial,

we will use the lwt client but the example should be easily translatable to Async.



To create a simple request, use one of the methods in `Cohttp_lwt_unix.Client`.

`call` is the most general, there are also http method specialized such as

`get`, `post`, etc.



For example downloading the reddit frontpage:



```ocaml

open Lwt

open Cohttp

open Cohttp_lwt_unix



let body =

  Client.get (Uri.of_string "https://www.reddit.com/") >>= fun (resp, body) ->

  let code = resp |> Response.status |> Code.code_of_status in

  Printf.printf "Response code: %d\n" code;

  Printf.printf "Headers: %s\n" (resp |> Response.headers |> Header.to_string);

  body |> Cohttp_lwt.Body.to_string >|= fun body ->

  Printf.printf "Body of length: %d\n" (String.length body);

  body



let () =

  let body = Lwt_main.run body in

  print_endline ("Received body\n" ^ body)

```



There are a few things to notice:



* We open 2 modules. `Cohttp` contains the backend independent modules and

  `Cohttp_lwt_unix` the lwt + unix specific ones.

* `Client.get` accepts a `Uri.t` and makes an http request. `Client.get` also

  accepts optional arguments for things like header information.

* The http response is returned in a tuple. The first element of the tuple

  contains the response's status code, headers, http version, etc. The second

  element contains the body.

* The body is then converted to a string and is returned (after the length is

  printed). Note that `Cohttp_lwt.Body.to_string` hence it's up to us to keep

  a reference to the result.

* We must trigger lwt's event loop for the request to run. `Lwt_main.run` will

  run the event loop and return with final value of `body` which we then print.



Note that `Cohttp_lwt_unix`/`Cohttp_async` are able to request an HTTPS page

by default. For `Cohttp_lwt_unix` users can use [ocaml-tls](https://github.com/mirleft/ocaml-tls.git) by installing `tls-lwt` or [ocaml-ssl](https://github.com/savonet/ocaml-ssl) by installing `lwt_ssl`. The latter is the default if both are installed but it is possible to force the selection of tls with the environment variable `CONDUIT_TLS=native`. For `Cohttp_async` the default is to use

`async_ssl` (but users are able to use `ocaml-tls` with some modifications).



Consult the following modules for reference:



* [Cohttp_lwt.Client](https://github.com/mirage/ocaml-cohttp/blob/master/cohttp-lwt/src/s.ml)

* [Cohttp_async.Client](https://github.com/mirage/ocaml-cohttp/blob/master/cohttp-async/src/client.mli)



The full documentation for the latest published version of the library is

available on the [repository github pages](https://mirage.github.io/ocaml-cohttp/).



### Compile and execute with dune



Create this `dune` file

```

cat - > dune <|= fun v -> `Done v)

    ; (Lwt_unix.sleep time >|= fun () -> `Timeout)

    ]



let body =

  let get () = Client.get (Uri.of_string "https://www.reddit.com/") in

  compute ~time:0.1 ~f:get >>= function

  | `Timeout -> failwith "Timeout expired"

  | `Done (resp, body) -> Lwt.return (resp, body)

```



Executing the code, which you can actually try by calling

```

$ dune exec examples/lwt_unix_doc/client_lwt_timeout.exe

```

the call will most likely fail with the following output

```

Fatal error: exception (Failure "Timeout expired")

```



Similarly, in the case of `cohttp-async` you can directly use Async's

[`with_timeout`](https://ocaml.janestreet.com/ocaml-core/latest/doc/async_unix/Async_unix/Clock/index.html#val-with_timeout) function.

For example,



```ocaml

let get_body ~uri ~timeout =

    let%bind _, body = Cohttp_async.Client.get ~interrupt:(after (sec timeout)) uri in

    Body.to_string body



let body =

  let uri = Uri.of_string "https://www.reddit.com/" in

  let timeout = 0.1 in

  Clock.with_timeout (sec timeout) (get_body ~uri ~timeout)

  >>| function

  | `Result body -> Log.debug logger "body: %s" body

  | `Timeout  -> Log.debug logger "Timeout with url:%s" url

```



## Managing sessions



Managing sessions and saving cookies across requests is not directly supported by

`cohttp`. It is not hard to roll out a custom solution, but an alternative is

to use the [`session`](https://github.com/inhabitedtype/ocaml-session) library,

which is compatible with `cohttp`.



## Multipart form data



Multipart form data is not supported out of the box but is provided by external libraries:

- [`multipart_form`](https://github.com/dinosaure/multipart_form) which has bounded memory consumption even when transferring large amount of data

- [`multipart-form-data`](https://github.com/cryptosense/multipart-form-data)

- [`http-multipart-formdata`](https://github.com/lemaetech/http-multipart-formdata) which however does not support streaming



## Creating custom resolver: a Docker Socket Client example



Cohttp provides a lot of utilities out of the box, but does not prevent the users

to dig in and customise it for their needs. The following is an example of a

[unix socket client to communicate with Docker](https://discuss.ocaml.org/t/how-to-write-a-simple-socket-based-web-client-for-docker/1760/3).



```ocaml

open Lwt.Infix

open Cohttp



let ctx =

  let resolver =

    let h = Hashtbl.create 1 in

    Hashtbl.add h "docker" (`Unix_domain_socket "/var/run/docker.sock");

    Resolver_lwt_unix.static h

  in

  Cohttp_lwt_unix.Client.custom_ctx ~resolver ()



let t =

  Cohttp_lwt_unix.Client.get ~ctx (Uri.of_string "http://docker/version")

  >>= fun (resp, body) ->

  let open Cohttp in

  let code = resp |> Response.status |> Code.code_of_status in

  Printf.printf "Response code: %d\n" code;

  Printf.printf "Headers: %s\n" (resp |> Response.headers |> Header.to_string);

  body |> Cohttp_lwt.Body.to_string >|= fun body ->

  Printf.printf "Body of length: %d\n" (String.length body);

  print_endline ("Received body\n" ^ body)



let _ = Lwt_main.run t

```



The main issue there is there no way to resolve a socket address, so you need to

create a custom resolver to map a hostname to the Unix domain socket.



To build and execute with `dune`, first create the following `dune` file

```

$ cat - > dune < `OK then Cohttp_lwt.Body.drain_body body else Lwt.return_unit

  in

  match status with

  | `OK -> Lwt.return (response, body)

  | `Permanent_redirect | `Moved_permanently ->

      handle_redirect ~permanent:true ~max_redirects request_uri response

  | `Found | `Temporary_redirect ->

      handle_redirect ~permanent:false ~max_redirects request_uri response

  | `Not_found | `Gone -> failwith "Not found"

  | status ->

      Printf.ksprintf failwith "Unhandled status: %s"

          (Cohttp.Code.string_of_status status)



and handle_redirect ~permanent ~max_redirects request_uri response =

  if max_redirects <= 0 then failwith "Too many redirects"

  else

    let headers = Http.Response.headers response in

    let location = Http.Header.get headers "location" in

    match location with

    | None -> failwith "Redirection without Location header"

    | Some url ->

        let open Lwt.Syntax in

        let uri = Uri.of_string url in

        let* () =

          if permanent then

            Logs_lwt.warn (fun m ->

                m "Permanent redirection from %s to %s"

                  (Uri.to_string request_uri)

                  url)

          else Lwt.return_unit

        in

        http_get_and_follow uri ~max_redirects:(max_redirects - 1)

```



The following example, adapted from [blue-http](https://github.com/brendanlong/blue-http/blob/master/src/redirect.ml), does a similar thing with `cohttp-async` (and [ppx_let](https://github.com/janestreet/ppx_let)).



```ocaml

open Core_kernel

open Async_kernel



let with_redirects ~max_redirects uri f =

  let seen_uris = Hash_set.create (module String) in

  let rec loop ~max_redirects uri =

    Hash_set.add seen_uris (Uri.to_string uri);

    let%bind ((response, response_body) as res) = f uri in

    let status_code =

      Cohttp.(Response.status response |> Code.code_of_status)

    in

    if Cohttp.Code.is_redirection status_code then (

      match Cohttp.(Response.headers response |> Header.get_location) with

      | Some new_uri when Uri.to_string new_uri |> Hash_set.mem seen_uris ->

          return res

      | Some new_uri ->

          if max_redirects > 0 then

            (* Cohttp leaks connections if we don't drain the response body *)

            Cohttp_async.Body.drain response_body >>= fun () ->

            loop ~max_redirects:(max_redirects - 1) new_uri

          else (

            Log.Global.debug ~tags:[]

              "Ignoring %d redirect from %s to %s: redirect limit exceeded"

              status_code (Uri.to_string uri) (Uri.to_string new_uri);

            return res)

      | None ->

          Log.Global.debug ~tags:[]

            "Ignoring %d redirect from %s: there is no Location header"

            status_code (Uri.to_string uri);

          return res)

    else return res

  in

  loop ~max_redirects uri

```



You can read a bit more on the rationale behind the absence of this functionality in the API [here](https://github.com/mirage/ocaml-cohttp/issues/76).



## Basic Server Tutorial



Implementing a server in cohttp using the Lwt backend (for Async is very similar)

is mostly equivalent to implementing a function of type :



```

conn -> Http.Request.t -> Cohttp_lwt.Body.t -> (Http.Response.t * Cohttp_lwt.Body.t) Lwt.t

```



The parameters are self explanatory but we'll summarize them quickly here:



* `conn` - contains connection information

* `Http.Request.t` - Request information such as method, uri, headers, etc.

* `Cohttp_lwt.Body.t` - Contains the request body. You must manually decode the

  request body into json, form encoded pairs, etc. For cohttp, the body is

  simply binary data.



Here's an example of a simple cohttp server that outputs back request

information.



```ocaml

open Lwt

open Cohttp

open Cohttp_lwt_unix



let server =

  let callback _conn req body =

    let uri = req |> Request.uri |> Uri.to_string in

    let meth = req |> Request.meth |> Code.string_of_method in

    let headers = req |> Request.headers |> Header.to_string in

    ( body |> Cohttp_lwt.Body.to_string >|= fun body ->

      Printf.sprintf "Uri: %s\nMethod: %s\nHeaders\nHeaders: %s\nBody: %s" uri

        meth headers body )

    >>= fun body -> Server.respond_string ~status:`OK ~body ()

  in

  Server.create ~mode:(`TCP (`Port 8000)) (Server.make ~callback ())



let () = ignore (Lwt_main.run server)

```



### Compile and execute with dune



Create this `dune` file

```

cat - > dune < with_metadata header tags k ppf fmt in

  { Logs.report }



let () =

  Fmt_tty.setup_std_outputs ~style_renderer:`Ansi_tty ~utf_8:true ();

  Logs.set_reporter (reporter Fmt.stderr);

  Logs.set_level ~all:true (Some Logs.Debug)

```



Note that you can selectively filter out the logs produced by `cohttp-lwt` and `cohttp-lwt-unix` internals as follows.



```ocaml

let () =

  (* Set log level v for all loggers, this does also affect cohttp internal loggers *)

  Logs.set_level ~all:true level;

  (* Disable all cohttp-lwt and cohttp-lwt-unix logs *)

  List.iter (fun src ->

      match Logs.Src.name src with

      | "cohttp.lwt.io" | "cohttp.lwt.server" -> Logs.Src.set_level src None

      | _ -> ())

  @@ Logs.Src.list ()

```



## Important Links



- [Cohttp API Documentation](https://mirage.github.io/ocaml-cohttp/)

- [Conduit API Documentation](https://mirage.github.io/ocaml-conduit/)