{"id":13395030,"url":"https://github.com/typhoeus/typhoeus","last_synced_at":"2025-05-12T03:36:38.287Z","repository":{"id":504740,"uuid":"132108","full_name":"typhoeus/typhoeus","owner":"typhoeus","description":" Typhoeus wraps libcurl in order to make fast and reliable requests.","archived":false,"fork":false,"pushed_at":"2025-03-12T17:13:03.000Z","size":1491,"stargazers_count":4092,"open_issues_count":112,"forks_count":438,"subscribers_count":56,"default_branch":"master","last_synced_at":"2025-05-09T12:04:01.580Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://rubydoc.info/github/typhoeus/typhoeus","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/typhoeus.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2009-02-18T23:14:50.000Z","updated_at":"2025-05-08T11:37:49.000Z","dependencies_parsed_at":"2024-11-05T18:36:27.039Z","dependency_job_id":"e789b3e7-e3de-444e-8c8e-22befbe6b260","html_url":"https://github.com/typhoeus/typhoeus","commit_stats":{"total_commits":1201,"total_committers":157,"mean_commits":7.649681528662421,"dds":0.6053288925895087,"last_synced_commit":"f5c5751df49089da89fc2683a23df04850a45604"},"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typhoeus%2Ftyphoeus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typhoeus%2Ftyphoeus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typhoeus%2Ftyphoeus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typhoeus%2Ftyphoeus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/typhoeus","download_url":"https://codeload.github.com/typhoeus/typhoeus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253418118,"owners_count":21905326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T17:01:39.781Z","updated_at":"2025-05-12T03:36:38.263Z","avatar_url":"https://github.com/typhoeus.png","language":"Ruby","readme":"# Typhoeus [![CI](https://github.com/typhoeus/typhoeus/actions/workflows/ci.yml/badge.svg)](https://github.com/typhoeus/typhoeus/actions/workflows/ci.yml) [![Experimental](https://github.com/typhoeus/typhoeus/actions/workflows/experimental.yml/badge.svg)](https://github.com/typhoeus/typhoeus/actions/workflows/experimental.yml) [![Code Climate](https://img.shields.io/codeclimate/maintainability/typhoeus/typhoeus.svg)](https://codeclimate.com/github/typhoeus/typhoeus) [![Gem Version](https://img.shields.io/gem/v/typhoeus.svg)](https://rubygems.org/gems/typhoeus)\n\nLike a modern code version of the mythical beast with 100 serpent heads, Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling logic.\n\n## Example\n\nA single request:\n\n```ruby\nTyphoeus.get(\"www.example.com\", followlocation: true)\n```\n\nParallel requests:\n\n```ruby\nhydra = Typhoeus::Hydra.new\n10.times.map{ hydra.queue(Typhoeus::Request.new(\"www.example.com\", followlocation: true)) }\nhydra.run\n```\n\n## Installation\nRun:\n```\nbundle add typhoeus\n```\n\nOr install it yourself as:\n\n```\ngem install typhoeus\n```\n\n## Project Tracking\n\n* [API Documentation](https://rubydoc.info/github/typhoeus/typhoeus/frames/Typhoeus) (GitHub master)\n\n## Usage\n\n### Introduction\n\nThe primary interface for Typhoeus is comprised of three classes: Request, Response, and Hydra. Request represents an HTTP request object, response represents an HTTP response, and Hydra manages making parallel HTTP connections.\n\n```ruby\nrequest = Typhoeus::Request.new(\n  \"www.example.com\",\n  method: :post,\n  body: \"this is a request body\",\n  params: { field1: \"a field\" },\n  headers: { Accept: \"text/html\" }\n)\n```\n\nWe can see from this that the first argument is the url. The second is a set of options.\nThe options are all optional. The default for `:method` is `:get`.\n\nWhen you want to send URL parameters, you can use `:params` hash to do so. Please note that in case of you should send a request via `x-www-form-urlencoded` parameters, you need to use `:body` hash instead. `params` are for URL parameters and `:body` is for the request body.\n\n#### Sending requests through the proxy\n\nAdd a proxy url to the list of options:\n\n```ruby\noptions = {proxy: 'http://myproxy.org'}\nreq = Typhoeus::Request.new(url, options)\n```\n\nIf your proxy requires authentication, add it with `proxyuserpwd` option key:\n\n```ruby\noptions = {proxy: 'http://proxyurl.com', proxyuserpwd: 'user:password'}\nreq = Typhoeus::Request.new(url, options)\n```\n\nNote that `proxyuserpwd` is a colon-separated username and password, in the vein of basic auth `userpwd` option.\n\n\nYou can run the query either on its own or through the hydra:\n\n``` ruby\nrequest.run\n#=\u003e \u003cTyphoeus::Response ... \u003e\n```\n\n```ruby\nhydra = Typhoeus::Hydra.hydra\nhydra.queue(request)\nhydra.run\n```\n\nThe response object will be set after the request is run.\n\n```ruby\nresponse = request.response\nresponse.code\nresponse.total_time\nresponse.headers\nresponse.body\n```\n\n### Making Quick Requests\n\nTyphoeus has some convenience methods for performing single HTTP requests. The arguments are the same as those you pass into the request constructor.\n\n```ruby\nTyphoeus.get(\"www.example.com\")\nTyphoeus.head(\"www.example.com\")\nTyphoeus.put(\"www.example.com/posts/1\", body: \"whoo, a body\")\nTyphoeus.patch(\"www.example.com/posts/1\", body: \"a new body\")\nTyphoeus.post(\"www.example.com/posts\", body: { title: \"test post\", content: \"this is my test\"})\nTyphoeus.delete(\"www.example.com/posts/1\")\nTyphoeus.options(\"www.example.com\")\n```\n#### Sending params in the body with PUT\nWhen using POST the content-type is set automatically to 'application/x-www-form-urlencoded'. That's not the case for any other method like PUT, PATCH, HEAD and so on,  irrespective of whether you are using body or not. To get the same result as POST, i.e. a hash in the body coming through as params in the receiver, you need to set the content-type as shown below:\n```ruby\nTyphoeus.put(\"www.example.com/posts/1\",\n        headers: {'Content-Type'=\u003e \"application/x-www-form-urlencoded\"},\n        body: {title:\"test post updated title\", content: \"this is my updated content\"}\n    )\n```\n\n### Handling HTTP errors\n\nYou can query the response object to figure out if you had a successful\nrequest or not. Here’s some example code that you might use to handle errors.\nThe callbacks are executed right after the request is finished, make sure to define\nthem before running the request.\n\n```ruby\nrequest = Typhoeus::Request.new(\"www.example.com\", followlocation: true)\n\nrequest.on_complete do |response|\n  if response.success?\n    # hell yeah\n  elsif response.timed_out?\n    # aw hell no\n    log(\"got a time out\")\n  elsif response.code == 0\n    # Could not get an http response, something's wrong.\n    log(response.return_message)\n  else\n    # Received a non-successful http response.\n    log(\"HTTP request failed: \" + response.code.to_s)\n  end\nend\n\nrequest.run\n```\n\nThis also works with serial (blocking) requests in the same fashion. Both\nserial and parallel requests return a Response object.\n\n### Handling file uploads\n\nA File object can be passed as a param for a POST request to handle uploading\nfiles to the server. Typhoeus will upload the file as the original file name\nand use Mime::Types to set the content type.\n\n```ruby\nTyphoeus.post(\n  \"http://localhost:3000/posts\",\n  body: {\n    title: \"test post\",\n    content: \"this is my test\",\n    file: File.open(\"thesis.txt\",\"r\")\n  }\n)\n```\n\n### Streaming the response body\n\nTyphoeus can stream responses. When you're expecting a large response,\nset the `on_body` callback on a request. Typhoeus will yield to the callback\nwith chunks of the response, as they're read. When you set an `on_body` callback,\nTyphoeus will not store the complete response.\n\n```ruby\ndownloaded_file = File.open 'huge.iso', 'wb'\nrequest = Typhoeus::Request.new(\"www.example.com/huge.iso\")\nrequest.on_headers do |response|\n  if response.code != 200\n    raise \"Request failed\"\n  end\nend\nrequest.on_body do |chunk|\n  downloaded_file.write(chunk)\nend\nrequest.on_complete do |response|\n  downloaded_file.close\n  # Note that response.body is \"\"\nend\nrequest.run\n```\n\nIf you need to interrupt the stream halfway,\nyou can return the `:abort` symbol from the `on_body` block, example:\n\n```ruby\nrequest.on_body do |chunk|\n  buffer \u003c\u003c chunk\n  :abort if buffer.size \u003e 1024 * 1024\nend\n```\n\nThis will properly stop the stream internally and avoid any memory leak which\nmay happen if you interrupt with something like a `return`, `throw` or `raise`.\n\n### Making Parallel Requests\n\nGenerally, you should be running requests through hydra. Here is how that looks:\n\n```ruby\nhydra = Typhoeus::Hydra.hydra\n\nfirst_request = Typhoeus::Request.new(\"http://example.com/posts/1\")\nfirst_request.on_complete do |response|\n  third_url = response.body\n  third_request = Typhoeus::Request.new(third_url)\n  hydra.queue third_request\nend\nsecond_request = Typhoeus::Request.new(\"http://example.com/posts/2\")\n\nhydra.queue first_request\nhydra.queue second_request\nhydra.run # this is a blocking call that returns once all requests are complete\n```\n\nThe execution of that code goes something like this. The first and second requests are built and queued. When hydra is run the first and second requests run in parallel. When the first request completes, the third request is then built and queued, in this example based on the result of the first request. The moment it is queued Hydra starts executing it.  Meanwhile the second request would continue to run (or it could have completed before the first). Once the third request is done, `hydra.run` returns.\n\nHow to get an array of response bodies back after executing a queue:\n\n```ruby\nhydra = Typhoeus::Hydra.new\nrequests = 10.times.map {\n  request = Typhoeus::Request.new(\"www.example.com\", followlocation: true)\n  hydra.queue(request)\n  request\n}\nhydra.run\n\nresponses = requests.map { |request|\n  request.response.body\n}\n```\n`hydra.run` is a blocking request. You can also use the `on_complete` callback to handle each request as it completes:\n\n```ruby\nhydra = Typhoeus::Hydra.new\n10.times do\n  request = Typhoeus::Request.new(\"www.example.com\", followlocation: true)\n  request.on_complete do |response|\n    #do_something_with response\n  end\n  hydra.queue(request)\nend\nhydra.run\n```\n\n### Making Parallel Requests with Faraday + Typhoeus\n\n```ruby\nrequire 'faraday'\n\nconn = Faraday.new(:url =\u003e 'http://httppage.com') do |builder|\n  builder.request  :url_encoded\n  builder.response :logger\n  builder.adapter  :typhoeus\nend\n\nconn.in_parallel do\n  response1 = conn.get('/first')\n  response2 = conn.get('/second')\n\n  # these will return nil here since the\n  # requests have not been completed\n  response1.body\n  response2.body\nend\n\n# after it has been completed the response information is fully available\n# response1.status, etc\nresponse1.body\nresponse2.body\n```\n\n### Specifying Max Concurrency\n\nHydra will also handle how many requests you can make in parallel. Things will get flakey if you try to make too many requests at the same time. The built in limit is 200. When more requests than that are queued up, hydra will save them for later and start the requests as others are finished. You can raise or lower the concurrency limit through the Hydra constructor.\n\n```ruby\nTyphoeus::Hydra.new(max_concurrency: 20)\n```\n\n### Memoization\n\nHydra memoizes requests within a single run call. You have to enable memoization.\nThis will result in a single request being issued. However, the on_complete handlers of both will be called.\n\n```ruby\nTyphoeus::Config.memoize = true\n\nhydra = Typhoeus::Hydra.new(max_concurrency: 1)\n2.times do\n  hydra.queue Typhoeus::Request.new(\"www.example.com\")\nend\nhydra.run\n```\n\nThis will result in two requests.\n\n```ruby\nTyphoeus::Config.memoize = false\n\nhydra = Typhoeus::Hydra.new(max_concurrency: 1)\n2.times do\n  hydra.queue Typhoeus::Request.new(\"www.example.com\")\nend\nhydra.run\n```\n\n### Caching\n\nTyphoeus includes built in support for caching. In the following example, if there is a cache hit, the cached object is passed to the on_complete handler of the request object.\n\n```ruby\nclass Cache\n  def initialize\n    @memory = {}\n  end\n\n  def get(request)\n    @memory[request]\n  end\n\n  def set(request, response)\n    @memory[request] = response\n  end\nend\n\nTyphoeus::Config.cache = Cache.new\n\nTyphoeus.get(\"www.example.com\").cached?\n#=\u003e false\nTyphoeus.get(\"www.example.com\").cached?\n#=\u003e true\n```\n\nFor use with [Dalli](https://github.com/mperham/dalli):\n\n```ruby\nrequire \"typhoeus/cache/dalli\"\n\ndalli = Dalli::Client.new(...)\nTyphoeus::Config.cache = Typhoeus::Cache::Dalli.new(dalli)\n```\n\nFor use with Rails:\n\n```ruby\nrequire \"typhoeus/cache/rails\"\n\nTyphoeus::Config.cache = Typhoeus::Cache::Rails.new\n```\n\nFor use with [Redis](https://github.com/redis/redis-rb):\n\n```ruby\nrequire \"typhoeus/cache/redis\"\n\nredis = Redis.new(...)\nTyphoeus::Config.cache = Typhoeus::Cache::Redis.new(redis)\n```\n\nAll three of these adapters take an optional keyword argument `default_ttl`, which sets a default\nTTL on cached responses (in seconds), for requests which do not have a cache TTL set.\n\nYou may also selectively choose not to cache by setting `cache` to `false` on a request or to use\na different adapter.\n\n```ruby\ncache = Cache.new\nTyphoeus.get(\"www.example.com\", cache: cache)\n```\n\n### Direct Stubbing\n\nHydra allows you to stub out specific urls and patterns to avoid hitting\nremote servers while testing.\n\n```ruby\nresponse = Typhoeus::Response.new(code: 200, body: \"{'name' : 'paul'}\")\nTyphoeus.stub('www.example.com').and_return(response)\n\nTyphoeus.get(\"www.example.com\") == response\n#=\u003e true\n```\n\nThe queued request will hit the stub. You can also specify a regex to match urls.\n\n```ruby\nresponse = Typhoeus::Response.new(code: 200, body: \"{'name' : 'paul'}\")\nTyphoeus.stub(/example/).and_return(response)\n\nTyphoeus.get(\"www.example.com\") == response\n#=\u003e true\n```\n\nYou may also specify an array for the stub to return sequentially.\n\n```ruby\nTyphoeus.stub('www.example.com').and_return([response1, response2])\n\nTyphoeus.get('www.example.com') == response1 #=\u003e true\nTyphoeus.get('www.example.com') == response2 #=\u003e true\n```\n\nWhen testing make sure to clear your expectations or the stubs will persist between tests. The following can be included in your spec_helper.rb file to do this automatically.\n\n```ruby\nRSpec.configure do |config|\n  config.before :each do\n    Typhoeus::Expectation.clear\n  end\nend\n```\n\n### Timeouts\n\nNo exceptions are raised on HTTP timeouts. You can check whether a request timed out with the following method:\n\n```ruby\nTyphoeus.get(\"www.example.com\", timeout: 1).timed_out?\n```\n\nTimed out responses also have their success? method return false.\n\nThere are two different timeouts available: [`timeout`](http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTTIMEOUT)\nand [`connecttimeout`](http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUT).\n`timeout` is the time limit for the entire request in seconds.\n`connecttimeout` is the time limit for just the connection phase, again in seconds.\n\nThere are two additional more fine grained options `timeout_ms` and\n`connecttimeout_ms`. These options offer millisecond precision but are not always available (for instance on linux if `nosignal` is not set to true).\n\nWhen you pass a floating point `timeout` (or `connecttimeout`) Typhoeus will set `timeout_ms` for you if it has not been defined. The actual timeout values passed to curl will always be rounded up.\n\nDNS timeouts of less than one second are not supported unless curl is compiled with an asynchronous resolver.\n\nThe default `timeout` is 0 (zero) which means curl never times out during transfer. The default `connecttimeout` is 300 seconds. A `connecttimeout` of 0 will also result in the default `connecttimeout` of 300 seconds.\n\n### Following Redirections\n\nUse `followlocation: true`, eg:\n\n```ruby\nTyphoeus.get(\"www.example.com\", followlocation: true)\n```\n\n### Basic Authentication\n\n```ruby\nTyphoeus::Request.get(\"www.example.com\", userpwd: \"user:password\")\n```\n\n### Compression\n\n```ruby\nTyphoeus.get(\"www.example.com\", accept_encoding: \"gzip\")\n```\n\nThe above has a different behavior than setting the header directly in the header hash, eg:\n```ruby\nTyphoeus.get(\"www.example.com\", headers: {\"Accept-Encoding\" =\u003e \"gzip\"})\n```\n\nSetting the header hash directly will not include the `--compressed` flag in the libcurl command and therefore libcurl will not decompress the response.  If you want the `--compressed` flag to be added automatically, set `:accept_encoding` Typhoeus option.\n\n\n### Cookies\n\n```ruby\nTyphoeus::Request.get(\"www.example.com\", cookiefile: \"/path/to/file\", cookiejar: \"/path/to/file\")\n```\n\nHere, `cookiefile` is a file to read cookies from, and `cookiejar` is a file to write received cookies to.\nIf you just want cookies enabled, you need to pass the same filename for both options.\n\n### Other CURL options\n\nAre available and documented [here](http://rubydoc.info/github/typhoeus/ethon/Ethon/Easy/Options)\n\n### SSL\n\nSSL comes built in to libcurl so it’s in Typhoeus as well. If you pass in a\nurl with \"https\" it should just work assuming that you have your [cert\nbundle](http://curl.haxx.se/docs/caextract.html) in order and the server is\nverifiable. You must also have libcurl built with SSL support enabled. You can\ncheck that by doing this:\n\n```\ncurl --version\n```\n\nNow, even if you have libcurl built with OpenSSL you may still have a messed\nup cert bundle or if you’re hitting a non-verifiable SSL server then you’ll\nhave to disable peer verification to make SSL work. Like this:\n\n```ruby\nTyphoeus.get(\"https://www.example.com\", ssl_verifypeer: false)\n```\n\nIf you are getting \"SSL: certificate subject name does not match target host\nname\" from curl (ex:- you are trying to access to b.c.host.com when the\ncertificate subject is \\*.host.com). You can disable host verification. Like\nthis:\n\n```ruby\n# host checking enabled\nTyphoeus.get(\"https://www.example.com\", ssl_verifyhost: 2)\n# host checking disabled\nTyphoeus.get(\"https://www.example.com\", ssl_verifyhost: 0)\n```\n\n### Verbose debug output\n\nIt’s sometimes useful to see verbose output from curl. You can enable it on a per-request basis:\n\n```ruby\nTyphoeus.get(\"http://example.com\", verbose: true)\n```\n\nor globally:\n\n```ruby\nTyphoeus::Config.verbose = true\n```\n\nJust remember that libcurl prints it’s debug output to the console (to\nSTDERR), so you’ll need to run your scripts from the console to see it.\n\n### Default User Agent Header\n\nIn many cases, all HTTP requests made by an application require the same User-Agent header set. Instead of supplying it on a per-request basis by supplying a custom header, it is possible to override it for all requests using:\n\n\n```ruby\nTyphoeus::Config.user_agent = \"custom user agent\"\n```\n\n### Running the specs\n\nRunning the specs should be as easy as:\n\n```\nbundle install\nbundle exec rake\n```\n## Semantic Versioning\n\nThis project conforms to [semver](http://semver.org/).\n\n## LICENSE\n\n(The MIT License)\n\nCopyright © 2009-2010 [Paul Dix](http://www.pauldix.net/)\n\nCopyright © 2011-2012 [David Balatero](https://github.com/dbalatero/)\n\nCopyright © 2012-2016 [Hans Hasselberg](http://github.com/i0rek/)\n\nPermission is hereby granted, free of charge, to any person obtaining a\ncopy of this software and associated documentation files (the \"Software\"),\nto deal in the Software without restriction, including without\nlimitation the rights to use, copy, modify, merge, publish, distribute,\nsublicense, and/or sell copies of the Software, and to permit persons\nto whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included\nin all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS\nOR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\nTHE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR\nOTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,\nARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR\nOTHER DEALINGS IN THE SOFTWARE.\n","funding_links":[],"categories":["Web Apps, Services \u0026 Interaction","Ruby","HTTP","HTTP Clients and tools","HarmonyOS"],"sub_categories":["HTTP clients","Windows Manager"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftyphoeus%2Ftyphoeus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftyphoeus%2Ftyphoeus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftyphoeus%2Ftyphoeus/lists"}