{"id":16889741,"url":"https://github.com/benoitc/hackney_pooler","last_synced_at":"2025-08-12T00:07:40.187Z","repository":{"id":23745499,"uuid":"27119471","full_name":"benoitc/hackney_pooler","owner":"benoitc","description":"Experiment an API to limit the number of hackney requests launched concurrently","archived":false,"fork":false,"pushed_at":"2015-06-25T15:10:16.000Z","size":393,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-08-10T00:38:54.318Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benoitc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-11-25T09:58:05.000Z","updated_at":"2018-09-14T18:50:51.000Z","dependencies_parsed_at":"2022-08-22T04:30:24.942Z","dependency_job_id":null,"html_url":"https://github.com/benoitc/hackney_pooler","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/benoitc/hackney_pooler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoitc%2Fhackney_pooler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoitc%2Fhackney_pooler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoitc%2Fhackney_pooler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoitc%2Fhackney_pooler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benoitc","download_url":"https://codeload.github.com/benoitc/hackney_pooler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoitc%2Fhackney_pooler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269976794,"owners_count":24506467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T16:58:41.647Z","updated_at":"2025-08-12T00:07:40.165Z","avatar_url":"https://github.com/benoitc.png","language":"Erlang","readme":"# hackney_pooler\n\n## Goal:\n\nExperiment an API to limit the number of requests launched concurrently.\n\nRight now hackney is launching as many requests possible whcih means that a\nlot of FDs can be used. Also the pool can become a bottleneck when it get too\nmuch requests.\n\nIn hackney pooler you will have N concurrent requests where N is the number of\nworkers in a pool. A pool group is sharing a pool and a pool can be shared\nbetween multiples groups if given in the pool config.\n\nInternally a pool is maintained using\n[worker_pool](https://github.com/inaka/worker_pool).\n\n## Example of a synchronous request:\n\n    1\u003e application:ensure_all_started(hackney_pooler).\n    {ok,[asn1,crypto,public_key,ssl,hackney_pooler]}\n    2\u003e hackney_pooler:new_pool(test, [{workers,1000}, {concurrency, 4},\n                                      {max_connections, 150}]).\n    {ok,\u003c0.54.0\u003e}\n    4\u003e hackney_pooler:request(test, get, \u003c\u003c\"https://friendpaste.com\"\u003e\u003e).\n    {ok,200,\n        [{\u003c\u003c\"Server\"\u003e\u003e,\u003c\u003c\"nginx/0.7.62\"\u003e\u003e},\n         {\u003c\u003c\"Date\"\u003e\u003e,\u003c\u003c\"Tue, 25 Nov 2014 09:42:41 GMT\"\u003e\u003e},\n         {\u003c\u003c\"Content-Type\"\u003e\u003e,\u003c\u003c\"text/html; charset=utf-8\"\u003e\u003e},\n         {\u003c\u003c\"Transfer-Encoding\"\u003e\u003e,\u003c\u003c\"chunked\"\u003e\u003e},\n         {\u003c\u003c\"Connection\"\u003e\u003e,\u003c\u003c\"keep-alive\"\u003e\u003e},\n         {\u003c\u003c\"Set-Cookie\"\u003e\u003e,\n          \u003c\u003c\"FRIENDPASTE_SID=d7ae3781285eb1ec3598a5b220ea78c90e430cb7; expires=Tue, 0\"...\u003e\u003e},\n         {\u003c\u003c\"Access-Control-Allow-Origin\"\u003e\u003e,\u003c\u003c\"None\"\u003e\u003e},\n         {\u003c\u003c\"Access-Control-Allow-Credentials\"\u003e\u003e,\u003c\u003c\"true\"\u003e\u003e},\n         {\u003c\u003c\"Access-Control-Allow-Methods\"\u003e\u003e,\n          \u003c\u003c\"POST, GET, PUT, DELETE, OPTIONS\"\u003e\u003e},\n         {\u003c\u003c\"Access-Control-Allow-Headers\"\u003e\u003e,\n          \u003c\u003c\"X-Requested-With, X-HTTP-Method-Override, Content-Type, \"...\u003e\u003e},\n         {\u003c\u003c\"Access-Control-Max-Age\"\u003e\u003e,\u003c\u003c\"86400\"\u003e\u003e}],\n        \u003c\u003c\"\u003c!DOCTYPE html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\u003chead\u003e\\n    \u003cmeta charset=\\\"utf-8\\\"/\u003e\\n    \u003ctitle\u003eFriendpaste - Welcome\"...\u003e\u003e}\n\n\n\u003e Note: by default a pooler is launch with only 1 connection pool. Using the\n\u003e `concurrency` option will create N pools of connections whre 2 * N + 1 =\n\u003e number of i/o threads. You can force the number of connections pools using\n\u003e `{concurrency, N}`.\n\n## Example of an asynchronous request:\n\nAn asynchronous request send is like a cast, it handle the request in the\nworker and can send the result to a Pid or a function. If a function is given,\nit will be handled in the worker.\n\n    1\u003e application:ensure_all_started(hackney_pooler).\n    {ok,[asn1,crypto,public_key,ssl,idna,hackney,pooler,\n        hackney_pooler]}\n        2\u003e  hackney_pooler:new_pool(test, [{group, testing}, {max_count, 50}, {init_count, 50} ]).\n    {ok,\u003c0.61.0\u003e}\n    3\u003e  hackney_pooler:async_request(test, self(), get, \u003c\u003c\"https://friendpaste.com\"\u003e\u003e, [], \u003c\u003c\u003e\u003e, []).\n    ok\n    4\u003e flush().\n    Shell got {hpool,{test,{ok,200,\n        [{\u003c\u003c\"Server\"\u003e\u003e,\u003c\u003c\"nginx/0.7.62\"\u003e\u003e},\n        {\u003c\u003c\"Date\"\u003e\u003e,\u003c\u003c\"Wed, 26 Nov 2014 11:00:37 GMT\"\u003e\u003e},\n        {\u003c\u003c\"Content-Type\"\u003e\u003e,\n            \u003c\u003c\"text/html; charset=utf-8\"\u003e\u003e},\n        {\u003c\u003c\"Transfer-Encoding\"\u003e\u003e,\u003c\u003c\"chunked\"\u003e\u003e}, [...]\n\n\u003e An async request can send requests to a pid, a function (arity 1 or 2) or\n\u003e nothing if nil is given.\n\n## Pool Configuration via application environment\n\n    % hackneypooler.config\n    % Start Erlang as: erl -config pooler\n    % -*- mode: erlang -*-\n    % hackney_pooler app config\n    [\n     {hackney_pooler, [\n             {pools, [\n                      [{name, test},\n                       {workers, 1000},\n                       {concurrency, true}]\n            ]}]}\n    ]\n\n\nYou can also pass default settinsg using the `default_conf` env setting.\n\n## Known limitations\n\n- Streams are not handled. A body is fetched entirely when the worker return.\n- Config is not validated.\n- no REST api\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenoitc%2Fhackney_pooler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenoitc%2Fhackney_pooler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenoitc%2Fhackney_pooler/lists"}