{"id":13527405,"url":"https://github.com/node-modules/urllib","last_synced_at":"2025-05-13T16:11:37.132Z","repository":{"id":1499815,"uuid":"1753353","full_name":"node-modules/urllib","owner":"node-modules","description":"Request HTTP(s) URLs in a complex world.","archived":false,"fork":false,"pushed_at":"2025-03-27T02:56:32.000Z","size":1686,"stargazers_count":734,"open_issues_count":8,"forks_count":125,"subscribers_count":36,"default_branch":"master","last_synced_at":"2025-04-23T20:59:22.647Z","etag":null,"topics":["curl","curl-library","http-client","httpclient","request","undici","urllib","urllib2"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/node-modules.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-05-16T02:04:01.000Z","updated_at":"2025-04-23T14:00:05.000Z","dependencies_parsed_at":"2023-07-05T20:46:40.818Z","dependency_job_id":"fb9595c0-5260-4f65-8b87-3855a8616d6d","html_url":"https://github.com/node-modules/urllib","commit_stats":{"total_commits":615,"total_committers":59,"mean_commits":"10.423728813559322","dds":"0.42601626016260163","last_synced_commit":"ef17c34cd71c1526a9da8f8ed244704d13a5bf6e"},"previous_names":[],"tags_count":227,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/node-modules%2Furllib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/node-modules%2Furllib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/node-modules%2Furllib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/node-modules%2Furllib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/node-modules","download_url":"https://codeload.github.com/node-modules/urllib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250514767,"owners_count":21443208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["curl","curl-library","http-client","httpclient","request","undici","urllib","urllib2"],"created_at":"2024-08-01T06:01:47.414Z","updated_at":"2025-04-23T20:59:46.505Z","avatar_url":"https://github.com/node-modules.png","language":"TypeScript","readme":"# urllib\n\n[![NPM version][npm-image]][npm-url]\n[![Node.js CI](https://github.com/node-modules/urllib/actions/workflows/nodejs.yml/badge.svg)](https://github.com/node-modules/urllib/actions/workflows/nodejs.yml)\n[![Test coverage][codecov-image]][codecov-url]\n[![Known Vulnerabilities][snyk-image]][snyk-url]\n[![npm download][download-image]][download-url]\n[![Node.js Version](https://img.shields.io/node/v/urllib.svg?style=flat)](https://nodejs.org/en/download/)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](https://makeapullrequest.com)\n![CodeRabbit Pull Request Reviews](https://img.shields.io/coderabbit/prs/github/node-modules/urllib)\n\n[npm-image]: https://img.shields.io/npm/v/urllib.svg?style=flat-square\n[npm-url]: https://npmjs.org/package/urllib\n[codecov-image]: https://codecov.io/gh/node-modules/urllib/branch/master/graph/badge.svg\n[codecov-url]: https://codecov.io/gh/node-modules/urllib\n[snyk-image]: https://snyk.io/test/npm/urllib/badge.svg?style=flat-square\n[snyk-url]: https://snyk.io/test/npm/urllib\n[download-image]: https://img.shields.io/npm/dm/urllib.svg?style=flat-square\n[download-url]: https://npmjs.org/package/urllib\n\nRequest HTTP URLs in a complex world — basic\nand digest authentication, redirections, timeout and more.\n\n## Install\n\n```bash\nnpm install urllib\n```\n\n## Usage\n\n### TypeScript and ESM\n\n```ts\nimport { request } from 'urllib';\n\nconst { data, res } = await request('http://cnodejs.org/');\n// result: { data: Buffer, res: Response }\nconsole.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);\n```\n\n### CommonJS\n\n```js\nconst { request } = require('urllib');\n\nconst { data, res } = await request('http://cnodejs.org/');\n// result: { data: Buffer, res: Response }\nconsole.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);\n```\n\n## API Doc\n\n### Method: `async request(url[, options])`\n\n#### Arguments\n\n- **url** String | Object - The URL to request, either a String or a Object that return by [url.parse](https://nodejs.org/api/url.html#url_url_parse_urlstr_parsequerystring_slashesdenotehost).\n- ***options*** Object - Optional\n  - ***method*** String - Request method, defaults to `GET`. Could be `GET`, `POST`, `DELETE` or `PUT`. Alias 'type'.\n  - ***data*** Object - Data to be sent. Will be stringify automatically.\n  - ***content*** String | [Buffer](https://nodejs.org/api/buffer.html) - Manually set the content of payload. If set, `data` will be ignored.\n  - ***stream*** [stream.Readable](https://nodejs.org/api/stream.html#stream_class_stream_readable) - Stream to be pipe to the remote. If set, `data` and `content` will be ignored.\n  - ***writeStream*** [stream.Writable](https://nodejs.org/api/stream.html#stream_class_stream_writable) - A writable stream to be piped by the response stream. Responding data will be write to this stream and `callback` will be called with `data` set `null` after finished writing.\n  - ***files*** {Array\u003cReadStream|Buffer|String\u003e | Object | ReadStream | Buffer | String - The files will send with `multipart/form-data` format, base on `formstream`. If `method` not set, will use `POST` method by default.\n  - ***contentType*** String - Type of request data. Could be `json` (**Notes**: not use `application/json` here). If it's `json`, will auto set `Content-Type: application/json` header.\n  - ***dataType*** String - Type of response data. Could be `text` or `json`. If it's `text`, the `callback`ed `data` would be a String. If it's `json`, the `data` of callback would be a parsed JSON Object and will auto set `Accept: application/json` header. Default `callback`ed `data` would be a `Buffer`.\n  - ***fixJSONCtlChars*** Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is `false`.\n  - ***headers*** Object - Request headers.\n  - ***timeout*** Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Default is `5000`. You can use `timeout: 5000` to tell urllib use same timeout on two phase or set them separately such as `timeout: [3000, 5000]`, which will set connecting timeout to 3s and response 5s.\n  - ***keepAliveTimeout*** `number | null` - Default is `4000`, 4 seconds - The timeout after which a socket without active requests will time out. Monitors time between activity on a connected socket. This value may be overridden by *keep-alive* hints from the server. See [MDN: HTTP - Headers - Keep-Alive directives](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive#directives) for more details.\n  - ***auth*** String - `username:password` used in HTTP Basic Authorization.\n  - ***digestAuth*** String - `username:password` used in HTTP [Digest Authorization](https://en.wikipedia.org/wiki/Digest_access_authentication).\n  - ***followRedirect*** Boolean - follow HTTP 3xx responses as redirects. defaults to true.\n  - ***maxRedirects*** Number - The maximum number of redirects to follow, defaults to 10.\n  - ***formatRedirectUrl*** Function - Format the redirect url by yourself. Default is `url.resolve(from, to)`.\n  - ***beforeRequest*** Function - Before request hook, you can change every thing here.\n  - ***streaming*** Boolean - lets you get the `res` object when request connected, default `false`. alias `customResponse`\n  - ***compressed*** Boolean - Accept `gzip, br` response content and auto decode it, default is `false`.\n  - ***timing*** Boolean - Enable timing or not, default is `true`.\n  - ***socketPath*** String | null - request a unix socket service, default is `null`.\n  - ***highWaterMark*** Number - default is `67108864`, 64 KiB.\n\n#### Options: `options.data`\n\nWhen making a request:\n\n```js\nawait request('https://example.com', {\n  method: 'GET',\n  data: {\n    'a': 'hello',\n    'b': 'world',\n  },\n});\n```\n\nFor `GET` request, `data` will be stringify to query string, e.g. `http://example.com/?a=hello\u0026b=world`.\n\nFor others like `POST`, `PATCH` or `PUT` request,\nin defaults, the `data` will be stringify into `application/x-www-form-urlencoded` format\nif `content-type` header is not set.\n\nIf `content-type` is `application/json`, the `data` will be `JSON.stringify` to JSON data format.\n\n#### Options: `options.content`\n\n`options.content` is useful when you wish to construct the request body by yourself,\nfor example making a `content-type: application/json` request.\n\nNotes that if you want to send a JSON body, you should stringify it yourself:\n\n```js\nawait request('https://example.com', {\n  method: 'POST',\n  headers: {\n    'Content-Type': 'application/json',\n  },\n  content: JSON.stringify({\n    a: 'hello',\n    b: 'world',\n  }),\n});\n```\n\nIt would make a HTTP request like:\n\n```bash\nPOST / HTTP/1.1\nhost: example.com\ncontent-type: application/json\n\n{\n  \"a\": \"hello\",\n  \"b\": \"world\"\n}\n```\n\nThis exmaple can use `options.data` with `application/json` content type:\n\n```js\nawait request('https://example.com', {\n  method: 'POST',\n  headers: {\n    'content-type': 'application/json'\n  },\n  data: {\n    a: 'hello',\n    b: 'world',\n  }\n});\n```\n\n#### Options: `options.files`\n\nUpload a file with a `hello` field.\n\n```js\nawait request('https://example.com/upload', {\n  method: 'POST',\n  files: __filename,\n  data: {\n    hello: 'hello urllib',\n  },\n});\n```\n\nUpload multi files with a `hello` field.\n\n```js\nawait request('https://example.com/upload', {\n  method: 'POST',\n  files: [\n    __filename,\n    fs.createReadStream(__filename),\n    Buffer.from('mock file content'),\n  ],\n  data: {\n    hello: 'hello urllib with multi files',\n  },\n});\n```\n\nCustom file field name with `uploadfile`.\n\n```js\nawait request('https://example.com/upload', {\n  method: 'POST',\n  files: {\n    uploadfile: __filename,\n  },\n});\n```\n\n### Response Object\n\nResponse is normal object, it contains:\n\n- `status` or `statusCode`: response status code.\n  - `-1` meaning some network error like `ENOTFOUND`\n  - `-2` meaning ConnectionTimeoutError\n- `headers`: response http headers, default is `{}`\n- `size`: response size\n- `aborted`: response was aborted or not\n- `rt`: total request and response time in ms.\n- `timing`: timing object if timing enable.\n- `socket`: socket info\n\n## Run test with debug log\n\n```bash\nNODE_DEBUG=urllib:* npm test\n```\n\n## Request with HTTP2\n\nCreate a HttpClient with `options.allowH2 = true`\n\n```ts\nimport { HttpClient } from 'urllib';\n\nconst httpClient = new HttpClient({\n  allowH2: true,\n});\n\nconst response = await httpClient.request('https://node.js.org');\nconsole.log(response.status);\nconsole.log(response.headers);\n```\n\n## Mocking Request\n\nexport from [undici](https://undici.nodejs.org/#/docs/best-practices/mocking-request)\n\n```ts\nimport { strict as assert } from 'assert';\nimport { MockAgent, setGlobalDispatcher, request } from 'urllib';\n\nconst mockAgent = new MockAgent();\nsetGlobalDispatcher(mockAgent);\n\nconst mockPool = mockAgent.get('http://localhost:7001');\n\nmockPool.intercept({\n  path: '/foo',\n  method: 'POST',\n}).reply(400, {\n  message: 'mock 400 bad request',\n});\n\nconst response = await request('http://localhost:7001/foo', {\n  method: 'POST',\n  dataType: 'json',\n});\nassert.equal(response.status, 400);\nassert.deepEqual(response.data, { message: 'mock 400 bad request' });\n```\n\n## Request through a http proxy\n\nexport from [undici](https://undici.nodejs.org/#/docs/best-practices/proxy)\n\n```ts\nimport { ProxyAgent, request } from 'urllib';\n\nconst proxyAgent = new ProxyAgent('http://my.proxy.com:8080');\nconst response = await request('https://www.npmjs.com/package/urllib', {\n  dispatcher: proxyAgent,\n});\nconsole.log(response.status, response.headers);\n```\n\n## Benchmarks\n\nFork [undici benchmarks script](https://github.com/fengmk2/undici/blob/urllib-benchmark/benchmarks/benchmark.js)\n\n\u003e undici@6.19.2\n\n```bash\nNode.js v18.20.3\n\n┌─────────┬───────────────────────┬─────────┬────────────────────┬─────────────┬─────────────────────────┐\n│ (index) │         Tests         │ Samples │       Result       │  Tolerance  │ Difference with slowest │\n├─────────┼───────────────────────┼─────────┼────────────────────┼─────────────┼─────────────────────────┤\n│    0    │  'urllib2 - request'  │   10    │  '321.53 req/sec'  │ '± 0.38 %'  │           '-'           │\n│    1    │ 'http - no keepalive' │   10    │  '607.77 req/sec'  │ '± 0.80 %'  │       '+ 89.02 %'       │\n│    2    │         'got'         │   101   │ '7929.51 req/sec'  │ '± 4.46 %'  │      '+ 2366.15 %'      │\n│    3    │     'node-fetch'      │   40    │ '8651.95 req/sec'  │ '± 2.99 %'  │      '+ 2590.84 %'      │\n│    4    │       'request'       │   101   │ '8864.09 req/sec'  │ '± 7.81 %'  │      '+ 2656.82 %'      │\n│    5    │   'undici - fetch'    │   101   │ '9607.01 req/sec'  │ '± 4.23 %'  │      '+ 2887.87 %'      │\n│    6    │        'axios'        │   55    │ '10378.80 req/sec' │ '± 2.94 %'  │      '+ 3127.91 %'      │\n│    7    │     'superagent'      │   75    │ '11286.74 req/sec' │ '± 2.90 %'  │      '+ 3410.29 %'      │\n│    8    │  'http - keepalive'   │   60    │ '11288.96 req/sec' │ '± 2.95 %'  │      '+ 3410.98 %'      │\n│    9    │  'urllib4 - request'  │   101   │ '11352.65 req/sec' │ '± 10.20 %' │      '+ 3430.79 %'      │\n│   10    │  'urllib3 - request'  │   40    │ '13831.19 req/sec' │ '± 2.89 %'  │      '+ 4201.64 %'      │\n│   11    │  'undici - pipeline'  │   60    │ '14562.44 req/sec' │ '± 2.91 %'  │      '+ 4429.06 %'      │\n│   12    │  'undici - request'   │   70    │ '19630.64 req/sec' │ '± 2.87 %'  │      '+ 6005.32 %'      │\n│   13    │   'undici - stream'   │   55    │ '20843.50 req/sec' │ '± 2.90 %'  │      '+ 6382.54 %'      │\n│   14    │  'undici - dispatch'  │   55    │ '21233.10 req/sec' │ '± 2.82 %'  │      '+ 6503.70 %'      │\n└─────────┴───────────────────────┴─────────┴────────────────────┴─────────────┴─────────────────────────┘\n\nNode.js v20.15.0\n\n┌─────────┬───────────────────────┬─────────┬────────────────────┬────────────┬─────────────────────────┐\n│ (index) │ Tests                 │ Samples │ Result             │ Tolerance  │ Difference with slowest │\n├─────────┼───────────────────────┼─────────┼────────────────────┼────────────┼─────────────────────────┤\n│ 0       │ 'urllib2 - request'   │ 10      │ '332.91 req/sec'   │ '± 1.13 %' │ '-'                     │\n│ 1       │ 'http - no keepalive' │ 10      │ '615.50 req/sec'   │ '± 2.25 %' │ '+ 84.88 %'             │\n│ 2       │ 'got'                 │ 55      │ '7658.39 req/sec'  │ '± 2.98 %' │ '+ 2200.42 %'           │\n│ 3       │ 'node-fetch'          │ 30      │ '7832.96 req/sec'  │ '± 2.96 %' │ '+ 2252.86 %'           │\n│ 4       │ 'axios'               │ 40      │ '8607.27 req/sec'  │ '± 2.79 %' │ '+ 2485.44 %'           │\n│ 5       │ 'request'             │ 35      │ '8703.49 req/sec'  │ '± 2.84 %' │ '+ 2514.35 %'           │\n│ 6       │ 'undici - fetch'      │ 65      │ '9971.24 req/sec'  │ '± 2.96 %' │ '+ 2895.15 %'           │\n│ 7       │ 'superagent'          │ 30      │ '11006.46 req/sec' │ '± 2.90 %' │ '+ 3206.11 %'           │\n│ 8       │ 'http - keepalive'    │ 55      │ '11610.14 req/sec' │ '± 2.87 %' │ '+ 3387.44 %'           │\n│ 9       │ 'urllib3 - request'   │ 25      │ '13873.38 req/sec' │ '± 2.96 %' │ '+ 4067.27 %'           │\n│ 10      │ 'urllib4 - request'   │ 25      │ '14291.36 req/sec' │ '± 2.92 %' │ '+ 4192.82 %'           │\n│ 11      │ 'undici - pipeline'   │ 45      │ '14617.69 req/sec' │ '± 2.84 %' │ '+ 4290.85 %'           │\n│ 12      │ 'undici - dispatch'   │ 101     │ '18716.29 req/sec' │ '± 3.97 %' │ '+ 5521.98 %'           │\n│ 13      │ 'undici - request'    │ 101     │ '19165.16 req/sec' │ '± 3.25 %' │ '+ 5656.81 %'           │\n│ 14      │ 'undici - stream'     │ 30      │ '21816.28 req/sec' │ '± 2.99 %' │ '+ 6453.15 %'           │\n└─────────┴───────────────────────┴─────────┴────────────────────┴────────────┴─────────────────────────┘\n\nNode.js v22.3.0\n\n┌─────────┬───────────────────────┬─────────┬────────────────────┬────────────┬─────────────────────────┐\n│ (index) │ Tests                 │ Samples │ Result             │ Tolerance  │ Difference with slowest │\n├─────────┼───────────────────────┼─────────┼────────────────────┼────────────┼─────────────────────────┤\n│ 0       │ 'urllib2 - request'   │ 15      │ '297.46 req/sec'   │ '± 2.65 %' │ '-'                     │\n│ 1       │ 'http - no keepalive' │ 10      │ '598.25 req/sec'   │ '± 1.94 %' │ '+ 101.12 %'            │\n│ 2       │ 'axios'               │ 30      │ '8487.94 req/sec'  │ '± 2.91 %' │ '+ 2753.52 %'           │\n│ 3       │ 'got'                 │ 50      │ '10054.46 req/sec' │ '± 2.89 %' │ '+ 3280.16 %'           │\n│ 4       │ 'request'             │ 45      │ '10306.02 req/sec' │ '± 2.87 %' │ '+ 3364.73 %'           │\n│ 5       │ 'node-fetch'          │ 55      │ '11160.02 req/sec' │ '± 2.87 %' │ '+ 3651.83 %'           │\n│ 6       │ 'superagent'          │ 80      │ '11302.28 req/sec' │ '± 2.85 %' │ '+ 3699.66 %'           │\n│ 7       │ 'undici - fetch'      │ 60      │ '11357.87 req/sec' │ '± 2.89 %' │ '+ 3718.35 %'           │\n│ 8       │ 'http - keepalive'    │ 60      │ '13782.10 req/sec' │ '± 2.97 %' │ '+ 4533.34 %'           │\n│ 9       │ 'urllib4 - request'   │ 70      │ '15965.62 req/sec' │ '± 2.88 %' │ '+ 5267.40 %'           │\n│ 10      │ 'urllib3 - request'   │ 55      │ '16010.37 req/sec' │ '± 2.90 %' │ '+ 5282.45 %'           │\n│ 11      │ 'undici - pipeline'   │ 35      │ '17969.37 req/sec' │ '± 2.95 %' │ '+ 5941.03 %'           │\n│ 12      │ 'undici - dispatch'   │ 101     │ '18765.50 req/sec' │ '± 3.01 %' │ '+ 6208.68 %'           │\n│ 13      │ 'undici - request'    │ 85      │ '20091.12 req/sec' │ '± 2.95 %' │ '+ 6654.33 %'           │\n│ 14      │ 'undici - stream'     │ 45      │ '21599.12 req/sec' │ '± 2.81 %' │ '+ 7161.30 %'           │\n└─────────┴───────────────────────┴─────────┴────────────────────┴────────────┴─────────────────────────┘\n```\n\n## License\n\n[MIT](LICENSE)\n\n## Contributors\n\n[![Contributors](https://contrib.rocks/image?repo=node-modules/urllib)](https://github.com/node-modules/urllib/graphs/contributors)\n\nMade with [contributors-img](https://contrib.rocks).\n","funding_links":[],"categories":["Repository","请求处理","Uncategorized","TypeScript","1. 后端开发"],"sub_categories":["HTTP","redux 扩展","Uncategorized","1.1 HTTP"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnode-modules%2Furllib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnode-modules%2Furllib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnode-modules%2Furllib/lists"}