https://github.com/idiocc/nicer
An Http Multipart/Form-Data Request Body Parser.
https://github.com/idiocc/nicer
Last synced: about 1 year ago
JSON representation
An Http Multipart/Form-Data Request Body Parser.
- Host: GitHub
- URL: https://github.com/idiocc/nicer
- Owner: idiocc
- License: mit
- Created: 2019-07-06T11:57:11.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-07-29T17:57:37.000Z (almost 7 years ago)
- Last Synced: 2024-10-22T21:00:06.510Z (over 1 year ago)
- Language: JavaScript
- Homepage: https://www.idio.cc
- Size: 3.37 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# nicer
[](https://npmjs.org/package/nicer)
`nicer` is An Http _Multipart/Form-Data_ Request Body Parser. It can receive form data fields and files headers and stream their data.
```sh
yarn add nicer
```
| Library | 1 | 2 | 3 |
| ---------- | ---------- | ---------- | ---------- |
| dicer | 137.21mb/s | 100.65mb/s | 165.95mb/s |
| multiparty | 25.78mb/s | 29.48mb/s | 29.60mb/s |
| nicer | 106.55mb/s | 112.46mb/s | 134.24mb/s |
| nicerc | 114.42mb/s | 115.04mb/s | 116.57mb/s |
Stable Benchmark (18 Jul)
| Library | Max Speed |
| ---------- | --------- |
| (1) dicer | 92.93mb/s |
| (2) nicerc | 79.95mb/s |
| nicer | 72.23mb/s |
| (3) multiparty | 28.79mb/s |
Nicer is comparable to the faster streaming parser, Dicer since the real-world data (uploading 2 fields, 2 text files and 50 photos) is processed at speeds that are close to max 90mb/s. In the benchmark, nicer is the source code of this package, whereas nicerc is the compiled JavaScript optimised Closure Compiler, which probably increases the speed by 5-10%.
## Table Of Contents
* [Stable Benchmark](#stable-benchmark)
- [Table Of Contents](#table-of-contents)
- [API](#api)
- [`constructor(boundary: string): Nicer`](#constructorboundary-string-nicer)
* [`Nicer`](#type-nicer)
* [`Part`](#type-part)
- [Errors](#errors)
* [Extra Buffer](#extra-buffer)
- [Debug](#debug)
- [Copyright](#copyright)
## API
The package is available by importing its default constructor function:
```js
import Nicer from 'nicer'
```
## `constructor(`
`boundary: string,`
`): Nicer`
Creates a transform that emits objects with a header buffer and the body stream. The body stream is a pass-through so all data must be written as it comes, the request doesn't pause for data to be consumed. The header is a buffer which can be parsed more and/or decrypted, but it does not stream. The assumption is the headers are short therefore a header buffer is accumulated until `\r\n` is found. Just make sure to run behind _NginX_ then it should be alright.
__`Nicer`__: A stream that emits objects with a header buffer and the body PassThrough stream.
| Name | Type | Description |
| ------------- | --------------- | ------------------------------ |
| __boundary*__ | string | The mandatory field separator. |
```js
import { Transform } from 'stream'
import Nicer from 'nicer'
const detected = []
await http.startPlain((req, res) => {
const boundary = getBoundary(req, res)
console.log('Boundary detected: %s', boundary)
req.pipe(new Nicer({ boundary })).pipe(new Transform({
objectMode: true,
transform(obj, enc, next) {
const { header: HEADER, stream: STREAM } = obj
// to print in sync have to wait for all data
// since STREAM is a pass-through
let d = []
detected.push(['%s\n====\n', HEADER, d])
STREAM.on('data', (data) => {
d.push(data)
})
next()
},
final() {
res.statusCode = 200
res.end(JSON.stringify(detected))
},
}))
})
```
A new instance of Nicer can be piped into by an http.IncomingMessage stream in the Node.JS server. Then a transform stream must be created to listen for the data emitted by Nicer in object mode.
__`Part`__: A part that gets emitted by _Nicer_.
| Name | Type | Description |
| ----------- | ---------------------------- | ------------------------------ |
| __stream*__ | !stream.PassThrough | The mandatory field separator. |
| __header*__ | !Buffer | The header found before data. |
```
Boundary detected: u2KxIV5yF1y+xUspOQCCZopaVgeV6Jxihv35XQJmuTx8X3sh
Content-Disposition: form-data; name="key"
====
[ 'value' ]
Content-Disposition: form-data; name="alan"
====
[ 'watts' ]
Content-Disposition: form-data; name="file"; filename="test/fixture/test.txt"
Content-Type: application/octet-stream
====
[ 'a test file\n' ]
```
The data received by the 'transform' method, contains the { header, stream } properties. The data from the stream must be accumulated.
## Errors
The errors are spawned when the buffer remaining the stream after the `final` event, and processed to extract the rest of the fields, still contains symbols different from -- (`[45,45]`).
```http
-------example
Content-Disposition: form-data; name="key"
data
-------exampleWAT
```
The data remaining after the last boundary detected after the final method is called does not have any meaning and is discarded. This is not the case with parts that arrived before the stream was closed, i.e., the file limit is not implemented.
```fs
Boundary detected: -----example
[!] Error Unexpected end of request body, wanted to see "--" but saw WA.
Detected Data:
Content-Disposition: form-data; name="key"
====
[ 'data' ]
```
The parser will always check for the closing -- and emit an error in the end, however the headers and data streams emitted by it, would have been all closed, i.e., the data can still be used.
## Debug
The software can write debug information, when the `DEBUG=nicer` env variable is set.
Debug (Source)
```js
import { Writable } from 'stream'
import Nicer from '../src'
const detected = []
await http.startPlain((req, res) => {
const boundary = getBoundary(req, res)
console.log('Boundary detected: %s', boundary)
const nicer = new Nicer({ boundary })
const bt = new BufferTransform(50)
req.pipe(bt).pipe(nicer).pipe(new Writable({
objectMode: true,
write(obj, enc, next) {
const { header: HEADER, stream: STREAM } = obj
next()
},
final() {
res.statusCode = 200
res.end(JSON.stringify(detected))
},
}))
})
```
The transform method appends data to the left-over buffer (which can usually be small enough to accommodate [--boundary.length-1] symbols) and consumes data. The data is consumed by first trying to find the boundary in the new buffer. If this is possible, then depending on the state of the parser, the data found before the separator is either flushed in an existing data stream, or appended to the existing header, which can then lead to body-flushing.
```sh
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔎 Finished boundary scan, buffer of length 50B left, separators found: 0 +2ms
nicer one consume safe consumed 0B and left 50B +1ms
nicer +1ms
nicer 100B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer ⭐ Found starting boundary at index 2 +0ms
nicer 🔎 Finished boundary scan, buffer of length 48B left, separators found: 1 +0ms
nicer one consume safe consumed 52B and left 48B +1ms
nicer +0ms
nicer 98B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔎 Finished boundary scan, buffer of length 98B left, separators found: 0 +0ms
nicer one consume safe consumed 46B and left 52B +0ms
nicer +0ms
nicer 102B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔛 Found boundary, data size 7B +0ms
nicer 🗒 Found header and data of size <53B> +1ms
nicer Content-Disposition: form-data; name="key" +0ms
nicer value +0ms
nicer 🔎 Finished boundary scan, buffer of length 43B left, separators found: 1 +2ms
nicer one consume safe consumed 59B and left 43B +0ms
nicer +0ms
nicer 93B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔎 Finished boundary scan, buffer of length 93B left, separators found: 0 +0ms
nicer one consume safe consumed 41B and left 52B +0ms
nicer +1ms
nicer 102B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔛 Found boundary, data size 13B +0ms
nicer 🗒 Found header and data of size <54B> +0ms
nicer Content-Disposition: form-data; name="alan" +0ms
nicer watts +0ms
nicer 🔎 Finished boundary scan, buffer of length 37B left, separators found: 1 +0ms
nicer one consume safe consumed 65B and left 37B +1ms
nicer +0ms
nicer 87B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔎 Finished boundary scan, buffer of length 87B left, separators found: 0 +0ms
nicer one consume safe consumed 35B and left 52B +0ms
nicer +0ms
nicer 102B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +11ms
nicer 🔎 Finished boundary scan, buffer of length 102B left, separators found: 0 +0ms
nicer +0ms
nicer 85B +1ms
nicer one consume safe consumed 50B and left 52B +0ms
nicer +0ms
nicer 102B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +1ms
nicer 🔛 Found boundary, data size 50B +0ms
nicer 🗒 Found header and data of size <135B> +0ms
nicer Content-Disposition: form-data; name="file"; filename="test/fixture/test.txt" +0ms
nicer Content-Type: ap... +0ms
nicer 🔎 Finished boundary scan, buffer of length 0B left, separators found: 1 +1ms
nicer one consume safe consumed 102B and left 0B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +0ms
nicer 🔎 Finished boundary scan, buffer of length 4B left, separators found: 0 +0ms
nicer one consume safe consumed 0B and left 4B +0ms
nicer 🔍 Staring boundary --u2KxIV5yF1y+x... scan +2ms
nicer 🔎 Finished boundary scan, buffer of length 4B left, separators found: 0 +1ms
nicer one consume safe consumed 0B and left 4B +0ms
```
After knowing what's left after the last found boundary, the Nicer parser takes only the safe amount of data to consume more which equals to the length of the boundary (including prior --), otherwise there might be a partial boundary leaking into the data stream. The remainder is saved as the new buffer, to which the following chunk in the transform method will be appended, and so on.
## Copyright
(c) [Idio][1] 2019
[1]: https://idio.cc
