https://github.com/shelfio/array-chunk-by-size
Chunk array of objects by their size in JSON
https://github.com/shelfio/array-chunk-by-size
arrays chunk chunking node-module npm-package splitting
Last synced: 3 months ago
JSON representation
Chunk array of objects by their size in JSON
- Host: GitHub
- URL: https://github.com/shelfio/array-chunk-by-size
- Owner: shelfio
- License: mit
- Created: 2017-11-02T15:57:01.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2025-05-31T00:29:34.000Z (4 months ago)
- Last Synced: 2025-05-31T11:07:27.767Z (4 months ago)
- Topics: arrays, chunk, chunking, node-module, npm-package, splitting
- Language: TypeScript
- Homepage:
- Size: 68.4 KB
- Stars: 4
- Watchers: 13
- Forks: 3
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: license
Awesome Lists containing this project
README
# array-chunk-by-size [](https://circleci.com/gh/shelfio/array-chunk-by-size)
> Chunk array of objects by their size in JSON
## Install
```
$ yarn add @shelf/array-chunk-by-size
```
## Usage
Useful if you want to split large array into smaller, but limited by JSON size.
Each array chunk will be up to specified amount of bytes when stringified into JSON.
```js
import {chunkArray} from '@shelf/array-chunk-by-size';
const bigArray = [{a: 1}, {b: 2}, {c: 3}];
const twoKilobytes = 2 * 1024;
const smallerArrays = chunkArray({input: bigArray, bytesSize: twoKilobytes});
// => [ [ ... ], [ ... ] ] and so on
```
Alternatively, you might pass a custom size calculation function.
For example, to chunk array by LLM tokens size:
```js
import {chunkArray} from '@shelf/array-chunk-by-size';
import {encode} from 'gpt-3-encoder';
const bigArray = ['msg-1', 'msg-2'];
const gpt3MaxTokens = 4000;
const smallerArrays = chunkArray({
input: bigArray,
bytesSize: gpt3MaxTokens,
sizeCalcFunction: item => encode(item).length,
});
```
## See Also
- [fast-chunk-string](https://github.com/shelfio/fast-chunk-string)
- [fast-normalize-spaces](https://github.com/shelfio/fast-normalize-spaces)
- [fast-natural-order-by](https://github.com/shelfio/fast-natural-order-by)
- [fast-uslug](https://github.com/shelfio/fast-uslug)
## Publish
```sh
$ git checkout master
$ yarn version
$ yarn publish
$ git push origin master --tags
```
## License
MIT © [Shelf](https://shelf.io)