Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bidoubiwa/turbo-json.js
turbo-json.js is a tool that concatenates all json files found in a directory into one big json file using streaming both during read as during write.
https://github.com/bidoubiwa/turbo-json.js
Last synced: 25 days ago
JSON representation
turbo-json.js is a tool that concatenates all json files found in a directory into one big json file using streaming both during read as during write.
- Host: GitHub
- URL: https://github.com/bidoubiwa/turbo-json.js
- Owner: bidoubiwa
- License: mit
- Created: 2020-02-04T13:38:34.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2023-03-27T04:58:28.000Z (almost 2 years ago)
- Last Synced: 2024-12-15T12:06:52.982Z (26 days ago)
- Language: JavaScript
- Homepage:
- Size: 668 KB
- Stars: 3
- Watchers: 2
- Forks: 2
- Open Issues: 8
-
Metadata Files:
- Readme: README.MD
- License: LICENSE
Awesome Lists containing this project
README
Turbo 🐡 JSON
turbo-json.js is a tool that combines all json files found in a directory into one big json file using **streaming** to avoid out-of-memory.
```bash
npx turbo-json.js data/
# Outputs `combined.json` file containing the content of all json files found in the data/ directory
```
All json files found in `data` directory will be combined into one file that is named `combined.json` par default.
Both `read` and `write` actions are done using `streaming`. The maximum data stored in memory is the buffer size.
## Table of Contents
- [Installation](#installation)
- [Usage](#usage)
- [How is it combined](#how-is-it-combined)
- [Validate JSON format](#validate-json-format)
- [Example](#example)
## Installation
### CLI:
Global install to have the CLI accessible from everywhere on your operating system.
```bash
npm install -g turbo-json.js # install globaly
```
No installation needed when using [npx](https://www.npmjs.com/package/npx).
```bash
npx turbo-json.js [options]
```
### Library:
```bash
# yarn
yarn add turbo-json.js
# npm
npm install turbo-json.js
```
## Usage
turbo-json.js takes one argument:
- Input directory: the path to the directory containing all the jsons.
and options:
- Output file _optional_ : path to the output file (default: `combined.json`).
- Validate Input JSON's: All input JSON files are validated to be a correct [JSON](https://www.json.org/json-en.html) (default: `true`).
- Validate Output JSON's: Validate if the outputed JSON is a correct [JSON](https://www.json.org/json-en.html) (default: `false`).
- Quiet mode : no logs are outputed (default: `false`).
It accepts relative path but also fully qualified paths.
### CLI usage:
Usage:
```bash
turbo-json [options]
```
Options:
```
-o, --output-file File name in which all the json files will be merged (default: "combined.json")
-I, --validate-input Check if input JSON files are valid (default: true)
-O, --validate-output Check if output JSON is a valid JSON (default: false)
-q, --quiet Quiet mode, no logs are outputed (default: false)
```
**Example**
```bash
turbo-json /data -o combined_data.json
```
### Library usage
```js
const { combineJson } = require('./src');
(async () => {
await combineJson('misc', { outputFile: 'combined_data.json', validateInput: true, validateOutput: false, quiet: false });
})();
```
## How is it combined
The tool requires the path to a directory to work. Inside that directory all json files are read and merged into one big json file.
The JSON file outputed by this tool contains an array with all the inputed JSON files.
**Example:**
If your JSON file contained `{ "id": 1 }` it will be stored in the output file like this `[{"id": 1}]`
Some examples:
Input files:
```json
{ "id": 1 }
```
```json
{ "id": 2 }
```
Output file:
```json
[
{ "id": 1 },
{ "id": 2 }
]
```
**Array exception**:
There is one exception to this rule. If your JSON file contains an array, it will be deconstructed/flattened in the final file (_could become an option please make an issue if you'd like that_).
Input files:
```json
[ 1, 2, 3 ]
```
```json
{ "id": 1 }
```
Output file:
```json
[
1,
2,
3,
{ "id": 1 }
]
```
## Validate Json Format
By default the tool supposes that your JSON files are correctly formated. If you'd like to check if it is well formated before concatenation the `validateInput` options should be set to true.
By doing so, only the well formated JSON's are concatenated but this comes at the price that every file will be stream-read two times. Once for validation and once for writing.
The same is possible to validate the output file using `validateOutput`.
Validity is based on the description provided by [json.org](https://www.json.org/json-en.html).
## Example
Using the JSON files present in `misc`, you can observe the outputed file [misc_output.json](./misc_output.json)
At the root of the repository use the following:
```bash
npx turbo-json.js misc
```
it will generate a combined.json file with all the JSON objects found in misc.