https://github.com/data-fair/processing-json-file
https://github.com/data-fair/processing-json-file
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/data-fair/processing-json-file
- Owner: data-fair
- License: agpl-3.0
- Created: 2025-11-28T16:13:05.000Z (8 months ago)
- Default Branch: master
- Last Pushed: 2026-06-24T10:32:23.000Z (about 2 months ago)
- Last Synced: 2026-06-24T10:44:09.080Z (about 2 months ago)
- Language: TypeScript
- Size: 294 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 
@data-fair/processing-json-file
A plugin that creates and updates a data-fair dataset from one or more remote JSON or CSV files.
## Features
- **Multiple protocols** — download source files over HTTP(S), SFTP or FTP.
- **JSON or CSV** — pick the source format; a JSON file (or folder of files) is flattened through a configurable path mapping, while a CSV file is loaded as-is (its columns become the dataset's columns).
- **Single file or whole folder** — point at a file matching the chosen format or at a folder (URL ending with `/`); every matching file in the folder is downloaded and concatenated.
- **Reused SFTP connection** — a single SFTP connection is opened for the whole batch (listing, downloads and deletions) instead of reconnecting per file.
- **Convert and load** — each file is converted according to the configuration and pushed to the target dataset with `_bulk_lines`.
- **Process and delete** — optionally delete each source file on the remote after a successful import, or move it to a `backup` folder instead (FTP/SFTP only).
## Configuration
| Field | Description |
| ----- | ----------- |
| `dataset` | The target data-fair dataset to fill. |
| `url` | URL of the source. A path ending with `.json`/`.csv` (depending on the chosen format) targets a single file; a path ending with `/` targets every matching file of the folder. |
| `username` / `password` | Optional credentials for the remote (HTTP basic auth, SFTP or FTP). |
| `drop` | Whether to drop the existing lines before loading. |
| `processAndDelete` | Delete each source file on the remote once it has been imported (FTP/SFTP only). |
| `processAndMove` | Move each source file to a `backup` folder after import instead of deleting it — next to the file for a single-file URL, or inside the imported folder for a folder URL. Independent checkbox; takes precedence over `processAndDelete` when both are set (FTP/SFTP only). |
| `format` | `json` (default) or `csv`. JSON files go through the path mapping below; CSV files are loaded as-is, with their delimiter auto-detected (all CSV files of a run must share the same delimiter). |
## Release
Publishing is handled automatically by CI: the plugin is pushed to the data-fair registry (`@data-fair/registry`), not to the public npm registry — there is no manual `npm publish`. A push to `main`/`master` publishes to the staging registry; pushing a `v*` tag publishes to production:
```bash
npm version minor # version bump + v* tag
git push --follow-tags # CI publishes to the production registry
```