https://github.com/tmck-code/target-ndjson

NDJSON Target for Singer Taps
https://github.com/tmck-code/target-ndjson

Last synced: 22 days ago
JSON representation

NDJSON Target for Singer Taps

• Host: GitHub
• URL: https://github.com/tmck-code/target-ndjson
• Owner: tmck-code
• License: agpl-3.0
• Created: 2020-07-30T06:32:36.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2020-09-23T07:52:45.000Z (over 4 years ago)
• Last Synced: 2024-10-25T00:09:41.196Z (2 months ago)
• Language: Python
• Size: 45.9 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# target-ndjson

This is a [Singer](https://singer.io) target that reads JSON-formatted data

following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).

- [target-ndjson](#target-ndjson)

  - [Behaviour](#behaviour)

    - [Performance](#performance)

  - [Testing](#testing)

    - [With Docker](#with-docker)

    - [Without Docker](#without-docker)

---

## Behaviour

This target writes to files in NDJSON (aka "Newline-Delimited JSON" files). This

essentially unpacks each `"RECORD"` log recived, verbatim, and writes to file.

This target is intended for use in systems that implement their own data processing

layers, or that need un-transformed output.

### Performance

In addition to producing "un-transformed" output, there are some additional changes

in this tap to improve overall latency.

- Using `ujson` over standard-lib `json` for JSON serde.

- Keeping output files open, rather than re-opening in append mode per-line

This tap is designed for circumstances in which large amounts of data must be ingested

from a tap and written to file, as fast and efficiently as (reasonably) possible.

## Testing

This repository can be tested using `pytest`, with or with out a docker container

### With Docker

```shell

# This Makefile contains handy helper commands for using docker.

# You can also use the docker commands directly (see Makefile for examples)

make build test

```

### Without Docker

```shell

pip install -r requirements.txt

pytest

```

---