Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/edgarrmondragon/tap-neon
Singer tap for the Neon.tech API
https://github.com/edgarrmondragon/tap-neon
neon-database postgresql singer-io singer-sdk
Last synced: 2 months ago
JSON representation
Singer tap for the Neon.tech API
- Host: GitHub
- URL: https://github.com/edgarrmondragon/tap-neon
- Owner: edgarrmondragon
- License: apache-2.0
- Created: 2022-12-03T05:15:06.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-10-07T22:46:41.000Z (3 months ago)
- Last Synced: 2024-10-07T22:50:10.415Z (3 months ago)
- Topics: neon-database, postgresql, singer-io, singer-sdk
- Language: Python
- Homepage: https://pypi.org/p/tap-neon
- Size: 312 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# tap-neon
Singer tap for Neon Serverless Postgres.
Built with the [Meltano Singer SDK](https://sdk.meltano.com).
## Capabilities
* `catalog`
* `state`
* `discover`
* `about`
* `stream-maps`
* `schema-flattening`
## Settings
| Setting | Required | Default | Description |
|:--------------------|:--------:|:-------:|:------------|
| api_key | True | None | API Key for Neon Serverless Postgres |
| start_date | False | None | Earliest datetime to get data from |
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth| False | None | The max depth to flatten schemas. |
A full list of supported settings and capabilities is available by running: `tap-neon --about`
### Source Authentication and Authorization
Read the [docs](https://neon.tech/docs/get-started-with-neon/using-api-keys/) to learn how to generate an API key.
## Usage
You can easily run `tap-neon` by itself or in a pipeline using [Meltano](https://meltano.com/).
### Executing the Tap Directly
```bash
tap-neon --version
tap-neon --help
tap-neon --config CONFIG --discover > ./catalog.json
```
## Developer Resources
### Initialize your Development Environment
```bash
pipx install hatch
```
### Create and Run Tests
Run integration tests:
```bash
hatch run test:integration
```
You can also test the `tap-neon` CLI interface directly:
```bash
hatch run sync:console -- --about --format=json
```
### Testing with [Meltano](https://www.meltano.com)
_**Note:** This tap will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios._
Your project comes with a custom `meltano.yml` project file already created. Go ahead and [install Meltano](https://docs.meltano.com/getting-started/installation/) if you haven't already.
1. Install all plugins
```bash
meltano install
```
1. Check that the extractor is working properly
```bash
meltano invoke tap-neon --version
```
1. Execute an ELT pipeline
```bash
meltano run tap-neon target-jsonl
```