Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/datamill-co/target-postgres

A Singer.io Target for Postgres
https://github.com/datamill-co/target-postgres

etl json-schema postgres singer stream

Last synced: about 1 month ago
JSON representation

A Singer.io Target for Postgres

Host: GitHub
URL: https://github.com/datamill-co/target-postgres
Owner: datamill-co
License: mit
Created: 2018-03-09T03:34:12.000Z (over 6 years ago)
Default Branch: master
Last Pushed: 2024-03-08T13:49:25.000Z (4 months ago)
Last Synced: 2024-05-02T02:24:26.962Z (2 months ago)
Topics: etl, json-schema, postgres, singer, stream
Language: Python
Homepage:
Size: 877 KB
Stars: 99
Watchers: 10
Forks: 75
Open Issues: 30
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Lists

jimsghstars - datamill-co/target-postgres - A Singer.io Target for Postgres (Python)

README

# Target Postgres

[![CircleCI](https://circleci.com/gh/datamill-co/target-postgres.svg?style=svg)](https://circleci.com/gh/datamill-co/target-postgres)

[![PyPI version](https://badge.fury.io/py/singer-target-postgres.svg)](https://pypi.org/project/singer-target-postgres/)

[![](https://img.shields.io/librariesio/github/datamill-co/target-postgres.svg)](https://libraries.io/github/datamill-co/target-postgres)

A [Singer](https://singer.io/) postgres target, for use with Singer streams generated by Singer taps.

## Features

- Creates SQL tables for [Singer](https://singer.io) streams
- Denests objects flattening them into the parent object's table
- Denests rows into separate tables
- Adds columns and sub-tables as new fields are added to the stream [JSON Schema](https://json-schema.org/)
- Full stream replication via record `version` and `ACTIVATE_VERSION` messages.

## Install

1. Add `libpq` dependency

```sh
# macos
brew install postgresql
```
```sh
# ubuntu
sudo apt install libpq-dev
```

1. install `singer-target-postgres`

```sh
pip install singer-target-postgres
```

## Usage

1. Follow the
[Singer.io Best Practices](https://github.com/singer-io/getting-started/blob/master/docs/RUNNING_AND_DEVELOPING.md#running-a-singer-tap-with-a-singer-target)
for setting up separate `tap` and `target` virtualenvs to avoid version
conflicts.

1. Create a [config file](#configjson) at
`~/singer.io/target_postgres_config.json` with postgres connection
information and target postgres schema.

```json
{
"postgres_host": "localhost",
"postgres_port": 5432,
"postgres_database": "my_analytics",
"postgres_username": "myuser",
"postgres_password": "1234",
"postgres_schema": "mytapname"
}
```

1. Run `target-postgres` against a [Singer](https://singer.io) tap.

```bash
~/.virtualenvs/tap-something/bin/tap-something \
| ~/.virtualenvs/target-postgres/bin/target-postgres \
--config ~/singer.io/target_postgres_config.json >> state.json
```

If you are running windows, the following is equivalent:

```
venvs\tap-exchangeratesapi\Scripts\tap-exchangeratesapi.exe | ^
venvs\target-postgresql\Scripts\target-postgres.exe ^
--config target_postgres_config.json
```

### Config.json

The fields available to be specified in the config file are specified
here.

| Field
| ---------------------------
| `postgres_host`
| `postgres_port`
| `postgres_database`
| `postgres_username`
| `postgres_password`
| `postgres_schema`
| `postgres_sslmode`
| `postgres_sslcert`
| `postgres_sslkey`
| `postgres_sslrootcert`
| `postgres_sslcrl`
| `invalid_records_detect`
| `invalid_records_threshold`
| `disable_collection`
| `logging_level`
| `persist_empty_tables`
| `max_batch_rows`
| `max_buffer_size`
| `batch_detection_threshold`
| `state_support`
| `add_upsert_indexes`
| `before_run_sql`
| `after_run_sql`
| `before_run_sql_file`
| `after_run_sql_file`
| `application_name` | Type | Default | Details | | --------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `["string", "null"]` | `"localhost"` | | | `["integer", "null"]` | `5432` | | | `["string"]` | `N/A` | | | `["string", "null"]` | `N/A` | | | `["string", "null"]` | `null` | | | `["string", "null"]` | `"public"` | | | `["string", "null"]` | `"prefer"` | Refer to the [libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS) docs for more information about SSL | | `["string", "null"]` | `"~/.postgresql/postgresql.crt"` | Only used if a SSL request w/ a client certificate is being made | | `["string", "null"]` | `"~/.postgresql/postgresql.key"` | Only used if a SSL request w/ a client certificate is being made | | `["string", "null"]` | `"~/.postgresql/root.crt"` | Used for authentication of a server SSL certificate | | `["string", "null"]` | `"~/.postgresql/root.crl"` | Used for authentication of a server SSL certificate | | `["boolean", "null"]` | `true` | Include `false` in your config to disable `target-postgres` from crashing on invalid records | | `["integer", "null"]` | `0` | Include a positive value `n` in your config to allow for `target-postgres` to encounter at most `n` invalid records per stream before giving up. | | `["string", "null"]` | `false` | Include `true` in your config to disable [Singer Usage Logging](#usage-logging). | | `["string", "null"]` | `"INFO"` | The level for logging. Set to `DEBUG` to get things like queries executed, timing of those queries, etc. See [Python's Logger Levels](https://docs.python.org/3/library/logging.html#levels) for information about valid values. | | `["boolean", "null"]` | `False` | Whether the Target should create tables which have no records present in Remote. | | `["integer", "null"]` | `200000` | The maximum number of rows to buffer in memory before writing to the destination table in Postgres | | `["integer", "null"]` | `104857600` (100MB in bytes) | The maximum number of bytes to buffer in memory before writing to the destination table in Postgres | | `["integer", "null"]` | `5000`, or 1/40th `max_batch_rows` | How often, in rows received, to count the buffered rows and bytes to check if a flush is necessary. There's a slight performance penalty to checking the buffered records count or bytesize, so this controls how often this is polled in order to mitigate the penalty. This value is usually not necessary to set as the default is dynamically adjusted to check reasonably often. | | `["boolean", "null"]` | `True` | Whether the Target should emit `STATE` messages to stdout for further consumption. In this mode, which is on by default, STATE messages are buffered in memory until all the records that occurred before them are flushed according to the batch flushing schedule the target is configured with. | | `["boolean", "null"]` | `True` | Whether the Target should create column indexes on the important columns used during data loading. These indexes will make data loading slightly slower but the deduplication phase much faster. Defaults to on for better baseline performance. | | `["string", "null"]` | `None` | Raw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target. Useful for setup like `SET ROLE` or other connection state that is important. | | `["string", "null"]` | `None` | Raw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target. Useful for setup like `SET ROLE` or other connection state that is important. | | `["string", "null"]` | `None` | Similar to `before_run_sql` but reads an external file instead of SQL in the JSON config file. | | `["string", "null"]` | `None` | Similar to `after_run_sql` but reads an external file instead of SQL in the JSON config file. | | `["string", "null"]` | `None` | Set the postgresql `application_name` connection option to help with debugging, etc... |

### Supported Versions

`target-postgres` only supports [JSON Schema Draft4](http://json-schema.org/specification-links.html#draft-4).
While declaring a schema _is optional_, any input schema which declares a version
other than 4 will be rejected.

`target-postgres` supports all versions of PostgreSQL which are presently supported
by the PostgreSQL Global Development Group. Our [CI config](https://github.com/datamill-co/target-postgres/blob/master/.circleci/config.yml) defines all versions we are currently supporting.

| Version | Current minor | Supported | First Release | Final Release |
| ------- | ------------- | --------- | ------------------ | ----------------- |
| 15 | 15.0 | Yes | October 13, 2022 | November 11, 2027 |
| 14 | 14.5 | Yes | September 30, 2021 | November 12, 2026 |
| 13 | 13.8 | Yes | September 24, 2020 | November 13, 2025 |
| 12 | 12.12 | Yes | October 3, 2019 | November 14, 2024 |
| 11 | 11.17 | Yes | October 18, 2018 | November 9, 2023 |
| 10 | 10.22 | Yes | October 5, 2017 | November 10, 2022 |

_The above is copied from the [current list of versions](https://www.postgresql.org/support/versioning/) on Postgresql.org_

## Known Limitations

- Requires a [JSON Schema](https://json-schema.org/) for every stream.
- Only string, string with date-time format, integer, number, boolean,
object, and array types with or without null are supported. Arrays can
have any of the other types listed, including objects as types within
items.
- Example of JSON Schema types that work
- `['number']`
- `['string']`
- `['string', 'null']`
- Exmaple of JSON Schema types that **DO NOT** work
- `['string', 'integer']`
- `['integer', 'number']`
- `['any']`
- `['null']`
- JSON Schema combinations such as `anyOf` and `oneOf` are not supported.
- JSON Schema \$ref is partially supported:
- **_NOTE:_** The following limitations are known to **NOT** fail gracefully
- Presently you cannot have any circular or recursive `$ref`s
- `$ref`s must be present within the schema:
- URI's do not work
- if the `$ref` is broken, the behaviour is considered unexpected
- Any values which are the `string` `NULL` will be streamed to PostgreSQL as the literal `null`
- Table names are restricted to:
- 63 characters in length
- can only be composed of `_`, lowercase letters, numbers, `$`
- cannot start with `$`
- ASCII characters
- Field/Column names are restricted to:
- 63 characters in length
- ASCII characters

## Indexes

If the `add_upsert_indexes` config option is enabled, which it is by default, `target-postgres` adds indexes on the tables it creates for its own queries to be more performant. Specifically, `target-postgres` automatically adds indexes to the `_sdc_sequence` column and the `_sdc_level__id` columns which are used heavily when inserting and upserting.

`target-postgres` doesn't have any facilities for adding other indexes to the managed tables, so if there are more indexes required, they should be added by another downstream tool, or can just be added by an administrator when necessary. Note that these indexes incur performance overhead to maintain as data is inserted, These indexes can also prevent `target-postgres` from dropping columns in the future if the schema of the table changes, in which case an administrator should drop the index so `target-postgres` is able to drop the columns it needs to.

**Note**: Index adding is new as of version `0.2.1`, and `target-postgres` does not retroactively create indexes for tables it created before that time. If you want to add indexes to older tables `target-postgres` is loading data into, they should be added manually.

## Usage Logging

[Singer.io](https://www.singer.io/) requires official taps and targets to collect anonymous usage data. This data is only used in aggregate to report on individual tap/targets, as well as the Singer community at-large. IP addresses are recorded to detect unique tap/targets users but not shared with third-parties.

To disable anonymous data collection set `disable_collection` to `true` in the configuration JSON file.

## Developing

`target-postgres` utilizes [poetry](https://python-poetry.org/docs/) for package
management, and [PyTest](https://docs.pytest.org/en/latest/contents.html) for testing.

### Documentation