Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/stephane-klein/tap-gitlab-playground
Import data from GitLab to PostgreSQL with singer tap-gitlab
https://github.com/stephane-klein/tap-gitlab-playground
etl etl-components gitlab playground singer
Last synced: 8 days ago
JSON representation
Import data from GitLab to PostgreSQL with singer tap-gitlab
- Host: GitHub
- URL: https://github.com/stephane-klein/tap-gitlab-playground
- Owner: stephane-klein
- Created: 2020-08-12T13:22:30.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2020-08-21T10:06:46.000Z (over 4 years ago)
- Last Synced: 2024-11-29T22:16:09.667Z (2 months ago)
- Topics: etl, etl-components, gitlab, playground, singer
- Language: Shell
- Homepage:
- Size: 4.88 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Tap-GitLab playground
Project status: Work in progress
In this project I use `tap-gitlab` to extract GitLab data via API and import this data in PostreSQL database.
Question: [Should I use https://github.com/singer-io/tap-gitlab or https://gitlab.com/meltano/tap-gitlab ?](https://github.com/singer-io/tap-gitlab/issues/19)
In this playground, I will use:
- [`meltano/tap-gitlab`](https://gitlab.com/meltano/tap-gitlab)
- [`datamill-co/target-postgres`](https://github.com/datamill-co/target-postgres)
## Prerequisites
On macOS, install with Brew:
```
$ brew install python
$ brew cask install docker
```
## Install and configure tap-gitlab
```
$ python3 -m venv .venv/tap-gitlab/
$ ./.venv/tap-gitlab/bin/pip install git+https://gitlab.com/meltano/tap-gitlab.git
```
## Install target-postgres
```
$ python3 -m venv .venv/target-postgres/
```
```
$ ./.venv/target-postgres/bin/pip install singer-target-postgres==0.2.4
```
## Start PostgreSQL database
```
$ ./scripts/up.sh
```
## Execute GitLab importation to PostgreSQL
The first time, when `state.json` don't exists, execute:
```
$ ./.venv/tap-gitlab/bin/tap-gitlab \
-c tap-gitlab-config.json \
| ./.venv/target-postgres/bin/target-postgres \
--config target_postgres_config.json \
> state.json
```
Next execute this commands to use `state.json` [Singer State](https://github.com/singer-io/getting-started/blob/master/docs/CONFIG_AND_STATE.md) file:
```
$ ./.venv/tap-gitlab/bin/tap-gitlab \
-c tap-gitlab-config.json \
-s state.json \
| ./.venv/target-postgres/bin/target-postgres \
--config target_postgres_config.json \
> state-target.json
$ tail -1 state-target.json > state.json
```