https://github.com/transferwise/pipelinewise-tap-oracle
Singer.io Tap for Oracle - PipelineWise compatible
https://github.com/transferwise/pipelinewise-tap-oracle
Last synced: 5 months ago
JSON representation
Singer.io Tap for Oracle - PipelineWise compatible
- Host: GitHub
- URL: https://github.com/transferwise/pipelinewise-tap-oracle
- Owner: transferwise
- License: agpl-3.0
- Archived: true
- Fork: true (singer-io/tap-oracle)
- Created: 2019-09-06T07:45:13.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2024-09-20T00:41:08.000Z (over 1 year ago)
- Last Synced: 2025-09-25T11:54:27.256Z (9 months ago)
- Language: Python
- Homepage: https://transferwise.github.io/pipelinewise/
- Size: 188 KB
- Stars: 3
- Watchers: 2
- Forks: 8
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Notice
To better serve Wise business and customer needs, the PipelineWise codebase needs to shrink.
We have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo.
The last version before this decision is [v0.64.1](https://github.com/transferwise/pipelinewise/tree/v0.64.1)
We thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily
# pipelinewise-tap-oracle
[](https://badge.fury.io/py/pipelinewise-tap-oracle)
[](https://pypi.org/project/pipelinewise-tap-oracle/)
[](https://opensource.org/licenses/GPL-3.0)
[Singer](https://www.singer.io/) tap that extracts data from a [Oracle](https://www.oracle.com/database/) database and produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
This is a [PipelineWise](https://transferwise.github.io/pipelinewise) compatible tap connector.
## How to use it
The recommended method of running this tap is to use it from [PipelineWise](https://transferwise.github.io/pipelinewise). When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at [Tap Oracle](https://transferwise.github.io/pipelinewise/connectors/taps/oracle.html)
If you want to run this [Singer Tap](https://singer.io) independently please read further.
## Log based replication
Tap-Oracle Log-based replication requires some configuration changes in Oracle database:
* Enable `ARCHIVELOG` mode
* Set retention period a reasonable and long enough period, ie. 1 day, 3 days, etc.
* Enable Supplemental logging
### Setting up Log-based replication on a self hosted Oracle Database:
To verify the current archiving mode, if the result is `ARCHIVELOG`, archiving is enabled:
```
SQL> SELECT LOG_MODE FROM V$DATABASE
```
To enable `ARCHIVELOG` mode (if not enabled yet):
```
SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP MOUNT
SQL> ALTER DATABASE ARCHIVELOG
SQL> ALTER DATABASE OPEN
```
To set retention period, use RMAN:
```
RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;
```
To enable supplemental logging:
```
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS
```
### Setting up Log-based replication on Oracle on Amazon RDS
To set retention period:
```
begin
rdsadmin.rdsadmin_util.set_configuration(
name => 'archivelog retention hours',
value => '24');
end;
```
To enable supplemental logging:
```
begin
rdsadmin.rdsadmin_util.alter_supplemental_logging(p_action => 'ADD');
end;
```
### Install and Run
First, make sure Python 3 is installed on your system or follow these
installation instructions for [Mac](http://docs.python-guide.org/en/latest/starting/install3/osx/) or
[Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04).
It's recommended to use a virtualenv:
```bash
python3 -m venv venv
pip install pipelinewise-tap-oracle
```
or
```bash
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
```
### Configuration
Running the the tap requires a `config.json` file. Example with the minimal settings:
```json
{
"host": "foo.com",
"port": 1521,
"user": "my_user",
"password": "password",
"sid": "ORCL",
"filter_schemas": "MY_USER" # optional
}
```
### To run tests:
Tests require Oracle on Amazon RDS >= 12.1, and a user called `ROOT`.
1. Define environment variables that requires running the tests.
```
export TAP_ORACLE_HOST=
export TAP_ORACLE_PORT=
export TAP_ORACLE_USER=ROOT
export TAP_ORACLE_PASSWORD=
export TAP_ORACLE_SID=
```
1. Install python dependencies in a virtual env and run nose unit and integration tests
```
python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip
pip install .
pip install nose
```
3. To run unit tests:
```
nosetests
```