https://github.com/airbnb/omniduct

A toolkit providing a uniform interface for connecting to and extracting data from a wide variety of (potentially remote) data stores (including HDFS, Hive, Presto, MySQL, etc).
https://github.com/airbnb/omniduct

Last synced: 3 months ago
JSON representation

A toolkit providing a uniform interface for connecting to and extracting data from a wide variety of (potentially remote) data stores (including HDFS, Hive, Presto, MySQL, etc).

• Host: GitHub
• URL: https://github.com/airbnb/omniduct
• Owner: airbnb
• License: mit
• Created: 2017-02-22T23:35:26.000Z (over 7 years ago)
• Default Branch: main
• Last Pushed: 2024-06-01T00:07:39.000Z (5 months ago)
• Last Synced: 2024-07-21T22:27:06.356Z (3 months ago)
• Language: Python
• Homepage:
• Size: 577 KB
• Stars: 251
• Watchers: 30
• Forks: 46
• Open Issues: 14
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
#  Omniduct

[![PyPI - Version](https://img.shields.io/pypi/v/omniduct.svg)](https://pypi.org/project/omniduct/)

![PyPI - Python Version](https://img.shields.io/pypi/pyversions/omniduct.svg)

![PyPI - Status](https://img.shields.io/pypi/status/omniduct.svg)

[![Build Status](https://img.shields.io/github/actions/workflow/status/airbnb/omniduct/tests.yml?branch=main)](https://github.com/airbnb/omniduct/actions?query=workflow%3A%22Run+Tests%22)

[![Documentation Status](https://readthedocs.org/projects/omniduct/badge/?version=latest)](http://omniduct.readthedocs.io/en/latest/?badge=latest)

`omniduct` provides uniform interfaces for connecting to and extracting data from a wide variety of (potentially remote) data stores (including HDFS, Hive, Presto, MySQL, etc).

- **Documentation:** http://omniduct.readthedocs.io

- **Source:** https://github.com/airbnb/omniduct

- **Bug reports:** https://github.com/airbnb/omniduct/issues

It provides:

- A generic plugin-based programmatic API to access data in a consistent manner across different services (see [supported protocols](http://omniduct.readthedocs.io/en/latest/protocols.html)).

- A framework for lazily connecting to data sources and maintaining these connections during the entire lifetime of the relevant Python session.

- Automatic port forwarding of remote services over SSH where connections cannot be made directly.

- Convenient IPython magic functions for interfacing with data providers from within IPython and Jupyter Notebook sessions.

- Utility classes and methods to assist in maintaining registries of useful services.