https://github.com/mitdbg/aurum-datadiscovery
https://github.com/mitdbg/aurum-datadiscovery
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/mitdbg/aurum-datadiscovery
- Owner: mitdbg
- License: mit
- Created: 2016-04-27T12:45:00.000Z (about 10 years ago)
- Default Branch: master
- Last Pushed: 2023-03-06T05:16:32.000Z (over 3 years ago)
- Last Synced: 2025-02-21T18:35:06.963Z (over 1 year ago)
- Language: Python
- Size: 43.5 MB
- Stars: 75
- Watchers: 16
- Forks: 49
- Open Issues: 69
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Aurum: Discovering Data in Lakes, Clouds and Databases
Webpage version of this documentation: [http://mitdbg.github.io/aurum-datadiscovery/](http://mitdbg.github.io/aurum-datadiscovery/)
Aurum helps users identify relevant content among multiple data
sources that may consist of tabular files, such as CSV, and relational tables.
These may be stored in relational database management systems (RDBMS), file
systems, and they may live in cloud services, data lakes or other on-premise
repositories.
Aurum helps you find data through different interfaces. The most flexible one is
an API of primitives that can be composed to build queries that describe the
data of interest. For example, you can write a query that says "find tables that
contain a column with name 'ID' and have at least one column that looks like
an input column". You can also query with very simple primitives, such as "find
columns that contain the keyword 'caffeine'". You can also do more complex
queries, such as figuring out what tables join with a table of interest. The
idea is that the API is flexible enough to allow a wide range of use cases, and
that it works over all data you feed to the system, regardless where these live.
* [**Why do I need Aurum?**](docs/why_aurum.md) We show you various scenarios in which Aurum has proven useful.
* [**Design Rationale**](docs/design_rationale.md) A brief explanation of the system architecture and
design rationale.
* [**Quick Start**](docs/quick_start.md) A guide to setup Aurum and start running some discovery queries.
* [**Tutorial**](docs/tutorial.md) A tutorial that walks you through the different aspects of Aurum, from how
to write queries using the discovery API, to how to create new connectors to read data from different
data sources to how to store data in different stores.
* [**FAQ**](docs/faq.md) Collection of frequent questions
Aurum is a work in progress, we expect to release its first open-source version in the 4th quarter of 2018.
We are happy to accept contributions of the community. If you are interested in contributing take a look at
the [CONTRIBUTING](../CONTRIBUTING.md) and feel free to email raulcf@csail.mit.edu
We also have a code of conduct:
### Code of Conduct
Check the code of conduct for Aurum here:
https://github.com/mitdbg/aurum-datadiscovery/blob/master/CODE_OF_CONDUCT.md
Please, report violations of the code of conduct by sending an email to
raulcf@csail.mit.edu