Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/finos/datahub

DataHub - Synthetic data library
https://github.com/finos/datahub

data library pandas python sklearn synthetic

Last synced: about 1 month ago
JSON representation

DataHub - Synthetic data library

Awesome Lists containing this project

README 

        
[![FINOS - Archived](https://cdn.jsdelivr.net/gh/finos/contrib-toolbox@master/images/badge-archived.svg)](https://community.finos.org/docs/governance/Software-Projects/stages/archived)



This project is archived, which means that it's in read-only state; you can download and use this code, but please be aware that it may be buggy and may also contain security vulnerabilities. If you're interested to restore development activities on this project, please email [email protected]



DataHub
 



![DataHub logo](https://raw.githubusercontent.com/finos/datahub/master/docs/logo.png) 



_Synthetic data generation_



DataHub is a set of python libraries dedicated to the production of synthetic data to be used in tests, machine learning training, statistical analysis, and other use cases [wiki](https://en.wikipedia.org/wiki/Synthetic_data). DataHub uses existing datasets to generate synthetic models. If no existing data is available it will use user-provided scripts and data rules to generate synthetic data using out-of-the-box helper datasets.



Synthetic datasets are simply artificiality manufactured sets, produced to a desired degree of accuracy. Real Data does play a part in synthetic generation, all depending on the realism you require. The product roadmaps details out the functionality planned in this respect.



DataHub's core is predominantly based around pandas data frames and object generation.

A common question: Now that I have a data frame of synthetic-data, what do I do with it? The Pandas library comes with an array of options here - so for the time being sinking to databases is out of the scope of the core library, however see that examples in the test folder for some common patterns.



**note** As we build out a config based synthetic spec generator, we will bring this back into scope - please see our roadmap/issue list and get involved in the discussion.



## Key documents



1. For information on how to get started with DataHub see our [Getting Started Guide](https://github.com/finos/datahub/blob/master/docs/GettingStarted.md)

2. For more technical information about DataHub and how to customize it, see the [Developer Guide](https://github.com/finos/datahub/blob/master/docs/DeveloperGuide.md)

3. For high-level project direction see [Road Map](https://github.com/finos/datahub/blob/master/docs/synthetic-data-roadmap/roadmap.md), [Requirements Gathering Approach](https://github.com/finos/datahub/blob/master/docs/synthetic-data-roadmap/synthetic-data-requirements-gathering.md) and [Delegated Action Groups](https://github.com/finos/datahub/tree/master/docs/delegated-action-groups).

4. For Feature Development, Good First Issues, Help Wanted and Bug Tracking see [DataHub GitHub Issues](https://github.com/finos/datahub/issues). 

5. This project uses [Gravizo](https://g.gravizo.com) for all diagrams and charts as highlighted in [DataHub Issue 41](https://github.com/finos/datahub/issues/41).   



## Overview of Synthetic data



- Synthetic data is information that's is artificially manufactured rather than generated by *real-world events.

- Synthetic data is created algorithmically, and can be used as a stand-in for  test datasets of production data

- **Real data** does play a part in synthetic data generation - depending on how

realistic you want the output



## License



Copyright 2020 Citigroup



Distributed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).



SPDX-License-Identifier: [Apache-2.0](https://spdx.org/licenses/Apache-2.0)