https://github.com/munichpavel/fake-data-for-learning
Sample interesting fake data for machine and human learning
https://github.com/munichpavel/fake-data-for-learning
data-generation data-science fake-data fake-data-generator machine-learning python statistics
Last synced: 3 months ago
JSON representation
Sample interesting fake data for machine and human learning
- Host: GitHub
- URL: https://github.com/munichpavel/fake-data-for-learning
- Owner: munichpavel
- License: mit
- Created: 2019-06-16T18:54:51.000Z (about 7 years ago)
- Default Branch: main
- Last Pushed: 2025-02-27T10:45:13.000Z (over 1 year ago)
- Last Synced: 2025-12-17T02:32:58.988Z (6 months ago)
- Topics: data-generation, data-science, fake-data, fake-data-generator, machine-learning, python, statistics
- Language: Python
- Homepage: https://munichpavel.github.io/fake-data-for-learning
- Size: 475 KB
- Stars: 8
- Watchers: 3
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# fake-data-for-learning
Interesting fake multivariate data is harder to generate than it should be. Textbooks typically give definitions, two standard examples (multinomial and multivariate normal) and then proceed to proving theorems and propositions. True, one dimensional distributions can be combined, but here as well the source of examples is also sparse, e.g. products of distributions or copulas (typically Gaussian or t-copulas) applied to these 1-d examples.
For machine learning experimentation, it is useful to have an unlimited supply of interesting fake data, where by interesting I mean that we know certain properties of the data and want to test if the algorithm can pick this up. A great potential source of such data is graphical models.
The goal of this package is to make it easy to generate interesting fake data. In the current release, we generate fake data with discrete Bayesian networks (also known as directed graphical models).
* **Website**: [https://munichpavel.github.io/fake-data-for-learning](https://munichpavel.github.io/fake-data-for-learning)
* **Documentation**: [https://munichpavel.github.io/fake-data-docs/](https://munichpavel.github.io/fake-data-docs/)
## Basic usage
The methods and interfaces for `fake_data_for_learning` largely follow those of [scipy](https://scipy.org), e.g. the method `rvs` to generate random samples, and `pmf` for the probability mass function, with extensions to handle non-integer sample values.
Defining and sampling from (discrete) conditional random variables:
```python
import numpy as np
from fake_data_for_learning.fake_data_for_learning import BayesianNodeRV, SampleValue
# Gender -> Y
# Define Gender with probability table, node label and value labels
Gender = BayesianNodeRV('Gender', np.array([0.55, 0.45]), values=['female', 'male'])
# Define Y with conditional probability table, node, value and parent labels
pt_YcGender = np.array([
[0.9, 0.1],
[0.4, 0.6],
])
Y = BayesianNodeRV('Y', pt_YcGender, parent_names=['Gender'])
# Evaluate probability mass function for given parent values
Y.pmf(0, parent_values={'Gender': SampleValue('male', label_encoder=Gender.label_encoder)})
# 0.4
# Sample from Y given Gender
Y.rvs({'Gender': SampleValue('male', label_encoder=Gender.label_encoder)}, seed=42)
# array([0])
```
Combine into a Bayesian network; sample and calculate the probability mass function of each sample:
```python
from fake_data_for_learning.fake_data_for_learning import FakeDataBayesianNetwork
samples = bn.rvs(size=5)
# Rounding of pmf is only for display purposes
samples['pmf'] = samples[['Gender', 'Y']].apply(lambda sample: round(bn.pmf(sample), 3), axis=1)
```
Visualize the Bayesian network:
```python
bn.draw_graph()
```
See the demo notebook [notebooks/bayesian-network.ipynb](notebooks/bayesian-network.ipynb) for feature examples.
To avoid having to enter all each value of a conditional probability array, there are also two methods to generate random conditional probability tables.
The method `fake_data_for_learning.utils.RandomCpt()` gives a random conditional probability table, but if you want to constrain the entries to satisfy constraints on expectation values, this is done in the class `fake_data_for_learning.utils.ProbabilityPolytope`; see the example notebook [notebooks/conditional-probability-tables-with-constraints.ipynb](notebooks/conditional-probability-tables-with-constraints.ipynb). See also [Optional Dependencies](#optional-dependencies) below.
## Installation
Install from [pypi](https://pypi.org/project/fake-data-for-learning/): `pip install fake-data-for-learning`
### Optional dependencies
Note that the methods of `utils.ProbabilityPolytope` that use polytope calculations to generate conditional probability tables subject to constraints on expectation value require the non-pure-python library [pypoman](https://github.com/stephane-caron/pypoman). System dependencies include [`cython`](https://cython.org) and [`glpk`](https://www.gnu.org/software/glpk/). See the [installation instructions](https://github.com/stephane-caron/pypoman#installation) for external dependency instructions.
On Mac OS X, these can be installed with
```console
brew install cython glpk cddlib gmp
```
(see also the [pycddlib](https://github.com/mcmtroffaes/pycddlib) issue [Installation using pip on MacOS 11.6 #49](https://github.com/mcmtroffaes/pycddlib/issues/49), but beware the typo in the first comment dependency name).
Follow the instructions regarding setting environment variables (e.g. in your `.bashrc` or `.zshrc` files). For example, your `.zshrc` file on Mac OS X might contain
```bash
# openblas
export LDFLAGS="-L/opt/homebrew/opt/openblas/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openblas/include"
# pycddlib
export CFLAGS="-I$(brew --prefix)/include -L$(brew --prefix)/lib"
export PATH="/opt/homebrew/opt/cython/bin:$PATH"
# For pkg-config to find openblas you may need to set:
export PKG_CONFIG_PATH="/opt/homebrew/opt/openblas/lib/pkgconfig"
```
By default the python dependencies for `utils.ProbabilityPolytope` are not installed; to do so, run from your virtual environment `pip install 'fake-data-for-learning[probability_polytope]'`
### Local development
* ``git clone`` the repository and ``cd`` into the project directory
* Create a virtual environment from the included ``requirements.txt`` file
## Documentation
To generate your own [Sphinx documentation](http://sphinx-doc.org/), you must set the environment variable ``LOCAL_BUILDDIR``.
Convenience scripts for the case of a separate build directories (locally and remotely) are in [docs/scripts](https://github.com/munichpavel/fake-data-for-learning/tree/master/docs/scripts).
## Related packages
This package exists because I became tired of googling for existing implementations of how I wanted to generate fake data. In the development process, however, I found other packages for generating interesting fake data, notably
* [pyro](https://pyro.ai/) is convenient for generating a wide variety of interesting fake data. It is easy to generate fake data from Bayesian networks joined by link functions; see e.g. [the introductory tutorial](http://pyro.ai/examples/intro_part_i.html).
* [pgmpy](http://pgmpy.org/index.html) has a large amount of overlapping functionality, noting that `pgmpy` has a significantly larger scope. One difference is the bookkeeping convention for conditional probability tables: `pgmpy` represents conditional probability tables as 2d matrices, whereas we give each of the *n*-1 conditioned variables its own dimension, resulting in an *n* dimensional matrix.
* [pyagrum](https://pyagrum.readthedocs.io) is a Python wrapper around the C++ library [aGrUM](http://agrum.org/), and has similar funcionality with a larger scope. Unlike `pgmpy`, `pyagrum` has a similar API for specifying conditional probability tables to the one used here.
* [causalgraphicalmodels](https://github.com/ijmbarr/causalgraphicalmodels)'s class `StructuralCausalModel` allows sampling from Bayesian network where the variables are related as functions of one another, rather than via the conditional probability tables used here.
## Change log
### v0.4.5
Add to Mac OS X optional install instructions.
### v0.4.4
Fix missing usage of optional dependency specification
### v0.4.3
Make non-python-dependencies from `utils.ProbabilityPolytope` an optional install.
### v0.4.2
Fix mac os x dependency install issue.
### v0.4.1
Fix dependencies' API changes.
### v0.4.0
This release adds a method for generating categorical data whose (multidimensional) contingency table equals a given one. The motivation is to generate fake data exhibiting [Simpson's paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox).