Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/nucleuscloud/neosync

Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.
https://github.com/nucleuscloud/neosync

benthos docker etl faker fine-tuning golang hacktoberfest kubernetes nextjs open-source orchestration protobuf react reactjs self-hosted synthetic-data synthetic-data-generation test-data-generator testing typescript

Last synced: 4 days ago
JSON representation

Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.

Host: GitHub
URL: https://github.com/nucleuscloud/neosync
Owner: nucleuscloud
License: other
Created: 2023-08-24T21:41:12.000Z (about 1 year ago)
Default Branch: main
Last Pushed: 2024-10-29T20:07:01.000Z (6 days ago)
Last Synced: 2024-10-29T20:24:45.894Z (6 days ago)
Topics: benthos, docker, etl, faker, fine-tuning, golang, hacktoberfest, kubernetes, nextjs, open-source, orchestration, protobuf, react, reactjs, self-hosted, synthetic-data, synthetic-data-generation, test-data-generator, testing, typescript
Language: Go
Homepage: https://www.neosync.dev
Size: 121 MB
Stars: 3,279
Watchers: 18
Forks: 118
Open Issues: 110
Metadata Files:
- Readme: README.md
- License: LICENSE.md
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md

Awesome Lists containing this project

AiTreasureBox - nucleuscloud/neosync - 11-02_3300_1](https://img.shields.io/github/stars/nucleuscloud/neosync.svg)|Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.| (Repos)

README

Open Source Data Anonymization and Synthetic Data Orchestration

## Introduction

[Neosync](https://www.neosync.dev) is an open-source, developer-first way to anonymize PII, generate synthetic data and sync environments for better testing, debugging and developer experience.

Companies use Neosync to:

1. **Safely test code against production data** - Anonymize sensitive production data in order to safely use it locally for a better testing and developer experience
2. **Easily reproduce production bugs locally** - Anonymize and subset production data to get a safe, representative data set that you can use to locally reproduce production bugs quickly and efficiently
3. **High quality data for lower-level environments** - Catch bugs before they hit production when you hydrate your staging and QA environments with production-like data
4. **Solve GDPR, DPDP, FERPA, HIPAA and more** - Use anonymized and synthetic data to reduce your compliance scope and easily comply with laws like HIPAA, GDPR, and DPDP
5. **Seed development databases** - Easily seed development databases with synthetic data for unit testing, demos and more

## Features

- **Generate synthetic data** based on your schema
- **Anonymize existing production-data** for a better developer experience
- **Subset your production database** for local and CI testing using any SQL query
- **Complete async pipeline** that automatically handles job retries, failures and playback using an event-sourcing model
- **Referential integrity** for your data automatically
- **Declarative, GitOps based configs** as a step in your CI pipeline to hydrate your CI DB
- **Pre-built data transformers** for all major data types
- **Custom data transformers** using javascript or LLMs
- **Pre-built integrations** with Postgres, Mysql, S3

## Getting started

Neosync is a fully dockerized setup which makes it easy to get up and running.

A [compose.yml](./compose.yml) file at the root contains production image refs that allow you to get up and running with just a few commands without having to build anything on your system.

Neosync uses the newer `docker compose` command, so be sure to have that installed on your machine.

To start Neosync, clone the repo into a local directory, be sure to have docker installed and running, and then run:

```sh
make compose/up
```

To stop, run:

```sh
make compose/down
```

Neosync will now be available on [http://localhost:3000](http://localhost:3000).

The production compose pre-seeds with connections and jobs to get you started! Simply run the generate and sync job to watch Neosync in action!

## Kubernetes, Auth Mode and more

For more in-depth details on environment variables, Kubernetes deployments, and a production-ready guide, check out the [Deploy Neosync](https://docs.neosync.dev/deploy/introduction) section of our Docs.

## Resources

Some resources to help you along the way:

- [Docs](https://docs.neosync.dev) for comprehensive documentation and guides
- [Discord](https://discord.com/invite/MFAMgnp4HF) for discussion with the community and Neosync team
- [X](https://x.com/neosynccloud) for the latest updates

## Contributing

We love contributions big and small. Here are just a few ways that you can contribute to Neosync.

- Join our [Discord](https://discord.com/invite/MFAMgnp4HF) channel and ask us any questions there
- Open a PR (see our instructions on [developing with Neosync locally](https://docs.neosync.dev/guides/neosync-local-dev))
- Submit a [feature request](https://github.com/nucleuscloud/neosync/issues/new?assignees=&labels=enhancement%2C+feature&template=feature_request.md) or [bug report](https://github.com/nucleuscloud/neosync/issues/new?assignees=&labels=bug&template=bug_report.md)

## Licensing

We strongly believe in free and open source software and make this repo is available under the [MIT expat license](./LICENSE.md).