Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/z3z1ma/cdf
A framework to manage data, continuously
https://github.com/z3z1ma/cdf
data framework pipelines transformation
Last synced: about 2 months ago
JSON representation
A framework to manage data, continuously
- Host: GitHub
- URL: https://github.com/z3z1ma/cdf
- Owner: z3z1ma
- License: apache-2.0
- Created: 2023-10-26T05:31:25.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-09-01T16:04:08.000Z (4 months ago)
- Last Synced: 2024-10-14T07:28:15.550Z (2 months ago)
- Topics: data, framework, pipelines, transformation
- Language: Python
- Homepage: https://z3z1ma.github.io/cdf/
- Size: 913 KB
- Stars: 25
- Watchers: 2
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
CDF (Continuous Data Framework)
Craft end-to-end data pipelines and manage them continuously
---
## 📖 Table of Contents
- [📖 Table of Contents](#-table-of-contents)
- [📍 Overview](#-overview)
- [📦 Features](#-features)
- [🚀 Getting Started](#-getting-started)
- [📚 Documentation](#-documentation)
- [🤝 Contributing](#-contributing)
- [🛣 Roadmap](#-roadmap)
- [📄 License](#-license)
- [👏 Acknowledgments](#-acknowledgments)
---
## 📍 Overview
CDF (Continuous Data Framework) is an integrated framework designed to manage data across the entire lifecycle, from ingestion through transformation to publishing. It is built on top of two open-source projects, `sqlmesh` and `dlt`, providing a unified interface for complex data operations. CDF simplifies data engineering workflows, offering scalable solutions from small to large projects through an opinionated project structure that supports both multi-workspace and single-workspace layouts.
> [!WARNING]
> The repo is currently under ACTIVE development with multiple large refactors already having been completed. As such, you must be aware that the codebase is not yet stable and is subject to change. Furthermore, you must look to the code (or tests) itself for the most accurate and up-to-date information until this disclaimer is removed.
## Features
...
## Getting Started
1. **Installation**:
CDF requires Python 3.9 or newer. Install CDF using pip:
```bash
pip install python-cdf
```
## Documentation
For detailed documentation, including API references and tutorials, visit [CDF Documentation](#).
## Contributing
Contributions to CDF are welcome! Please refer to the [contributing guidelines](CONTRIBUTING.md) for more information on how to submit pull requests, report issues, or suggest enhancements.
## License
CDF is licensed under [Apache 2.0 License](LICENSE).
---
This README provides an overview of the CDF tool, highlighting its primary features, installation steps, basic usage examples, and contribution guidelines. It serves as a starting point for users to understand the capabilities of CDF and how it can be integrated into their data engineering workflows.
### 🧪 Tests
Run the tests with `pytest`:
```sh
pytest tests
```
## 🛣 Project Roadmap
TODO: Add a roadmap for the project.
## 🤝 Contributing
Contributions are welcome! Here are several ways you can contribute:
- **[Submit Pull Requests](https://github.com/z3z1ma/cdf/blob/main/CONTRIBUTING.md)**: Review open PRs, and submit your own PRs.
- **[Join the Discussions](https://github.com/z3z1ma/cdf/discussions)**: Share your insights, provide feedback, or ask questions.
- **[Report Issues](https://github.com/z3z1ma/cdf/issues)**: Submit bugs found or log feature requests for z3z1ma.
#### *Contributing Guidelines*
Click to expand
1. **Fork the Repository**: Start by forking the project repository to your GitHub account.
2. **Clone Locally**: Clone the forked repository to your local machine using a Git client.
```sh
git clone
```
3. **Create a New Branch**: Always work on a new branch, giving it a descriptive name.
```sh
git checkout -b new-feature-x
```
4. **Make Your Changes**: Develop and test your changes locally.
5. **Commit Your Changes**: Commit with a clear and concise message describing your updates.
```sh
git commit -m 'Implemented new feature x.'
```
6. **Push to GitHub**: Push the changes to your forked repository.
```sh
git push origin new-feature-x
```
7a. **Submit a Pull Request**: Create a PR against the original project repository. Clearly describe the changes and their motivations.
Once your PR is reviewed and approved, it will be merged into the main branch.
---
## 📄 License
This project is distributed under the [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) License. For more details, refer to the [LICENSE](https://github.com/z3z1ma/cdf/blob/main/LICENSE) file.
---
## 👏 Acknowledgments
- Harness (https://harness.io/) for being the proving grounds in which the initial concept of this project was born.
- SQLMesh (https://sqlmesh.com) for being a foundational pillar of this project as well as the team for their support,
advice, and guidance.
- DLT (https://dlthub.com) for being the other foundational pillar of this project as well as the team for their
support, advice, and guidance.
[**Return**](#Top)
---