Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sayantikabanik/datajourney
Open-source Data Management Framework
https://github.com/sayantikabanik/datajourney
allthingsopen dagster data-engineering flask gha holoviews intake llm mito open-source panel pytest vale
Last synced: about 18 hours ago
JSON representation
Open-source Data Management Framework
- Host: GitHub
- URL: https://github.com/sayantikabanik/datajourney
- Owner: sayantikabanik
- License: cc0-1.0
- Created: 2022-03-02T14:02:56.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-10-25T03:21:04.000Z (3 months ago)
- Last Synced: 2024-10-25T04:21:19.181Z (3 months ago)
- Topics: allthingsopen, dagster, data-engineering, flask, gha, holoviews, intake, llm, mito, open-source, panel, pytest, vale
- Language: HTML
- Homepage: https://sayantikabanik.github.io/DataJourney/
- Size: 80.1 MB
- Stars: 13
- Watchers: 2
- Forks: 2
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
[](https://creativecommons.org/publicdomain/zero/1.0/)
[](https://www.contributor-covenant.org/version/2/0/code_of_conduct/)\
[](https://github.com/sayantikabanik/DataJourney/actions/workflows/CI.yml)
[](https://github.com/sayantikabanik/DataJourney/actions/workflows/github-repo-stats.yml)
[](https://github.com/sayantikabanik/DataJourney/actions/workflows/static.yml)
[](https://github.com/sayantikabanik/DataJourney/actions/workflows/review.yml)
[](https://github.com/sayantikabanik/DataJourney/actions/workflows/rate-limit-monitor.yml)
### 🚌 DataJourney
#### 🪶Short version
Design- first Open Source Data Management Toolkit. Simplifies data workflows with modular, reproducible solutions
#### 🌲Long version
DataJourney demonstrates how organizations can effectively manage and utilize data by harnessing the power of open-source technologies. It's designed to help navigate the complex landscape of data tools, offering a structured approach to building **scalable**, and **reproducible** data workflows.
Built on open-source principles, the framework guides users through essential steps—from **identifying** goals and **selecting tools** to **testing** and **customising** workflows. With its flexible, modular design, DataJourney can be tailored to individual needs, making it an invaluable toolkit for data professionals.
### 🧱 Design Philosophy (LEGO)
Built with additive, subtractive capabilities glued with open source.
Each layer has a certain strength of communication inbuilt
- PO (Base): Static home(s) to keep it together `(GitHub)`
- P1 (Tooling): Tooling, strings `(Powered by open source)`
- P2 (Maintenance + Monitoring): Env, automations `(Pixi + GHA)`
- P3 (Abstraction): Layer(s), CLI/task manager for users to interact with `(Pixi)`
### 🛠 Current workflows covered
{✨= Experimental,
✅ = Implemented}
| Status | Workflow Description |
|--------|------------------------------------------------------------------------------------------------------------------|
| ✅ | `Python Packaging framework` design principles |
| ✅ | `GitHub actions` configured |
| ✅ | `Vale.sh` configured at PR level |
| ✅ | `Pre-commit hooks` configured for code linting/formatting |
| ✨ | `Hello world` LLM design example based on [LangChain](https://python.langchain.com/) |
| ✅ | Environment management via [pixi](https://prefix.dev/) |
| ✅ | Reading data from online sources using [intake](https://github.com/intake/intake) |
| ✅ | Sample pipeline built using [Dagster](https://github.com/dagster-io/dagster) |
| ✅ | Building Dashboard using [holoviews](https://holoviews.org/gallery/index.html) + [panel](https://panel.holoviz.org/reference/index.html) |
| ✅ | Exploratory data analysis (EDA) using [mito](https://www.trymito.io/) |
| ✅ | Web UI build on [Flask](https://flask.palletsprojects.com/en/3.0.x/) |
| ✅ | Web UI re-done and expanded with [FastHTML](https://docs.fastht.ml/) |
| ✅ | Leverage AI models to analyse data [GitHub AI models Beta](https://docs.github.com/en/github-models/prototyping-with-ai-models) |
### ☕️ Quickly getting started with DataJourney
- Clone DJ `[email protected]:sayantikabanik/DataJourney.git`
- Generate & add `GITHUB_TOKEN`, instructions [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic)
- Added requirement to run the LLM workflows
- Switch directory `cd DataJourney`
- Download pixi : [prefix.dev](https://prefix.dev/)
- Activate env: `pixi shell`
- Install DJ framework locally `pixi run DJ_package`
- List all the tasks: `pixi task list`
- Execute a task from the list: `pixi run `
- Execute a task with verbosity enabled: `pixi run -v `
### 🏃🏽♀️ Active `tasks` under DJ
- GIT_TOKEN_CHECK
- DJ_package
- DJ_pre_commit
- DJ_dagster
- DJ_fasthtml_app
- DJ_flask_app
- DJ_mito_app
- DJ_panel_app
- DJ_llm_analysis
- DJ_hello_world_langchain
### 🔌 About pre-commit-hooks and activating
Just like the name suggests, pre-commit-hooks are designed to format the code based on PEP standards before committing. [More details](https://pre-commit.com/)
```shell
pixi run DJ_pre_commit
```
### 🦭 Executing LLM script: Generate stock price recommendations
```shell
pixi run DJ_llm_analysis
```
### 🪼 Execute pre-configured Dagster pipeline
```shell
pixi run DJ_dagster
```
### 🐙 Panel app
```shell
pixi run DJ_panel_app
```
*NOTE:*
The dashboard generated is exported into HTML format and saved as [stock_price_twilio_dashboard](analytics_framework%2Fdashboard%2Fstock_price_twilio_dashboard.html)
### 🐵 Mito
To explore further visit [trymito.io](https://docs.trymito.io/)
```shell
pixi run DJ_mito_app
```
[//]: # ( )
### 🦋 Display all data sources present via web UI
```shell
# Run FastHTML app
pixi run DJ_fasthtml_app
```
