Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/datonic/datadex

📦 Serverless and local-first Open Data Platform
https://github.com/datonic/datadex

dbt duckdb open-data quarto sql

Last synced: 13 days ago
JSON representation

📦 Serverless and local-first Open Data Platform

Awesome Lists containing this project

README 

        





  
D A T A D E X


  
The Open Data Platform for your community Open Data 





  GitHub

  GitHub Workflow Status

  GitHub Repo stars








Datadex is a fully open-source, serverless, and local-first Data Platform that improves how communities collaborate on Open Data. Datadex is not a new tool, it is a pattern showing an opinionated bridge between existing ones.



> [!TIP]

> Datasets generated by this project are ready to explore and consume at HuggingFace.

>

> [Check them out](https://huggingface.co/datonic)!



[Learn more about the approach in this post](https://davidgasquez.com/modern-open-data-portals/) or check other real-world production implementations of the Datadex pattern working in the following repositories:



- [LUNG-SARG](https://github.com/open-radiogenomics/lung-sarg). The Open Data Platform for Sustainable, Accessible Lung Radiogenomics.

- [Gitcoin Grants Data Portal](https://github.com/davidgasquez/gitcoin-grants-data-portal). Data hub for Gitcoin Grants data. Improves data access and empowers data scientists to conduct research and helps to guide community-driven analysis and decisions.

- [Filecoin Data Portal](https://github.com/davidgasquez/filecoin-data-portal/). Data hub for Filecoin data! Like Dune, but in your laptop.



### 💡 Principles



- **Open**: Code, standards, infrastructure, and data, are public and open source.

- **Modular and Interoperable**: Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), can be deployed to many places (S3 + GH Pages, IPFS, ...) and integrates with multiple tools (thanks to the Arrow ecosystem). Use open tools, standards, infrastructure, and share data in [accessible formats](https://voltrondata.com/codex/a-new-frontier).

- **Permissionless**: Don't ask, fork it and improve the models, add a new source or update any script. No API limits, just plain files.

- **Data as Code**: Declarative stateless transformations tracked in `git`. Improves data access and empowers data scientists to conduct research and helps to guide community-driven analysis and decisions. Version your data as code! Publish and share your reusable models for others to build on top. Datasets should be both reproducible and accessible!

- **Glue**: Be a bridge between tools and approaches. E.g: Use software engineering good practices like types, tests, materialized views, and more.



### 🚀 What can you do with Datadex?



- Add new data sources. Bring data locally and work with it!

- Model existing datasets using `Python` and `SQL`.

- Explore your data wherever you want. Use Jupyter Notebooks, BI Tools, Excel, ....

- [Share your findings with others](https://datadex.datonic.io/notebooks/2023-01-01-Datadex) by publishing them online as beautiful static websites.



## ⚙️ Setup



There are two ways to get started with Datadex or your own Open Data Portal/Platform based on the Datadex pattern, Python Virtual Environment or Docker / Dev Containers.



If you hit any issue, please [open an issue](https:github.com/datonic/datadex/issues/new)!



### 🐍 Python Virtual Environment



You can install all the dependencies inside a Python virtual environment by running `make setup`. To do that, clone the repository and run the following commands from the root folder.



```bash

make setup

```



Alternatively, you can rely on your system's Python installation to create a virtual environment and install the dependencies.



```bash

# Create a virtual environment

python3 -m venv .venv

source .venv/bin/activate



# Install the package and dependencies

pip install -e ".[dev]"

```



Now, you should be able to spin up Dagster UI (`make dev` or `dagster dev`) and [access it locally](http://127.0.0.1:3000).



### 🐳 Docker / Dev Containers



Using [VSCode Remote Containers](https://code.visualstudio.com/docs/remote/containers) is the fastest and smoother way to start using Datadex, but requires you to have Docker running. Open the project in VSCode and click on the bottom right corner to open the project in a container.



Once inside the develpment environment, you'll only need to run `make dev` to spin up the [Dagster UI locally](http://127.0.0.1:3000). You'll also have a few extra extensions installed and configured to work with the project.



The development environment can also run in your browser thanks to GitHub Codespaces!



[![badge](https://github.com/codespaces/badge.svg)](https://codespaces.new/davidgasquez/datadex)



## 🎯 Motivation



This project started after [thinking how an Open Data Protocol could look like](https://publish.obsidian.md/davidgasquez/Open+Data)!



## 👏 Acknowledgements



- Datadex is possible thanks to amazing open source projects like [DuckDB](https://www.duckdb.org/), [dbt](https://getdbt.com), [Dagster](https://dagster.io/), [Quarto](https://quarto.org/), and many others...

- Datadex name was inspired by [Juan Benet's `data` projects](https://juan.benet.ai/blog/2014-03-11-discussion-scienceexchange/).