Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dagster-io/mdsfest-opensource-mds
Demo Project for Open Source MDS
https://github.com/dagster-io/mdsfest-opensource-mds
dagster data duckdb mds modern stack
Last synced: about 1 month ago
JSON representation
Demo Project for Open Source MDS
- Host: GitHub
- URL: https://github.com/dagster-io/mdsfest-opensource-mds
- Owner: dagster-io
- Created: 2023-08-23T00:12:30.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-01-12T22:15:32.000Z (5 months ago)
- Last Synced: 2024-04-18T14:35:04.021Z (about 2 months ago)
- Topics: dagster, data, duckdb, mds, modern, stack
- Language: Python
- Homepage: https://dagster.io
- Size: 700 KB
- Stars: 140
- Watchers: 6
- Forks: 23
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Lists
- awesome-dbt - mdsfest-opensource-mds - Demo project for open source MDS. (Sample Projects)
- awesome-public-dbt-projects - Bird Data - Bird Data with [Dagster](https://dagster.io/), [Evidence](https://evidence.dev/) and duckdb. (Projects)
- awesome-dbt - mdsfest-opensource-mds - Demo project for open source MDS. (Sample Projects)
README
## Open Source MDS
## The Talk
You can watch a recording of [the talk on Youtube](https://www.youtube.com/watch?v=JHCDgRbWWq0)
This stack is built on a combination of tools including:
- [Dagster](https://dagster.io)
- [DuckDB](https://duckdb.org)
- [dbt](https://www.getdbt.com)
- [dbt-duckdbt](https://github.com/jwills/dbt-duckdb)
- [Evidence](https://evidence.dev)
- [Sling](https://slingdata.io)
- [Steampipe](https://steampipe.io)
## Requirements
You will need Python installed. This was all tested on Python 3.10.12
From a virtual environment, run
```python
pip install -e .'[dev]'
```
Most of the dependencies will be installed through Python.
For Evidence.dev, you will need [nodejs](https://nodejs.org/en/download) installed
Install [Sling](https://docs.slingdata.io/sling-cli/getting-started) for getting data from Postgres.
```
# On Mac, view the website for other platforms
brew install slingdata-io/sling/sling
```
Steampipe is a separate requirement for the Mastodon API, and can be installed by following the [instructions here](https://steampipe.io/downloads)
On Apple, run:
```shell
brew install turbot/tap/steampipe
steampipe plugin install turbot/mastodon
```
For the Mastodon API, create an Access Token. I used the [birds.town](https://birds.town/settings/applications)
instance.
Update `~/.steampipe/config/mastodon.spc` with your token and instance:
```
connection "mastodon" {
plugin = "mastodon"
server = "https://birds.town"
access_token = "abcd12345supersecretpassword"
max_toots = -1
}
```
And run
```shell
dagster dev
```
Load up dagster at http://localhost:3000/asset-groups/
And click Materialize all to run the end-to-end pipeline.
## Visualization
Evidence.dev is used for visualization.
First, go the `dbt_project` folder
```
cd dbt_project
npm --prefix ./reports install
npm --prefix ./reports run dev -- --port 4000
```