{"id":13704068,"url":"https://github.com/dagster-io/mdsfest-opensource-mds","last_synced_at":"2025-12-29T23:35:03.910Z","repository":{"id":190464812,"uuid":"681870533","full_name":"dagster-io/mdsfest-opensource-mds","owner":"dagster-io","description":"Demo Project for Open Source MDS","archived":false,"fork":false,"pushed_at":"2024-01-12T22:15:32.000Z","size":717,"stargazers_count":160,"open_issues_count":2,"forks_count":24,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-13T16:10:24.295Z","etag":null,"topics":["dagster","data","duckdb","mds","modern","stack"],"latest_commit_sha":null,"homepage":"https://dagster.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dagster-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-23T00:12:30.000Z","updated_at":"2024-10-05T00:43:22.000Z","dependencies_parsed_at":"2023-08-24T20:01:52.518Z","dependency_job_id":"5ae99a09-b400-4b14-81d3-5c71f2927beb","html_url":"https://github.com/dagster-io/mdsfest-opensource-mds","commit_stats":{"total_commits":10,"total_committers":4,"mean_commits":2.5,"dds":0.4,"last_synced_commit":"31c02ca1cb53387f950de635939ae7c9aa0a993b"},"previous_names":["dagster-io/mdsfest-opensource-mds"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagster-io%2Fmdsfest-opensource-mds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagster-io%2Fmdsfest-opensource-mds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagster-io%2Fmdsfest-opensource-mds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dagster-io%2Fmdsfest-opensource-mds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dagster-io","download_url":"https://codeload.github.com/dagster-io/mdsfest-opensource-mds/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224439683,"owners_count":17311501,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dagster","data","duckdb","mds","modern","stack"],"created_at":"2024-08-02T21:01:03.778Z","updated_at":"2025-12-29T23:35:03.876Z","avatar_url":"https://github.com/dagster-io.png","language":"Python","funding_links":[],"categories":["Sample Projects","Projects","data"],"sub_categories":[],"readme":"## Open Source MDS\n\n![](dbt_project/reports/static/asset_graph.png)\n\n## The Talk\n\nYou can watch a recording of [the talk on Youtube](https://www.youtube.com/watch?v=JHCDgRbWWq0)\n\nThis stack is built on a combination of tools including:\n\n- [Dagster](https://dagster.io)\n- [DuckDB](https://duckdb.org)\n- [dbt](https://www.getdbt.com)\n- [dbt-duckdbt](https://github.com/jwills/dbt-duckdb)\n- [Evidence](https://evidence.dev)\n- [Sling](https://slingdata.io)\n- [Steampipe](https://steampipe.io)\n\n\n\n## Requirements\n\nYou will need Python installed. This was all tested on Python 3.10.12\nFrom a virtual environment, run\n\n```python\npip install -e .'[dev]'\n```\nMost of the dependencies will be installed through Python.\n\nFor Evidence.dev, you will need [nodejs](https://nodejs.org/en/download) installed\n\nInstall [Sling](https://docs.slingdata.io/sling-cli/getting-started) for getting data from Postgres.\n\n```\n# On Mac, view the website for other platforms\nbrew install slingdata-io/sling/sling\n```\n\n\nSteampipe is a separate requirement for the Mastodon API, and can be installed by following the [instructions here](https://steampipe.io/downloads)\n\nOn Apple, run:\n\n```shell\nbrew install turbot/tap/steampipe\nsteampipe plugin install turbot/mastodon\n```\n\nFor the Mastodon API, create an Access Token. I used the [birds.town](https://birds.town/settings/applications)\ninstance.\n\nUpdate `~/.steampipe/config/mastodon.spc` with your token and instance:\n\n```\nconnection \"mastodon\" {\n    plugin = \"mastodon\"\n    server = \"https://birds.town\"\n    access_token = \"abcd12345supersecretpassword\"\n    max_toots = -1\n}\n```\n\nAnd run\n\n```shell\ndagster dev\n```\n\nLoad up dagster at http://localhost:3000/asset-groups/\n\nAnd click Materialize all to run the end-to-end pipeline.\n\n\n## Visualization\n\nEvidence.dev is used for visualization.\n\nFirst, go the `dbt_project` folder\n\n```\ncd dbt_project\n\nnpm --prefix ./reports install\nnpm --prefix ./reports run dev -- --port 4000\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdagster-io%2Fmdsfest-opensource-mds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdagster-io%2Fmdsfest-opensource-mds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdagster-io%2Fmdsfest-opensource-mds/lists"}