{"id":20405742,"url":"https://github.com/cloudquery/cq_dagster_embedded_elt","last_synced_at":"2026-06-04T12:31:35.335Z","repository":{"id":225362686,"uuid":"762512799","full_name":"cloudquery/cq_dagster_embedded_elt","owner":"cloudquery","description":"An example repository of how to configure a dagster workflow to use CloudQuery running directly inside (i.e \"embedded ELT\")","archived":false,"fork":false,"pushed_at":"2024-02-24T16:24:29.000Z","size":4,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-25T19:41:51.900Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudquery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-23T23:49:41.000Z","updated_at":"2024-03-01T15:25:08.000Z","dependencies_parsed_at":"2024-03-01T17:53:11.140Z","dependency_job_id":null,"html_url":"https://github.com/cloudquery/cq_dagster_embedded_elt","commit_stats":null,"previous_names":["cloudquery/cq_dagster_embedded_elt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fcq_dagster_embedded_elt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fcq_dagster_embedded_elt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fcq_dagster_embedded_elt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudquery%2Fcq_dagster_embedded_elt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudquery","download_url":"https://codeload.github.com/cloudquery/cq_dagster_embedded_elt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241950200,"owners_count":20047599,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T05:12:50.945Z","updated_at":"2026-06-04T12:31:35.329Z","avatar_url":"https://github.com/cloudquery.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CloudQuery Dagster Embedded Example\n\nThis example contains an example of how to run CloudQuery as ingestion step\ninside a dagster asset utilizing solely dagster orchestrator and resource managemetn without relying on other cloud providers and orchestrators.\n\nThis example pipeline will be fully runnable both locally and in the cloud using the same configuration, code and queries! Utilizing Dagster, DuckDB (MotherDuck) and CloudQuery local \u0026 cloud capabilities.\n\nThis is a [Dagster](https://dagster.io/) project scaffolded with [`dagster project scaffold`](https://docs.dagster.io/getting-started/create-new-project).\n\n## Setup\n\nTo run this example locally\n\n```\ngit clone https://github.com/cloudquery/cq_dagster_embedded\ncd cq_dagster_embedded\npip install -e \".[dev]\"\n\n# Load it in the web UI\ndagster-webserver\n```\n\n## Getting started\n\nFirst, install your Dagster code location as a Python package. By using the --editable flag, pip will install your Python package in [\"editable mode\"](https://pip.pypa.io/en/latest/topics/local-project-installs/#editable-installs) so that as you develop, local code changes will automatically apply.\n\n```bash\npip install -e \".[dev]\"\n```\n\nThen, start the Dagster UI web server:\n\n```bash\ndagster dev\n```\n\nOpen http://localhost:3000 with your browser to see the project.\n\nYou can start writing assets in `cq_dagster_embedded/assets.py`. The assets are automatically loaded into the Dagster code location as you define them.\n\n## Development\n\n### Adding new Python dependencies\n\nYou can specify new Python dependencies in `setup.py`.\n\n### Unit testing\n\nTests are in the `cq_dagster_embedded_tests` directory and you can run tests using `pytest`:\n\n```bash\npytest cq_dagster_embedded_tests\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudquery%2Fcq_dagster_embedded_elt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudquery%2Fcq_dagster_embedded_elt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudquery%2Fcq_dagster_embedded_elt/lists"}