{"id":41463337,"url":"https://github.com/jairus-m/dagster-sdlc","last_synced_at":"2026-04-04T16:16:12.251Z","repository":{"id":261009850,"uuid":"882969720","full_name":"jairus-m/dagster-sdlc","owner":"jairus-m","description":"An e2e pipeline using dlt, dagster, duckdb, and dbt-core","archived":false,"fork":false,"pushed_at":"2025-07-29T16:17:02.000Z","size":28324,"stargazers_count":18,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-02-02T11:44:58.997Z","etag":null,"topics":["dagster","dbt","dlthub","duckdb","sklearn","snowflake","software-development-lifecycle","uv"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jairus-m.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-04T06:22:04.000Z","updated_at":"2025-12-23T04:49:33.000Z","dependencies_parsed_at":"2024-12-04T06:18:19.484Z","dependency_job_id":"87088ffb-2cec-4213-84ca-d996543cde65","html_url":"https://github.com/jairus-m/dagster-sdlc","commit_stats":null,"previous_names":["jairus-m/strava-duckdb","jairus-m/dagster-dlt","jairus-m/dagster-sdlc"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jairus-m/dagster-sdlc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jairus-m%2Fdagster-sdlc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jairus-m%2Fdagster-sdlc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jairus-m%2Fdagster-sdlc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jairus-m%2Fdagster-sdlc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jairus-m","download_url":"https://codeload.github.com/jairus-m/dagster-sdlc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jairus-m%2Fdagster-sdlc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31405699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dagster","dbt","dlthub","duckdb","sklearn","snowflake","software-development-lifecycle","uv"],"created_at":"2026-01-23T16:00:39.984Z","updated_at":"2026-04-04T16:16:12.202Z","avatar_url":"https://github.com/jairus-m.png","language":"Python","funding_links":[],"categories":["Learning Material"],"sub_categories":["Projects"],"readme":"# The Analytics Development Lifecycle within a Modern Data Engineering Framework\n\nUtilizing dltHub, dbt, + Dagster as a framework for developing data products with software engineering best practices. \n\n![Slide1](https://github.com/user-attachments/assets/4f0025b5-c203-424f-96a6-5be81f61c844)\n\nWhile the short-term goal is to learn these tools, the greater goal is to understand and flesh out what the full development and deployment cycle can look like for orchestrating a data platform and deploying custom pipelines. There is a great process in the transformation layer using dbt where we have local development, testing, versioning/branching, CICD, code-review, separation of dev and prod, project structure/cohesion etc., but how can we apply that to the entire data platform and espeacially, the 10-20% of ingestion jobs that cannot be done in a managed tool like Airbyte and/or is best done using a custom solution?\n\n# Current Status\n\u003cimg width=\"1317\" alt=\"Screenshot 2025-01-05 at 11 51 00 AM\" src=\"https://github.com/user-attachments/assets/a40c230d-1634-46ca-9210-d7847f487323\" /\u003e\n\n### Dagster\n- Orchestrated ingest, transformation, and downstream dependecies (ML/Analytics) with Dagster - https://github.com/jairus-m/dagster-dlt/pull/2, https://github.com/jairus-m/dagster-dlt/pull/6\n  - Developed in dev environment and materaizlied in `dagster dev` server\n  - Configured resources / credentials in a root `.env` file\n  - Current Dagster folder structure (dependencies managed by UV) - https://github.com/jairus-m/dagster-dlt/pull/15\n    - One code location: `dagster_proj/` \n      - Assets: `dagster_proj/assets/`\n      - Resources: `dagster_proj/resources/__init__.py`\n      - Jobs: `dagster_proj/jobs/__init__.py`\n      - Schedules: `dagster_proj/schedules/__init__.py`\n      - Utils: `dagster_proj/utils/__init__.py`\n      - Definitions: `dagster_proj/__init__.py`\n    - The structure is experimental and based on the DagsterU courses\n### dltHub\n- Built a dltHub EL pipeline via the RESTAPIConfig class in `dagster_proj/assets/dlt/activities.py`\n  - Declaratively extracts my raw activity/stats data from Strava's REST API and loads it into DuckDB/Snowflake\n    - Added mulitple Strava endpoints - https://github.com/jairus-m/dagster-dlt/pull/18\n  - Created a custom configurable resource for Strava API - https://github.com/jairus-m/dagster-dlt/pull/5, https://github.com/jairus-m/dagster-dlt/pull/11\n### dbt-core\n- Built a dbt-core project to transform the activities data in `analytics_dbt/models`\n### Sklearn ML Pipeline\n- Created an Sklearn ML pipeline to predict energy expenditure for a given cycling activity\n  - WIP but the general flow of preprocessing, building the ML model, training, testing/evaluation, and prediction can be found in `dagster_proj/assets/ml_analytics/energy_prediction.py`\n  - This a downstream dependency of a dbt asset materialized in duckdb\n### Analytics\n- Created a Plotly analytics dashboard + an ML results related visulization - https://github.com/jairus-m/dagster-dlt/pull/14\n  - In `dagster_proj/assets/ml_analytics/weekly_totals.py` \n\n## Deployment Status\n- Deployed this project to Dagster+ \n  - CICD w/ branching deployments for every PR\n- Seperated execution environments - https://github.com/jairus-m/dagster-dlt/pull/13\n  - dev (DuckDB)\n  - branch (Snowflake)\n  - prod (Snowflake)\n- Configured pre-commits / CI checks and added unit tests - https://github.com/jairus-m/dagster-dlt/pull/16\n  - Added `ruff` Python linter - https://github.com/jairus-m/dagster-dlt/pull/8\n  - Astral `uv` for Python dependency management - https://github.com/jairus-m/dagster-dlt/pull/1\n\n## TODO:\n- Beef up the ML pipeline with `dagster-mlflow` for experiment tracking, model versioning, better model observability, etc\n- Utilize Snowflake Cloning/dbt Slim CI for CI\n- Implement partitions/backfilling with dlt/Dagster\n\n# Getting Started:\n\nFor local development only:\n\n1. Clone this repo locally\n2. Create a `.env` file at the root of the directory:\n  ```\n  # these are the config values for local dev and will change in branch/prod deployment\n  DBT_TARGET=dev\n  DAGSTER_ENVIRONMENT=dev\n  DUCKDB_DATABASE=data/dev/strava.duckdb\n\n  #strava\n  CLIENT_ID= \n  CLIENT_SECRET=\n  REFRESH_TOKEN=\n  ```\n3. Download `uv` and run `uv sync`\n4. Build the Python package in developer mode via `uv pip install -e \".[dev]\"`\n5. Run the dagster daemon locally via `dagster dev`\n6. Materialize the pipeline!\n\n__Additional Notes:__ \n- The `refresh_token` in the Strava UI produces an `access_token` that is limited in scope. Please follow these [Strava Dev Docs](https://developers.strava.com/docs/getting-started/#oauth) to generate the proper `refresh_token` which will then produce an `access_token` with the proper scopes.\n- If you want to run the dbt project locally, outside of dagster, you need to add a `DBT_PROFILES_DIR` environment variable to the .env file and export it\n  - For example, my local env var is: `DBT_PROFILES_DIR=/Users/jairusmartinez/Desktop/dlt-strava/analytics_dbt`\n  - Yours will be: `DBT_PROFILES_DIR=/PATH_TO_YOUR_CLONED_REPO_DIR/analytics_dbt`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjairus-m%2Fdagster-sdlc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjairus-m%2Fdagster-sdlc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjairus-m%2Fdagster-sdlc/lists"}