{"id":20201004,"url":"https://github.com/openclimatefix/dagster-dags","last_synced_at":"2025-10-14T13:34:05.905Z","repository":{"id":192780961,"uuid":"687376023","full_name":"openclimatefix/dagster-dags","owner":"openclimatefix","description":"Dags for running jobs on Leonardo","archived":false,"fork":false,"pushed_at":"2025-07-08T11:08:06.000Z","size":360,"stargazers_count":5,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-08T12:23:13.047Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2023-09-05T08:26:58.000Z","updated_at":"2025-07-08T11:07:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"7876729a-f324-417a-9f2b-34d9a6b810e1","html_url":"https://github.com/openclimatefix/dagster-dags","commit_stats":{"total_commits":67,"total_committers":2,"mean_commits":33.5,"dds":0.3880597014925373,"last_synced_commit":"c068a262057b2f3dc56d1f91edc88e4bd854bd20"},"previous_names":["openclimatefix/dagster-dags"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/openclimatefix/dagster-dags","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdagster-dags","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdagster-dags/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdagster-dags/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdagster-dags/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/dagster-dags/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2Fdagster-dags/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018706,"owners_count":26086609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T04:46:27.872Z","updated_at":"2025-10-14T13:34:05.862Z","avatar_url":"https://github.com/openclimatefix.png","language":"Python","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# Dagster Dags\n\n**Orchestrate data pipelines for ML dataset creation**\n\n[![tags badge](https://img.shields.io/github/v/tag/openclimatefix/dagster-dags?include_prereleases\u0026sort=semver\u0026color=7BCDF3)](https://github.com/openclimatefix/dagster-dags/tags)\n[![contributors badge](https://img.shields.io/github/contributors/openclimatefix/dagster-dags?color=FFFFFF)](https://github.com/openclimatefix/dagster-dags/graphs/contributors)\n[![workflows badge](https://img.shields.io/github/actions/workflow/status/openclimatefix/dagster-dags/branch_ci.yml?branch=main\u0026color=FFD053)](https://github.com/openclimatefix/dagster-dags/actions/workflows/branch_ci.yml)\n[![ease of contribution: easy](https://img.shields.io/badge/ease%20of%20contribution:%20easy-32bd50)](https://github.com/openclimatefix/ocf-meta-repo?tab=readme-ov-file#overview-of-ocfs-nowcasting-repositories)\n\nIn order to train and evaluate an ML model, datasets must be created consistently and reproducibly.\n\nForecasting renewable energy generation depends on large-timescale weather data:\nNumerical Weather Prediction (NWP) data; satellite imagery;\natmospheric quality data. Dagster helps to these datasets organised and up to date.\n\nThis repository contains the Dagster definitions that orchestrate the creation of these datasets.\n\n## Installation\n\nThe repository is packaged as a Docker image that can be used as a Dagster\n[code server](https://docs.dagster.io/concepts/code-locations/workspace-files#running-your-own-grpc-server)\n\n```bash\n$ docker pull ghcr.io/openclimatefix/dagster-dags\n```\n\n## Example Usage\n\n**To add as a code location in an existing Dagster setup:**\n\n```bash\n$ docker run -d \\\n    -p 4266:4266 \\\n    -e DAGSTER_CURRENT_IMAGE=ghcr.io/openclimatefix/dagster-dags \\\n    ghcr.io/openclimatefix/dagster-dags\n```\n\n```yaml\n# $DAGSTER_HOME/workspace.yaml\n\nload_from:\n  - grpc_server:\n      host: localhost\n      port: 4266\n      location_name: \"dagster-dags\" # Name of the module\n```\n\n\u003e [!Note]\n\u003e Setting `DAGSTER_CURRENT_IMAGE` environment variable is necessary to tell Dagster\n\u003e to spawn jobs using the set container image. Since the Containerfile has all the\n\u003e required dependencies for the user code, it makes sense to set it to itself.\n\n**To deploy the entire Dagster multi-container stack:**\n\n```bash\n$ docker compose up -f infrastructure/docker-compose.yml\n```\n\n\u003e [!Note]\n\u003e This will start a full Dagster setup with a web UI, a Postgres database,\n\u003e and a QueuedRunCoordinator. This might be overkill for some setups.\n\n## Documentation\n\nThe repository is split into folders covering the basic concepts of Dagster:\n\n- Top-level [Definitions](https://docs.dagster.io/concepts/code-locations) defining the code location are defined in `src/dagster_dags/definitions.py`\n- [Assets](https://docs.dagster.io/concepts/assets/software-defined-assets) are in `src/dagster_dags/assets`\n- [Resources](https://docs.dagster.io/concepts/resources#resources) are in `src/dagster_dags/resources`\n\nThey are then subdivided by module into data-type-specific folders.\n\n## Development\n\nTo run a development Dagster server, install the required dependencies in a virtual environment,\nactivate it, and run the server:\n\n```bash\n$ cd scr \u0026\u0026 dagster dev --module-name=dagster_dags\n```\n\nThis should spawn a UI at `localhost:3000` where you can interact with the Dagster webserver.\n\n### Linting and static type checking\n\nThis project uses [MyPy](https://mypy.readthedocs.io/en/stable/) for static type checking\nand [Ruff](https://docs.astral.sh/ruff/) for linting.\nInstalling the development dependencies makes them available in your virtual environment.\n\nUse them via:\n\n```bash\n$ python -m mypy .\n$ python -m ruff check .\n```\n\nBe sure to do this periodically while developing to catch any errors early\nand prevent headaches with the CI pipeline. It may seem like a hassle at first,\nbut it prevents accidental creation of a whole suite of bugs.\n\n### Running the test suite\n\nRun the unittests with:\n\n```bash\n$ python -m unittest discover -s tests\n```\n\n## Further Reading\n\nOn running your own GRPC code server as a code location in Dagster:\n- Dagster guide on [running a GRPC server](https://docs.dagster.io/concepts/code-locations/workspace-files#running-your-own-grpc-server).\n- Creating a GRPC code server container as part of a [multi-container Dagster stack](https://docs.dagster.io/deployment/guides/docker#multi-container-docker-deployment).\n\n\n---\n\n## Contributing and community\n\n[![issues badge](https://img.shields.io/github/issues/openclimatefix/dagster-dags?color=FFAC5F)](https://github.com/openclimatefixdagster-dags/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)\n\n- PR's are welcome! See the [Organisation Profile](https://github.com/openclimatefix) for details on contributing\n- Find out about our other projects in the [OCF Meta Repo](https://github.com/openclimatefix/ocf-meta-repo)\n- Check out the [OCF blog](https://openclimatefix.org/blog) for updates\n- Follow OCF on [LinkedIn](https://uk.linkedin.com/company/open-climate-fix)\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n---\n\n*Part of the [Open Climate Fix](https://github.com/orgs/openclimatefix/people) community.*\n\n[![OCF Logo](https://cdn.prod.website-files.com/62d92550f6774db58d441cca/6324a2038936ecda71599a8b_OCF_Logo_black_trans.png)](https://openclimatefix.org)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fdagster-dags","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fdagster-dags","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fdagster-dags/lists"}