{"id":48873567,"url":"https://github.com/continuous-dems/dem-recipes","last_synced_at":"2026-04-15T23:09:11.516Z","repository":{"id":340634187,"uuid":"1166890527","full_name":"continuous-dems/dem-recipes","owner":"continuous-dems","description":"Recipes for DEM generation","archived":false,"fork":false,"pushed_at":"2026-03-31T16:58:57.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-31T18:41:01.365Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/continuous-dems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-25T18:09:35.000Z","updated_at":"2026-03-31T16:54:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/continuous-dems/dem-recipes","commit_stats":null,"previous_names":["continuous-dems/dem-recipes"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/continuous-dems/dem-recipes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/continuous-dems%2Fdem-recipes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/continuous-dems%2Fdem-recipes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/continuous-dems%2Fdem-recipes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/continuous-dems%2Fdem-recipes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/continuous-dems","download_url":"https://codeload.github.com/continuous-dems/dem-recipes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/continuous-dems%2Fdem-recipes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31863520,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-15T23:09:10.941Z","updated_at":"2026-04-15T23:09:11.498Z","avatar_url":"https://github.com/continuous-dems.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🗺️ DEM Recipes (The Globato/Fetchez Cookbook)\nWelcome to the **DEM Recipes** repository. This is the central archive for our Continuous-DEM project configurations.\n\nInstead of storing massive, static DEM files, we store the **recipes** used to create them. By treating our Digital Elevation Models as \"Infrastructure as Code,\" we ensure that any team or community member can reproduce, update, or audit a DEM surface from scratch using the `fetchez` and `globato` engines.\n\n# 🚀 How to Launch a Recipe\nRecipes are written in standard YAML. To execute a recipe and build the DEM, simply pass the YAML file to the `fetchez` CLI:\n\n```bash\nfetchez recipes/socal_template.yaml\n```\n\nAlternatively, you can load and launch recipes directly within a Python driver script using the `fetchez.pipeline` API:\n\n```python\nfrom fetchez.recipe import Recipe\n\n# Load the engine with your recipe and launch\nRecipe.from_file(\"recipes/socal_template.yaml\").run()\n```\n\n# 📖 Anatomy of a Recipe\nA `fetchez` YAML configuration is broken down into specific operational blocks. Here is the generalized structure:\n\n* **1. Project \u0026 Execution Metadata**\nDefines what you are building and how much compute power to use.\n\n```yaml\nproject:\n  name: \"Project_Name\"\n  description: \"Description of the DEM.\"\n\nexecution:\n  threads: 4 # Number of parallel download/processing streams\n\nregion: [-120.0, -119.0, 33.0, 34.0] # The bounding box: [West, East, South, North]\n```\n\n* **2. Modules (The Data Sources)**\nThe `modules` block lists the data sources `fetchez` will query and ingest. Modules are evaluated in order.\n\nCrucially, you are not limited to remote APIs! You can seamlessly inject your own local or proprietary data into the pipeline using the `local_fs` module.\n\n```yaml\nmodules:\n  # Remote module with arguments\n  - module: nos_hydro\n    args:\n      datatype: \"xyz\" # Only get the legacy NOS Hydro surveys\n      where: \"date \u003e= 2000\" # Arguments specific to the NOS module\n    hooks:\n      # These hooks ONLY apply to NOS Hydro data\n      - name: stream_data\n\n  # Single Files\n  - module: local_fs\n    args:\n      path: \"../local_surveys/new_dredge_project.xyz\"\n      data_type: \"xyz\"\n    hooks:\n      - name: stream_data\n\n  # Directory scan\n  - module: local_fs\n    args:\n      path: \"../local_surveys/my_cleaned_multibeam/\"\n      ext: \".xyz\"\n      data_type: \"multibeam\"\n      gen_inf: True\n    hooks:\n      - name: stream_data\n\n  # Simple remote module\n  - module: tnm\n    args:\n      formats: \"GeoTIFF\"\n    # Notice this module has no specific hooks; it will just download the files.\n```\n\n* **3. Global Hooks (The Assembly Line)**\nThe `global_hooks` block defines the processing pipeline. While module hooks only touch specific data, **Global Hooks process the combined pool of data from all modules.**\n\n```yaml\nglobal_hooks:\n  - name: set_weight\n    args:\n      rules:\n        nos_hydro: 10.0\n        tnm: 2.0\n\n  - name: multi_stack\n    args:\n      res: \"1s\"\n      output: \"final_stack.tif\"\n```\n\n# 🪝 Understanding Hooks and the Lifecycle\nHooks are the specialized tools that process data. It is critical to understand when they run. `fetchez` processes hooks in three distinct stages:\n\n* **1. MANIFEST Stage:** (*pre*) Runs before downloads begin.\n\n*Use case*: Filtering the list of URLs, assigning stack weights (set_weight), or generating boundary masks (osm_landmask) before processing starts.\n\n* **2. FILE Stage:** Runs during the download loop on each individual file.\n\n*Use case*: Unzipping archives, converting formats, or streaming point clouds directly into a grid accumulator (multi_stack).\n\n* **3. COLLECTION Stage:** (*post*) Runs after all files have been downloaded and streamed.\n\n*Use case*: Spatial interpolation (sm_cudem), blending seams (sm_blend), and final output cropping.\n\n**Global vs. Module Hooks**\n* **Module Hooks (modules.hooks):** Only execute on the files fetched by that specific module.\n\n* **Global Hooks (global_hooks):** Execute on the entire, aggregated dataset from all modules simultaneously.\n\n# 💡 Pro-Tips for Recipe Writers\n* **1. Keep it DRY with YAML Anchors**\nIf multiple modules require the exact same set of hooks (e.g., streaming and cropping), do not copy and paste. Define an anchor (\u0026) and alias it (*):\n\n```yaml\n_definitions:\n  standard_stream: \u0026standard_stream\n    - name: stream_data\n    - name: spatial_crop\n      args: { buffer: 0.05 }\n\nmodules:\n  - module: dav\n    hooks: *standard_stream\n  - module: csb\n    hooks: *standard_stream\n```\n\n* **2. Path Resolution is Automatic**\nWhen you define a file output in a hook (e.g., output: \"stack.tif\"), `fetchez` automatically saves it relative to where the YAML file is located. You do not need to hardcode absolute paths!\n\n* **3. Inspect Available Tools**\nNot sure what a hook does or what arguments it takes? Ask the CLI:\n\n* List all hooks: `fetchez --list-hooks`\n\n* Get specific hook documentation: `fetchez --hook-info ms_cudem`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontinuous-dems%2Fdem-recipes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontinuous-dems%2Fdem-recipes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontinuous-dems%2Fdem-recipes/lists"}