{"id":37320020,"url":"https://github.com/csiro/cmextremes-schism_tiling","last_synced_at":"2026-01-16T03:17:00.187Z","repository":{"id":301499068,"uuid":"1007411678","full_name":"csiro/cmextremes-schism_tiling","owner":"csiro","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-27T06:23:39.000Z","size":35,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-10T06:42:12.156Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/csiro.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-24T00:45:08.000Z","updated_at":"2025-06-27T04:44:19.000Z","dependencies_parsed_at":"2025-06-27T07:34:45.828Z","dependency_job_id":"3a599c7c-0150-4c52-ba27-ccb1b0aa85f4","html_url":"https://github.com/csiro/cmextremes-schism_tiling","commit_stats":null,"previous_names":["csiro/cmextremes-schism_tiling"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/csiro/cmextremes-schism_tiling","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csiro%2Fcmextremes-schism_tiling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csiro%2Fcmextremes-schism_tiling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csiro%2Fcmextremes-schism_tiling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csiro%2Fcmextremes-schism_tiling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/csiro","download_url":"https://codeload.github.com/csiro/cmextremes-schism_tiling/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/csiro%2Fcmextremes-schism_tiling/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28477087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T03:13:13.607Z","status":"ssl_error","status_checked_at":"2026-01-16T03:11:47.863Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-16T03:16:59.534Z","updated_at":"2026-01-16T03:17:00.179Z","avatar_url":"https://github.com/csiro.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SCHISM rechunking\n\n## Introduction\n\nScripts written in snakemake to tile and chunk output from the SCHISM model.\n\nIn order to achieve fast time series access, it is required to concatenate the NetCDF files and apply chunking in the time dimension. For practical reasons involving file sizes, it becomes necessary to \"tile\" the data, or split it up spatially.\n\nBecause SCHISM is on an irregular grid indexed by node number, the divisions are not tiles in the traditional sense (lon/lat).\n\nIn the SCHISM output, variables can be a function of triangular grid element, and sometimes a function of node (vertex).\n\nAs it is tricky to cope with both of these in the chunking, it was decided to focus on the node variables. In the grid, there are 155083 nodes. We split these up (almost) equally into 103 groups (or \"tiles\").\n\nThe process of forming a tile is as follows.\n\n1. For each year \u0026 month: extract the data for tile *n* using `ncks`\n\n2. Concatenate over time all data for tile *n* using `ncrcat`\n\n3. Apply chunking in time dimension using `nccopy`\n\nFor the baseline SCHISM scenario there are are (35 years) X (12 months) X (103 tiles) , so that is approximately 43000 extract tasks, plus 103 concatenations and 103 chunking jobs. As this could quickly become unmanageable ( and slow without parallelisation ), [Snakemake workflow tool](https://snakemake.readthedocs.io/) has been deployed for the task.\n\nSnakemake allows the work to be specified in compact form. It will also compute all dependencies and apply the appropriate rule to do any work that needs to be done. It can easily be interrupted and resumed.\n\nJobs can be deployed in SLURM on the cluster and resources allocated to it can be adjusted (i.e. max number of simultaneous running jobs).\n\nSnakemake has its own syntax although it is python-based, and python can be freely mixed with it. In addition, the rules may be carried out using python or bash shell.\n\n## Preparing the environment\n\nAlthough I created my own [miniconda environment](https://docs.conda.io/en/latest/miniconda.html)with snakemake installed in it, snakemake has now been installed in HPC python environments. Therefore, you can run\n\n```\nmodule load python/3.6.1\n```\n\n## Quick run through the Snakefile\n\nLooking at the Snakefiles, it is conceptually easier to work backwards starting from the desired file (tiled and chunked). These are specified under *rule all*.\n\nVia the *output:* specification of *rule chunk*, this creates a need/dependency for concatenated tile files as inputs. The *shell:* directive specifies how to create chunked files from concatenated ones.\n\nThis brings us to *rule concat* which describes how to make concatenated files from individual monthly extracts. As inputs, it expects to see extracts for every tile, year and month.\n\nThis creates a dependency on *rule extract*. Here there expected SCHISM files are specified as inputs. The shell commands to perform the extracts is specified.\n\n## Running on the cluster\n\nIn order to run on the cluster, it should be run on one of the pearcey interactive nodes as a supervisor. This could be run under a vnc session so as not to terminate on logout. Use this command\n\n```bash\nmodule load python/3.6.1\nsnakemake -s Snakefile.baseline  --printshellcmds --cluster 'sbatch -t {params.time} --mem={params.mem}' -j 50\n```\n\nThis will trigger slurm jobs, maximum 50 at a time, that will perform the work.\n\nThe extract jobs are generally short, often 10 minutes or less, so this gets done fairly quickly, but it does take a few days to work through tens of thousands of tasks. If the cluster looks quiet, the 50 maximum could be increased, but you might run into disk bandwidth issues on the OSM.\n\n## Quota issues\n\nThe snakemake job should be run on /scratch1 as a working directory due to space, bandwidth and file count quotas.\n\nSnakemake will put many thousands of files into the *.snakemake* directory, more than your quota under /home.\nIt will also write thousands of slurm job output files. Sometimes you might have to stop the snakemake task (ctrl-C), delete these, then restart. The task remains to automatically clean up these files as the jobs progress.\n\n# License\n\nThis source code is licensed under a modified form of the BSD / MIT open source license. Please see the LICENSE.md file for a copy, and make sure you understand and agree to the terms before using this code.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsiro%2Fcmextremes-schism_tiling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcsiro%2Fcmextremes-schism_tiling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsiro%2Fcmextremes-schism_tiling/lists"}