{"id":26573213,"url":"https://github.com/linsalrob/pawsey","last_synced_at":"2026-07-02T09:32:43.346Z","repository":{"id":232497545,"uuid":"784498200","full_name":"linsalrob/pawsey","owner":"linsalrob","description":"Code for running lots of different things on pawsey. This is a bit of a generic bucket and some of the code will be duplicated elsewhere in different projects","archived":false,"fork":false,"pushed_at":"2026-06-29T07:53:17.000Z","size":227,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-29T09:25:29.225Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linsalrob.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-10T00:55:51.000Z","updated_at":"2026-06-29T07:53:21.000Z","dependencies_parsed_at":"2024-12-15T02:18:07.820Z","dependency_job_id":"572002d4-baf7-4a9f-bc61-44ed5d1ee92d","html_url":"https://github.com/linsalrob/pawsey","commit_stats":null,"previous_names":["linsalrob/pawsey"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/linsalrob/pawsey","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Fpawsey","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Fpawsey/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Fpawsey/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Fpawsey/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linsalrob","download_url":"https://codeload.github.com/linsalrob/pawsey/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Fpawsey/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35041996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-23T00:39:47.285Z","updated_at":"2026-07-02T09:32:43.305Z","avatar_url":"https://github.com/linsalrob.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cluster information\n\n[Pawsey partitions](https://pawsey.atlassian.net/wiki/spaces/US/pages/51929058/Running+Jobs+on+Setonix)\n\nOn the wk queue there is 1840M per core, so either multiple the # cores by 1840M or divide the memory by 1840M to get the number of cores you should request.\n\n4G = 3 cores\n8G = 5 cores\n16G = 9 cores\n32G = 18 cores\n\n### Actual memory/cores requested:\n\n15G -\u003e 18 cores (Job 32732860)\n\n# User accounts, adding people, allocations, and acacia keys\n\nCheck https://portal.pawsey.org.au\n\nKeys are stored in rclone.conf already (~/.config/rclone/rclone.conf)\n\n# Conda\n\nInstall conda in\n\n```\n/software/projects/$PAWSEY_PROJECT/$USER/miniconda3\n```\n\nand then set your package directory to be on `/scratch/`\n\n```\nconda config --add pkgs_dirs /scratch/$PAWSEY_PROJECT/$USER/miniconda3/pkg_dir\n```\n\n## Create a temporary conda installation. \n\nThis will get deleted after (at most) 30 days, but is good if you want to install \nsomething to test. \n\n\n```\nTMP=$(for i in {1..12}; do printf \"%x\" $((RANDOM % 16)); done)\nmamba create -y --prefix=/scratch/pawsey1018/edwa0468/software/miniconda3/$TMP  python=3.12\nmamba activate /scratch/pawsey1018/edwa0468/software/miniconda3/$TMP\n```\n\nAlternatively, make a TMP miniconda:\n```\nmamba create -y --prefix=/scratch/pawsey1018/edwa0468/software/miniconda3/megahit  'python\u003e=3.12'\nmamba activate /scratch/pawsey1018/edwa0468/software/miniconda3/megahit\n```\n\n# Partitions\n\n\nMore information [Pawsey partitions](https://pawsey.atlassian.net/wiki/spaces/US/pages/51929058/Running+Jobs+on+Setonix)\n\nwork partition (316 nodes, 128 cores per node)\n- Default partition, used for regular compute jobs\n- 230 GB available memory per node, which is 1.8 GB per core\n- max time 24 hours\ndebug partition (8 nodes, 128 cores per node)\n- Intended for short interactive jobs\n- Use for benchmarking, testing, debugging, profiling and development\ncopy partition (8 nodes, 32 cores per node)\n- Use for data transfers to/from Setonix\n- 89 GB available memory per node, which is approximately 2.78 GB per core \nhighmem partition (8 nodes, 128 cores per node)\n- Use for large memory workflows that require more than 230 GB RAM per node\n- Supports up to 980 GB of available memory per node, which is 7.65 GB per core\nlong partition (8 nodes, 128 cores per node)\n- Use for jobs that require wall times between 24 and 96 hours\n\n# Data Moving\n\nUse  data-mover.pawsey.org.au  to put data onto/off of pawsey\n\n# acacia\n\nBucket names must be unique.\nBucket names must all be lowercase\nNames probably have to be unique across all S3, so databases doesn't work!\n\ne.g. \n  VALID: rclone mkdir pawsey1018:fame \nINVALID: rclone mkdir pawsey1018:FAME\n\n# Account Balance\n\nYou can check the status of your accounts with\n\npawseyAccountBalance -mygroups\n\n(note the -storage option does not appear to work)\n\n# Disk quotas\n\nYou can check the disk quotas for either your PROJECT or USER using these two command:\n\n```\nlfs quota -g $PAWSEY_PROJECT -h /software\nlfs quota -u $USER -h /software\n```\n\nIn particular, pay attention to the number of files, usually limited to 100,000\n\n\n# Conda clean\n\nUsing \n\n```\nconda clean -af\n```\n\nwil clean up some of the conda cache files.\n\n\n# squeue_format\n\nBy default, things like array jobs don't show up. Set the $SQUEUE_FORMAT env variable to set the columns to show\n\n\n# create a mamba (conda) environment in scratch. Note you probably have to delete this since you've named it!\nmamba env create --file atavide_lite.yaml --prefix /scratch/pawsey1018/edwa0468/software/miniconda3/atavide_lite\n\nHere is the TMP version\nTMP=$(for i in {1..12}; do printf \"%x\" $((RANDOM % 16)); done)\nmamba create -y --prefix=/scratch/pawsey1018/edwa0468/software/miniconda3/$TMP  python=3.12\nmamba activate /scratch/pawsey1018/edwa0468/software/miniconda3/$TMP\n\n# rclone\n\nMake sure that you add `--local-no-set-modtime` to any rclone command.\n\nOrdinarily, `rclone` syncs the local timestamp so that it is the same as the remote file. However, if you download a file later, that time stamp will be \u003e30 days old and the file will be deleted. \ne.g.\n\n```\nrclone copy --local-no-set-modtime A18:fame/CF/JessCarlsonJones/WorldWideCF/fasta/fasta_subsampled fasta_subsampled\n```\n\n# installing node and nvm\n\nI currently have these installed on `/scratch` with a symlink from `$HOME/.nvm` to the directory on `/scratch`, so they sometimes disappear!\n\nDelete and redownload nvm:\n\n```\ncd ~/.nvm\nrm -rf alias versions test\ncurl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash\nsource ~/.bashrc\nnvm install node\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Fpawsey","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinsalrob%2Fpawsey","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Fpawsey/lists"}