{"id":23673821,"url":"https://github.com/carpentries-incubator/hpc-intro","last_synced_at":"2025-12-23T03:30:11.507Z","repository":{"id":37988134,"uuid":"86095619","full_name":"carpentries-incubator/hpc-intro","owner":"carpentries-incubator","description":"An Introduction to High Performance Computing","archived":false,"fork":false,"pushed_at":"2025-01-13T19:29:09.000Z","size":31473,"stargazers_count":146,"open_issues_count":69,"forks_count":142,"subscribers_count":24,"default_branch":"gh-pages","last_synced_at":"2025-01-13T20:26:38.643Z","etag":null,"topics":["alpha","carpentries-incubator","carpentry-lesson","english","hpc-carpentry","lesson"],"latest_commit_sha":null,"homepage":"https://carpentries-incubator.github.io/hpc-intro/","language":"Vim Snippet","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/carpentries-incubator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION","codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["carpentries"],"custom":["https://give.communityin.org/TheCarpentries"]}},"created_at":"2017-03-24T17:47:33.000Z","updated_at":"2025-01-13T19:29:13.000Z","dependencies_parsed_at":"2024-08-07T17:16:43.589Z","dependency_job_id":"6bf3ce8a-b360-4152-903c-13f7ea58a435","html_url":"https://github.com/carpentries-incubator/hpc-intro","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-intro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-intro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-intro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/carpentries-incubator%2Fhpc-intro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/carpentries-incubator","download_url":"https://codeload.github.com/carpentries-incubator/hpc-intro/tar.gz/refs/heads/gh-pages","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239705959,"owners_count":19683753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alpha","carpentries-incubator","carpentry-lesson","english","hpc-carpentry","lesson"],"created_at":"2024-12-29T12:56:38.079Z","updated_at":"2025-12-23T03:30:11.419Z","avatar_url":"https://github.com/carpentries-incubator.png","language":"Vim Snippet","funding_links":["https://github.com/sponsors/carpentries","https://give.communityin.org/TheCarpentries"],"categories":[],"sub_categories":[],"readme":"# Intro to HPC\n\nThis lesson teaches the basics of interacting with high-performance computing\n(HPC) clusters through the command line\n\n[![Build Status][badge-img]][badge-lnk]\n\n## Using this material\n\nNOTE: This is _not_ Carpentries boilerplate! Please read carefully.\n\n1. Follow the instructions found in The Carpentries' [example lesson][ex-lesson]\n   to create a repository for your lesson. Install Ruby, Make, and Jekyll\n   following the instructions [here][setup].\n\n1. For easier portability, we use snippets of text and code to capture inputs\n   and outputs that are host- or site-specific and cannot be scripted. These\n   are stored in a library\n   [_includes/snippets_library](_includes/snippets_library), with\n   subdirectories matching the pattern `InstitutionName_ClusterName_scheduler`.\n   If your cluster is not already present, please copy (`cp -r`) the _closest\n   match_ as a new folder under `snippets_library`.\n\n   * We have placed snippets in files with the `.snip` extension, to make\n     tracking easier. These files contain Markdown-formatted text, and will\n     render to HTML when the lesson is built.\n   * Code snippets are placed in subdirectories that are named according to the\n     episode they appear in. For example, if the snippet is for episode 12, then\n     it will be in a subdirectory called `12`.\n   * In the episodes source, snippets are included using [Liquid][liquid]\n     scripting `include` statements. For example, the first snippet in episode\n     12 is included using `{% include /snippets/12/info.snip %}`.\n\n1. Edit `_config_options.yml` in your snippets folder. These options set such\n   things as the address of the host to log in to, definitions of the command\n   prompt, and scheduler names. You can also change the order of the episodes,\n   or omit episodes, by editing the configuration block under `episode_names`\n   in this file.\n\n1. Set the environment variable `HPC_JEKYLL_CONFIG` to the relative path of the\n   configuration file in your snippets folder:\n\n   ```bash\n   export HPC_JEKYLL_CONFIG=_includes/snippets_library/.../_config_options.yml\n   ```\n\n1. Preview the lesson locally, by running `make serve`. You can then view the\n   website in your browser, following the links in the output (usually,\n   \u003chttps://localhost:4000\u003e). Pages will be automatically regenerated every\n   time you write to them.\n\n1. If there are discrepancies in the output, edit the snippet file containing\n   it, or create a new one and customize.\n\n1. Add your snippet directory name to the GitHub Actions configuration file,\n   [.github/workflows/test_and_build.yml](.github/workflows/test_and_build.yml).\n\n1. Check out a new branch(`git checkout -b new_branch_name`), commit your\n   changes, and push to your fork of the repository. If you're comfortable\n   sharing, please file a Pull Request against our [upstream repo][upstream].\n   We would love to have your site config for the Library.\n\n1. To maintain compatibility, please _do not_ merge your new branch into your\n   fork's `gh-pages` branch. Instead, wait until your pull request has been\n   merged upstream, then pull down the upstream version. Otherwise, your\n   repository will _diverge_ from ours, and pull requests you make in the\n   future will probably not be accepted.\n\n## Deploying a Customized Lesson\n\nThe steps above will help you port the default HPC Intro lesson to your\nspecific cluster, but the changes will only be visible on your local machine.\nTo build a website for a specific workshop or instance of the lesson, you'll\nwant to make a stand-alone copy.\n\n### Template Your Customized Repository\n\nThis will let you create an exact duplicate of your fork. Without this, GitHub\nwon't let you create a second fork of a repository on the same account.\n\n1. On GitHub, go to your repository's __Settings__.\n1. Under the repository name, check the \"Template Repository\" box.\n1. Go to the __Code__ tab.\n1. Click the new button to __Use This Template__.\n1. Fill in a name, like `yyyy-mm-dd-hpc-intro`.\n1. Check the __Include all branches__ box.\n1. Go!\n\n### Merge Your Customized Branch\n\nIf your snippets are already included in the snippet library, skip this step.\n\n1. On GitHub, find the drop-down menu of branches. It should be all the way to\n   the left of the \"Use This Template\" button.\n1. From the list, select the branch containing your site customization.\n1. There should be a bar above the list of repository contents with the branch\n   name, stating \"This branch is _x_ commits ahead, _y_ commits behind\n   `gh-pages`\" or similar. To the right of that, click the button to __Create\n   Pull Request__.\n1. Make sure that the source and destination repositories at the top of the new\n   PR are _both_ your current duplicate of hpc-intro, not the upstream.\n1. Create the pull request, then click the __Merge__ button. You can delete the\n   customization branch when it's done.\n\n### Modify `_config.yml`\n\nGitHub builds sites using the top-level [`_config.yml`](_config.yml), only, but\nyou want the values set in the snippet library.\n\n1. Open a copy of your\n  `_includes/snippet_library/Institution_Cluster_scheduler/_config_options.yml`\n1. On GitHub, open the top-level `_config.yml` for editing.\n1. Copy your `_config_options.yml`, overwriting the values under the __SITE\n   specific configuration__ section of the top-level `_config.yml`. Leave the\n   rest as-is.\n1. Commit the change.\n1. Back on the __Code__ tab, there should be a timer icon, a green check, or a\n   red X next to the latest commit hash. If it's a timer, the site is building;\n   give it time.\n1. If the symbol is a red x, something went wrong. Click it to open the build\n   log, and attempt to correct the error. Follow GitHub's troubleshooting\n   guide, and double-check the values in `_config.yml` ar ecorrect and\n   complete.\n1. Once you see a green check, your website will be available for viewing at\n   `https://your-github-account.github.io/name-of-the-repository`.\n\n## Lesson Outlines\n\nThe following list of items is meant as a guide on what content should go where\nin this repo. This should work as a guide where you can contribute. If a bullet\npoint is prefixed by a file name, this is the lesson where the listed content\nshould go into. This document is meant as a concept map converted into a flow\nof learning goals and questions. Note, again, that it is possible, when\nbuilding your actual lesson, to re-order these files, or omit one or more of\nthem.\n\n[User profiles](_extras/learner-profiles.md) of people approaching\nhigh-performance computing from an academic and/or commercial background are\nprovided to help guide planning and decision-making.\n\n1. [Why use a cluster?](_episodes/11-hpc-intro.md) (20 minutes)\n\n   * Brief, concentrate on the concepts not details like interconnect type, etc.\n   * Be able to describe what a compute cluster (HPC/HTC system) is\n   * Explain how a cluster differs from a laptop, desktop, cloud, or \"server\"\n   * Identify how an compute cluster could benefit you.\n   * Jargon busting\n\n1. [Working on a remote HPC system](_episodes/12-cluster.md) (35 minutes)\n\n   * Understand the purpose of using a terminal program and SSH\n   * Learn the basics of working on a remote system\n   * Know the differences of between login and compute nodes\n   * Objectives: Connect to a cluster using ssh; Transfer files to and from the\n     cluster; Run the hostname command on a compute node of the cluster.\n   * Potential tools: `ssh`, `ls`, `hostname`, `logout`, `nproc`, `free`,\n     `scp`, `man`, `wget`\n\n1. [Working with the scheduler](_episodes/13-scheduler.md) (1 hour 15 minutes)\n\n   * Know how to submit a program and batch script to the cluster (interactive \u0026\n     batch)\n   * Use the batch system command line tools to monitor the execution of your\n     job.\n   * Inspect the output and error files of your jobs.\n   * Potential tools: shell script, `sbatch`, `squeue -u`, `watch`, `-N`, `-n`,\n     `-c`, `--mem`, `--time`, `scancel`, `srun`, `--x11 --pty`,\n   * Extras: `--mail-user`, `--mail-type`,\n   * Remove? `watch`\n   * Later lessons? `-N` `-n` `-c`\n\n1. [Accessing software via Modules](_episodes/14-modules.md) (45 minutes)\n\n   * Understand the runtime environment at login\n   * Learn how software modules can modify your environment\n   * Learn how modules prevent problems and promote reproducibility\n   * Objectives: how to load and use a software package.\n   * Tools: `module avail`, `module load`, `which`, `echo $PATH`, `module\n     list`, `module unload`, `module purge`, `.bashrc`, `.bash_profile`, `git\n     clone`, `make`\n   * Remove: `make`, `git clone`,\n   * Extras: `.bashrc`, `.bash_profile`\n\n1. [Transferring files with remote computers](\n   _episodes/15-transferring-files.md) (30 minutes)\n\n   * Understand the (cognitive) limitations that remote systems don't\n     necessarily have local Finder/Explorer windows\n   * Be mindful of network and speed restrictions (e.g. cannot push from\n     cluster; many files vs one archive)\n   * Know what tools can be used for file transfers, and transfer modes (binary\n     vs text)\n   * Objective: Be able to transfer files to and from a computing cluster.\n   * Tools: `wget`, `scp`, `rsync` (callout), `mkdir`, FileZilla,\n   * Remove: `dos2unix`, `unix2dos`,\n   * Bonus: `gzip`, `tar`, `dos2unix`, `cat`, `unix2dos`, `sftp`, `pwd`,\n     `lpwd`, `put`, `get`\n\n1. [Running a parallel job](_episodes/16-parallel.md) (1 hour)\n\n   * Introduce message passing and MPI as the fundamental engine of parallel\n     software\n   * Walk through a simple Python program for estimation of π\n   * Use [mpi4py][mpi4py] to parallelize the program\n   * Write job submission scripts \u0026 run the job on a cluster node\n   * Tools: `nano`, `sbatch`, `squeue`\n\n1. [Using resources effectively](_episodes/17-resources.md) (40 minutes)\n\n   * Understand how to look up job statistics\n   * Learn how to use job statistics to understand the health of your jobs\n   * Learn some very basic techniques to monitor / profile code execution.\n   * Understand job size and resource request implications.\n   * Tools: `fastqc`, `sacct`, `ssh`, `top`, `free`, `ps`, `kill`, `killall`\n     (note that some of these may not be appropriate on shared systems)\n\n1. [Using shared resources responsibly](_episodes/18-responsibility.md) (20\n   minutes)\n\n   * Discuss the ways some activities can affect everyone else on the system\n\n### Nascent lesson ideas\n\n1. Playing friendly in the cluster (psteinb: the following is very tricky as it\n   is site dependent, I personally would like to see it in [_extras][extras])\n\n   * Understanding resource utilisation\n   * Profiling code - time, size, etc.\n   * Getting system stats\n   * Consequences of going over\n\n1. Filesystems and Storage: objectives likely include items from @psteinb's\n   [Shared Filesystem lesson][hpcday-fs]:\n\n   * Understand the difference between a local and shared / network filesystem\n   * Learn about high performance / scratch filesystems\n   * Raise attention that misuse (intentional or not) of a common file system\n     negatively affects all users very quickly.\n   * Possible tools: `echo $TEMP`, `ls -al /tmp`, `df`, `quota`\n\n1. Advanced Job Scripting and Submission:\n\n   * Checking status of jobs (`squeue`, `bjobs` etc.), explain different job\n     states and relate to scheduler basics\n   * Cancelling/deleting a job (`scancel`, `bkill` etc.)\n   * Passing options to the scheduler (log files)\n   * Callout: Changing a job's name\n   * Optional Callout: Send an email once the job completes (not all sites\n     support sending emails)\n   * for a starting point, see [this][hpcday-sched] for reference\n\n1. Filesystem Zoo:\n\n   * execute a job that collects node information and stores the output to\n     `/tmp`\n   * ask participants where the output went and why they can't see it\n   * execute a job that collects node information and stores the output to\n     `/shared` or however your shared file system is called\n   * for a starting point, see [this][hpcday-fs]\n\n\u003c!-- links --\u003e\n[badge-img]: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fcarpentries-incubator%2Fhpc-intro%2Fbadge%3Fref%3Dgh-pages\u0026style=flat\n[badge-lnk]: https://actions-badge.atrox.dev/carpentries-incubator/hpc-intro/goto?ref=gh-pages\n[ex-lesson]: https://github.com/carpentries/lesson-example\n[extras]: https://github.com/carpentries-incubator/hpc-intro/tree/gh-pages/_extras\n[hpcday-fs]: https://github.com/psteinb/hpc-in-a-day/blob/gh-pages/_episodes/01-04-shared-filesystem.md\n[hpcday-sched]: https://psteinb.github.io/hpc-in-a-day/02-02-advanced-job-scheduling\n[liquid]: https://shopify.github.io/liquid/\n[mpi4py]: https://mpi4py.readthedocs.io\n[setup]: http://carpentries.github.io/lesson-example/setup.html\n[upstream]: https://github.com/carpentries-incubator/hpc-intro\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarpentries-incubator%2Fhpc-intro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcarpentries-incubator%2Fhpc-intro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarpentries-incubator%2Fhpc-intro/lists"}