{"id":19209160,"url":"https://github.com/hrolive/large-language-models-on-supercomputers","last_synced_at":"2026-05-18T10:11:34.989Z","repository":{"id":261701089,"uuid":"863420818","full_name":"HROlive/Large-Language-Models-on-Supercomputers","owner":"HROlive","description":"Comprehensive exploration of LLMs, including cutting-edge techniques and tools such as parameter-efficient fine-tuning (PEFT), quantization, zero redundancy optimizers (ZeRO), fully sharded data parallelism (FSDP), DeepSpeed, and Huggingface accelerate.","archived":false,"fork":false,"pushed_at":"2024-11-07T23:41:58.000Z","size":11521,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-14T10:03:41.575Z","etag":null,"topics":["deepspeed","evaluation-metrics","fsdp","high-performance-computing","hpc","huggingface","huggingface-transformers","jupyter","llm","llm-inference","llm-training","monitoring","peft","python","quantization","slurm","tokenization","transformer","unsloth"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HROlive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-26T09:08:32.000Z","updated_at":"2024-11-07T23:42:01.000Z","dependencies_parsed_at":"2024-11-08T00:29:01.181Z","dependency_job_id":"ad8f94bd-9685-411a-b291-fcc8c3ecc32a","html_url":"https://github.com/HROlive/Large-Language-Models-on-Supercomputers","commit_stats":null,"previous_names":["hrolive/large-language-models-on-supercomputers"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HROlive/Large-Language-Models-on-Supercomputers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FLarge-Language-Models-on-Supercomputers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FLarge-Language-Models-on-Supercomputers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FLarge-Language-Models-on-Supercomputers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FLarge-Language-Models-on-Supercomputers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HROlive","download_url":"https://codeload.github.com/HROlive/Large-Language-Models-on-Supercomputers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FLarge-Language-Models-on-Supercomputers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33174091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-18T09:27:30.708Z","status":"ssl_error","status_checked_at":"2026-05-18T09:27:28.300Z","response_time":71,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepspeed","evaluation-metrics","fsdp","high-performance-computing","hpc","huggingface","huggingface-transformers","jupyter","llm","llm-inference","llm-training","monitoring","peft","python","quantization","slurm","tokenization","transformer","unsloth"],"created_at":"2024-11-09T13:29:19.635Z","updated_at":"2026-05-18T10:11:29.980Z","avatar_url":"https://github.com/HROlive.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Course](images/banner.png)\n\n## Table of Contents\n1. [Description](#description)\n2. [Information](#information)\n3. [License](#license)\n\n\u003ca name=\"descripton\"\u003e\u003c/a\u003e\n## Description\n\nDuring the last three years, interest in Large Language Models (LLMs) has experienced a meteoric rise, leaving virtually no domain untouched. The complexity of the models themselves, however, has increased to such an extent, that access to powerful computing resources has become a requirement for anyone wanting to develop products with this novel approach.\n\nIn this intensive two half-day course, participants will dive into the world of LLMs and their development on supercomputers. From covering the fundamentals to hands-on implementations, this course offers a comprehensive exploration of LLMs, including cutting-edge techniques and tools such as parameter-efficient fine-tuning (PEFT), quantization, zero redundancy optimizers (ZeRO), fully sharded data parallelism (FSDP), DeepSpeed, and Huggingface accelerate.\n\nBy the end of this course, participants will have gained the understanding, knowledge and practical skills to develop LLMs effectively on supercomputers, empowering them to tackle challenging natural language processing tasks across various domains.\n\nThis course is jointly organized by the VSC Research Center, TU Wien, and EuroCC Austria.\n\n\u003ca name=\"information\"\u003e\u003c/a\u003e\n## Information\nThe overall goals of this course were the following:\n\u003e - Introduction to LLMs (Overview, Huggingface Ecosystem, Transformer Anatomy, Tokenization \u0026 Embeddings);\n\u003e - Memory-efficient Training (Quantization, PEFT, unsloth, Hands-on example);\n\u003e - Distributed Training (Huggingface Accelerate, ZeRO, FSDP \u0026 DeepSpeed);\n\u003e - Evaluation (Methods \u0026 Metrics. Monitoring, Inference);\n\nMore detailed information and links for the course can be found on the [course website](https://events.vsc.ac.at/event/136/).\n\n\u003ca name=\"license\"\u003e\u003c/a\u003e\n## License\n\nLicense: CC BY-SA 4.0 (Attribution-ShareAlike), see https://creativecommons.org/licenses/by-sa/4.0/legalcode\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrolive%2Flarge-language-models-on-supercomputers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhrolive%2Flarge-language-models-on-supercomputers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrolive%2Flarge-language-models-on-supercomputers/lists"}