{"id":19209181,"url":"https://github.com/hrolive/upscaling-ai-worflows","last_synced_at":"2026-05-04T15:39:30.662Z","repository":{"id":104688615,"uuid":"486262288","full_name":"HROlive/Upscaling-AI-Worflows","owner":"HROlive","description":"How to use Docker and Singularity containers in conjunction with TensorFlow and Horovod to do distributed training and upscale an AI app.","archived":false,"fork":false,"pushed_at":"2022-08-20T11:00:02.000Z","size":851,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-04T16:23:48.119Z","etag":null,"topics":["containers","deep-learning","docker","high-performance-computing","horovod","machine-learning","natural-language-processing","notebook","singularity","tensorflow","transfer-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HROlive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-27T16:04:12.000Z","updated_at":"2024-11-21T11:14:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"a2371e00-662c-499a-9edd-dadecd7d5666","html_url":"https://github.com/HROlive/Upscaling-AI-Worflows","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FUpscaling-AI-Worflows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FUpscaling-AI-Worflows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FUpscaling-AI-Worflows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HROlive%2FUpscaling-AI-Worflows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HROlive","download_url":"https://codeload.github.com/HROlive/Upscaling-AI-Worflows/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240281017,"owners_count":19776429,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","deep-learning","docker","high-performance-computing","horovod","machine-learning","natural-language-processing","notebook","singularity","tensorflow","transfer-learning"],"created_at":"2024-11-09T13:29:24.700Z","updated_at":"2026-05-04T15:39:25.640Z","avatar_url":"https://github.com/HROlive.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Workshop](images/upscaling-ai-workflows.jpg)\n\n## Table of Contents\n1. [Description](#description)\n2. [Instructions](#instructions)\n3. [Getting access to HPC resources](#installation)\n4. [Certificate](#certificate)\n\n\u003ca name=\"descripton\"\u003e\u003c/a\u003e\n## Description\n\nIn this workshop, sponsored by The Swedish EuroCC Hub for High-Performance Computing, we started by overviewing the basics of Docker and Singularity. Then, we used resources from the Slovenian supercomputer, Vega, to do distributed training using TensorFlow and Horovod frameworks. \nMoreover, we also used Docker and Singularity containers in conjunction with TensorFlow and Horovod to upscale an AI app.\n\nArtificial Intelligence (AI) has become a foundational building block of our modern world. Accordingly, a vast effort has been put into bringing AI to researchers and practitioners of a wide range of fields. Nonetheless, the computationally intensive task of training an AI increasingly requires more computational power than what our laptops and PCs can offer. Therefore, the ability to develop and train a neural network on large clusters seems imperative. This workshop teaches us how to scale an AI-powered applications in large clusters, i.e., supercomputers.\n\nThe outcomes of the workshop are:\n\u003e - Create, deploy, and update containers locally on a supercomputer\n\u003e - Upscale the transfer learning of an NLP model in TensorFlow\n\u003e - Upscale the transfer learning of an NLP model using Horovod\n\u003e - Upscale the transfer learning of a containerized NLP model\n\n\u003ca name=\"instructions\"\u003e\u003c/a\u003e\n## Instructions\n\nAll necessary information and links for the workshop, QA, exercises, tutorials, etc, can be found in the [workshop website](https://hackmd.io/@enccs/upscaling-april2022).\n\n\u003ca name=\"installation\"\u003e\u003c/a\u003e\n## Getting access to HPC resources\n\nDetailed instructions on how to access Vega's resources can be found on [The Swedish EuroCC Hub for High-Performance Computing website](https://enccs.se/).\n\n![Workshop](images/distributed-training.png)\n\nHow to reserve nodes and launch a jupyter notebook on VEGA:\n\nRunning Jupyter on Vega and connect to it\n\nTo book a node:\n```\nsalloc -n 1 --gres=gpu:1 --reservation=enccs-day1 --mem-per-gpu=40GB --ntasks 4 --cpus-per-task 1 -t 01:00:00\n```\nYou will see something like salloc: Nodes gn38 are ready for job\n\nLogin to compute node from login node (replace gn38 with your compute node ID):\n```\nssh gn38\n```\nLoad modules on compute node:\n```\nmodule load Anaconda3/2020.11\nmodule load scikit-learn\nmodule load TensorFlow/2.5.0-fosscuda-2020b\n```\nPick a random port between 8000 and 9000, and replace gnXX\n```\njupyter-notebook --no-browser --port=8123 --ip=gnXX\n```\nFrom local machine (replace gnXX with name of compute node, and replace 8123 with the port you selected when running jupyter-notebook):\n```\nssh -L localhost:9999:gnXX:8123 username@vglogin0005.vega.izum.si\n```\nOpen in your browser (port 9999 should match what you selected in the ssh-tunnel command above):\n```\nhttp://localhost:9999/\n```\n\nDetailed instructions can be found on the [Vega website](https://doc.vega.izum.si/jupyter/).\n\n\u003ca name=\"certificate\"\u003e\u003c/a\u003e\n## Certificate\nThe certificate can be found [here](https://github.com/HROlive/Upscaling-AI-Worflows/blob/main/images/certificate-hugo-oliveira.pdf).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrolive%2Fupscaling-ai-worflows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhrolive%2Fupscaling-ai-worflows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrolive%2Fupscaling-ai-worflows/lists"}