{"id":17632703,"url":"https://github.com/matthewhaynesonline/ai-server-setup","last_synced_at":"2025-04-12T20:25:02.740Z","repository":{"id":254711833,"uuid":"838043021","full_name":"matthewhaynesonline/ai-server-setup","owner":"matthewhaynesonline","description":"Setup AWS EC2 instance from scratch with NVIDIA CUDA, Docker, Packer for AI / ML.","archived":false,"fork":false,"pushed_at":"2024-09-08T21:15:50.000Z","size":16,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-09T23:01:50.371Z","etag":null,"topics":["ai","ami","aws","cuda","devops","docker","ml","mlops","packer"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matthewhaynesonline.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-04T19:14:28.000Z","updated_at":"2024-12-19T07:32:11.000Z","dependencies_parsed_at":"2024-10-23T07:18:55.429Z","dependency_job_id":null,"html_url":"https://github.com/matthewhaynesonline/ai-server-setup","commit_stats":null,"previous_names":["matthewhaynesonline/ai-server-setup"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewhaynesonline%2Fai-server-setup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewhaynesonline%2Fai-server-setup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewhaynesonline%2Fai-server-setup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewhaynesonline%2Fai-server-setup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matthewhaynesonline","download_url":"https://codeload.github.com/matthewhaynesonline/ai-server-setup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248626955,"owners_count":21135765,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ami","aws","cuda","devops","docker","ml","mlops","packer"],"created_at":"2024-10-23T01:45:11.373Z","updated_at":"2025-04-12T20:25:02.707Z","avatar_url":"https://github.com/matthewhaynesonline.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI and ML Server Setup\n\nThis is the code that accompanies the [AI Server from Scratch in AWS video](https://www.youtube.com/watch?v=N_KFYqvEZvU).\n\nThis project walks through setting up an AWS EC2 instance optimized for generative AI and machine learning tasks, using NVIDIA and Docker on Ubuntu.\n\n### Prerequisites\n\n- An AWS account\n- Basic knowledge of:\n  - AWS (EC2)\n  - Virtual Machines\n  - Linux / Command Line / SSH\n- Familiarity with Docker and containerization concepts\n\n### Goals\n\n1. Configure a compute-optimized VM from scratch (starting with a blank Ubuntu image)\n2. Ensure portability to avoid vendor lock-in and reduce dependence on platform-specific tools\n3. Create a flexible environment suitable for various AI and ML frameworks\n4. Streamline deployments by creating a custom image with Packer\n\n#### Why EC2 vs Bedrock, HuggingFace Endpoints, etc.\n\nAn EC2 is not the perfect choice for all projects. There may be cases where a hosted API / an API that charges a price per token is cheaper and / or easier to get up an running (an API will almost always be easier to get up and running).\n\nHere are the reason this guide uses an EC2:\n\n1. Price ceiling: the price for the EC2 and storage has a predictable ceiling, so there shouldn't be any surprise bills\n2. Versatile: since an EC2 is ostensibly a VM (with a GPU), we can run all kinds of projects on it, not just the use cases that are supported by an API / platform provider\n3. Portable: the steps to configure the EC2 should generally apply to other Linux instances\n\n### Setup / Instructions / Notes\n\n#### NVIDIA Drivers / CUDA\n\nIf you run into issue like `RuntimeError: No CUDA GPUs are available` or `Failed to initialize NVML: Driver/library version mismatch` just try rebooting.\n\n[https://stackoverflow.com/a/43023000](https://stackoverflow.com/a/43023000)\n\n#### AWS\n\n**Make sure to create an aws.env file with your settings.** `cd aws; cp aws.env.example aws.env`\n\nSee the aws directory. There is an example setup script that will configure the specified EC2 and a deploy script that copy the code to the EC2 and start the stack.\n\n**NOTE: if running on a non standard port (not 80 or 443) make sure to allow inbound traffic to the EC2 on that port.**\n\n##### Lambda Auto Stop EC2\n\nAn example Lambda function is provided to stop all EC2 instances. The demo'd use for this is to stop all EC2s at midnight to prevent long running instances from incurring a higher bill.\n\n##### Packer\n\nAn example Packer file is provided to generate a custom AMI from the setup steps in this guide. This will allow you to quickly deploy a baseline instance with NVIDIA / Docker support without having to reconfigure from scratch every time.\n\n#### llama.cpp\n\nllama.cpp is included just to test that the server is working as expected, both natively and in a container. It is added as a [git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules).\n\n##### Compile with CUDA and Support for NVIDIA T4 (nvidia 7.5)\n\n- https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#cuda\n- https://developer.nvidia.com/cuda-gpus\n\n```sh\nmake clean\nCUDA_DOCKER_ARCH=compute_75 GGML_CUDA=1 make\n```\n\n#### Download Model\n\n```sh\ncd models\n\n# Use Bartowski gguf to fix issues with llama.cpp\ncurl -O -L \"https://huggingface.co/bartowski/Phi-3.1-mini-4k-instruct-GGUF/resolve/main/Phi-3.1-mini-4k-instruct-Q6_K_L.gguf\"\n```\n\n### Resources\n\n- [Auto Stop EC2 Using AWS Lambda](https://repost.aws/knowledge-center/start-stop-lambda-eventbridge)\n- [Install Docker Compose on Linux](https://docs.docker.com/compose/install/linux/)\n- [Ubuntu NVIDIA Driver Docs](https://ubuntu.com/server/docs/nvidia-drivers-installation)\n- [NVIDIA CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)\n  - [CUDA GPU Compatibility List](https://developer.nvidia.com/cuda-gpus)\n- [NVIDIA Container Toolkit Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)\n- [Elasticsearch / Opensearch virtual memory](https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html)\n  - [https://stackoverflow.com/q/51445846](https://stackoverflow.com/q/51445846)\n- [llama.cpp](https://github.com/ggerganov/llama.cpp)\n- [Distantmagic's Paddler Installation on AWS EC2 CUDA Instances Notes](https://github.com/distantmagic/paddler/blob/v0.3.0/infra/tutorial-installing-llamacpp-aws-cuda.md)\n  - This has been deprecated in favor of [LLMOps Handbook](https://github.com/distantmagic/llmops-handbook)\n- [Packer Machine Image Builder](https://www.packer.io/)\n  - [Packer AWS Tutorial](https://developer.hashicorp.com/packer/tutorials/aws-get-started)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthewhaynesonline%2Fai-server-setup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatthewhaynesonline%2Fai-server-setup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthewhaynesonline%2Fai-server-setup/lists"}