{"id":26304367,"url":"https://github.com/neuralwork/build-cog-inference-container","last_synced_at":"2026-04-20T03:32:52.580Z","repository":{"id":222332241,"uuid":"756828066","full_name":"neuralwork/build-cog-inference-container","owner":"neuralwork","description":"Containerized model deployment of neuralwork/instruct-finetune-mistral to AWS EC2 with Cog.","archived":false,"fork":false,"pushed_at":"2024-02-13T15:00:15.000Z","size":6,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-03T15:44:17.735Z","etag":null,"topics":["aws","cog","deep","deployment","docker","ec2","generative-ai","llm","mistral"],"latest_commit_sha":null,"homepage":"https://blog.neuralwork.ai/deploying-llms-on-aws-ec2-using-cog-a-complete-guide/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neuralwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-13T11:43:11.000Z","updated_at":"2024-07-30T20:22:41.000Z","dependencies_parsed_at":"2024-02-13T17:57:54.035Z","dependency_job_id":null,"html_url":"https://github.com/neuralwork/build-cog-inference-container","commit_stats":null,"previous_names":["neuralwork/build-cog-inference-container"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/neuralwork/build-cog-inference-container","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralwork%2Fbuild-cog-inference-container","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralwork%2Fbuild-cog-inference-container/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralwork%2Fbuild-cog-inference-container/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralwork%2Fbuild-cog-inference-container/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neuralwork","download_url":"https://codeload.github.com/neuralwork/build-cog-inference-container/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralwork%2Fbuild-cog-inference-container/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32031576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cog","deep","deployment","docker","ec2","generative-ai","llm","mistral"],"created_at":"2025-03-15T08:16:05.475Z","updated_at":"2026-04-20T03:32:52.554Z","avatar_url":"https://github.com/neuralwork.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Build a Dockerized Inference API using Cog\nThis repository contains the code and instructions to build a Dockerized Inference API for an LLM using Cog. For detailed tutorial of building the docker image and deploying to it to AWS EC2, please refer to [our blog](https://blog.neuralwork.ai/).\nThe LLM is the mistral-7b finetuned on the style instruct dataset and named mistral-7b-style-instruct. Training code and instructions of the model can be found in the [instruct-finetune-mistral](https://github.com/neuralwork/instruct-finetune-mistral) repository, its detailed tutotial can be found in [our blog post](https://blog.neuralwork.ai/deploying-llms-on-aws-ec2-using-cog-a-complete-guide/).\n\n## Pre-requisites\n- Nvidia GPU with CUDA support.\n- [Docker](https://www.docker.com/) installed.\n- [Cog](https://github.com/replicate/cog) installed.\n- [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed.\n\n## Build the Docker Image\nTo build the Docker image, run the following in the cloned directory:\n```bash\ncog build -t mistral-7b-style-instruct\n```\nThis will build the Docker image with the name mistral-7b-style-instruct.\n\n## Run the Docker Image\nTo run the Docker image, run the following in the cloned directory:\n```bash\ndocker run -p 5000:5000 mistral-7b-style-instruct\n```\n## Test the Inference API\nTo test the Inference API, you can use the following curl command:\n\n```bash\ncurl http://localhost:5000/predictions -X POST -H \"Content-Type: application/json\" -d '{\"input\": {\"prompt\":\"I am an athletic and 180cm tall man in my mid twenties, I have a rectangle shaped body with slightly broad shoulders and have a sleek,casual style. I usually prefer darker colors.\", \"event\": \"I am going to a wedding.\"}}'\n```\nOr you can use the following python code:\n```python\nimport requests\n\nurl = 'http://localhost:5000/predictions'\ndata = {\"input\": {\"prompt\":\"I am an athletic and 180cm tall man in my mid twenties, I have a rectangle shaped body with slightly broad shoulders and have a sleek,casual style. I usually prefer darker colors.\", \"event\": \"I am going to a wedding.\"}}\nresponse = requests.post(url, json=data)\nprint(response.json())\n```\n\nFrom [neuralwork](https://neuralwork.ai/) with :heart:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneuralwork%2Fbuild-cog-inference-container","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneuralwork%2Fbuild-cog-inference-container","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneuralwork%2Fbuild-cog-inference-container/lists"}