{"id":49635408,"url":"https://github.com/renswickd/deploy-gpt-oss-in-ec2","last_synced_at":"2026-05-05T14:34:43.531Z","repository":{"id":313484616,"uuid":"1051070866","full_name":"renswickd/deploy-gpt-oss-in-ec2","owner":"renswickd","description":"Local deployment of gpt-oss-20b model in AWS EC2 instance.","archived":false,"fork":false,"pushed_at":"2025-09-06T10:15:22.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-06T12:09:52.912Z","etag":null,"topics":["docker","ec2-instance","fastapi","gpt-oss-20b","gpu","ollama"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/renswickd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-05T11:53:51.000Z","updated_at":"2025-09-06T10:22:25.000Z","dependencies_parsed_at":"2025-09-06T12:09:55.467Z","dependency_job_id":"1cbcfae9-ab6a-4fdd-bdd5-1e010cd678c5","html_url":"https://github.com/renswickd/deploy-gpt-oss-in-ec2","commit_stats":null,"previous_names":["renswickd/deploy-gpt-oss-in-ec2"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/renswickd/deploy-gpt-oss-in-ec2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renswickd%2Fdeploy-gpt-oss-in-ec2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renswickd%2Fdeploy-gpt-oss-in-ec2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renswickd%2Fdeploy-gpt-oss-in-ec2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renswickd%2Fdeploy-gpt-oss-in-ec2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/renswickd","download_url":"https://codeload.github.com/renswickd/deploy-gpt-oss-in-ec2/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/renswickd%2Fdeploy-gpt-oss-in-ec2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32653658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-05T11:29:49.557Z","status":"ssl_error","status_checked_at":"2026-05-05T11:29:48.587Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","ec2-instance","fastapi","gpt-oss-20b","gpu","ollama"],"created_at":"2026-05-05T14:34:42.760Z","updated_at":"2026-05-05T14:34:43.521Z","avatar_url":"https://github.com/renswickd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Local LLM FastAPI App (Ollama)\n\nA FastAPI service that calls a locally hosted Ollama model for chat-style interactions. Uses the official `ollama` Python client. Designed with simple, clean structure and reasonable best practices.\n\n## Prerequisites\n\n- Python 3.10+\n- [Ollama](https://ollama.com) running within container (default `http://localhost:11434`)\n- Considering the extensive resource usage, `llama3:8b` model available in local Ollama. (for testing the app functionalities)\n\n## Configuration\n\nThe app reads settings from `.env` in the project root.\n\nRequired:\n\n- Instance sizing for 20B (recommended):\n    - GPU: 24 GB VRAM fits 4-bit quantization comfortably (e.g., g5.2xlarge).\n    - CPU-only: favor 32–64 GB RAM; e.g., c7i.4xlarge (16 vCPU, 32 GB) for better headroom.\n\nOptional:\n\n- `OLLAMA_HOST` — default `http://ollama:11434`\n- `APP_NAME` — default `Local gpt-oss-20b Chat API`\n- `DEBUG` — default `false`\n- `CORS_ORIGINS` — comma-separated list (defaults to `*`)\n\n## Install \u0026 Run\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate  # Windows: .venv\\\\Scripts\\\\activate\npip install -r requirements.txt\nuvicorn app.main:app --reload --port 8000\n```\n\n## Docker\n\nBuild the image:\n\n```bash\ndocker build -t local-llm-api:latest .\n```\n\n### Docker Compose\n\nRun an Ollama container too (optional):\n\n```bash\ndocker compose --profile ollama up --build\n```\n\nThe API is available at `http://localhost:8000`. Health check: `GET /api/health`.\n\n### Docker/Compose host configuration\n\n- If both run in compose, use the Ollama service name, e.g. `OLLAMA_HOST=http://ollama:11434`.\n\n## AWS EC2 Deployment\n\nTo deploy this app with the `gpt-oss-20b` model on AWS EC2 (instance sizing, GPU options, Compose configurations, and hardening), see:\n\n- `docs/aws-ec2-deployment.md`\n\nQuick starts on EC2:\n\n- Host Ollama (CPU or GPU on host):\n- Compose-managed Ollama (same box):\n  - `export OLLAMA_HOST=http://ollama:11434 \u0026\u0026 COMPOSE_PROFILES=ollama docker compose up --build -d`\n  - `docker compose --profile ollama exec ollama ollama pull gpt-oss-20b`\n\nHelper scripts:\n\n- `bash scripts/ollama_container_setup.sh --model gpt-oss-20b [--gpus]` to run Ollama in a container and pull the model.\n\n## Helper Script\n\nRun a single script to build the image, pull the model, and start services:\n\n```bash\n# Host Ollama (default):\nbash scripts/build_and_pull.sh --model \"$OLLAMA_MODEL\"\n```\n\nNotes:\n- If `--model` is omitted, the script reads `OLLAMA_MODEL` from `.env`.\n\n## Endpoints\n\n- `GET /` — basic info\n- `GET /api/health` — app + Ollama health\n- `POST /api/chat` — chat completion\n\n### POST /api/chat\n\nBody:\n\n```json\n{\n  \"messages\": [\n    {\"role\": \"system\", \"content\": \"You are helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"What is Langchain in one line?\"}\n  ],\n  \"stream\": false,\n  \"temperature\": 0.2\n}\n```\n\n- If `stream: true`, the response is newline-delimited JSON (`application/x-ndjson`) compatible with Ollama streaming chunks.\n- If `stream: false` or omitted, returns a single JSON object from Ollama.\n\n## Notes\n\n- Startup performs a best-effort health check but does not fail the app if Ollama is down.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frenswickd%2Fdeploy-gpt-oss-in-ec2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frenswickd%2Fdeploy-gpt-oss-in-ec2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frenswickd%2Fdeploy-gpt-oss-in-ec2/lists"}