{"id":19128752,"url":"https://github.com/indiu6/generative-ai","last_synced_at":"2026-04-04T08:40:10.644Z","repository":{"id":260031011,"uuid":"880097026","full_name":"indiu6/generative-ai","owner":"indiu6","description":"A Generative AI microservice using Node.js TypeScript Python for backend services, leverages Apache Kafka for messaging, Postgres for structured data storage, Pinecone for vector storage, ElasticSearch for search functionality, OpenAI's API for LLMs - stateful, event-based Realtime API communicates over a WebSocket","archived":false,"fork":false,"pushed_at":"2024-12-14T16:04:33.000Z","size":707,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-03T10:45:03.682Z","etag":null,"topics":["datadog","docker","elasticsearch","kafka","kubernetes","llm","nodejs","openai","pinecone","postgres","python","reactjs","typescript","websocket"],"latest_commit_sha":null,"homepage":"http://34.107.191.55/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/indiu6.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-29T05:25:56.000Z","updated_at":"2024-12-14T16:04:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"7d0d083d-4a4b-45d4-b6d5-64477c8dc400","html_url":"https://github.com/indiu6/generative-ai","commit_stats":null,"previous_names":["indiu6/generative-ai"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indiu6%2Fgenerative-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indiu6%2Fgenerative-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indiu6%2Fgenerative-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indiu6%2Fgenerative-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/indiu6","download_url":"https://codeload.github.com/indiu6/generative-ai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240199243,"owners_count":19763825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datadog","docker","elasticsearch","kafka","kubernetes","llm","nodejs","openai","pinecone","postgres","python","reactjs","typescript","websocket"],"created_at":"2024-11-09T06:05:27.861Z","updated_at":"2025-12-30T19:59:09.003Z","avatar_url":"https://github.com/indiu6.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Generative-AI-Playground ![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)\n\nIntegrates Node.js with TypeScript and Python for backend services.\n\nLeverages Apache Kafka for messaging.\n\nPostgres for structured data storage, Pinecone for vector storage.\n\nElasticSearch for search functionality, OpenAI's API for LLMs.\n\nThe OpenAI Realtime API is a stateful, event-based API that communicates over a WebSocket.\n\nThe app will be deployed using Kubernetes, AWS EKS, ECR and monitored with DataDog, while React handles the frontend interface.\n\n## Architecture Overview\n\n1. **Frontend**: The _React_ UI sends prompt requests to the backend and establishes a _WebSocket_ connection to receive _AI_ responses in real-time.\n2. **Backend**: _Node.js_ (_TypeScript_) microservice acts as the _API_ gateway, processing requests and responses.\n   - **API Endpoint**: The backend’s HTTP endpoint accepts prompt requests and produces them to _Kafka_.\n   - **WebSocket Server**: A _WebSocket_ server runs alongside the HTTP server, forwarding responses from Kafka’s `response-topic` to the frontend.\n3. **LLM Integration**: A _Python_-based microservice interacts with _OpenAI API_ for LLM queries.\n   - This service consumes prompts from Kafka’s `generate-text` topic, processes them with OpenAI, and produces the response to `response-topic`.\n   - Integrate _WebSocket_ and configure code based on [OpenAI Realtime API (beta) doc](https://platform.openai.com/docs/guides/realtime/overview)\n   - Check usage, bill, balance of my API key, [OpenAI API Usage check](https://platform.openai.com/settings/organization/usage)\n4. **Messaging**: _Apache Kafka_ to handle message queues and asynchronous tasks.\n5. **Authentication**: User authentication using _JWT_ or _OAuth2_ - Save their message history and ensure a personalized experience while maintaining secure access to saved data. (_TBD_)\n6. **Storage**: (_TBD_)\n   - _Postgres_ for structured data like user profiles, logs, and configuration.\n   - _Pinecone_ for vector embeddings to store and retrieve contextual information.\n   - _ElasticSearch_ for fast, keyword-based search.\n7. **Deployment**: _Kubernetes_ for _Docker_ container orchestration, _AWS_ EKS, ECR for infrastructure, and _DataDog_ (_TBD_) for monitoring\n\n## Docker and Docker-Compose Setup for _Dev_\n\nTo containerize the three microservices and test their communication:\n\n1. **Dockerize Microservices**: Create Dockerfiles for each of the three microservices (Frontend, Backend, LLM Integration).\n2. **Docker-Compose Configuration**: Set up a `docker-compose.yml` file to define and manage the multi-container application.\n3. **Testing Communication**: Use Docker-Compose to build and run the containers, ensuring they can communicate with each other.\n\n### Steps to Test\n\n1. **Build and Start Containers**: Run `docker-compose up --build` to build and start all services.\n2. **Verify Communication**: Ensure that the frontend can send requests to the backend, and the backend can communicate with the LLM service.\n\nRefer to the respective Dockerfile and service documentation for detailed setup instructions.\n\n## Deployment on AWS EKS for _PROD_ \u003e\u003e Google K8s Engine\n\n- Changed due to the cost: GKE + GCE \u003c AWS EKS + EC2\n- Deploying 3 Dockerized microservices using AWS Elastic Kubernetes Service (EKS) leverages Kubernetes for orchestration and AWS for infrastructure.\n\n### By automating the building, tagging, and pushing process, it can streamline the deployment pipeline, ensuring consistency and efficiency when working with multiple Docker images in Google Artifact Registry\n\n```bash\n./build_tag_push_docker_images.sh\n\nkubectl apply -f k8s-config-gke/\nkubectl rollout restart deployment llm-service\n```\n\n1. **Setup AWS EKS Cluster**:\n\n   - Create an EKS cluster using the AWS Management Console, AWS CLI, or Infrastructure as Code (IaC) tools like Terraform.\n   - Ensure cluster has the necessary IAM roles and policies for EKS to manage resources.\n\n2. **Configure kubectl**:\n\n   - Install and configure `kubectl` to interact with your EKS cluster.\n   - Update kubeconfig file to include the EKS cluster context.\n\n3. **Create Kubernetes Manifests**:\n\n   - Define Kubernetes manifests for my microservices, including Deployments, Services, ConfigMaps, and Secrets.\n   - Ensure each microservice has a corresponding Deployment and Service for load balancing.\n\n4. **Deploy Microservices**:\n\n   - Apply Kubernetes manifests using `kubectl apply -f \u003cmanifest-file\u003e.yaml`.\n   - Verify that the pods are running and services are correctly exposed.\n\n5. **Setup AWS ECR**: \u003e\u003e Google Artifact Registry\n\n   - Push Docker images to AWS Elastic Container Registry (ECR).\n   - Update Kubernetes manifests to pull images from ECR.\n\n6. **Monitor and Scale**:\n   - Use Kubernetes Horizontal Pod Autoscaler (HPA) to scale your microservices based on demand.\n   - Monitor your cluster and applications using AWS CloudWatch and DataDog.\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findiu6%2Fgenerative-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Findiu6%2Fgenerative-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findiu6%2Fgenerative-ai/lists"}