{"id":22313247,"url":"https://github.com/lreimer/k8s-native-java-ai","last_synced_at":"2025-10-29T20:34:52.556Z","repository":{"id":266070848,"uuid":"897265157","full_name":"lreimer/k8s-native-java-ai","owner":"lreimer","description":"Demos for Java-based AI on Kubernetes from Development to Deployment","archived":false,"fork":false,"pushed_at":"2025-03-31T15:40:20.000Z","size":526,"stargazers_count":2,"open_issues_count":2,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T22:51:34.286Z","etag":null,"topics":["envoy","kubernetes","langchain4j","llm","quarkus","weaviate"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lreimer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-02T10:31:48.000Z","updated_at":"2024-12-10T15:24:27.000Z","dependencies_parsed_at":"2024-12-02T12:40:59.927Z","dependency_job_id":null,"html_url":"https://github.com/lreimer/k8s-native-java-ai","commit_stats":null,"previous_names":["lreimer/k8s-native-java-ai"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lreimer/k8s-native-java-ai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lreimer%2Fk8s-native-java-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lreimer%2Fk8s-native-java-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lreimer%2Fk8s-native-java-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lreimer%2Fk8s-native-java-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lreimer","download_url":"https://codeload.github.com/lreimer/k8s-native-java-ai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lreimer%2Fk8s-native-java-ai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017942,"owners_count":26086213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["envoy","kubernetes","langchain4j","llm","quarkus","weaviate"],"created_at":"2024-12-03T22:06:40.978Z","updated_at":"2025-10-14T04:30:25.938Z","avatar_url":"https://github.com/lreimer.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Java-based AI on Kubernetes\n\nDemo repository for Java-based AI on Kubernetes from Development to Deployment.\n\n## Setup\n\n```bash\n# create the Kubernetes cluster in GCP with GPU support\n# bootstrap AI platform components and services using Flux2\nmake create-gke-cluster\nmake bootstrap-flux2\n\n# required to configure Config Connector with Google Cloud ProjectID\nkubectl annotate namespace default cnrm.cloud.google.com/project-id=\"cloud-native-experience-lab\"\n```\n\n## External Secrets using Google Cloud Security Manager\n\n```bash\n# credentials to access certain GCP infrastructure components are stored externally\n# make sure that the Google Cloud Security Manager API is enabled in your project\nmake create-gke-es-sa\n\n# if required change and apply the ClusterSecretStore CRD\n# kubectl apply -f infrastructure/platform/external-secrets/secret-store.yaml\n\n# this is how to create secrets in the Security Manager\ngcloud secrets create external-secrets-sa --data-file=external-secrets-sa.json --replication-policy=automatic\n\n# to demonstrate the secrets synchronization\nkubectl apply -f infrastructure/platform/external-secrets/sa-secret.yaml\nkubectl get secret gcp-sa-credentials -o jsonpath='{.data.external-secrets-sa\\.json}' | base64 -d\n```\n\n## Building a chat service with Quarkus and OpenAI\n\n```bash\n# use the Quarkus starter to create a service skeleton\n# select desired build system and dependencies\nopen https://code.quarkus.io\n\n# for local development use the following commands \ncd openai-chat-service\nexport QUARKUS_LANGCHAIN4J_OPENAI_API_KEY=$OPENAI_API_KEY\n./gradlew quarkusDev\n\n# interact with the service locally\nhttp get localhost:8080/api/ask q==\"Was macht QAware?\"\nhttp get localhost:8080/api/ask q==\"What does QAware do?\"\nhttp get localhost:8080/api/ask q==\"Was macht Microsoft?\"\nhttp get localhost:8080/api/ask q==\"What is the sum of 40 and 2?\"\nhttp get localhost:8080/api/ask q==\"What does QAware do? Send email to mlr@qaware.de with subject Information and response as message.\"\n\n# this here is managed by Flux2\nkubectl apply -k infrastructure/services/openai-chat-service/\nkubectl get all\n```\n\n## Building an OpenAI Proxy using Envoy\n\nThe access to the OpenAI API is provided using a cluster internal Envoy based proxy.\n\n```bash\n# in order for the proxy to work ou have to manually create a Kubernetes secret\n# that contains an OPENAI_API_KEY environment variable\nkubectl create secret generic openai-api-key --from-literal=OPENAI_API_KEY=$OPENAI_API_KEY\n\n# to test the proxy, issue the following curl command\n# exchange localhost with the actual LoadBalancer IP\ncurl http://localhost:10000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n     \"model\": \"gpt-4o-mini\",\n     \"messages\": [{\"role\": \"user\", \"content\": \"Say this is a test!\"}],\n     \"temperature\": 0.7\n   }'\n```\n\n## Building a chat service with Quarkus and Ollama\n\n```bash\n# this is 99% similar to the instructions of using Quarkus and OpenAI\n# the only difference, use\n#    'io.quarkiverse.langchain4j:quarkus-langchain4j-ollama:0.22.0'\n# instead of \n#    'io.quarkiverse.langchain4j:quarkus-langchain4j-openai:0.22.0'\n\n# for local development use the following commands \nollama serve\nollama run llama3.1\n\ncd ollama-chat-service\n./gradlew quarkusDev\n\n# interact with the service locally\nhttp get localhost:8080/api/ask q==\"Was macht die QAware GmbH?\"\n\n# this here is managed by Flux2\nkubectl apply -k infrastructure/services/openai-chat-service/\nkubectl get all\n```\n\n## Building AI Agent using Google ADK (Java)\n\n```bash\ncd hello-time-agent\n\nmvn compile exec:java -Dexec.mainClass=\"com.example.agent.AgentCliRunner\"\n\nmvn compile exec:java \\\n    -Dexec.mainClass=\"com.google.adk.web.AdkWebServer\" \\\n    -Dexec.args=\"--adk.agents.source-dir=target --server.port=8080\"\n```\n\n## Deploying custom LLMs using Ollama Operator\n\n```bash\n# model deployment using CLI\nkollama deploy llama3.1\nkollama expose llama3.1 --service-name=ollama-model-llama31-lb --service-type=LoadBalancer\n\n# model deployment via CRD\nkubectl apply -f infrastructure/models/phi3.yaml\nkollama expose phi3 --service-type LoadBalancer\n\n# to start a chat with ollama\n# exchange localhost with the actual LoadBalancer IP\nOLLAMA_HOST=localhost:11434 ollama run phi3\nOLLAMA_HOST=localhost:11434 ollama run llama3.1\n\n# call the chat API of Ollama or OpenAI\n# curl http://localhost:11434/v1/chat/completions\ncurl http://localhost:11434/api/chat  \\\n  -H \"Content-Type: application/json\"  \\\n  -d '{\n    \"model\": \"llama3.1\",\n    \"messages\": [\n      {\n        \"role\": \"user\",\n        \"content\": \"Say this is a test!\"\n      }\n    ]\n  }'\n```\n\n## Deploying Langchain4J Easy RAG Chatbot\n\n```bash\n# see https://docs.quarkiverse.io/quarkus-langchain4j/dev/easy-rag.html\n# see https://github.com/quarkiverse/quarkus-langchain4j/tree/main/samples/chatbot-easy-rag\n```\n\n## Deploying Airbyte\n\n```bash\n# make sure to create the Airbyte service account\nmake create-gke-airbyte-sa\n\n# bind the SA to your personal gCP user (optional)\ngcloud iam service-accounts add-iam-policy-binding airbyte@$GCP_PROJECT.iam.gserviceaccount.com --member=\"user:mario-leander.reimer@qaware.de\" --role=\"roles/iam.serviceAccountUser\"\n\n# now create the Airbyte secret with the relevant GCP credentials.\nkubectl create secret generic airbyte-config-secrets -n airbyte --from-file=airbyte.json\n```\n\n## Planing Kubernetes Cluster Topology\n\n```bash\n# useful information to setup K8s cluster topology with GPUs\n\n# see https://cloud.google.com/compute/docs/gpus?hl=de\n# see https://cloud.google.com/compute/all-pricing?hl=de\n# see https://cloud.google.com/compute/gpus-pricing?hl=de#other-gpu-models\n\n# see https://cloud.google.com/compute/docs/gpus/create-gpu-vm-general-purpose?hl=de\n# see https://cloud.google.com/compute/docs/disks/local-ssd?hl=de\n```\n\n## Maintainer\n\nM.-Leander Reimer (@lreimer), \u003cmario-leander.reimer@qaware.de\u003e\n\n## License\n\nThis software is provided under the MIT open source license, read the `LICENSE`\nfile for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flreimer%2Fk8s-native-java-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flreimer%2Fk8s-native-java-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flreimer%2Fk8s-native-java-ai/lists"}