{"id":48601813,"url":"https://github.com/rogervinas/spring-boot-ai-langfuse-evaluations","last_synced_at":"2026-04-08T22:30:59.864Z","repository":{"id":342875106,"uuid":"1175343752","full_name":"rogervinas/spring-boot-ai-langfuse-evaluations","owner":"rogervinas","description":"🏦 PoC using Spring AI, RAG and MCP fully instrumented with Langfuse for precision evaluation and hallucination detection.","archived":false,"fork":false,"pushed_at":"2026-03-23T19:53:37.000Z","size":59,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-24T17:53:03.839Z","etag":null,"topics":["evaluation","hallucination","langfuse","mcp","rag","spring-ai"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rogervinas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-07T15:28:03.000Z","updated_at":"2026-03-23T19:53:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rogervinas/spring-boot-ai-langfuse-evaluations","commit_stats":null,"previous_names":["rogervinas/spring-boot-ai-langfuse-evaluations"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rogervinas/spring-boot-ai-langfuse-evaluations","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-boot-ai-langfuse-evaluations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-boot-ai-langfuse-evaluations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-boot-ai-langfuse-evaluations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-boot-ai-langfuse-evaluations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rogervinas","download_url":"https://codeload.github.com/rogervinas/spring-boot-ai-langfuse-evaluations/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogervinas%2Fspring-boot-ai-langfuse-evaluations/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31577444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evaluation","hallucination","langfuse","mcp","rag","spring-ai"],"created_at":"2026-04-08T22:30:58.192Z","updated_at":"2026-04-08T22:30:59.849Z","avatar_url":"https://github.com/rogervinas.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI](https://github.com/rogervinas/spring-boot-ai-langfuse-evaluations/actions/workflows/ci.yml/badge.svg)](https://github.com/rogervinas/spring-boot-ai-langfuse-evaluations/actions/workflows/ci.yml)\n![Java](https://img.shields.io/badge/Java-21-blue?labelColor=black)\n![Kotlin](https://img.shields.io/badge/Kotlin-2.x-blue?labelColor=black)\n![SpringBoot](https://img.shields.io/badge/SpringBoot-4.x-blue?labelColor=black)\n![SpringAI](https://img.shields.io/badge/SpringAI-2.x-blue?labelColor=black)\n![Langfuse](https://img.shields.io/badge/Langfuse-3.x-blue?labelColor=black)\n\n![Gemini](https://img.shields.io/badge/Gemini-✓-4285F4?labelColor=black)\n![Bedrock](https://img.shields.io/badge/Bedrock-✓-FF9900?labelColor=black)\n![Ollama](https://img.shields.io/badge/Ollama-✓-FFFFFF?labelColor=black)\n\n# Spring AI Langfuse Evaluations\n\n\u003e **A multi-model RAG and Tool-Calling assistant for modern fintech, built with Spring AI and evaluated with Langfuse for observability, RAG faithfulness, and tool-call accuracy.**\n\n## Stack\n\n- **Spring Boot** + **Spring AI**\n- **LLM Providers**: AWS Bedrock, Google Gemini, or local Ollama\n- **PGVector** as the vector database for RAG\n- **Langfuse** for tracing and evaluation via OpenTelemetry\n\n## Configuration\n\nYou can configure the application using environment variables or a `system.properties` file in the root directory. This file is ignored by Git and is loaded by both `./gradlew bootRun` and tests.\n\nExample `system.properties`:\n\n```properties\n# AWS Bedrock\nAWS_ACCESS_KEY_ID=...\nAWS_SECRET_ACCESS_KEY=...\nAWS_REGION=eu-central-1\nAWS_BEDROCK_CHAT_MODEL=...\nAWS_BEDROCK_EMBEDDING_MODEL=...\n\n# Google Gemini\nGOOGLE_API_KEY=...\n```\n\n## Running the local vector database\n\nStart PGVector for the RAG vector database:\n\n```bash\ndocker compose -f docker-compose-vectordb.yml up -d\n```\n\nThis starts a PostgreSQL instance with the `pgvector` extension on port `5432`.\n\nTo stop it:\n\n```bash\ndocker compose -f docker-compose-vectordb.yml down\n```\n\nTo stop it and remove all volumes (removes all vector database data):\n\n```bash\ndocker compose -f docker-compose-vectordb.yml down -v\n```\n\n## Running Ollama locally\n\nIf you want to use local LLMs, you can run Ollama either via Docker Compose or as a native application (more info at [ollama.com](https://ollama.com/)).\n\n### Using Docker Compose\n\n```bash\ndocker compose -f docker-compose-ollama.yml up -d\n```\n\nTo stop it:\n\n```bash\ndocker compose -f docker-compose-ollama.yml down\n```\n\n\n## Running Langfuse locally\n\nThe `docker-compose-langfuse.yml` is based on the [official Langfuse docker-compose.yml](https://github.com/langfuse/langfuse/blob/main/docker-compose.yml) with two modifications:\n- All ports except `3000` (the main UI/API) are commented out to avoid collisions with other local containers\n- `LANGFUSE_INIT_*` environment variables are set to auto-provision an organization, project, and user on first startup\n\nStart the Langfuse stack (includes PostgreSQL, ClickHouse, Redis, MinIO):\n\n```bash\ndocker compose -f docker-compose-langfuse.yml up -d\n```\n\nThis auto-provisions:\n- **Organization**: `rogervinas-bank`\n- **Project**: `banking-sentinel`\n- **API keys**: `publickey-local` / `secretkey-local`\n- **User**: `admin@local.dev` / `password`\n\nLangfuse UI is available at http://localhost:3000.\n\nTo stop it:\n\n```bash\ndocker compose -f docker-compose-langfuse.yml down\n```\n\nTo start from scratch (removes all data including traces, users, and projects):\n\n```bash\ndocker compose -f docker-compose-langfuse.yml down -v\n```\n\n## Running the application\n\nThe `application.yml` is pre-configured to send traces to the local Langfuse instance using the auto-provisioned API keys.\n\nYou can run the application using one of the following profiles:\n\n### 1. Ollama Profile (Local)\n\nRequires [Ollama](#running-ollama-locally) and [Vector database](#running-the-local-vector-database) to be running.\n\n```bash\nSPRING_PROFILES_ACTIVE=ollama ./gradlew bootRun\n```\n\n### 2. AWS Bedrock Profile\n\nRequires [Vector database](#running-the-local-vector-database) to be running. You need to configure AWS credentials and models via environment variables or `system.properties`:\n\n```bash\n# Set environment variables or use system.properties\nSPRING_PROFILES_ACTIVE=bedrock ./gradlew bootRun\n```\n\n### 3. Google Gemini Profile\n\nRequires [Vector database](#running-the-local-vector-database) to be running. You need to configure your Google AI API key via environment variables or `system.properties`:\n\n```bash\n# Set environment variables or use system.properties\nSPRING_PROFILES_ACTIVE=gemini ./gradlew bootRun\n```\n\n\n## Spring AI + Spring Boot 4.x observability workarounds\n\nSpring AI 2.0.0-M2 observation handlers (`ChatModelPromptContentObservationHandler`, `ChatModelCompletionObservationHandler`, etc.) only log to SLF4J instead of adding key values to the observation context, which means the data doesn't reach OTel-based backends like Langfuse.\n\nAdditionally, some auto-configurations depend on a Micrometer `Tracer` bean that Spring Boot 4.x no longer provides (it was part of `spring-boot-actuator-autoconfigure` in Spring Boot 3.x).\n\nThis project provides custom `ObservationFilter` implementations in `com.rogervinas.bank.observation` that add the data as high-cardinality key values using [OTel semantic conventions for GenAI](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/):\n\n| Filter | Replaces | OTel attributes |\n|---|---|---|\n| `ChatModelObservationFilter` | `ChatModelPromptContentObservationHandler` + `ChatModelCompletionObservationHandler` | `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.tool.definitions`, `gen_ai.response.model` (fix) |\n| `ChatClientObservationFilter` | `ChatClientPromptContentObservationHandler` + `ChatClientCompletionObservationHandler` | `gen_ai.input.messages`, `gen_ai.output.messages` |\n| `ImageModelObservationFilter` | `ImageModelPromptContentObservationHandler` | `gen_ai.input.messages` |\n| `VectorStoreObservationFilter` | `VectorStoreQueryResponseObservationHandler` | `gen_ai.retrieval.documents` |\n| `ToolCallingObservationFilter` | `ToolCallingContentObservationFilter` | `gen_ai.tool.name`, `gen_ai.tool.description`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result` |\n\n## Langfuse\n\n## Requirements\n\n[Langfuse CLI](https://langfuse.com/docs/api-and-data-platform/features/cli)\nhttps://github.com/langfuse/langfuse-cli?tab=readme-ov-file#install\n\n## Start Tracing ✅\n\n## Prompt Management\n\nhttps://langfuse.com/docs/prompt-management/overview\n\nVia OTEL:\n```\nlangfuse.observation.prompt.name: The name of your managed prompt\nlangfuse.observation.prompt.version: The version of the prompt\n```\n\n## Structured Input and Output\n\nWe use structured output (`ChatResponse` with `answer` + `suggestedActions` enum) instead of raw text for several reasons:\n\n- **Deterministic evaluation of actions**: with an enum like `SuggestedAction.FREEZE_CARD` we can assert exact values in tests, no need for an LLM-as-judge to verify if \"the agent offered to freeze the card\"\n- **LLM-as-judge only where it adds value**: the free-text `answer` still needs subjective evaluation (tone, completeness, correctness of explanation), so we keep LLM-as-judge for that part only\n- **Better Langfuse traces**: structured JSON in trace output is filterable and aggregatable in dashboards, instead of opaque text blobs\n- **Langfuse evaluators can target specific fields**: e.g. score `suggestedActions` correctness separately from `answer` quality\n- **Datasets and experiments**: when creating Langfuse datasets, structured expected output (e.g. `expectedActions: [FREEZE_CARD, OPEN_DISPUTE]`) makes it trivial to compare against actual output\n- **Frontend can render structured data**: action buttons, transaction tables, etc. instead of parsing markdown\n- **Caveat — \"deterministic\" is relative**: while the output *format* is deterministic (enum values, not free text), the LLM's *choice* of which actions to suggest is still non-deterministic. For example, asked about an unauthorized charge, the LLM might return `[FREEZE_CARD, GET_TRANSACTIONS]` instead of `[FREEZE_CARD, OPEN_DISPUTE]` because it wants to retrieve transactions first. So we assert deterministically only on the safety-critical action (`FREEZE_CARD`) and use LLM-as-judge for the rest of the reasoning.\n- **Unit tests vs full-flow evaluation**: deterministic assertions on a single response are inherently flaky because the LLM may spread actions across multiple turns. The real question is not \"did this one response contain the right actions?\" but \"did the whole conversation lead to the right outcome?\". This is where Langfuse evaluations (datasets + experiments) are more appropriate — they can evaluate the full trace including all tool calls and actions taken, rather than a snapshot of one turn.\n\nWe use Spring AI's `.entity(ChatResponse::class.java)` which leverages `BeanOutputConverter` to instruct the LLM to respond in the expected JSON schema and automatically deserialize it.\n\n## Evals\n\n### [Create a dataset](https://langfuse.com/docs/evaluation/experiments/datasets) to measure your LLM application's performance consistently\n\n```bash\nexport LANGFUSE_PUBLIC_KEY=publickey-local\nexport LANGFUSE_SECRET_KEY=secretkey-local\nexport LANGFUSE_HOST=http://localhost:3000\n```\n\nSynthetic or from Production\n\n### [Run an experiment](https://langfuse.com/docs/evaluation/core-concepts#experiments) get an overview of how your application is doing\n\n### [Set up a live evaluator](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge) to monitor your live traces\n\n### User feedback\n\nhttps://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk\n\n```bash\ncurl -X POST http://localhost:3000/api/public/scores \\\n  -u \"publickey-local:secretkey-local\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"traceId\": \"91d4fa4042cc6f3a3fc3b47c2a846331\",\n    \"name\": \"correctness\",\n    \"value\": 0.9,\n    \"dataType\": \"NUMERIC\",\n    \"comment\": \"Factually correct\"\n  }'\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogervinas%2Fspring-boot-ai-langfuse-evaluations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frogervinas%2Fspring-boot-ai-langfuse-evaluations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogervinas%2Fspring-boot-ai-langfuse-evaluations/lists"}