Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/langfuse/langfuse

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
https://github.com/langfuse/langfuse

analytics evaluation gpt hacktoberfest langchain large-language-models llama-index llm llm-evaluation llm-observability llmops monitoring observability open-source openai playground prompt-engineering prompt-management self-hosted ycombinator

Last synced: 5 days ago
JSON representation

🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

Host: GitHub
URL: https://github.com/langfuse/langfuse
Owner: langfuse
License: other
Created: 2023-05-18T17:47:09.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-10-29T09:53:17.000Z (2 months ago)
Last Synced: 2024-10-29T09:57:16.598Z (2 months ago)
Topics: analytics, evaluation, gpt, hacktoberfest, langchain, large-language-models, llama-index, llm, llm-evaluation, llm-observability, llmops, monitoring, observability, open-source, openai, playground, prompt-engineering, prompt-management, self-hosted, ycombinator
Language: TypeScript
Homepage: https://langfuse.com/docs
Size: 14.5 MB
Stars: 6,151
Watchers: 23
Forks: 595
Open Issues: 147
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md

Awesome Lists containing this project

Awesome-LLM-Productization - Langfuse - Open source observability and analytics for LLM applications (Models and Tools / LLM Deployment)
awesome-ai-sdks - GitHub
awesome-llmops - Langfuse - square) | (LLMOps / Observability)
awesome-open-data-centric-ai - langfuse
awesome-production-machine-learning - Langfuse - Langfuse is an observability & analytics solution for LLM-based applications. (Evaluation and Monitoring)
StarryDivineSky - langfuse/langfuse - 适用于 Typescript、Python、OpenAI、Langchain、Litellm、Flowise、Superagent 和 Langflow 的稳定 SDK + 集成 (A01_文本生成_文本对话 / 大语言对话模型及数据)
Awesome-LLM - Langfuse - Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground. (LLM Deployment)
awesome - langfuse/langfuse - 🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 (TypeScript)
project-awesome - langfuse/langfuse - 🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 (TypeScript)
AiTreasureBox - langfuse/langfuse - 01-07_7637_15](https://img.shields.io/github/stars/langfuse/langfuse.svg)|🪢 Open source LLM engineering platform. Observability, metrics, evals, prompt management, testing, prompt playground, datasets, LLM evaluations -- 🍊YC W23 🤖 integrate via Typescript, Python / Decorators, OpenAI, Langchain, LlamaIndex, Litellm, Instructor, Mistral, Perplexity, Claude, Gemini, Vertex| (Repos)
Awesome-LLM-RAG-Application - langfuse
Awesome-RAG - LangFuse - source tool for tracking LLM metrics, observability, and prompt management. (📊 Metrics / Response Evaluation Metrics)
awesome-ChatGPT-repositories - langfuse - 🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 (Prompts)
awesome_ai_agents - Langfuse - 🪢 Open source LLM engineering platform - LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with Llama… (Building / Prompt Engineering)
awesome_ai_agents - Langfuse - 🪢 Open source LLM engineering platform - LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with Llama… (Building / Prompt Engineering)
jimsghstars - langfuse/langfuse - 🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 (TypeScript)

README

        ![Langfuse GitHub Banner](https://github.com/langfuse/langfuse/assets/121163007/6035f0f3-d691-4963-b5d0-10cf506e9d42)

Langfuse: Open Source LLM Engineering Platform

LLM Observability, Prompt Management, LLM Evaluations,
Datasets, LLM Metrics, and Prompt Playground




   

      

         

            Sign up

          · 

         

            Self Host

          · 

         

            Demo (live data)

         

      

   

   

      Docs ·

      Report Bug ·

      Feature Request ·

      Changelog ·

      Roadmap ·

      Discord 

   

   Langfuse uses Github Discussions  for Support and Feature Requests.

   


   We're hiring. Join us in Product Engineering and Developer Relations.

   


   


   

      

      

      

      

      

   



## Langfuse Overview

[![Langfuse Overview Video](https://github.com/user-attachments/assets/3926b288-ff61-4b95-8aa1-45d041c70866)](https://langfuse.com/watch-demo)

### Develop

- **LLM Observability:** Instrument your app and start ingesting traces to Langfuse ([Quickstart](https://langfuse.com/docs/get-started), [Integrations](https://langfuse.com/docs/integrations) [Tracing](https://langfuse.com/docs/tracing))

- **Langfuse UI:** Inspect and debug complex logs ([Demo](https://langfuse.com/docs/demo), [Tracing](https://langfuse.com/docs/tracing))

- **Prompt Management:** Manage, version and deploy prompts from within Langfuse ([Prompt Management](https://langfuse.com/docs/prompts/get-started))

- **Prompt Engineering:** Test and iterate on your prompts with the [LLM Playground](https://langfuse.com/docs/playground)

### Monitor

- **LLM Analytics:** Track metrics (cost, latency, quality) and gain insights from dashboards & data exports ([Analytics](https://langfuse.com/docs/analytics))

- **LLM Evaluations:** Collect and calculate scores for your LLM completions ([Scores & Evaluations](https://langfuse.com/docs/scores))

  - Run ([Model-based evaluations](https://langfuse.com/docs/scores/model-based-evals)) and LLM-as-a-Judge within Langfuse

  - Collect user feedback ([User Feedback](https://langfuse.com/docs/scores/user-feedback))

  - Manually score LLM outputs in Langfuse ([Manual Scores](https://langfuse.com/docs/scores/manually))

### Test

- **Experiments:** Track and test app behaviour before deploying a new version

  - Datasets let you test expected in and output pairs and benchmark performance before deploying ([Datasets](https://langfuse.com/docs/datasets))

  - Track versions and releases in your application ([Experimentation](https://langfuse.com/docs/experimentation), [Prompt Management](https://langfuse.com/docs/prompts))

## Get started

### Langfuse Cloud

Managed deployment by the Langfuse team, generous free-tier (hobby plan), no credit card required.

**[» Langfuse Cloud](https://cloud.langfuse.com)**

## Self-Hosting Open Source LLM Observability with Langfuse

### Localhost (docker)

```bash

# Clone repository

git clone https://github.com/langfuse/langfuse.git

cd langfuse

# Run server and database

docker compose up -d

```

[→ Learn more about deploying locally](https://langfuse.com/docs/deployment/local)

### Self-host (docker)

Langfuse is simple to self-host and keep updated. It currently requires only a single docker container and a postgres database.

[→ Self Hosting Instructions](https://langfuse.com/docs/deployment/self-host)

Templated deployments: [Railway, GCP, AWS, Azure, Kubernetes and others](https://langfuse.com/docs/deployment/self-host#platform-specific-information)

## Get Started

### API Keys

You need a Langfuse public and secret key to get started. Sign up [here](https://cloud.langfuse.com) and find them in your project settings.

### Ingesting Data · Instrumenting Your Application · LLM Observability with Langfuse

Note: We recommend using our fully async, typed [SDKs](https://langfuse.com/docs/sdk) that allow you to instrument any LLM application with any underlying model. They are available in [Python (Decorators)](https://langfuse.com/docs/sdk/python) & [JS/TS](https://langfuse.com/docs/sdk/typescript). The SDKs will always be the most fully featured and stable way to ingest data into Langfuse.

See the [→ Quickstart](https://langfuse.com/docs/get-started) to integrate Langfuse.

### LLM Observability Integrations

| Integration                                                                  | Supports                   | Description                                                                                                                                      |

| ---------------------------------------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |

| [SDK](https://langfuse.com/docs/sdk)                                         | Python, JS/TS              | Manual instrumentation using the SDKs for full flexibility.                                                                                      |

| [OpenAI](https://langfuse.com/docs/integrations/openai)                      | Python, JS/TS              | Automated instrumentation using drop-in replacement of OpenAI SDK.                                                                               |

| [Langchain](https://langfuse.com/docs/integrations/langchain)                | Python, JS/TS              | Automated instrumentation by passing callback handler to Langchain application.                                                                  |

| [LlamaIndex](https://langfuse.com/docs/integrations/llama-index/get-started) | Python                     | Automated instrumentation via LlamaIndex callback system.                                                                                        |

| [Haystack](https://langfuse.com/docs/integrations/haystack)                  | Python                     | Automated instrumentation via Haystack content tracing system.                                                                                   |

| [LiteLLM](https://langfuse.com/docs/integrations/litellm)                    | Python, JS/TS (proxy only) | Use any LLM as a drop in replacement for GPT. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs). |

| [Vercel AI SDK](https://langfuse.com/docs/integrations/vercel-ai-sdk)        | JS/TS                      | TypeScript toolkit designed to help developers build AI-powered applications with React, Next.js, Vue, Svelte, Node.js.                          |

| [API](https://langfuse.com/docs/api)                                         |                            | Directly call the public API. OpenAPI spec available.                                                                                            |

Packages integrated with Langfuse:

| Name                                                            | Description                                                                                                             |

| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |

| [Instructor](https://langfuse.com/docs/integrations/instructor) | Library to get structured LLM outputs (JSON, Pydantic)                                                                  |

| [Dify](https://langfuse.com/docs/integrations/dify)             | Open source LLM app development platform with no-code builder.                                                          |

| [Ollama](https://langfuse.com/docs/integrations/ollama)         | Easily run open source LLMs on your own machine.                                                                        |

| [Mirascope](https://langfuse.com/docs/integrations/mirascope)   | Python toolkit for building LLM applications.                                                                           |

| [Flowise](https://langfuse.com/docs/integrations/flowise)       | JS/TS no-code builder for customized LLM flows.                                                                         |

| [Langflow](https://langfuse.com/docs/integrations/langflow)     | Python-based UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows. |

## Questions and feedback

### Ideas and roadmap

- [Roadmap](https://langfuse.com/roadmap)

- [GitHub Discussions](https://github.com/orgs/langfuse/discussions)

- [Feature Requests](https://langfuse.com/ideas)

### Support and feedback

In order of preference the best way to communicate with us:

- [GitHub Discussions](https://github.com/orgs/langfuse/discussions) (preferred): Contribute [ideas](https://langfuse.com/ideas), [support requests](https://langfuse.com/gh-support) and [report bugs](https://langfuse.com/issues)

- [Discord](https://langfuse.com/discord): community support

- Privately: contact at langfuse dot com

## Contributing to Langfuse

- Vote on [Ideas](https://github.com/orgs/langfuse/discussions/categories/ideas)

- Raise and comment on [Issues](https://github.com/langfuse/langfuse/issues)

- Open a PR - see [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to setup a development environment.

## License

This repository is MIT licensed, except for the `ee` folders. See [LICENSE](LICENSE) and [docs](https://langfuse.com/docs/open-source) for more details.

## Misc

### GET API to export your data

[**GET routes**](https://langfuse.com/docs/integrations/api) to use data in downstream applications (e.g. embedded analytics). You can also access them conveniently via the SDKs ([docs](https://langfuse.com/docs/query-traces)).

### Security & Privacy

We take data security and privacy seriously. Please refer to our [Security and Privacy](https://langfuse.com/security) page for more information.

### Telemetry

By default, Langfuse automatically reports basic usage statistics of self-hosted instances to a centralized server (PostHog).

This helps us to:

1. Understand how Langfuse is used and improve the most relevant features.

2. Track overall usage for internal and external (e.g. fundraising) reporting.

None of the data is shared with third parties and does not include any sensitive information. We want to be super transparent about this and you can find the exact data we collect [here](/web/src/features/telemetry/index.ts).

You can opt-out by setting `TELEMETRY_ENABLED=false`.

### Star History



 

   

   

   

 



### Open Source Projects Using Langfuse

Top open-source Python projects that use Langfuse, ranked by stars ([Source](https://github.com/langfuse/langfuse-docs/blob/main/components-mdx/dependents)):

| Repository 
| :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 
|    [langgenius](https://github.com/langgenius) / [dify](https://github.com/langgenius/dify) 
|    [open-webui](https://github.com/open-webui) / [open-webui](https://github.com/open-webui/open-webui) 
|    [lobehub](https://github.com/lobehub) / [lobe-chat](https://github.com/lobehub/lobe-chat) 
|    [langflow-ai](https://github.com/langflow-ai) / [langflow](https://github.com/langflow-ai/langflow) 
|    [run-llama](https://github.com/run-llama) / [llama_index](https://github.com/run-llama/llama_index) 
|    [chatchat-space](https://github.com/chatchat-space) / [Langchain-Chatchat](https://github.com/ 
|    [FlowiseAI](https://github.com/FlowiseAI) / [Flowise](https://github.com/FlowiseAI/Flowise) 
|    [mindsdb](https://github.com/mindsdb) / [mindsdb](https://github.com/mindsdb/mindsdb) 
|    [twentyhq](https://github.com/twentyhq) / [twenty](https://github.com/twentyhq/twenty) 
|    [PostHog](https://github.com/PostHog) / [posthog](https://github.com/PostHog/posthog) 
|    [BerriAI](https://github.com/BerriAI) / [litellm](https://github.com/BerriAI/litellm) 
|    [mediar-ai](https://github.com/mediar-ai) / [screenpipe](https://github.com/mediar-ai/screenpipe) 
|    [formbricks](https://github.com/formbricks) / [formbricks](https://github.com/formbricks/formbricks) 
|    [anthropics](https://github.com/anthropics) / [courses](https://github.com/anthropics/courses) 
|    [GreyDGL](https://github.com/GreyDGL) / [PentestGPT](https://github.com/GreyDGL/PentestGPT) 
|    [superagent-ai](https://github.com/superagent-ai) / [superagent](https://github.com/superagent 
|    [promptfoo](https://github.com/promptfoo) / [promptfoo](https://github.com/promptfoo/promptfoo) 
|    [onlook-dev](https://github.com/onlook-dev) / [onlook](https://github.com/onlook-dev/onlook) 
|    [Canner](https://github.com/Canner) / [WrenAI](https://github.com/Canner/WrenAI) 
|    [pingcap](https://github.com/pingcap) / [autoflow](https://github.com/pingcap/autoflow) 
|    [MLSysOps](https://github.com/MLSysOps) / [MLE-agent](https://github.com/MLSysOps/MLE-agent) 
|    [open-webui](https://github.com/open-webui) / [pipelines](https://github.com/open-webui/pipelines) 
|    [alishobeiri](https://github.com/alishobeiri) / [thread](https://github.com/alishobeiri/thread) 
|    [topoteretes](https://github.com/topoteretes) / [cognee](https://github.com/topoteretes/cognee) 
|    [bRAGAI](https://github.com/bRAGAI) / [bRAG-langchain](https://github.com/bRAGAI/bRAG-langchain) 
|    [opslane](https://github.com/opslane) / [opslane](https://github.com/opslane/opslane) 
|    [dynamiq-ai](https://github.com/dynamiq-ai) / [dynamiq](https://github.com/dynamiq-ai/dynamiq) 
|    [theopenconversationkit](https://github.com/theopenconversationkit) / [tock](https://github.com 
|    [andysingal](https://github.com/andysingal) / [llm-course](https://github.com/andysingal/llm-course) 
|    [phospho-app](https://github.com/phospho-app) / [phospho](https://github.com/phospho-app/phospho) 
|    [sentient-engineering](https://github.com/sentient-engineering) / [agent-q](https://github.com 
|    [sql-agi](https://github.com/sql-agi) / [DB-GPT](https://github.com/sql-agi/DB-GPT) 
|    [PostHog](https://github.com/PostHog) / [posthog-foss](https://github.com/PostHog/posthog-foss) 
|    [vespperhq](https://github.com/vespperhq) / [vespper](https://github.com/vespperhq/vespper) 
|    [block](https://github.com/block) / [goose](https://github.com/block/goose) 
|    [aorwall](https://github.com/aorwall) / [moatless-tools](https://github.com/aorwall/moatless-tools) 
|    [dmayboroda](https://github.com/dmayboroda) / [minima](https://github.com/dmayboroda/minima) 
|    [RobotecAI](https://github.com/RobotecAI) / [rai](https://github.com/RobotecAI/rai) 
|    [i-am-alice](https://github.com/i-am-alice) / [3rd-devs](https://github.com/i-am-alice/3rd-devs) 
|    [8090-inc](https://github.com/8090-inc) / [xrx-sample-apps](https://github.com/8090-inc/xrx-sample-apps) 
|    [babelcloud](https://github.com/babelcloud) / [LLM-RGB](https://github.com/babelcloud/LLM-RGB) 
|    [souzatharsis](https://github.com/souzatharsis) / [tamingLLMs](https://github.com/souzatharsis/tamingLLMs) 
|    [LibreChat-AI](https://github.com/LibreChat-AI) / [librechat.ai](https://github.com/LibreChat- 
|    [deepset-ai](https://github.com/deepset-ai) / [haystack-core-integrations](https://github.com/d

| Stars | ------------------------------------- | ----: | | 54865 | | 51531 | | 49003 | | 39093 | | 37368 | chatchat-space/Langchain-Chatchat)    | 32486 | | 32448 | | 26931 | | 24195 | | 22618 | | 15151 | | 11037 | |  9386 | |  8385 | |  7374 | -ai/superagent)                       |  5391 | |  4976 | |  4141 | |  2526 | |  2061 | |  1161 | |  1100 | |  1074 | |   971 | |   823 | |   677 | |   639 | /theopenconversationkit/tock)         |   514 | |   394 | |   384 | /sentient-engineering/agent-q)        |   370 | |   324 | |   305 | |   304 | |   295 | |   291 | |   221 | |   172 | |   148 | |   138 | |   135 | |   129 | AI/librechat.ai)                      |   128 | eepset-ai/haystack-core-integrations) |   126 |

Ecosyste.ms: Awesome

https://github.com/langfuse/langfuse

Awesome Lists containing this project

README

Langfuse: Open Source LLM Engineering Platform

LLM Observability, Prompt Management, LLM Evaluations,Datasets, LLM Metrics, and Prompt Playground

Sign up · Self Host · Demo (live data)

LLM Observability, Prompt Management, LLM Evaluations,
Datasets, LLM Metrics, and Prompt Playground

Sign up
·

Self Host
·

Demo (live data)