{"id":13532972,"url":"https://github.com/hegelai/prompttools","last_synced_at":"2025-05-14T14:07:52.968Z","repository":{"id":179220903,"uuid":"658453192","full_name":"hegelai/prompttools","owner":"hegelai","description":"Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).","archived":false,"fork":false,"pushed_at":"2024-08-15T22:20:40.000Z","size":34054,"stargazers_count":2848,"open_issues_count":41,"forks_count":238,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-05-10T17:48:26.368Z","etag":null,"topics":["deep-learning","developer-tools","embeddings","large-language-models","llms","machine-learning","prompt-engineering","python","vector-search"],"latest_commit_sha":null,"homepage":"http://prompttools.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hegelai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-25T19:33:00.000Z","updated_at":"2025-05-10T02:26:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"d5d09dbe-e3f4-4a4e-ad84-8be139d9a5c2","html_url":"https://github.com/hegelai/prompttools","commit_stats":null,"previous_names":["hegelai/prompttools"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hegelai%2Fprompttools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hegelai%2Fprompttools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hegelai%2Fprompttools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hegelai%2Fprompttools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hegelai","download_url":"https://codeload.github.com/hegelai/prompttools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159194,"owners_count":22024558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","developer-tools","embeddings","large-language-models","llms","machine-learning","prompt-engineering","python","vector-search"],"created_at":"2024-08-01T07:01:15.495Z","updated_at":"2025-05-14T14:07:52.947Z","avatar_url":"https://github.com/hegelai.png","language":"Python","funding_links":[],"categories":["LLMOps","Prompting libraries \u0026 tools","LLM应用","Python","A01_文本生成_文本对话","Observability","LLM Applications","Chatbots \u0026 Virtual Companions","HarmonyOS","Repos","python","✅ Evaluation, Testing \u0026 Benchmarking","Prompting libraries \u0026 tools (in alphabetical order)","4. Agentic AI \u0026 Multi-Agent Systems","*Ops for AI"],"sub_categories":["LLM Evaluation \u0026 Testing","LLM 评估工具","大语言对话模型及数据","Popular-LLM","Windows Manager","Observability","LLMOps"],"readme":"\u003ch1 align=\"center\"\u003e\n \u003ca href=\"https://hegel-ai.com\"\u003e\n \u003cpicture\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"img/hegel_ai_logo_dark.svg\"\u003e\n  \u003cimg height=\"70\" src=\"img/hegel_ai_logo.svg\"\u003e\n \u003c/picture\u003e\n \u003c/a\u003e\n \u003cbr\u003e\n PromptTools\n\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n:wrench: Test and experiment with prompts, LLMs, and vector databases. :hammer:\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"http://prompttools.readthedocs.io/\"\u003e\u003cimg src=\"https://img.shields.io/badge/View%20Documentation-Docs-yellow\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://discord.gg/7KeRPNHGdJ\"\u003e\u003cimg src=\"https://img.shields.io/badge/Join%20our%20community-Discord-blue\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pepy.tech/project/prompttools\" target=\"_blank\"\u003e\u003cimg src=\"https://static.pepy.tech/badge/prompttools\" alt=\"Total Downloads\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/hegelai/prompttools\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/hegelai/prompttools\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://twitter.com/hegel_ai\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/Hegel_AI?style=social\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\nWelcome to `prompttools` created by [Hegel AI](https://hegel-ai.com/)! This repo offers a set of open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. The core idea is to enable developers to evaluate using familiar interfaces like _code_, _notebooks_, and a local _playground_.\n\nIn just a few lines of code, you can test your prompts and parameters across different models (whether you are using\nOpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases.\n\n```python\nfrom prompttools.experiment import OpenAIChatExperiment\n\nmessages = [\n    [{\"role\": \"user\", \"content\": \"Tell me a joke.\"},],\n    [{\"role\": \"user\", \"content\": \"Is 17077 a prime number?\"},],\n]\n\nmodels = [\"gpt-3.5-turbo\", \"gpt-4\"]\ntemperatures = [0.0]\nopenai_experiment = OpenAIChatExperiment(models, messages, temperature=temperatures)\nopenai_experiment.run()\nopenai_experiment.visualize()\n```\n\n\n![image](img/demo.gif)\n\nTo stay in touch with us about issues and future updates, join the [Discord](https://discord.gg/7KeRPNHGdJ).\n\n## Quickstart\n\nTo install `prompttools`, you can use `pip`:\n\n```\npip install prompttools\n```\n\nYou can run a simple example of a `prompttools` locally with the following\n\n```\ngit clone https://github.com/hegelai/prompttools.git\ncd prompttools \u0026\u0026 jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynb\n```\n\nYou can also run the notebook in [Google Colab](https://colab.research.google.com/drive/1YVcpBew8EqbhXFN8P5NaFrOIqc1FKWeS?usp=sharing)\n\n## Playground\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"img/playground.gif\" width=\"1000\" height=\"500\"\u003e\n\u003c/p\u003e\n\nIf you want to interact with `prompttools` using our playground interface, you can launch it with the following commands.\n\nYou can run a simple example of a `prompttools` locally with the following\n\n```\npip install notebook  # If jupyter notebook has not been installed\npip install prompttools\n```\n\nThen, clone the git repo and launch the streamlit app:\n\n```\ngit clone https://github.com/hegelai/prompttools.git\ncd prompttools \u0026\u0026 streamlit run prompttools/playground/playground.py\n```\n\nYou can also access a hosted version of the playground on the [Streamlit Community Cloud](https://prompttools.streamlit.app/).\n\n\u003e Note: The hosted version does not support LlamaCpp\n\n## Documentation\n\nOur [documentation website](https://prompttools.readthedocs.io/en/latest/index.html) contains the full API reference\nand more description of individual components. Check it out!\n\n## Supported Integrations\n\nHere is a list of APIs that we support with our experiments:\n\nLLMs\n- OpenAI (Completion, ChatCompletion, Fine-tuned models) - **Supported**\n- LLaMA.Cpp (LLaMA 1, LLaMA 2) - **Supported**\n- HuggingFace (Hub API, Inference Endpoints) - **Supported**\n- Anthropic - **Supported**\n- Mistral AI - **Supported**\n- Google Gemini - **Supported**\n- Google PaLM (legacy) - **Supported**\n- Google Vertex AI - **Supported**\n- Azure OpenAI Service - **Supported**\n- Replicate - **Supported**\n- Ollama - _In Progress_\n\nVector Databases and Data Utility\n- Chroma - **Supported**\n- Weaviate - **Supported**\n- Qdrant - **Supported**\n- LanceDB - **Supported**\n- Milvus - Exploratory\n- Pinecone - **Supported**\n- Epsilla - _In Progress_\n\nFrameworks\n- LangChain - **Supported**\n- MindsDB - **Supported**\n- LlamaIndex - Exploratory\n\nComputer Vision\n- Stable Diffusion - **Supported**\n- Replicate's hosted Stable Diffusion - **Supported**\n\nIf you have any API that you'd like to see being supported soon, please open an issue or\na PR to add it. Feel free to discuss in our Discord channel as well.\n\n## Frequently Asked Questions (FAQs)\n\n1. Will this library forward my LLM calls to a server before sending it to OpenAI, Anthropic, and etc.?\n    - No, the source code will be executed on your machine. Any call to LLM APIs will be directly executed from your machine without any forwarding.\n\n2. Does `prompttools` store my API keys or LLM inputs and outputs to a server?\n    - No, all of those data stay on your local machine. We do not collect any PII (personally identifiable information).\n\n3. How do I persist my results?\n   -  To persist the results of your tests and experiments, you can export your `Experiment` with the methods `to_csv`,\n      `to_json`, `to_lora_json`, or `to_mongo_db`. We are building more persistence features and we will be happy to further discuss your use cases, pain points, and what export\n      options may be useful for you.\n\n## Sentry\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eUsage Tracking\u003c/b\u003e\u003c/summary\u003e\n\nSince we are changing our API rapidly, there are some errors caused by our negligence or out of date documentation.\nTo improve user experience, we collect data from normal package usage that helps us understand the\nerrors that are raised. This data is collected and sent to [Sentry](https://sentry.io/),\na third-party error tracking service, commonly used in open-source softwares. It only logs this library's own actions.\n\nYou can easily opt-out by defining an environment variable called `SENTRY_OPT_OUT`.\n\n\u003c/details\u003e\n\n## Contributing\n\nWe welcome PRs and suggestions! Don't hesitate to open a PR/issue or to reach out to us [via email](mailto:team@hegel-ai.com).\nPlease have a look at our [contribution guide](CONTRIBUTING.md) and\n[\"Help Wanted\" issues](https://github.com/hegelai/prompttools/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) to get started!\n\n## Usage and Feedback\n\nWe will be delighted to work with early adopters to shape our designs. Please reach out to us [via email](mailto:team@hegel-ai.com) if you're\ninterested in using this tooling for your project or have any feedback.\n\n## License\n\nWe will be gradually releasing more components to the open-source community. The current license can be found in the  [LICENSE](LICENSE) file. If there is any concern, please [contact us](mailto:eam@hegel-ai.com) and we will be happy to work with you.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhegelai%2Fprompttools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhegelai%2Fprompttools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhegelai%2Fprompttools/lists"}