{"id":18395852,"url":"https://github.com/deepset-ai/hayhooks","last_synced_at":"2025-04-07T03:35:13.857Z","repository":{"id":214173824,"uuid":"734975513","full_name":"deepset-ai/hayhooks","owner":"deepset-ai","description":"Deploy Haystack pipelines behind a REST Api.","archived":false,"fork":false,"pushed_at":"2024-05-22T08:58:47.000Z","size":116,"stargazers_count":24,"open_issues_count":7,"forks_count":5,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-05-22T15:40:20.116Z","etag":null,"topics":["ai","api-rest","haystack","llm"],"latest_commit_sha":null,"homepage":"https://haystack.deepset.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepset-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-23T08:03:58.000Z","updated_at":"2024-06-17T10:13:12.890Z","dependencies_parsed_at":"2024-03-19T09:29:52.289Z","dependency_job_id":"118c1ce9-e5ca-4f7d-ab51-b5697eaa5138","html_url":"https://github.com/deepset-ai/hayhooks","commit_stats":null,"previous_names":["masci/hayhooks","deepset-ai/hayhooks"],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fhayhooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fhayhooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fhayhooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepset-ai%2Fhayhooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepset-ai","download_url":"https://codeload.github.com/deepset-ai/hayhooks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247589793,"owners_count":20963022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","api-rest","haystack","llm"],"created_at":"2024-11-06T02:12:10.080Z","updated_at":"2025-04-07T03:35:13.848Z","avatar_url":"https://github.com/deepset-ai.png","language":"Python","funding_links":[],"categories":["📚 Projects (2474 total)"],"sub_categories":["MCP Servers"],"readme":"# Hayhooks\n\n**Hayhooks** makes it easy to deploy and serve [Haystack](https://haystack.deepset.ai/) pipelines as REST APIs.\n\nIt provides a simple way to wrap your Haystack pipelines with custom logic and expose them via HTTP endpoints, including OpenAI-compatible chat completion endpoints. With Hayhooks, you can quickly turn your Haystack pipelines into API services with minimal boilerplate code.\n\n[![PyPI - Version](https://img.shields.io/pypi/v/hayhooks.svg)](https://pypi.org/project/hayhooks)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/hayhooks.svg)](https://pypi.org/project/hayhooks)\n[![Docker image release](https://github.com/deepset-ai/hayhooks/actions/workflows/docker.yml/badge.svg)](https://github.com/deepset-ai/hayhooks/actions/workflows/docker.yml)\n[![Tests](https://github.com/deepset-ai/hayhooks/actions/workflows/tests.yml/badge.svg)](https://github.com/deepset-ai/hayhooks/actions/workflows/tests.yml)\n\n**Table of Contents**\n\n- [Quick Start with Docker Compose](#quick-start-with-docker-compose)\n- [Quick Start](#quick-start)\n- [Install the package](#install-the-package)\n- [Configuration](#configuration)\n  - [Environment Variables](#environment-variables)\n  - [CORS Settings](#cors-settings)\n- [CLI Commands](#cli-commands)\n- [Start hayhooks](#start-hayhooks)\n- [Deploy a pipeline](#deploy-a-pipeline)\n  - [PipelineWrapper](#why-a-pipeline-wrapper)\n  - [Setup Method](#setup)\n  - [Run API Method](#run_api)\n  - [PipelineWrapper development with `overwrite` option](#pipelinewrapper-development-with-overwrite-option)\n  - [Additional Dependencies](#additional-dependencies)\n- [Support file uploads](#support-file-uploads)\n- [Run pipelines from the CLI](#run-pipelines-from-the-cli)\n  - [Run a pipeline from the CLI JSON-compatible parameters](#run-a-pipeline-from-the-cli-json-compatible-parameters)\n  - [Run a pipeline from the CLI uploading files](#run-a-pipeline-from-the-cli-uploading-files)\n- [MCP support](#mcp-support)\n  - [MCP Server](#mcp-server)\n  - [Create a PipelineWrapper for exposing a Haystack pipeline as a MCP Tool](#create-a-pipelinewrapper-for-exposing-a-haystack-pipeline-as-a-mcp-tool)\n- [OpenAI Compatibility](#openai-compatibility)\n  - [OpenAI-compatible endpoints generation](#openai-compatible-endpoints-generation)\n  - [Using Hayhooks as `open-webui` backend](#using-hayhooks-as-open-webui-backend)\n  - [Run Chat Completion Method](#run_chat_completion)\n  - [Streaming Responses](#streaming-responses-in-openai-compatible-endpoints)\n  - [Integration with haystack OpenAIChatGenerator](#integration-with-haystack-openaichatgenerator)\n- [Advanced Usage](#advanced-usage)\n  - [Run Hayhooks Programmatically](#run-hayhooks-programmatically)\n- [Deployment Guidelines](#deployment-guidelines)\n- [Legacy Features](#legacy-features)\n  - [Deploy Pipeline Using YAML](#deploy-a-pipeline-using-only-its-yaml-definition)\n- [License](#license)\n\n## Quick start with Docker Compose\n\nTo quickly get started with Hayhooks, we provide a ready-to-use Docker Compose 🐳 setup with pre-configured integration with [open-webui](https://openwebui.com/).\n\nIt's available [here](https://github.com/deepset-ai/hayhooks-open-webui-docker-compose).\n\n## Quick start\n\n### Install the package\n\nStart by installing the package:\n\n```shell\npip install hayhooks\n```\n\nIf you want to use the [MCP server](#mcp-server), you need to install the `hayhooks[mcp]` package:\n\n```shell\npip install hayhooks[mcp]\n```\n\n**NOTE: You'll need to run at least Python 3.10+ to use the MCP server.**\n\n### Configuration\n\nCurrently, you can configure Hayhooks by:\n\n- Set the environment variables in an `.env` file in the root of your project.\n- Pass the supported arguments and options to `hayhooks run` command.\n- Pass the environment variables to the `hayhooks` command.\n\n#### Environment variables\n\nThe following environment variables are supported:\n\n- `HAYHOOKS_HOST`: The host on which the server will listen.\n- `HAYHOOKS_PORT`: The port on which the server will listen.\n- `HAYHOOKS_MCP_PORT`: The port on which the MCP server will listen.\n- `HAYHOOKS_MCP_HOST`: The host on which the MCP server will listen.\n- `HAYHOOKS_PIPELINES_DIR`: The path to the directory containing the pipelines.\n- `HAYHOOKS_ROOT_PATH`: The root path of the server.\n- `HAYHOOKS_ADDITIONAL_PYTHONPATH`: Additional Python path to be added to the Python path.\n- `HAYHOOKS_DISABLE_SSL`: Boolean flag to disable SSL verification when making requests from the CLI.\n- `HAYHOOKS_SHOW_TRACEBACKS`: Boolean flag to show tracebacks on errors during pipeline execution and deployment.\n\n##### CORS Settings\n\n- `HAYHOOKS_CORS_ALLOW_ORIGINS`: List of allowed origins (default: [\"*\"])\n- `HAYHOOKS_CORS_ALLOW_METHODS`: List of allowed HTTP methods (default: [\"*\"])\n- `HAYHOOKS_CORS_ALLOW_HEADERS`: List of allowed headers (default: [\"*\"])\n- `HAYHOOKS_CORS_ALLOW_CREDENTIALS`: Allow credentials (default: false)\n- `HAYHOOKS_CORS_ALLOW_ORIGIN_REGEX`: Regex pattern for allowed origins (default: null)\n- `HAYHOOKS_CORS_EXPOSE_HEADERS`: Headers to expose in response (default: [])\n- `HAYHOOKS_CORS_MAX_AGE`: Maxium age for CORS preflight responses in seconds (default: 600)\n\n### CLI commands\n\nThe `hayhooks` package provides a CLI to manage the server and the pipelines.\nAny command can be run with `hayhooks \u003ccommand\u003e --help` to get more information.\n\nCLI commands are basically wrappers around the HTTP API of the server. The full API reference is available at [//HAYHOOKS_HOST:HAYHOOKS_PORT/docs](http://HAYHOOKS_HOST:HAYHOOKS_PORT/docs) or [//HAYHOOKS_HOST:HAYHOOKS_PORT/redoc](http://HAYHOOKS_HOST:HAYHOOKS_PORT/redoc).\n\n```shell\nhayhooks run     # Start the server\nhayhooks status  # Check the status of the server and show deployed pipelines\n\nhayhooks pipeline deploy-files \u003cpath_to_dir\u003e   # Deploy a pipeline using PipelineWrapper\nhayhooks pipeline deploy \u003cpipeline_name\u003e       # Deploy a pipeline from a YAML file\nhayhooks pipeline undeploy \u003cpipeline_name\u003e     # Undeploy a pipeline\nhayhooks pipeline run \u003cpipeline_name\u003e          # Run a pipeline\n```\n\n### Start Hayhooks\n\nLet's start Hayhooks:\n\n```shell\nhayhooks run\n```\n\nThis will start the Hayhooks server on `HAYHOOKS_HOST:HAYHOOKS_PORT`.\n\n### Deploy a pipeline\n\nNow, we will deploy a pipeline to chat with a website. We have created an example in the [examples/chat_with_website_streaming](examples/chat_with_website_streaming) folder.\n\nIn the example folder, we have two files:\n\n- `chat_with_website.yml`: The pipeline definition in YAML format.\n- `pipeline_wrapper.py` (mandatory): A pipeline wrapper that uses the pipeline definition.\n\n#### Why a pipeline wrapper?\n\nThe pipeline wrapper provides a flexible foundation for deploying Haystack pipelines by allowing users to:\n\n- Choose their preferred pipeline initialization method (YAML files, Haystack templates, or inline code)\n- Define custom pipeline execution logic with configurable inputs and outputs\n- Optionally expose OpenAI-compatible chat endpoints with streaming support for integration with interfaces like [open-webui](https://openwebui.com/)\n\nThe `pipeline_wrapper.py` file must contain an implementation of the `BasePipelineWrapper` class (see [here](src/hayhooks/server/utils/base_pipeline_wrapper.py) for more details).\n\nA minimal `PipelineWrapper` looks like this:\n\n```python\nfrom pathlib import Path\nfrom typing import List\nfrom haystack import Pipeline\nfrom hayhooks import BasePipelineWrapper\n\nclass PipelineWrapper(BasePipelineWrapper):\n    def setup(self) -\u003e None:\n        pipeline_yaml = (Path(__file__).parent / \"chat_with_website.yml\").read_text()\n        self.pipeline = Pipeline.loads(pipeline_yaml)\n\n    def run_api(self, urls: List[str], question: str) -\u003e str:\n        result = self.pipeline.run({\"fetcher\": {\"urls\": urls}, \"prompt\": {\"query\": question}})\n        return result[\"llm\"][\"replies\"][0]\n```\n\nIt contains two methods:\n\n#### setup()\n\nThis method will be called when the pipeline is deployed. It should initialize the `self.pipeline` attribute as a Haystack pipeline.\n\nYou can initialize the pipeline in many ways:\n\n- Load it from a YAML file.\n- Define it inline as a Haystack pipeline code.\n- Load it from a [Haystack pipeline template](https://docs.haystack.deepset.ai/docs/pipeline-templates).\n\n#### run_api(...)\n\nThis method will be used to run the pipeline in API mode, when you call the `{pipeline_name}/run` endpoint.\n\n**You can define the input arguments of the method according to your needs**.\n\n```python\ndef run_api(self, urls: List[str], question: str, any_other_user_defined_argument: Any) -\u003e str:\n    ...\n```\n\nThe input arguments will be used to generate a Pydantic model that will be used to validate the request body. The same will be done for the response type.\n\n**NOTE**: Since Hayhooks will _dynamically_ create the Pydantic models, you need to make sure that the input arguments are JSON-serializable.\n\nTo deploy the pipeline, run:\n\n```shell\nhayhooks pipeline deploy-files -n chat_with_website examples/chat_with_website\n```\n\nThis will deploy the pipeline with the name `chat_with_website`. Any error encountered during development will be printed to the console and show in the server logs.\n\n#### PipelineWrapper development with `overwrite` option\n\nDuring development, you can use the `--overwrite` flag to redeploy your pipeline without restarting the Hayhooks server:\n\n```shell\nhayhooks pipeline deploy-files -n {pipeline_name} --overwrite {pipeline_dir}\n```\n\nThis is particularly useful when:\n\n- Iterating on your pipeline wrapper implementation\n- Debugging pipeline setup issues\n- Testing different pipeline configurations\n\nThe `--overwrite` flag will:\n\n1. Remove the existing pipeline from the registry\n2. Delete the pipeline files from disk\n3. Deploy the new version of your pipeline\n\nFor even faster development iterations, you can combine `--overwrite` with `--skip-saving-files` to avoid writing files to disk:\n\n```shell\nhayhooks pipeline deploy-files -n {pipeline_name} --overwrite --skip-saving-files {pipeline_dir}\n```\n\nThis is useful when:\n\n- You're making frequent changes during development\n- You want to test a pipeline without persisting it\n- You're running in an environment with limited disk access\n\n#### Additional dependencies\n\nAfter installing the Hayhooks package, it might happen that during pipeline deployment you need to install additional dependencies in order to correctly initialize the pipeline instance when calling the wrapper's `setup()` method. For instance, the `chat_with_website` pipeline requires the `trafilatura` package, which is **not installed by default**.\n\n⚠️ Sometimes you may need to enable tracebacks in hayhooks to see the full error message. You can do this by setting the `HAYHOOKS_SHOW_TRACEBACKS` environment variable to `true` or `1`.\n\nThen, assuming you've installed the Hayhooks package in a virtual environment, you will need to install the additional required dependencies yourself by running:\n\n```shell\npip install trafilatura\n```\n\n## Support file uploads\n\nHayhooks can easily handle uploaded files in your pipeline wrapper `run_api` method by adding `files: Optional[List[UploadFile]] = None` as an argument.\n\nHere's a simple example:\n\n```python\ndef run_api(self, files: Optional[List[UploadFile]] = None) -\u003e str:\n    if files and len(files) \u003e 0:\n        filenames = [f.filename for f in files if f.filename is not None]\n        file_contents = [f.file.read() for f in files]\n\n        return f\"Received files: {', '.join(filenames)}\"\n\n    return \"No files received\"\n```\n\nThis will make Hayhooks handle automatically the file uploads (if they are present) and pass them to the `run_api` method.\nThis also means that the HTTP request **needs to be a `multipart/form-data` request**.\n\nNote also that you can handle **both files and parameters in the same request**, simply adding them as arguments to the `run_api` method.\n\n```python\ndef run_api(self, files: Optional[List[UploadFile]] = None, additional_param: str = \"default\") -\u003e str:\n    ...\n```\n\nYou can find a full example in the [examples/rag_indexing_query](examples/rag_indexing_query) folder.\n\n## Run pipelines from the CLI\n\n### Run a pipeline from the CLI JSON-compatible parameters\n\nYou can run a pipeline by using the `hayhooks pipeline run` command. Under the hood, this will call the `run_api` method of the pipeline wrapper, passing parameters as the JSON body of the request.\nThis is convenient when you want to do a test run of the deployed pipeline from the CLI without having to write any code.\n\nTo run a pipeline from the CLI, you can use the following command:\n\n```shell\nhayhooks pipeline run \u003cpipeline_name\u003e --param 'question=\"is this recipe vegan?\"'\n```\n\n### Run a pipeline from the CLI uploading files\n\nThis is useful when you want to run a pipeline that requires a file as input. In that case, the request will be a `multipart/form-data` request. You can pass both files and parameters in the same request.\n\n**NOTE**: To use this feature, you need to deploy a pipeline which is handling files (see [Support file uploads](#support-file-uploads) and [examples/rag_indexing_query](examples/rag_indexing_query) for more details).\n\n```shell\n# Upload a whole directory\nhayhooks pipeline run \u003cpipeline_name\u003e --dir files_to_index\n\n# Upload a single file\nhayhooks pipeline run \u003cpipeline_name\u003e --file file.pdf\n\n# Upload multiple files\nhayhooks pipeline run \u003cpipeline_name\u003e --dir files_to_index --file file1.pdf --file file2.pdf\n\n# Upload a single file passing also a parameter\nhayhooks pipeline run \u003cpipeline_name\u003e --file file.pdf --param 'question=\"is this recipe vegan?\"'\n```\n\n## MCP support\n\n**NOTE: You'll need to run at least Python 3.10+ to use the MCP server.**\n\n### MCP Server\n\nHayhooks now supports the [Model Context Protocol](https://modelcontextprotocol.io/) and can act as a [MCP Server](https://modelcontextprotocol.io/docs/concepts/architecture).\n\nIt will automatically list the deployed pipelines as [MCP Tools](https://modelcontextprotocol.io/docs/concepts/tools), using [Server-Sent Events (SSE)](https://modelcontextprotocol.io/docs/concepts/transports#server-sent-events-sse) as **MCP Transport**.\n\nTo run the Hayhooks MCP server, you can use the following command:\n\n```shell\nhayhooks mcp run\n```\n\nThis will start the Hayhooks MCP server on `HAYHOOKS_MCP_HOST:HAYHOOKS_MCP_PORT`.\n\n### Create a PipelineWrapper for exposing a Haystack pipeline as a MCP Tool\n\nA [MCP Tool](https://modelcontextprotocol.io/docs/concepts/tools) requires the following properties:\n\n- `name`: The name of the tool.\n- `description`: The description of the tool.\n- `inputSchema`: A JSON Schema object describing the tool's input parameters.\n\nFor each deployed pipeline, Hayhooks will:\n\n- Use the pipeline wrapper `name` as MCP Tool `name` (always present).\n- Use the pipeline wrapper **`run_api` method docstring** as MCP Tool `description` (if present).\n- Generate a Pydantic model from the `inputSchema` using the **`run_api` method arguments as fields**.\n\nHere's an example of a PipelineWrapper implementation for the `chat_with_website` pipeline which can be used as a MCP Tool:\n\n```python\nfrom pathlib import Path\nfrom typing import List\nfrom haystack import Pipeline\nfrom hayhooks import BasePipelineWrapper\n\n\nclass PipelineWrapper(BasePipelineWrapper):\n    def setup(self) -\u003e None:\n        pipeline_yaml = (Path(__file__).parent / \"chat_with_website.yml\").read_text()\n        self.pipeline = Pipeline.loads(pipeline_yaml)\n\n    def run_api(self, urls: List[str], question: str) -\u003e str:\n        #\n        # NOTE: The following docstring will be used as MCP Tool description\n        #\n        \"\"\"\n        Ask a question about one or more websites using a Haystack pipeline.\n        \"\"\"\n        result = self.pipeline.run({\"fetcher\": {\"urls\": urls}, \"prompt\": {\"query\": question}})\n        return result[\"llm\"][\"replies\"][0]\n```\n\n### Skip MCP Tool listing\n\nYou can skip the MCP Tool listing by setting the `skip_mcp` class attribute to `True` in your PipelineWrapper class.\nThis way, the pipeline will be deployed on Hayhooks but **will not be listed as a MCP Tool** when you run the `hayhooks mcp run` command.\n\n```python\nclass PipelineWrapper(BasePipelineWrapper):\n    # This will skip the MCP Tool listing\n    skip_mcp = True\n\n    def setup(self) -\u003e None:\n        ...\n\n    def run_api(self, urls: List[str], question: str) -\u003e str:\n        ...\n```\n\n## OpenAI compatibility\n\n### OpenAI-compatible endpoints generation\n\nHayhooks now can automatically generate OpenAI-compatible endpoints if you implement the `run_chat_completion` method in your pipeline wrapper.\n\nThis will make Hayhooks compatible with fully-featured chat interfaces like [open-webui](https://openwebui.com/), so you can use it as a backend for your chat interface.\n\n### Using Hayhooks as `open-webui` backend\n\nRequirements:\n\n- Ensure you have [open-webui](https://openwebui.com/) up and running (you can do it easily using `docker`, check [their quick start guide](https://docs.openwebui.com/getting-started/quick-start)).\n- Ensure you have Hayhooks server running somewhere. We will run it locally on `http://localhost:1416`.\n\n#### Configuring `open-webui`\n\nFirst, you need to **turn off `tags` and `title` generation from `Admin settings -\u003e Interface`**:\n\n![open-webui-settings](./docs/assets/open-webui-settings.png)\n\nThen you have two options to connect Hayhooks as a backend.\n\nAdd a **Direct Connection** from `Settings -\u003e Connections`:\n\nNOTE: **Fill a random value as API key as it's not needed**\n\n![open-webui-settings-connections](./docs/assets/open-webui-settings-connections.png)\n\nAlternatively, you can add an additional **OpenAI API Connections** from `Admin settings -\u003e Connections`:\n\n![open-webui-admin-settings-connections](./docs/assets/open-webui-admin-settings-connections.png)\n\nEven in this case, remember to **Fill a random value as API key**.\n\n#### run_chat_completion(...)\n\nTo enable the automatic generation of OpenAI-compatible endpoints, you need only to implement the `run_chat_completion` method in your pipeline wrapper.\n\n```python\ndef run_chat_completion(self, model: str, messages: List[dict], body: dict) -\u003e Union[str, Generator]:\n    ...\n```\n\nLet's update the previous example to add a streaming response:\n\n```python\nfrom pathlib import Path\nfrom typing import Generator, List, Union\nfrom haystack import Pipeline\nfrom hayhooks import get_last_user_message, BasePipelineWrapper, log\n\n\nURLS = [\"https://haystack.deepset.ai\", \"https://www.redis.io\", \"https://ssi.inc\"]\n\n\nclass PipelineWrapper(BasePipelineWrapper):\n    def setup(self) -\u003e None:\n        ...  # Same as before\n\n    def run_api(self, urls: List[str], question: str) -\u003e str:\n        ...  # Same as before\n\n    def run_chat_completion(self, model: str, messages: List[dict], body: dict) -\u003e Union[str, Generator]:\n        log.trace(f\"Running pipeline with model: {model}, messages: {messages}, body: {body}\")\n\n        question = get_last_user_message(messages)\n        log.trace(f\"Question: {question}\")\n\n        # Plain pipeline run, will return a string\n        result = self.pipeline.run({\"fetcher\": {\"urls\": URLS}, \"prompt\": {\"query\": question}})\n        return result[\"llm\"][\"replies\"][0]\n```\n\nDifferently from the `run_api` method, the `run_chat_completion` has a **fixed signature** and will be called with the arguments specified in the OpenAI-compatible endpoint.\n\n- `model`: The `name` of the Haystack pipeline which is called.\n- `messages`: The list of messages from the chat in the OpenAI format.\n- `body`: The full body of the request.\n\nSome notes:\n\n- Since we have only the user messages as input here, the `question` is extracted from the last user message and the `urls` argument is hardcoded.\n- In this example, the `run_chat_completion` method is returning a string, so the `open-webui` will receive a string as response and show the pipeline output in the chat all at once.\n- The `body` argument contains the full request body, which may be used to extract more information like the `temperature` or the `max_tokens` (see the [OpenAI API reference](https://platform.openai.com/docs/api-reference/chat/create) for more information).\n\nFinally, to use non-streaming responses in `open-webui` you need also to turn of `Stream Chat Response` chat settings.\n\nHere's a video example:\n\n![chat-completion-example](./docs/assets/chat-completion.gif)\n\n### Streaming responses in OpenAI-compatible endpoints\n\nHayhooks now provides a `streaming_generator` utility function that can be used to stream the pipeline output to the client.\n\nLet's update the `run_chat_completion` method of the previous example:\n\n```python\nfrom pathlib import Path\nfrom typing import Generator, List, Union\nfrom haystack import Pipeline\nfrom hayhooks import get_last_user_message, BasePipelineWrapper, log, streaming_generator\n\n\nURLS = [\"https://haystack.deepset.ai\", \"https://www.redis.io\", \"https://ssi.inc\"]\n\n\nclass PipelineWrapper(BasePipelineWrapper):\n    def setup(self) -\u003e None:\n        ...  # Same as before\n\n    def run_api(self, urls: List[str], question: str) -\u003e str:\n        ...  # Same as before\n\n    def run_chat_completion(self, model: str, messages: List[dict], body: dict) -\u003e Union[str, Generator]:\n        log.trace(f\"Running pipeline with model: {model}, messages: {messages}, body: {body}\")\n\n        question = get_last_user_message(messages)\n        log.trace(f\"Question: {question}\")\n\n        # Streaming pipeline run, will return a generator\n        return streaming_generator(\n            pipeline=self.pipeline,\n            pipeline_run_args={\"fetcher\": {\"urls\": URLS}, \"prompt\": {\"query\": question}},\n        )\n```\n\nNow, if you run the pipeline and call one of the following endpoints:\n\n- `{pipeline_name}/chat`\n- `/chat/completions`\n- `/v1/chat/completions`\n\nYou will see the pipeline output being streamed [in OpenAI-compatible format](https://platform.openai.com/docs/api-reference/chat/streaming) to the client and you'll be able to see the output in chunks.\n\nSince output will be streamed to `open-webui` there's **no need to change `Stream Chat Response`** chat setting (leave it as `Default` or `On`).\n\nHere's a video example:\n\n![chat-completion-streaming-example](./docs/assets/chat-completion-streaming.gif)\n\n### Integration with haystack OpenAIChatGenerator\n\nSince Hayhooks is OpenAI-compatible, it can be used as a backend for the [haystack OpenAIChatGenerator](https://docs.haystack.deepset.ai/docs/openaichatgenerator).\n\nAssuming you have a Haystack pipeline named `chat_with_website_streaming` and you have deployed it using Hayhooks, here's an example script of how to use it with the `OpenAIChatGenerator`:\n\n```python\nfrom haystack.components.generators.chat.openai import OpenAIChatGenerator\nfrom haystack.utils import Secret\nfrom haystack.dataclasses import ChatMessage\nfrom haystack.components.generators.utils import print_streaming_chunk\n\nclient = OpenAIChatGenerator(\n    model=\"chat_with_website_streaming\",\n    api_key=Secret.from_token(\"not-relevant\"),  # This is not used, you can set it to anything\n    api_base_url=\"http://localhost:1416/v1/\",\n    streaming_callback=print_streaming_chunk,\n)\n\nclient.run([ChatMessage.from_user(\"Where are the offices or SSI?\")])\n# \u003e The offices of Safe Superintelligence Inc. (SSI) are located in Palo Alto, California, and Tel Aviv, Israel.\n\n# \u003e {'replies': [ChatMessage(_role=\u003cChatRole.ASSISTANT: 'assistant'\u003e, _content=[TextContent(text='The offices of Safe \u003eSuperintelligence Inc. (SSI) are located in Palo Alto, California, and Tel Aviv, Israel.')], _name=None, _meta={'model': \u003e'chat_with_website_streaming', 'index': 0, 'finish_reason': 'stop', 'completion_start_time': '2025-02-11T15:31:44.599726', \u003e'usage': {}})]}\n```\n\n## Advanced usage\n\n### Run Hayhooks programmatically\n\nA Hayhooks app instance can be run programmatically created by using the `create_app` function. This is useful if you want to add custom routes or middleware to Hayhooks.\n\nHere's an example script:\n\n```python\nimport uvicorn\nfrom hayhooks.settings import settings\nfrom fastapi import Request\nfrom hayhooks import create_app\n\n# Create the Hayhooks app\nhayhooks = create_app()\n\n\n# Add a custom route\n@hayhooks.get(\"/custom\")\nasync def custom_route():\n    return {\"message\": \"Hi, this is a custom route!\"}\n\n\n# Add a custom middleware\n@hayhooks.middleware(\"http\")\nasync def custom_middleware(request: Request, call_next):\n    response = await call_next(request)\n    response.headers[\"X-Custom-Header\"] = \"custom-header-value\"\n    return response\n\n\nif __name__ == \"__main__\":\n    uvicorn.run(\"app:hayhooks\", host=settings.host, port=settings.port)\n```\n\n### Deployment guidelines\n\n📦 For detailed deployment guidelines, see [deployment_guidelines.md](docs/deployment_guidelines.md).\n\n### Legacy Features\n\n#### Deploy a pipeline using only its YAML definition\n\n**⚠️ This way of deployment is not maintained anymore and will be deprecated in the future**.\n\nWe're still supporting the Hayhooks _former_ way to deploy a pipeline.\n\nThe former command `hayhooks deploy` is now changed to `hayhooks pipeline deploy` and can be used to deploy a pipeline only from a YAML definition file.\n\nFor example:\n\n```shell\nhayhooks pipeline deploy -n chat_with_website examples/chat_with_website/chat_with_website.yml\n```\n\nThis will deploy the pipeline with the name `chat_with_website` from the YAML definition file `examples/chat_with_website/chat_with_website.yml`. You then can check the generated docs at `http://HAYHOOKS_HOST:HAYHOOKS_PORT/docs` or `http://HAYHOOKS_HOST:HAYHOOKS_PORT/redoc`, looking at the `POST /chat_with_website` endpoint.\n\n### License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepset-ai%2Fhayhooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepset-ai%2Fhayhooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepset-ai%2Fhayhooks/lists"}