{"id":13418389,"url":"https://github.com/ucbrise/clipper","last_synced_at":"2025-05-15T07:06:44.577Z","repository":{"id":41173671,"uuid":"72132009","full_name":"ucbrise/clipper","owner":"ucbrise","description":"A low-latency prediction-serving system","archived":false,"fork":false,"pushed_at":"2021-04-26T17:27:01.000Z","size":24861,"stargazers_count":1413,"open_issues_count":120,"forks_count":279,"subscribers_count":84,"default_branch":"develop","last_synced_at":"2025-04-14T13:04:16.897Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://clipper.ai","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ucbrise.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-27T17:30:27.000Z","updated_at":"2025-04-14T12:49:12.000Z","dependencies_parsed_at":"2022-08-30T22:12:32.745Z","dependency_job_id":null,"html_url":"https://github.com/ucbrise/clipper","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fclipper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fclipper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fclipper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fclipper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ucbrise","download_url":"https://codeload.github.com/ucbrise/clipper/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254292042,"owners_count":22046426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T22:01:01.714Z","updated_at":"2025-05-15T07:06:39.535Z","avatar_url":"https://github.com/ucbrise.png","language":"C++","readme":"# Clipper\n\n[![Build Status](https://amplab.cs.berkeley.edu/jenkins/buildStatus/icon?job=Clipper)](https://amplab.cs.berkeley.edu/jenkins/job/Clipper/) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n\n\u003cimg src=\"images/clipper-logo.png\" width=\"200\"\u003e\n\n## Note: Clipper is not actively maintained currently. It is available as a research artifact.\n\n## What is Clipper?\n\nClipper is a prediction serving system that sits between user-facing applications and a wide range of commonly used machine learning models and frameworks. Learn more about Clipper and view documentation at our website \u003chttp://clipper.ai\u003e.\n\n## What does Clipper do?\n\n* Clipper **simplifies integration of machine learning techniques** into user facing applications by providing a simple standard REST interface for prediction and feedback across a wide range of commonly used machine learning frameworks.  *Clipper makes product teams happy.*\n\n\n* Clipper **simplifies model deployment** and **helps reduce common bugs** by using the same tools and libraries used in model development to render live predictions.  *Clipper makes data scientists happy.*\n\n* Clipper **improves throughput** and ensures **reliable millisecond latencies** by introducing adaptive batching, caching, and straggler mitigation techniques.  *Clipper makes the infra-team less unhappy.*\n\n* Clipper **improves prediction accuracy** by introducing state-of-the-art bandit and ensemble methods to intelligently select and combine predictions and achieve real-time personalization across machine learning frameworks.  *Clipper makes users happy.*\n\n\n## Quickstart\n\n**Note: This quickstart works for the latest version of code. For a quickstart that works with the released version of Clipper available on PyPi, go to our [website](http://clipper.ai)**\n\n\u003e This quickstart requires [Docker](https://www.docker.com/) and supports Python 2.7, 3.5, 3.6 and 3.7.\n\n\n#### Clipper Example Code\n* Basic query: https://github.com/ucbrise/clipper/tree/develop/examples/basic_query\n* Image query: https://github.com/ucbrise/clipper/tree/develop/examples/image_query\n\n* Examples including metrics: https://github.com/ucbrise/clipper/tree/develop/examples\n#### Start a Clipper Instance and Deploy a Model\n\n__Install Clipper__\n\nYou can either install Clipper directly from GitHub:\n```sh\npip install git+https://github.com/ucbrise/clipper.git@develop#subdirectory=clipper_admin\n```\nor by cloning Clipper and installing directly from the file system:\n```sh\npip install -e \u003c/path/to/clipper_repo\u003e/clipper_admin\n```\n\n\n__Start a local Clipper cluster__\n\nFirst start a Python interpreter session.\n\n```sh\n$ python\n\n# Or start one with iPython\n$ conda install ipython\n$ ipython\n```\n\nCreate a `ClipperConnection` object and start Clipper. Running this command for the first time will\ndownload several Docker containers, so it may take some time.\n\n```py\nfrom clipper_admin import ClipperConnection, DockerContainerManager\nclipper_conn = ClipperConnection(DockerContainerManager())\nclipper_conn.start_clipper()\n```\n\n```pycon\n17-08-30:15:48:41 INFO     [docker_container_manager.py:95] Starting managed Redis instance in Docker\n17-08-30:15:48:43 INFO     [clipper_admin.py:105] Clipper still initializing.\n17-08-30:15:48:44 INFO     [clipper_admin.py:107] Clipper is running\n```\n\nRegister an application called `\"hello-world\"`. This will create a prediction REST endpoint at `http://localhost:1337/hello-world/predict`\n\n```py\nclipper_conn.register_application(name=\"hello-world\", input_type=\"doubles\", default_output=\"-1.0\", slo_micros=100000)\n```\n\n```pycon\n17-08-30:15:51:42 INFO     [clipper_admin.py:182] Application hello-world was successfully registered\n```\n\nInspect Clipper to see the registered apps\n\n```py\nclipper_conn.get_all_apps()\n```\n\n```pycon\n[u'hello-world']\n```\n\nDefine a simple model that just returns the sum of each feature vector.\nNote that the prediction function takes a list of feature vectors as\ninput and returns a list of strings.\n\n```py\ndef feature_sum(xs):\n    return [str(sum(x)) for x in xs]\n```\n\nImport the python deployer package\n\n```py\nfrom clipper_admin.deployers import python as python_deployer\n```\n\nDeploy the `\"feature_sum\"` function as a model. Notice that the application and model\nmust have the same input type.\n\n```py\npython_deployer.deploy_python_closure(clipper_conn, name=\"sum-model\", version=1, input_type=\"doubles\", func=feature_sum)\n```\n\n```pycon\n17-08-30:15:59:56 INFO     [deployer_utils.py:50] Anaconda environment found. Verifying packages.\n17-08-30:16:00:04 INFO     [deployer_utils.py:150] Fetching package metadata .........\nSolving package specifications: .\n\n17-08-30:16:00:04 INFO     [deployer_utils.py:151]\n17-08-30:16:00:04 INFO     [deployer_utils.py:59] Supplied environment details\n17-08-30:16:00:04 INFO     [deployer_utils.py:71] Supplied local modules\n17-08-30:16:00:04 INFO     [deployer_utils.py:77] Serialized and supplied predict function\n17-08-30:16:00:04 INFO     [python.py:127] Python closure saved\n17-08-30:16:00:04 INFO     [clipper_admin.py:375] Building model Docker image with model data from /tmp/python_func_serializations/sum-model\n17-08-30:16:00:05 INFO     [clipper_admin.py:378] Pushing model Docker image to sum-model:1\n17-08-30:16:00:07 INFO     [docker_container_manager.py:204] Found 0 replicas for sum-model:1. Adding 1\n17-08-30:16:00:07 INFO     [clipper_admin.py:519] Successfully registered model sum-model:1\n17-08-30:16:00:07 INFO     [clipper_admin.py:447] Done deploying model sum-model:1.\n```\n\n__Possible Error__\nIf start_clipper() is stuck at this logs, try `pip install -U cloudpickle==0.5.3`\n```\n18-05-21:12:19:59 INFO     [deployer_utils.py:44] Saving function to /tmp/clipper/tmpx6d_zqeq\n18-05-21:12:19:59 INFO     [deployer_utils.py:54] Serialized and supplied predict function\n18-05-21:12:19:59 INFO     [python.py:192] Python closure saved\n18-05-21:12:19:59 INFO     [python.py:206] Using Python 3.6 base image\n18-05-21:12:19:59 INFO     [clipper_admin.py:451] Building model Docker image with model data from /tmp/clipper/tmpx6d_zqeq\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': 'Step 1/2 : FROM clipper/python36-closure-container:develop'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': '\\n'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': ' ---\u003e 1aaddfa3945e\\n'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': 'Step 2/2 : COPY /tmp/clipper/tmpx6d_zqeq /model/'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': '\\n'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': ' ---\u003e b7c29f531d2e\\n'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'aux': {'ID': 'sha256:b7c29f531d2eaf59dd39579dbe512538be398dcb5fdd182db14e4d58770d2055'}}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': 'Successfully built b7c29f531d2e\\n'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:455] {'stream': 'Successfully tagged sum-model:1\\n'}\n18-05-21:12:20:00 INFO     [clipper_admin.py:457] Pushing model Docker image to sum-model:1\n18-05-21:12:20:02 INFO     [docker_container_manager.py:247] Found 0 replicas for sum-model:1. Adding 1\n```\n\nIt is because of cloudpickle dependency version issue. You may see this error logs from model container docker log. \n```\n$ docker logs 439ba722d79a # model container logs. For this example, it will be simple-example model container\nStarting Python Closure container\nConnecting to Clipper with default port: 7000\nTraceback (most recent call last):\n  File \"/container/python_closure_container.py\", line 56, in \u003cmodule\u003e\n    rpc_service.get_input_type())\n  File \"/container/python_closure_container.py\", line 28, in __init__\n    self.predict_func = load_predict_func(predict_path)\n  File \"/container/python_closure_container.py\", line 17, in load_predict_func\n    return cloudpickle.load(serialized_func_file)\n  File \"/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py\", line 1060, in _make_skel_func\n    base_globals['__builtins__'] = __builtins__\nTypeError: 'str' object does not support item assignment\n```\n\nTell Clipper to route requests for the \"hello-world\" application to the \"sum-model\"\n\n```py\nclipper_conn.link_model_to_app(app_name=\"hello-world\", model_name=\"sum-model\")\n```\n\n```pycon\n17-08-30:16:08:50 INFO     [clipper_admin.py:224] Model sum-model is now linked to application hello-world\n```\n\nYour application is now ready to serve predictions\n\n#### Query Clipper for predictions\n\n\nNow that you've deployed your first model, you can start requesting predictions at the REST endpoint that clipper created for your application: `http://localhost:1337/hello-world/predict`\n\nWith cURL:\n\n\n```sh\n$ curl -X POST --header \"Content-Type:application/json\" -d '{\"input\": [1.1, 2.2, 3.3]}' 127.0.0.1:1337/hello-world/predict\n```\n\nWith Python:\n\n```py\nimport requests, json, numpy as np\nheaders = {\"Content-type\": \"application/json\"}\nrequests.post(\"http://localhost:1337/hello-world/predict\", headers=headers, data=json.dumps({\"input\": list(np.random.random(10))})).json()\n```\n\n#### Clean up\n\nIf you closed the Python REPL you were using to start Clipper, you will need to start a new Python REPL and create another connection to the Clipper cluster. If you still have the Python REPL session active from earlier, you can re-use your existing `ClipperConnection` object.\n\n\nCreate a new connection. If you have still have the Python REPL from earlier, you can skip this step.\n\n```py\nfrom clipper_admin import ClipperConnection, DockerContainerManager\nclipper_conn = ClipperConnection(DockerContainerManager())\nclipper_conn.connect()\n```\nStop all Clipper docker containers\n\n```py\nclipper_conn.stop_all()\n```\n\n```pycon\n17-08-30:16:15:38 INFO     [clipper_admin.py:1141] Stopped all Clipper cluster and all model containers\n```\n\n\n## Contributing\n\nTo file a bug or request a feature, please file a GitHub issue. Pull requests are welcome. Additional help and instructions for contributors can be found on our website at \u003chttp://clipper.ai/contributing\u003e.\n\n## The Team\n\n+ [Dan Crankshaw](https://github.com/dcrankshaw)\n+ [Corey Zumar](https://github.com/Corey-Zumar)\n+ [Joey Gonzalez](https://github.com/jegonzal)\n+ [Alexey Tumanov](https://github.com/atumanov)\n+ [Eyal Sela](https://github.com/EyalSel)\n+ [Simon Mo](https://github.com/simon-mo)\n+ [Rehan Durrani](https://github.com/RehanSD)\n+ [Eric Sheng](https://github/com/es1024)\n\nYou can contact us at \u003cclipper-dev@googlegroups.com\u003e\n\n## Acknowledgements\n\nThis research is supported in part by DHS Award HSHQDC-16-3-00083, DOE Award SN10040 DE-SC0012463, NSF CISE Expeditions Award CCF-1139158, and gifts from Ant Financial, Amazon Web Services, CapitalOne, Ericsson, GE, Google, Huawei, Intel, IBM, Microsoft and VMware.\n","funding_links":[],"categories":["TODO scan for Android support in followings","C++","Paper-Code","Model Deployment and Orchestration Frameworks","Software"],"sub_categories":["Serving-Inference","Deploying models"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucbrise%2Fclipper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fucbrise%2Fclipper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucbrise%2Fclipper/lists"}