{"id":15433016,"url":"https://github.com/simonw/llm-replicate","last_synced_at":"2025-04-14T03:25:16.219Z","repository":{"id":181898880,"uuid":"667629709","full_name":"simonw/llm-replicate","owner":"simonw","description":"LLM plugin for models hosted on Replicate","archived":false,"fork":false,"pushed_at":"2024-04-18T17:13:27.000Z","size":40,"stargazers_count":60,"open_issues_count":6,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-19T09:12:02.920Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-18T00:44:52.000Z","updated_at":"2024-12-28T08:47:44.000Z","dependencies_parsed_at":"2023-07-18T02:32:07.511Z","dependency_job_id":"ec8d690b-43d2-48ed-aec2-bf2b2c9ac0bc","html_url":"https://github.com/simonw/llm-replicate","commit_stats":{"total_commits":27,"total_committers":2,"mean_commits":13.5,"dds":0.07407407407407407,"last_synced_commit":"cb815419beb29a749432d244502d03616081c04a"},"previous_names":["simonw/llm-replicate"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-replicate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-replicate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-replicate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-replicate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/llm-replicate/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248814595,"owners_count":21165790,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T18:30:31.426Z","updated_at":"2025-04-14T03:25:16.199Z","avatar_url":"https://github.com/simonw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llm-replicate\n\n[![PyPI](https://img.shields.io/pypi/v/llm-replicate.svg)](https://pypi.org/project/llm-replicate/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/llm-replicate?include_prereleases\u0026label=changelog)](https://github.com/simonw/llm-replicate/releases)\n[![Tests](https://github.com/simonw/llm-replicate/workflows/Test/badge.svg)](https://github.com/simonw/llm-replicate/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/llm-replicate/blob/main/LICENSE)\n\n[LLM](https://llm.datasette.io/) plugin for models hosted on [Replicate](https://replicate.com/)\n\n## Installation\n\nFirst, [install the LLM command-line utility](https://llm.datasette.io/en/stable/setup.html).\n\nNow install this plugin in the same environment as LLM.\n```bash\nllm install llm-replicate\n```\n## Configuration\n\nYou will need an API key from Replicate. You can [obtain one here](https://replicate.com/account/api-tokens).\n\nYou can set that as an environment variable called `REPLICATE_API_TOKEN`, or add it to the `llm` set of saved keys using:\n\n```bash\nllm keys set replicate\n```\n```\nEnter key: \u003cpaste key here\u003e\n```\nTo fetch and save details of [the default collection](https://replicate.com/collections/language-models) of language models hosted on Replicate, run this:\n```bash\nllm replicate fetch-models\n```\nTo add specific models that aren't listed in that collection, use the `llm replicate add` command.\n\nFor the Llama 2 model from [a16z-infra/llama13b-v2-chat](https://replicate.com/a16z-infra/llama13b-v2-chat) run this:\n\n```bash\nllm replicate add a16z-infra/llama13b-v2-chat \\\n  --chat --alias llama2\n```\nThe `--chat` flag indicates that this is a chat model, which means it will be able to work with `-c` continue mode.\n\nHere's the [70b version](https://replicate.com/replicate/llama70b-v2-chat) of that model:\n```bash\nllm replicate add \\\n  replicate/llama70b-v2-chat \\\n  --chat --alias llama70b\n```\n## Usage\n\nTo run a prompt against a model, pass its name or an alias to `llm -m`:\n```bash\nllm -m llama2 \"Ten great names for a pet pelican\"\n```\n\n\u003e Sure, here are ten great names for a pet pelican:\n\u003e\n\u003e 1. Pelty\n\u003e 2. Peanut\n\u003e 3. Puddles\n\u003e 4. Nibbles\n\u003e 5. Fuzzy\n\u003e 6. Gizmo\n\u003e 7. Hank\n\u003e 8. Luna\n\u003e 9. Scooter\n\u003e 10. Splishy\n\u003e\n\u003e I hope these suggestions help you find the perfect name for your pet pelican! Do you have any other questions?\n\n\nChat models can support continuing conversations, for example:\n```bash\nllm -c \"Five more and make them more nautical\"\n```\n\u003e Ahoy matey! Here be five more nautical-themed names for yer pet pelican:\n\u003e\n\u003e 1. Captain Hook\n\u003e 2. Anchoryn\n\u003e 3. Seadog\n\u003e 4. Plunder\n\u003e 5. Pointe Pelican\n\u003e\n\u003e I hope these suggestions help ye find the perfect name for yer feathered friend! Do ye have any other questions, matey?\n\nRun `llm models list` to see the full list of models:\n\n```bash\nllm models list\n```\nYou should see something like this:\n```\nReplicate: replicate-flan-t5-xl\nReplicate: replicate-llama-7b\nReplicate: replicate-gpt-j-6b\nReplicate: replicate-dolly-v2-12b\nReplicate: replicate-oasst-sft-1-pythia-12b\nReplicate: replicate-stability-ai-stablelm-tuned-alpha-7b\nReplicate: replicate-vicuna-13b\nReplicate: replicate-replit-code-v1-3b\nReplicate: replicate-replit-replit-code-v1-3b\nReplicate: replicate-joehoover-falcon-40b-instruct (aliases: falcon)\nReplicate (chat): replicate-a16z-infra-llama13b-v2-chat (aliases: llama2)\n```\nThen run a prompt through a specific model like this:\n```bash\nllm -m replicate-vicuna-13b \"Five great names for a pet llama\"\n```\n\n## Registering extra models\n\nTo register additional models that are not included in the default [Language models collection](https://replicate.com/collections/language-models), find their ID on Replicate and use the `llm replicate add` command.\n\nFor example, to add the [joehoover/falcon-40b-instruct](https://replicate.com/joehoover/falcon-40b-instruct) model, run this:\n\n```bash\nllm replicate add joehoover/falcon-40b-instruct \\\n  --alias falcon\n```\nThis adds the model with the alias `falcon` - you can have 0 or more aliases for a model.\n\nNow you can run it like this:\n```bash\nllm -m replicate-joehoover-falcon-40b-instruct \\\n  \"Three reasons to get a pet falcon\"\n```\nOr using the alias like this:\n```bash\nllm -m falcon \"Three reasons to get a pet falcon\"\n```\nYou can edit the list of models you have registered using the default `$EDITOR` like this:\n```bash\nllm replicate edit-models\n```\nIf you register a model using the `--chat` option that model will be treated slightly differently. Prompts sent to the model will be formatted like this:\n```\nUser: user input here\nAssistant:\n```\nIf you use `-c` [conversation mode](https://llm.datasette.io/en/stable/usage.html#continuing-a-conversation) the prompt will include previous messages in the conversation, like this:\n```\nUser: Ten great names for a pet pelican\nAssistant: Sure, here are ten great names for a pet pelican:\n\n1. Pelty\n2. Peanut\n3. Puddles\n4. Nibbles\n5. Fuzzy\n6. Gizmo\n7. Hank\n8. Luna\n9. Scooter\n10. Splishy\n\nI hope these suggestions help you find the perfect name for your pet pelican! Do you have any other questions?\nUser: Five more and make them more nautical\nAssistant:\n```\n\n## Fetching all Replicate predictions\n\nReplicate logs all predictions made against models. You can fetch all of these predictions using the `llm replicate fetch-predictions` command:\n\n```bash\nllm replicate fetch-predictions\n```\nThis will create or populate a table in your LLM `logs.db` database called `replicate_predictions`.\n\nThe data in this table will cover ALL Replicate models, not just language models that have been queried using this tool.\n\nRunning `llm replicate fetch-predictions` multiple times will only fetch predictions that have been created since the last time the command was run.\n\nTo browse the resulting data in [Datasette](https://datasette.io/), run this:\n```bash\ndatasette \"$(llm logs path)\"\n```\nThe schema for that table will look like this:\n```sql\nCREATE TABLE [replicate_predictions] (\n   [id] TEXT PRIMARY KEY,\n   [_model_guess] TEXT,\n   [completed_at] TEXT,\n   [created_at] TEXT,\n   [error] TEXT,\n   [input] TEXT,\n   [logs] TEXT,\n   [metrics] TEXT,\n   [output] TEXT,\n   [started_at] TEXT,\n   [status] TEXT,\n   [urls] TEXT,\n   [version] TEXT,\n   [webhook_completed] TEXT\n)\n```\nThis schema may change if the Replicate API adds new fields in the future.\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n```bash\ncd llm-replicate\npython3 -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\npip install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-replicate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fllm-replicate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-replicate/lists"}