{"id":15433024,"url":"https://github.com/simonw/llm-embed-jina","last_synced_at":"2025-04-14T16:09:05.833Z","repository":{"id":203632544,"uuid":"710073803","full_name":"simonw/llm-embed-jina","owner":"simonw","description":"Embedding models from Jina AI","archived":false,"fork":false,"pushed_at":"2024-01-18T00:13:41.000Z","size":15,"stargazers_count":55,"open_issues_count":6,"forks_count":6,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-10-18T07:53:51.189Z","etag":null,"topics":["embeddings","jina","llm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-26T01:02:30.000Z","updated_at":"2024-07-28T05:12:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"0f2d1313-d269-4d36-9b43-7ec6ec3115b0","html_url":"https://github.com/simonw/llm-embed-jina","commit_stats":{"total_commits":9,"total_committers":1,"mean_commits":9.0,"dds":0.0,"last_synced_commit":"9aa8d989ff4fbedc7321a2da67d21a05f03a3382"},"previous_names":["simonw/llm-embed-jina"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-embed-jina","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-embed-jina/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-embed-jina/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-embed-jina/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/llm-embed-jina/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245972665,"owners_count":20702721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embeddings","jina","llm"],"created_at":"2024-10-01T18:30:35.257Z","updated_at":"2025-03-30T03:31:23.355Z","avatar_url":"https://github.com/simonw.png","language":"Python","funding_links":[],"categories":["Building"],"sub_categories":["LLM Models"],"readme":"# llm-embed-jina\n\n[![PyPI](https://img.shields.io/pypi/v/llm-embed-jina.svg)](https://pypi.org/project/llm-embed-jina/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/llm-embed-jina?include_prereleases\u0026label=changelog)](https://github.com/simonw/llm-embed-jina/releases)\n[![Tests](https://github.com/simonw/llm-embed-jina/workflows/Test/badge.svg)](https://github.com/simonw/llm-embed-jina/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/llm-embed-jina/blob/main/LICENSE)\n\nEmbedding models from Jina AI\n\n## Background\n\n[Jina AI Launches World's First Open-Source 8K Text Embedding, Rivaling OpenAI](https://jina.ai/news/jina-ai-launches-worlds-first-open-source-8k-text-embedding-rivaling-openai/) introduces these models.\n\nSee also [Embeddings: What they are and why they matter](https://simonwillison.net/2023/Oct/23/embeddings/) for background on embeddings and an explanation of the LLM embeddings tool.\n\nHere's [my blog post about how I built this plugin](https://simonwillison.net/2023/Oct/26/llm-embed-jina/).\n\n## Installation\n\nInstall this plugin in the same environment as [LLM](https://llm.datasette.io/).\n\n    llm install llm-embed-jina\n\n## Usage\n\nThis plugin adds support for three new embedding models:\n\n- [`jina-embeddings-v2-small-en`](https://huggingface.co/jinaai/jina-embeddings-v2-small-en): 33 million parameters.\n- [`jina-embeddings-v2-base-en`](https://huggingface.co/jinaai/jina-embeddings-v2-base-en): 137 million parameters.\n- [`jina-embeddings-v2-large-en`](https://huggingface.co/jinaai/jina-embeddings-v2-large-en): 435 million parameters - not yet released, but it will work once it has been released.\n\nThe models will be downloaded the first time you try to use them.\n\nSee [the LLM documentation](https://llm.datasette.io/en/stable/embeddings/index.html) for everything you can do.\n\nTo get started embedding a single string, run the following:\n\n```bash\nllm embed -m jina-embeddings-v2-small-en -c 'Hello world'\n```\nThis will output a JSON array of 512 floating point numbers to your terminal.\n\nTo calculate and store embeddings for every README in the current directory (try this somewhere with a `node_modules` directory to get lots of READMEs) run this:\n\n```bash\nllm embed-multi jina-readmes \\\n    -m jina-embeddings-v2-small-en \\\n    --files . '**/README.md' --store\n```\nThen you can run searches against them like this:\n```bash\nllm similar jina-readmes -c 'utility functions'\n```\nAdd `| jq` to pipe it through [jq](https://jqlang.github.io/jq/) for pretty-printed output, or ` | jq .id` to just see the matching filenames.\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n```bash\ncd llm-embed-jina\npython3 -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\nllm install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-embed-jina","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fllm-embed-jina","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-embed-jina/lists"}