{"id":18885536,"url":"https://github.com/togethercomputer/together-python","last_synced_at":"2025-04-12T15:40:25.093Z","repository":{"id":231772394,"uuid":"624113979","full_name":"togethercomputer/together-python","owner":"togethercomputer","description":"The Official Python Client for Together's API","archived":false,"fork":false,"pushed_at":"2025-04-10T10:11:03.000Z","size":2118,"stargazers_count":58,"open_issues_count":15,"forks_count":15,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-10T11:39:22.289Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pypi.org/project/together/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/togethercomputer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-05T19:19:13.000Z","updated_at":"2025-04-10T10:11:06.000Z","dependencies_parsed_at":"2024-05-23T07:23:50.693Z","dependency_job_id":"7535489a-672b-404a-b7ca-c0af6d40920a","html_url":"https://github.com/togethercomputer/together-python","commit_stats":null,"previous_names":["togethercomputer/together-python"],"tags_count":68,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/togethercomputer%2Ftogether-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/togethercomputer%2Ftogether-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/togethercomputer%2Ftogether-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/togethercomputer%2Ftogether-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/togethercomputer","download_url":"https://codeload.github.com/togethercomputer/together-python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590749,"owners_count":21129885,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T07:19:39.221Z","updated_at":"2025-04-12T15:40:25.088Z","avatar_url":"https://github.com/togethercomputer.png","language":"Python","funding_links":[],"categories":["Python","AI Cloud"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://www.together.ai/\"\u003e\n    \u003cimg alt=\"together.ai\" height=\"100px\" src=\"https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n# Together Python API library\n\n[![PyPI version](https://img.shields.io/pypi/v/together.svg)](https://pypi.org/project/together/)\n[![Discord](https://dcbadge.vercel.app/api/server/9Rk6sSeWEG?style=flat\u0026compact=true)](https://discord.com/invite/9Rk6sSeWEG)\n[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/togethercompute.svg?style=social\u0026label=Follow%20%40togethercompute)](https://twitter.com/togethercompute)\n\nThe [Together Python API Library](https://pypi.org/project/together/) is the official Python client for Together's API platform, providing a convenient way for interacting with the REST APIs and enables easy integrations with Python 3.10+ applications with easy to use synchronous and asynchronous clients.\n\n\n\n## Installation\n\n\u003e 🚧\n\u003e The Library was rewritten in v1.0.0 released in April of 2024. There were significant changes made.\n\nTo install Together Python Library from PyPI, simply run:\n\n```shell Shell\npip install --upgrade together\n```\n\n### Setting up API Key\n\n\u003e 🚧 You will need to create an account with [Together.ai](https://api.together.xyz/) to obtain a Together API Key.\n\nOnce logged in to the Together Playground, you can find available API keys in [this settings page](https://api.together.xyz/settings/api-keys).\n\n#### Setting environment variable\n\n```shell\nexport TOGETHER_API_KEY=xxxxx\n```\n\n#### Using the client\n\n```python\nfrom together import Together\n\nclient = Together(api_key=\"xxxxx\")\n```\n\nThis repo contains both a Python Library and a CLI. We'll demonstrate how to use both below.\n\n## Usage – Python Client\n\n### Chat Completions\n\n```python\nfrom together import Together\n\nclient = Together()\n\n# Simple text message\nresponse = client.chat.completions.create(\n    model=\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\n    messages=[{\"role\": \"user\", \"content\": \"tell me about new york\"}],\n)\nprint(response.choices[0].message.content)\n\n# Multi-modal message with text and image\nresponse = client.chat.completions.create(\n    model=\"meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"text\",\n                \"text\": \"What's in this image?\"\n            },\n            {\n                \"type\": \"image_url\",\n                \"image_url\": {\n                    \"url\": \"https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png\"\n                }\n            }\n        ]\n    }]\n)\nprint(response.choices[0].message.content)\n\n# Multi-modal message with multiple images\nresponse = client.chat.completions.create(\n    model=\"Qwen/Qwen2.5-VL-72B-Instruct\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"text\",\n                \"text\": \"Compare these two images.\"\n            },\n            {\n                \"type\": \"image_url\",\n                \"image_url\": {\n                    \"url\": \"https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png\"\n                }\n            },\n            {\n                \"type\": \"image_url\",\n                \"image_url\": {\n                    \"url\": \"https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/slack.png\"\n                }\n            }\n        ]\n    }]\n)\nprint(response.choices[0].message.content)\n\n# Multi-modal message with text and video\nresponse = client.chat.completions.create(\n    model=\"Qwen/Qwen2.5-VL-72B-Instruct\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"text\",\n                \"text\": \"What's happening in this video?\"\n            },\n            {\n                \"type\": \"video_url\",\n                \"video_url\": {\n                    \"url\": \"http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4\"\n                }\n            }\n        ]\n    }]\n)\nprint(response.choices[0].message.content)\n```\n\nThe chat completions API supports three types of content:\n- Plain text messages using the `content` field directly\n- Multi-modal messages with images using `type: \"image_url\"`\n- Multi-modal messages with videos using `type: \"video_url\"`\n\nWhen using multi-modal content, the `content` field becomes an array of content objects, each with its own type and corresponding data.\n\n#### Streaming\n\n```python\nimport os\nfrom together import Together\n\nclient = Together()\nstream = client.chat.completions.create(\n    model=\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\n    messages=[{\"role\": \"user\", \"content\": \"tell me about new york\"}],\n    stream=True,\n)\n\nfor chunk in stream:\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\n#### Async usage\n\n```python\nimport asyncio\nfrom together import AsyncTogether\n\nasync_client = AsyncTogether()\nmessages = [\n    \"What are the top things to do in San Francisco?\",\n    \"What country is Paris in?\",\n]\n\nasync def async_chat_completion(messages):\n    async_client = AsyncTogether()\n    tasks = [\n        async_client.chat.completions.create(\n            model=\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\n            messages=[{\"role\": \"user\", \"content\": message}],\n        )\n        for message in messages\n    ]\n    responses = await asyncio.gather(*tasks)\n\n    for response in responses:\n        print(response.choices[0].message.content)\n\nasyncio.run(async_chat_completion(messages))\n```\n\n#### Fetching logprobs\n\nLogprobs are logarithms of token-level generation probabilities that indicate the likelihood of the generated token based on the previous tokens in the context. Logprobs allow us to estimate the model's confidence in its outputs, which can be used to decide how to optimally consume the model's output (e.g. rejecting low confidence outputs, retrying or ensembling model outputs etc).\n\n```python\nfrom together import Together\n\nclient = Together()\n\nresponse = client.chat.completions.create(\n    model=\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\n    messages=[{\"role\": \"user\", \"content\": \"tell me about new york\"}],\n    logprobs=1\n)\n\nresponse_lobprobs = response.choices[0].logprobs\n\nprint(dict(zip(response_lobprobs.tokens, response_lobprobs.token_logprobs)))\n# {'New': -2.384e-07, ' York': 0.0, ',': 0.0, ' also': -0.20703125, ' known': -0.20214844, ' as': -8.34465e-07, ... }\n```\n\nMore details about using logprobs in Together's API can be found [here](https://docs.together.ai/docs/logprobs).\n\n\n### Completions\n\nCompletions are for code and language models shown [here](https://docs.together.ai/docs/inference-models). Below, a code model example is shown.\n\n```python\nfrom together import Together\n\nclient = Together()\n\nresponse = client.completions.create(\n    model=\"codellama/CodeLlama-34b-Python-hf\",\n    prompt=\"Write a Next.js component with TailwindCSS for a header component.\",\n    max_tokens=200,\n)\nprint(response.choices[0].text)\n```\n\n#### Streaming\n\n```python\nfrom together import Together\n\nclient = Together()\nstream = client.completions.create(\n    model=\"codellama/CodeLlama-34b-Python-hf\",\n    prompt=\"Write a Next.js component with TailwindCSS for a header component.\",\n    stream=True,\n)\n\nfor chunk in stream:\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\n#### Async usage\n\n```python\nimport asyncio\nfrom together import AsyncTogether\n\nasync_client = AsyncTogether()\nprompts = [\n    \"Write a Next.js component with TailwindCSS for a header component.\",\n    \"Write a python function for the fibonacci sequence\",\n]\n\nasync def async_chat_completion(prompts):\n    tasks = [\n        async_client.completions.create(\n            model=\"codellama/CodeLlama-34b-Python-hf\",\n            prompt=prompt,\n        )\n        for prompt in prompts\n    ]\n    responses = await asyncio.gather(*tasks)\n\n    for response in responses:\n        print(response.choices[0].text)\n\nasyncio.run(async_chat_completion(prompts))\n```\n\n### Image generation\n\n```python\nfrom together import Together\n\nclient = Together()\n\nresponse = client.images.generate(\n    prompt=\"space robots\",\n    model=\"stabilityai/stable-diffusion-xl-base-1.0\",\n    steps=10,\n    n=4,\n)\nprint(response.data[0].b64_json)\n```\n\n### Embeddings\n\n```python\nfrom typing import List\nfrom together import Together\n\nclient = Together()\n\ndef get_embeddings(texts: List[str], model: str) -\u003e List[List[float]]:\n    texts = [text.replace(\"\\n\", \" \") for text in texts]\n    outputs = client.embeddings.create(model=model, input = texts)\n    return [outputs.data[i].embedding for i in range(len(texts))]\n\ninput_texts = ['Our solar system orbits the Milky Way galaxy at about 515,000 mph']\nembeddings = get_embeddings(input_texts, model='togethercomputer/m2-bert-80M-8k-retrieval')\n\nprint(embeddings)\n```\n\n### Reranking\n\n```python\nfrom typing import List\nfrom together import Together\n\nclient = Together()\n\ndef get_reranked_documents(query: str, documents: List[str], model: str, top_n: int = 3) -\u003e List[str]:\n    outputs = client.rerank.create(model=model, query=query, documents=documents, top_n=top_n)\n    # sort by relevance score and returns the original docs\n    return [documents[i] for i in [x.index for x in sorted(outputs.results, key=lambda x: x.relevance_score, reverse=True)]]\n\nquery = \"What is the capital of the United States?\"\ndocuments = [\"New York\",\"Washington, D.C.\", \"Los Angeles\"]\n\nreranked_documents = get_reranked_documents(query, documents, model='Salesforce/Llama-Rank-V1', top_n=1)\n\nprint(reranked_documents)\n```\n\nRead more about Reranking [here](https://docs.together.ai/docs/rerank-overview).\n\n### Files\n\nThe files API is used for fine-tuning and allows developers to upload data to fine-tune on. It also has several methods to list all files, retrive files, and delete files. Please refer to our fine-tuning docs [here](https://docs.together.ai/docs/fine-tuning-python).\n\n```python\nfrom together import Together\n\nclient = Together()\n\nclient.files.upload(file=\"somedata.jsonl\") # uploads a file\nclient.files.list() # lists all uploaded files\nclient.files.retrieve(id=\"file-d0d318cb-b7d9-493a-bd70-1cfe089d3815\") # retrieves a specific file\nclient.files.retrieve_content(id=\"file-d0d318cb-b7d9-493a-bd70-1cfe089d3815\") # retrieves content of a specific file\nclient.files.delete(id=\"file-d0d318cb-b7d9-493a-bd70-1cfe089d3815\") # deletes a file\n```\n\n### Fine-tunes\n\nThe finetune API is used for fine-tuning and allows developers to create finetuning jobs. It also has several methods to list all jobs, retrive statuses and get checkpoints. Please refer to our fine-tuning docs [here](https://docs.together.ai/docs/fine-tuning-python).\n\n```python\nfrom together import Together\n\nclient = Together()\n\nclient.fine_tuning.create(\n  training_file = 'file-d0d318cb-b7d9-493a-bd70-1cfe089d3815',\n  model = 'mistralai/Mixtral-8x7B-Instruct-v0.1',\n  n_epochs = 3,\n  n_checkpoints = 1,\n  batch_size = \"max\",\n  learning_rate = 1e-5,\n  suffix = 'my-demo-finetune',\n  wandb_api_key = '1a2b3c4d5e.......',\n)\nclient.fine_tuning.list() # lists all fine-tuned jobs\nclient.fine_tuning.retrieve(id=\"ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\") # retrieves information on finetune event\nclient.fine_tuning.cancel(id=\"ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\") # Cancels a fine-tuning job\nclient.fine_tuning.list_events(id=\"ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\") #  Lists events of a fine-tune job\nclient.fine_tuning.download(id=\"ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\") # downloads compressed fine-tuned model or checkpoint to local disk\n```\n\n### Models\n\nThis lists all the models that Together supports.\n\n```python\nfrom together import Together\n\nclient = Together()\n\nmodels = client.models.list()\n\nfor model in models:\n    print(model)\n```\n\n## Usage – CLI\n\n### Chat Completions\n\n```bash\ntogether chat.completions \\\n  --message \"system\" \"You are a helpful assistant named Together\" \\\n  --message \"user\" \"What is your name?\" \\\n  --model mistralai/Mixtral-8x7B-Instruct-v0.1\n```\n\nThe Chat Completions CLI enables streaming tokens to stdout by default. To disable streaming, use `--no-stream`.\n\n### Completions\n\n```bash\ntogether completions \\\n  \"Large language models are \" \\\n  --model mistralai/Mixtral-8x7B-v0.1 \\\n  --max-tokens 512 \\\n  --stop \".\"\n```\n\nThe Completions CLI enables streaming tokens to stdout by default. To disable streaming, use `--no-stream`.\n\n### Image Generations\n\n```bash\ntogether images generate \\\n  \"space robots\" \\\n  --model stabilityai/stable-diffusion-xl-base-1.0 \\\n  --n 4\n```\n\nThe image is opened in the default image viewer by default. To disable this, use `--no-show`.\n\n### Files\n\n```bash\n# Help\ntogether files --help\n\n# Check file\ntogether files check example.jsonl\n\n# Upload file\ntogether files upload example.jsonl\n\n# List files\ntogether files list\n\n# Retrieve file metadata\ntogether files retrieve file-6f50f9d1-5b95-416c-9040-0799b2b4b894\n\n# Retrieve file content\ntogether files retrieve-content file-6f50f9d1-5b95-416c-9040-0799b2b4b894\n\n# Delete remote file\ntogether files delete file-6f50f9d1-5b95-416c-9040-0799b2b4b894\n```\n\n### Fine-tuning\n\n```bash\n# Help\ntogether fine-tuning --help\n\n# Create fine-tune job\ntogether fine-tuning create \\\n  --model togethercomputer/llama-2-7b-chat \\\n  --training-file file-711d8724-b3e3-4ae2-b516-94841958117d\n\n# List fine-tune jobs\ntogether fine-tuning list\n\n# Retrieve fine-tune job details\ntogether fine-tuning retrieve ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\n\n# List fine-tune job events\ntogether fine-tuning list-events ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\n\n# Cancel running job\ntogether fine-tuning cancel ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\n\n# Download fine-tuned model weights\ntogether fine-tuning download ft-c66a5c18-1d6d-43c9-94bd-32d756425b4b\n```\n\n### Models\n\n```bash\n# Help\ntogether models --help\n\n# List models\ntogether models list\n```\n\n## Contributing\n\nRefer to the [Contributing Guide](CONTRIBUTING.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftogethercomputer%2Ftogether-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftogethercomputer%2Ftogether-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftogethercomputer%2Ftogether-python/lists"}