{"id":13468792,"url":"https://github.com/refuel-ai/autolabel","last_synced_at":"2025-05-13T21:06:22.756Z","repository":{"id":175600604,"uuid":"617708951","full_name":"refuel-ai/autolabel","owner":"refuel-ai","description":"Label, clean and enrich text datasets with LLMs.","archived":false,"fork":false,"pushed_at":"2025-03-05T01:15:43.000Z","size":53851,"stargazers_count":2210,"open_issues_count":79,"forks_count":154,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-04-28T17:59:38.735Z","etag":null,"topics":["anthropic-claude","data-science","gpt-4","huggingface-transformers","langchain","large-language-models","llm","llms","machine-learning","openai","python"],"latest_commit_sha":null,"homepage":"https://docs.refuel.ai/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/refuel-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-23T00:11:13.000Z","updated_at":"2025-04-28T15:11:26.000Z","dependencies_parsed_at":"2023-09-29T00:37:43.943Z","dependency_job_id":"c645f089-e9a6-4321-82c7-d118a4203f2b","html_url":"https://github.com/refuel-ai/autolabel","commit_stats":{"total_commits":600,"total_committers":21,"mean_commits":"28.571428571428573","dds":0.8416666666666667,"last_synced_commit":"1f168281eb864474b5e6cdd5429ccefa2c9ed81b"},"previous_names":["refuel-ai/autolabel"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/refuel-ai%2Fautolabel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/refuel-ai%2Fautolabel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/refuel-ai%2Fautolabel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/refuel-ai%2Fautolabel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/refuel-ai","download_url":"https://codeload.github.com/refuel-ai/autolabel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251362144,"owners_count":21577400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic-claude","data-science","gpt-4","huggingface-transformers","langchain","large-language-models","llm","llms","machine-learning","openai","python"],"created_at":"2024-07-31T15:01:19.203Z","updated_at":"2025-04-28T17:59:50.441Z","avatar_url":"https://github.com/refuel-ai.png","language":"Python","funding_links":[],"categories":["Python","A01_文本生成_文本对话","数据 Data","Tools Made Using LLMs ([home](#awesome-llm))","Langchain","python","Data ([home](#awesome-llm))","Infrastructure and Tools"],"sub_categories":["大语言对话模型及数据","Courses ([home](#awesome-llm))","Data Labeling"],"readme":"\u003cimg alt=\"Refuel logo\" src=\"https://raw.githubusercontent.com/refuel-ai/autolabel/main/docs/assets/Autolabel_blk_w_background.png\"\u003e\n\n\u003ch4 align=\"center\"\u003e\n  \u003ca href=\"https://discord.gg/fweVnRx6CU\"\u003eDiscord\u003c/a\u003e |\n  \u003ca href=\"https://twitter.com/RefuelAI\"\u003eTwitter\u003c/a\u003e |\n  \u003ca href=\"https://www.refuel.ai/\"\u003eWebsite\u003c/a\u003e |\n  \u003ca href=\"https://www.refuel.ai/blog-posts/llm-labeling-technical-report\"\u003eBenchmark\u003c/a\u003e\n\u003c/h4\u003e\n\n\u003cdiv align=\"center\" style=\"width:800px\"\u003e\n\n[![lint](https://github.com/refuel-ai/autolabel/actions/workflows/black.yaml/badge.svg)](https://github.com/refuel-ai/autolabel/actions/workflows/black.yaml/badge.svg) ![Tests](https://github.com/refuel-ai/autolabel/actions/workflows/test.yaml/badge.svg) ![Commit Activity](https://img.shields.io/github/commit-activity/m/refuel-ai/autolabel) [![Discord](https://img.shields.io/discord/1098746693152931901)](https://discord.gg/fweVnRx6CU) [![open in colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1t-9vNLkyoyySAG_0w3eR98biBOXlMO-E?usp=sharing)\n\n\u003c/div\u003e\n\n## ⚡ Quick Install\n\n`pip install refuel-autolabel`\n\n## 📖 Documentation\n\n[https://docs.refuel.ai/](https://docs.refuel.ai/)\n\n## 🏷 What is Autolabel\n\nAccess to [large, clean and diverse](https://twitter.com/karpathy/status/1528443124577513472?lang=en) labeled datasets is a critical component for any machine learning effort to be successful. State-of-the-art LLMs like GPT-4 are able to [automatically label data](https://arxiv.org/abs/2303.15056) with [high accuracy](https://arxiv.org/abs/2303.16854), and at a fraction of the cost and time compared to manual labeling.\n\nAutolabel is a Python library to label, clean and enrich text datasets with any Large Language Models (LLM) of your choice.\n\n## 🌟 (New!) Benchmark models on Refuel's Benchmark\n\nCheck out our [technical report](https://refuel.ai/blog-posts/announcing-refuel-llm-2) to learn more about the performance of RefuelLLM-v2 on our benchmark. You can replicate the benchmark yourself by following the steps below\n\n```python\ncd autolabel/benchmark\ncurl https://autolabel-benchmarking.s3.us-west-2.amazonaws.com/data.zip -o data.zip\nunzip data.zip\npython benchmark.py --model $model --base_dir benchmark-results\npython results.py --eval_dir benchmark-results\ncat results.csv\n```\n\nYou can benchmark the relevant model by replacing $model with the name of the model needed to be benchmarked. If it is an API hosted model like `gpt-3.5-turbo`, `gpt-4-1106-preview`, `claude-3-opus-20240229`, `gemini-1.5-pro-preview-0409` or some other Autolabel supported model, just write the name of the model. If the model to be benchmarked is a [vLLM supported model](https://docs.vllm.ai/en/latest/models/supported_models.html) then pass the local path or the huggingface path corresponding to the model. This will run the benchmark along with the _same_ prompts for all models.\n\nThe `results.csv` will contain a row with every model that was benchmarked as a row. Look at `benchmark/results.csv` for an example.\n\n## 🚀 Getting started\n\nAutolabel provides a simple 3-step process for labeling data:\n\n1. Specify the labeling guidelines and LLM model to use in a JSON config.\n2. Dry-run to make sure the final prompt looks good.\n3. Kick off a labeling run for your dataset!\n\nLet's imagine we are building an ML model to analyze sentiment analysis of movie review. We have a dataset of movie reviews that we'd like to get labeled first. For this case, here's what the example dataset and configs will look like:\n\n```python\n{\n    \"task_name\": \"MovieSentimentReview\",\n    \"task_type\": \"classification\",\n    \"model\": {\n        \"provider\": \"openai\",\n        \"name\": \"gpt-3.5-turbo\"\n    },\n    \"dataset\": {\n        \"label_column\": \"label\",\n        \"delimiter\": \",\"\n    },\n    \"prompt\": {\n        \"task_guidelines\": \"You are an expert at analyzing the sentiment of movie reviews. Your job is to classify the provided movie review into one of the following labels: {labels}\",\n        \"labels\": [\n            \"positive\",\n            \"negative\",\n            \"neutral\"\n        ],\n        \"few_shot_examples\": [\n            {\n                \"example\": \"I got a fairly uninspired stupid film about how human industry is bad for nature.\",\n                \"label\": \"negative\"\n            },\n            {\n                \"example\": \"I loved this movie. I found it very heart warming to see Adam West, Burt Ward, Frank Gorshin, and Julie Newmar together again.\",\n                \"label\": \"positive\"\n            },\n            {\n                \"example\": \"This movie will be played next week at the Chinese theater.\",\n                \"label\": \"neutral\"\n            }\n        ],\n        \"example_template\": \"Input: {example}\\nOutput: {label}\"\n    }\n}\n```\n\nInitialize the labeling agent and pass it the config:\n\n```python\n\nfrom autolabel import LabelingAgent, AutolabelDataset\n\nagent = LabelingAgent(config='config.json')\n```\n\nPreview an example prompt that will be sent to the LLM:\n\n```python\nds = AutolabelDataset('dataset.csv', config = config)\nagent.plan(ds)\n```\n\nThis prints:\n\n```\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100/100 0:00:00 0:00:00\n┌──────────────────────────┬─────────┐\n│ Total Estimated Cost     │ $0.538  │\n│ Number of Examples       │ 200     │\n│ Average cost per example │ 0.00269 │\n└──────────────────────────┴─────────┘\n─────────────────────────────────────────\n\nPrompt Example:\nYou are an expert at analyzing the sentiment of movie reviews. Your job is to classify the provided movie review into one of the following labels: [positive, negative, neutral]\n\nSome examples with their output answers are provided below:\n\nExample: I got a fairly uninspired stupid film about how human industry is bad for nature.\nOutput:\nnegative\n\nExample: I loved this movie. I found it very heart warming to see Adam West, Burt Ward, Frank Gorshin, and Julie Newmar together again.\nOutput:\npositive\n\nExample: This movie will be played next week at the Chinese theater.\nOutput:\nneutral\n\nNow I want you to label the following example:\nInput: A rare exception to the rule that great literature makes disappointing films.\nOutput:\n\n─────────────────────────────────────────────────────────────────────────────────────────\n\n```\n\nFinally, we can run the labeling on a subset or entirety of the dataset:\n\n```python\nds = agent.run(ds)\n```\n\nThe output dataframe contains the label column:\n\n```python\nds.df.head()\n                                                text  ... MovieSentimentReview_llm_label\n0  I was very excited about seeing this film, ant...  ...                       negative\n1  Serum is about a crazy doctor that finds a ser...  ...                       negative\n4  I loved this movie. I knew it would be chocked...  ...                       positive\n...\n```\n\n## Features\n\n1. Label data for [NLP tasks](https://docs.refuel.ai/guide/tasks/classification_task/) such as classification, question-answering and named entity-recognition, entity matching and more.\n2. Use commercial or open source [LLMs](https://docs.refuel.ai/guide/llms/llms/) from providers such as OpenAI, Anthropic, HuggingFace, Google and more.\n3. Support for research-proven LLM techniques to boost label quality, such as few-shot learning and chain-of-thought prompting.\n4. [Confidence estimation](https://docs.refuel.ai/guide/accuracy/confidence/) and explanations out of the box for every single output label\n5. [Caching and state management](https://docs.refuel.ai/guide/reliability/state-management/) to minimize costs and experimentation time\n\n## Access to Refuel hosted LLMs\n\nRefuel provides access to hosted open source LLMs for labeling, and for estimating confidence This is helpful, because you can calibrate a confidence threshold for your labeling task, and then route less confident labels to humans, while you still get the benefits of auto-labeling for the confident examples.\n\nIn order to use Refuel hosted LLMs, you can [request access here](https://refuel-ai.typeform.com/llm-access).\n\n## 🛠️ Roadmap\n\nCheck out our [public roadmap](https://github.com/orgs/refuel-ai/projects/15) to learn more about ongoing and planned improvements to the Autolabel library.\n\nWe are always looking for suggestions and contributions from the community. Join the discussion on [Discord](https://discord.gg/fweVnRx6CU) or open a [Github issue](https://github.com/refuel-ai/autolabel/issues) to report bugs and request features.\n\n## 🙌 Contributing\n\nAutolabel is a rapidly developing project. We welcome contributions in all forms - bug reports, pull requests and ideas for improving the library.\n\n1. Join the conversation on [Discord](https://discord.gg/fweVnRx6CU)\n2. Open an [issue](https://github.com/refuel-ai/autolabel/issues) on Github for bugs and request features.\n3. Grab an open issue, and submit a [pull request](https://github.com/refuel-ai/autolabel/blob/main/CONTRIBUTING.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frefuel-ai%2Fautolabel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frefuel-ai%2Fautolabel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frefuel-ai%2Fautolabel/lists"}