{"id":32624577,"url":"https://github.com/milistu/callm","last_synced_at":"2026-04-29T16:04:21.041Z","repository":{"id":319484388,"uuid":"1056133059","full_name":"milistu/callm","owner":"milistu","description":"Keep callm and process thousands of requests without (rate) limits","archived":false,"fork":false,"pushed_at":"2025-12-17T18:04:03.000Z","size":194,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-21T02:15:34.694Z","etag":null,"topics":["anthropic","api","claude","cohere","deepseek","gemini","google","llm","openai","parallel","requests","voyageai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/milistu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-13T13:06:48.000Z","updated_at":"2025-12-17T18:01:25.000Z","dependencies_parsed_at":"2025-10-19T08:27:09.925Z","dependency_job_id":"f942cf2b-a307-4193-ae51-163aba762656","html_url":"https://github.com/milistu/callm","commit_stats":null,"previous_names":["milistu/callm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/milistu/callm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milistu%2Fcallm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milistu%2Fcallm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milistu%2Fcallm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milistu%2Fcallm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/milistu","download_url":"https://codeload.github.com/milistu/callm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milistu%2Fcallm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32432917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T13:34:34.882Z","status":"ssl_error","status_checked_at":"2026-04-29T13:34:29.830Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","api","claude","cohere","deepseek","gemini","google","llm","openai","parallel","requests","voyageai"],"created_at":"2025-10-30T20:02:30.071Z","updated_at":"2026-04-29T16:04:21.035Z","avatar_url":"https://github.com/milistu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003ecallm\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\u003cstrong\u003eKeep callm and process thousands of requests without (rate) limits\u003c/strong\u003e\u003c/p\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/callm-py/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/callm-py?color=blue\u0026label=PyPI\" alt=\"PyPI version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/callm-py/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/pyversions/callm-py\" alt=\"Python versions\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/milistu/callm/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"License\"\u003e\u003c/a\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e •\n  \u003ca href=\"#quick-start\"\u003eQuick Start\u003c/a\u003e •\n  \u003ca href=\"#supported-providers\"\u003eProviders\u003c/a\u003e •\n  \u003ca href=\"#examples\"\u003eExamples\u003c/a\u003e •\n  \u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## 😌 Why callm?\n\nBuilding LLM-powered applications often means processing **thousands of API requests**. You've probably experienced:\n\n| Problem | Without callm | With callm |\n|---------|---------------|------------|\n| **Rate limit errors** | Constant 429 errors, manual sleep/retry | Automatic RPM \u0026 TPM throttling |\n| **Retry logic** | Write custom backoff for each project | Built-in exponential backoff with jitter |\n| **Token tracking** | No visibility into usage | Real-time token consumption metrics |\n| **Boilerplate code** | Copy-paste the same async code everywhere | One function call, any provider |\n| **Waiting for batch APIs** | Provider batch APIs take up to 24 hours | Results in minutes, not hours |\n| **Multiple SDKs** | Install openai, anthropic, cohere, ... | One library, all providers |\n\n**Stop rewriting the same parallel processing code.** callm handles the infrastructure so you can focus on your application.\n\n\u003e *Testing multiple providers? Just swap the provider class—no new dependencies, no code changes. Find what works best for your use case.*\n\n## Installation\n\n```bash\npip install callm-py\n```\n\n**From source:**\n```bash\ngit clone https://github.com/milistu/callm.git\ncd callm\npip install -e .\n```\n\n## Quick Start\n\nProcess 1,000 product descriptions to extract structured data—in under a minute:\n\n```python\nimport asyncio\nfrom callm import process_requests, RateLimitConfig\nfrom callm.providers import OpenAIProvider\n\n# Configure your provider\nprovider = OpenAIProvider(\n    api_key=\"sk-...\",\n    model=\"gpt-5-mini\",\n    request_url=\"https://api.openai.com/v1/responses\",\n)\n\n# Your data processing requests\nproducts = [\n    {\"id\": 1, \"description\": \"Nike Air Max 90 - Classic sneakers in white/black, size 10\"},\n    {\"id\": 2, \"description\": \"Sony WH-1000XM5 Wireless Headphones - Noise cancelling, 30hr battery\"},\n    # ... thousands more\n]\n\nrequests = [\n    {\n        \"input\": f\"Extract brand, category, and key features from: {p['description']}\",\n        \"metadata\": {\"product_id\": p[\"id\"]},\n    }\n    for p in products\n]\n\nasync def main():\n    results = await process_requests(\n        provider=provider,\n        requests=requests,\n        rate_limit=RateLimitConfig(\n            max_requests_per_minute=5_000,    # Stay under your tier limit\n            max_tokens_per_minute=2_000_000,\n        ),\n    )\n\n    print(f\"Processed {results.stats.successful} requests in {results.stats.duration_seconds:.1f}s\")\n    print(f\"Tokens used: {results.stats.total_input_tokens + results.stats.total_output_tokens:,}\")\n\n    # Access results\n    for result in results.successes:\n        print(f\"Product {result.metadata['product_id']}: {result.response}\")\n\nasyncio.run(main())\n```\n\n## Features\n\n- **Precise Rate Limiting** — Token buckets for RPM and TPM, respects provider limits\n- **Smart Retries** — Exponential backoff with jitter, automatic 429/5xx handling\n- **Usage Tracking** — Metrics for input tokens and output tokens\n- **Flexible I/O** — Process from Python lists or JSONL files, output to memory or disk\n- **Structured Outputs** — Support for Pydantic models and JSON schemas\n- **Provider Agnostic** — Same API across OpenAI, Anthropic, Gemini, DeepSeek, and more\n\n## Supported Providers\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\" width=\"150\"\u003e\n      \u003cimg src=\"https://us1.discourse-cdn.com/openai1/original/4X/3/2/1/321a1ba297482d3d4060d114860de1aa5610f8a9.png\" width=\"60\" alt=\"OpenAI\"\u003e\u003cbr\u003e\n      \u003cstrong\u003eOpenAI\u003c/strong\u003e\u003cbr\u003e\n      \u003csub\u003eChat, Responses, Embeddings\u003c/sub\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\" width=\"150\"\u003e\n      \u003cimg src=\"https://cdn.worldvectorlogo.com/logos/anthropic-1.svg\" width=\"60\" alt=\"Anthropic\"\u003e\u003cbr\u003e\n      \u003cstrong\u003eAnthropic\u003c/strong\u003e\u003cbr\u003e\n      \u003csub\u003eMessages API\u003c/sub\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\" width=\"150\"\u003e\n      \u003cimg src=\"https://registry.npmmirror.com/@lobehub/icons-static-png/1.75.0/files/light/gemini-color.png\" width=\"60\" alt=\"Google Gemini\"\u003e\u003cbr\u003e\n      \u003cstrong\u003eGemini\u003c/strong\u003e\u003cbr\u003e\n      \u003csub\u003eGenerate, Embeddings\u003c/sub\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\" width=\"150\"\u003e\n      \u003cimg src=\"https://registry.npmmirror.com/@lobehub/icons-static-png/1.75.0/files/dark/deepseek-color.png\" width=\"60\" alt=\"DeepSeek\"\u003e\u003cbr\u003e\n      \u003cstrong\u003eDeepSeek\u003c/strong\u003e\u003cbr\u003e\n      \u003csub\u003eChat Completions\u003c/sub\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\" width=\"150\"\u003e\n      \u003cimg src=\"https://registry.npmmirror.com/@lobehub/icons-static-png/1.75.0/files/dark/cohere-color.png\" width=\"60\" alt=\"Cohere\"\u003e\u003cbr\u003e\n      \u003cstrong\u003eCohere\u003c/strong\u003e\u003cbr\u003e\n      \u003csub\u003eEmbed API\u003c/sub\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\" width=\"150\"\u003e\n      \u003cimg src=\"https://i0.wp.com/blog.voyageai.com/wp-content/uploads/2023/10/logo.png?quality=80\u0026ssl=1\" width=\"60\" alt=\"Voyage AI\"\u003e\u003cbr\u003e\n      \u003cstrong\u003eVoyage AI\u003c/strong\u003e\u003cbr\u003e\n      \u003csub\u003eEmbeddings\u003c/sub\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n## Examples\n\nExplore real-world use cases in the [`examples/`](examples/) directory:\n\n| Use Case | Description |\n|----------|-------------|\n| [**Data Extraction**](examples/data_extraction/) | Extract structured data from product listings, invoices |\n| [**Embeddings**](examples/embeddings/) | Generate embeddings for RAG and semantic search |\n| [**Evaluation**](examples/evaluation/) | Multi-judge consensus evaluation |\n| [**Synthetic Data**](examples/synthetic_data/) | Generate training data and evaluation sets |\n| [**Classification**](examples/classification/) | Sentiment analysis, content moderation |\n| [**Translation**](examples/translation/) | Dataset translation for multilingual evaluation |\n\n### Processing Modes\n\ncallm supports four processing modes depending on your input source and output destination:\n\n| Input | Output | Best For |\n|-------|--------|----------|\n| Python list | In-memory | Small batches, interactive use |\n| Python list | JSONL file | Medium batches, need persistence |\n| JSONL file | JSONL file | Large batches, low memory |\n| JSONL file | In-memory | Loading saved requests, testing |\n\n```python\n# 1. List → Memory (small batches)\nresults = await process_requests(\n    provider=provider,\n    requests=my_list,\n    rate_limit=rate_limit,\n)\n# Access: results.successes, results.failures\n\n# 2. List → File (persist results)\nresults = await process_requests(\n    provider=provider,\n    requests=my_list,\n    rate_limit=rate_limit,\n    output_path=\"results.jsonl\",\n)\n\n# 3. File → File (large batches, low memory)\nresults = await process_requests(\n    provider=provider,\n    requests=\"input.jsonl\",\n    rate_limit=rate_limit,\n    output_path=\"results.jsonl\",\n)\n\n# 4. File → Memory (reload saved requests)\nresults = await process_requests(\n    provider=provider,\n    requests=\"input.jsonl\",\n    rate_limit=rate_limit,\n)\n```\n\n### Configuration\n\n```python\nfrom callm import RateLimitConfig, RetryConfig\n\n# Rate limiting (required)\nrate_limit = RateLimitConfig(\n    max_requests_per_minute=1000,\n    max_tokens_per_minute=100_000,\n)\n\n# Retry behavior (optional, sensible defaults)\nretry = RetryConfig(\n    max_attempts=5,\n    base_delay_seconds=0.5,\n    max_delay_seconds=15.0,\n    jitter=0.1,\n)\n\nresults = await process_requests(\n    provider=provider,\n    requests=requests,\n    rate_limit=rate_limit,\n    retry=retry,\n)\n```\n\n## API Reference\n\n### `process_requests()`\n\nMain function for parallel API request processing.\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `provider` | `BaseProvider` | Provider instance (OpenAI, Anthropic, etc.) |\n| `requests` | `list[dict] \\| str` | List of request dicts or path to JSONL file |\n| `rate_limit` | `RateLimitConfig` | RPM and TPM limits |\n| `retry` | `RetryConfig` | Optional retry configuration |\n| `output_path` | `str` | Optional path for output JSONL (enables streaming) |\n| `errors_path` | `str` | Optional path for error JSONL |\n| `logging_level` | `int` | Logging verbosity (default: 20/INFO) |\n\n**Returns:** `ProcessingResults` with `successes`, `failures`, and `stats`.\n\n## Contributing\n\nContributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n```bash\n# Setup development environment\ngit clone https://github.com/milistu/callm.git\ncd callm\nuv sync --dev\nuv run pre-commit install\n\n# Run tests\nuv run nox\n```\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003csub\u003eBuilt with 🧡 for engineers who process data at scale\u003c/sub\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmilistu%2Fcallm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmilistu%2Fcallm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmilistu%2Fcallm/lists"}