{"id":50290047,"url":"https://github.com/digitaldreams/local-batch-api","last_synced_at":"2026-05-28T05:30:39.320Z","repository":{"id":360335372,"uuid":"1248849022","full_name":"digitaldreams/local-batch-api","owner":"digitaldreams","description":"Drop-in replacement for the Anthropic and OpenAI batch APIs — self-hosted, running against local Ollama or LM Studio. Point your existing SDK code at this server instead of the real APIs. No API keys, no cloud costs, no rate limits.","archived":false,"fork":false,"pushed_at":"2026-05-26T02:15:35.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-26T02:32:22.645Z","etag":null,"topics":["anthropic-batch-api","batch-api","laravel","lmstudio","ollama","openai-batch"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/digitaldreams.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-25T05:33:32.000Z","updated_at":"2026-05-26T02:15:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/digitaldreams/local-batch-api","commit_stats":null,"previous_names":["digitaldreams/local-batch-api"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/digitaldreams/local-batch-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitaldreams%2Flocal-batch-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitaldreams%2Flocal-batch-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitaldreams%2Flocal-batch-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitaldreams%2Flocal-batch-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/digitaldreams","download_url":"https://codeload.github.com/digitaldreams/local-batch-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digitaldreams%2Flocal-batch-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33596316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic-batch-api","batch-api","laravel","lmstudio","ollama","openai-batch"],"created_at":"2026-05-28T05:30:38.583Z","updated_at":"2026-05-28T05:30:39.314Z","avatar_url":"https://github.com/digitaldreams.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# local-batch-api\n\nDrop-in replacement for the **Anthropic** and **OpenAI** batch APIs — self-hosted, running against local [Ollama](https://ollama.ai) or [LM Studio](https://lmstudio.ai). No API keys, no cloud costs, no rate limits.\n\n---\n\n## Requirements\n\n- PHP 8.4+\n- Laravel 13+\n- Running [Ollama](https://ollama.ai/download) or [LM Studio](https://lmstudio.ai) instance\n- A queue worker (`php artisan queue:work`)\n\n---\n\n## Installation\n\n### Step 1 — Install the package\n\n```bash\ncomposer require digitaldreams/local-batch-api\n```\n\n### Step 2 — Run migrations\n\n```bash\nphp artisan migrate\n```\n\nThis creates two tables: `batches` and `batch_files`.\n\n### Step 3 — Configure your inference backend\n\nAdd to your `.env`:\n\n```env\n# 'ollama' (default) or 'lmstudio'\nINFERENCE_PROVIDER=ollama\n\n# Base URL of your local inference server\nINFERENCE_URL=http://localhost:11434   # Ollama default\n# INFERENCE_URL=http://localhost:1234  # LM Studio default\n\n# Default model (can be overridden per request)\nINFERENCE_MODEL=llama3.2\n\n# Seconds before a single request times out\nINFERENCE_TIMEOUT=120\n\n# Parallel requests per batch chunk — keep at 1 for CPU, raise to 3-5 for GPU\nINFERENCE_CONCURRENCY=1\n```\n\n### Step 4 — Start a queue worker\n\nBatch jobs run asynchronously. The worker must be running:\n\n```bash\nphp artisan queue:work\n```\n\n---\n\n## Two Ways to Use This Package\n\nThis package supports two independent usage patterns:\n\n| | Event-based | REST API |\n|---|---|---|\n| **Who calls it** | Your own Laravel code | Any HTTP client (SDK, curl, external app) |\n| **Auth** | Laravel's existing auth | Sanctum token (or your middleware) |\n| **Routes needed** | No | Yes |\n| **Best for** | Internal pipelines, jobs, commands | Replacing Anthropic/OpenAI SDK endpoints |\n\n---\n\n## Approach 1 — Event-based (Internal Usage)\n\nUse this when your own Laravel application needs to submit and process batches. No HTTP routes required.\n\n### Submitting an Anthropic-format batch\n\nFire a `SubmitAnthropicBatchEvent` event. The package listener picks it up and dispatches the processing job automatically.\n\n```php\nuse BatchApi\\Events\\SubmitAnthropicBatchEvent;\nuse BatchApi\\Data\\Input\\AnthropicBatchItemDto;\n\n$items = [\n    new AnthropicBatchItemDto(\n        customId: 'req-1',\n        maxTokens: 512,\n        messages: [\n            ['role' =\u003e 'user', 'content' =\u003e 'Summarise this article in one paragraph.'],\n        ],\n    ),\n    new AnthropicBatchItemDto(\n        customId: 'req-2',\n        maxTokens: 256,\n        messages: [\n            ['role' =\u003e 'user', 'content' =\u003e 'What is the capital of France?'],\n        ],\n        system: 'You are a geography expert.',\n    ),\n];\n\nevent(new SubmitAnthropicBatchEvent($items));\n```\n\n### Submitting an OpenAI-format batch\n\nThe OpenAI flow requires a file ID. Upload first using the `BatchService`, then fire the event.\n\n```php\nuse BatchApi\\BatchService;\nuse BatchApi\\Events\\SubmitOpenAiBatchEvent;\nuse BatchApi\\Data\\Input\\OpenAiBatchItemDto;\n\n$service = app(BatchService::class);\n\n// Build items from raw JSONL or manually\n$items = [\n    new OpenAiBatchItemDto(\n        customId: 'req-1',\n        messages: [['role' =\u003e 'user', 'content' =\u003e 'Hello']],\n        maxTokens: 512,\n    ),\n];\n\n// Create a file record (mirrors OpenAI's file upload step)\n$file = $service-\u003euploadFile(\n    collect($items)-\u003emap(fn ($item) =\u003e json_encode([\n        'custom_id' =\u003e $item-\u003ecustomId,\n        'method' =\u003e 'POST',\n        'url' =\u003e '/v1/chat/completions',\n        'body' =\u003e ['messages' =\u003e $item-\u003emessages, 'max_tokens' =\u003e $item-\u003emaxTokens],\n    ]))-\u003eimplode(\"\\n\")\n);\n\nevent(new SubmitOpenAiBatchEvent($file-\u003eid, $items));\n```\n\n### Listening for results\n\nListen to `BatchCompletedEvent` to act on results when processing finishes:\n\n```php\n// app/Listeners/HandleBatchCompletedListener.php\n\nuse BatchApi\\Events\\BatchCompletedEvent;\nuse BatchApi\\Data\\BatchResultDto;\n\nclass HandleBatchCompletedListener\n{\n    public function handle(BatchCompletedEvent $event): void\n    {\n        $batch = $event-\u003ebatch;\n\n        foreach ($event-\u003eresults as $result) {\n            /** @var BatchResultDto $result */\n            if ($result-\u003esucceeded) {\n                // $result-\u003ecustomId   — matches your request's custom_id\n                // $result-\u003econtent    — the model's response text\n                // $result-\u003emodel      — model used\n                // $result-\u003einputTokens / $result-\u003eoutputTokens\n            } else {\n                // $result-\u003eerror — failure message\n            }\n        }\n    }\n}\n```\n\nRegister it in `AppServiceProvider::boot()`:\n\n```php\n// app/Providers/AppServiceProvider.php\n\nuse BatchApi\\Events\\BatchCompletedEvent;\nuse App\\Listeners\\HandleBatchCompletedListener;\nuse Illuminate\\Support\\Facades\\Event;\n\npublic function boot(): void\n{\n    Event::listen(BatchCompletedEvent::class, HandleBatchCompletedListener::class);\n}\n```\n\n### All available events\n\n| Event | Properties | Fired when |\n|-------|-----------|------------|\n| `BatchCreatedEvent` | `$batch`, `$items`, `$provider` | Batch record saved, job dispatched |\n| `BatchProcessingEvent` | `$batch` | Queue worker picks up the job |\n| `BatchItemStartedEvent` | `$batch`, `$dto` | Single request about to fire |\n| `BatchItemCompletedEvent` | `$batch`, `$result` | Single request finished |\n| `BatchCompletedEvent` | `$batch`, `$results` | All requests done |\n| `BatchFailedEvent` | `$batch`, `$exception` | Job threw an unrecoverable error |\n| `BatchCancelledEvent` | `$batch` | Batch cancelled |\n\n---\n\n## Approach 2 — REST API (External HTTP Clients)\n\nUse this when you want to **point an existing Anthropic or OpenAI SDK** at your local server instead of the cloud. The API surface is identical to the real APIs.\n\n### Step 1 — Register routes with authentication\n\nDo **not** set `BATCH_API_EXPOSE_ROUTES=true`. Instead, register routes manually inside a protected middleware group so you control authentication.\n\nInstall Sanctum if you haven't already:\n\n```bash\ncomposer require laravel/sanctum\nphp artisan install:api\n```\n\nIn your `routes/api.php` (or a service provider), wrap `BatchApi::routes()` with Sanctum middleware:\n\n```php\nuse BatchApi\\Facades\\BatchApi;\n\nRoute::middleware('auth:sanctum')-\u003egroup(function () {\n    BatchApi::routes();\n});\n```\n\nThis registers all 11 endpoints, each requiring a valid Sanctum token.\n\n\u003e **Note:** `BatchApi::routes()` also applies the `api` middleware internally. Wrapping it with `auth:sanctum` stacks both, so your routes have `api` + `auth:sanctum`.\n\n### Step 2 — Issue a token\n\n```php\n// In a controller or seeder\n$token = $user-\u003ecreateToken('batch-api-client')-\u003eplainTextToken;\n// Pass this token to the HTTP client\n```\n\n### Step 3 — Call the API\n\nAll requests need the token in the `Authorization` header:\n\n```\nAuthorization: Bearer \u003ctoken\u003e\n```\n\n---\n\n### Anthropic Batch API — Step by Step\n\n#### 1. Submit a batch\n\n```http\nPOST /api/anthropic/v1/messages/batches\nContent-Type: application/json\nAuthorization: Bearer \u003ctoken\u003e\n```\n\n```json\n{\n  \"requests\": [\n    {\n      \"custom_id\": \"req-1\",\n      \"params\": {\n        \"model\": \"llama3.2\",\n        \"max_tokens\": 512,\n        \"messages\": [\n          { \"role\": \"user\", \"content\": \"Say hello in one sentence.\" }\n        ]\n      }\n    },\n    {\n      \"custom_id\": \"req-2\",\n      \"params\": {\n        \"model\": \"llama3.2\",\n        \"max_tokens\": 512,\n        \"system\": \"You are a pirate. Always respond like a pirate.\",\n        \"messages\": [\n          { \"role\": \"user\", \"content\": \"What is the capital of France?\" }\n        ]\n      }\n    }\n  ]\n}\n```\n\nResponse `202 Accepted`:\n\n```json\n{\n  \"id\": \"550e8400-e29b-41d4-a716-446655440000\",\n  \"type\": \"message_batch\",\n  \"processing_status\": \"in_progress\",\n  \"request_counts\": { \"processing\": 2, \"succeeded\": 0, \"errored\": 0, \"canceled\": 0, \"expired\": 0 },\n  \"created_at\": \"2026-05-25T10:00:00+00:00\",\n  \"expires_at\": \"2026-05-26T10:00:00+00:00\",\n  \"ended_at\": null,\n  \"cancel_initiated_at\": null,\n  \"results_url\": null\n}\n```\n\n#### 2. Poll until done\n\n```http\nGET /api/anthropic/v1/messages/batches/{id}\nAuthorization: Bearer \u003ctoken\u003e\n```\n\nKeep polling until `processing_status` is `\"ended\"`.\n\n#### 3. Fetch results (NDJSON)\n\n```http\nGET /api/anthropic/v1/messages/batches/{id}/results\nAccept: application/x-ndjson\nAuthorization: Bearer \u003ctoken\u003e\n```\n\nReturns `204 No Content` if still processing. When ready, streams one JSON object per line:\n\n```jsonl\n{\"custom_id\":\"req-1\",\"result\":{\"type\":\"succeeded\",\"message\":{\"id\":\"msg_abc\",\"type\":\"message\",\"role\":\"assistant\",\"model\":\"llama3.2\",\"content\":[{\"type\":\"text\",\"text\":\"Hello! Great to meet you.\"}],\"stop_reason\":\"end_turn\",\"usage\":{\"input_tokens\":12,\"output_tokens\":10}}}}\n{\"custom_id\":\"req-2\",\"result\":{\"type\":\"errored\",\"error\":{\"type\":\"server_error\",\"message\":\"Ollama timeout\"}}}\n```\n\n#### Other Anthropic endpoints\n\n```http\nGET  /api/anthropic/v1/messages/batches              # list (supports ?limit=\u0026before_id=\u0026after_id=)\nPOST /api/anthropic/v1/messages/batches/{id}/cancel  # cancel\n```\n\n---\n\n### OpenAI Batch API — Step by Step\n\n#### 1. Upload a JSONL file\n\nCreate a `.jsonl` file (one request per line):\n\n```jsonl\n{\"custom_id\":\"req-1\",\"method\":\"POST\",\"url\":\"/v1/chat/completions\",\"body\":{\"model\":\"llama3.2\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}],\"max_tokens\":512}}\n{\"custom_id\":\"req-2\",\"method\":\"POST\",\"url\":\"/v1/chat/completions\",\"body\":{\"model\":\"llama3.2\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 2+2?\"}],\"max_tokens\":256}}\n```\n\nUpload it:\n\n```http\nPOST /api/openai/v1/files\nContent-Type: multipart/form-data\nAuthorization: Bearer \u003ctoken\u003e\n\nfile=@requests.jsonl\npurpose=batch\n```\n\nResponse `201 Created`:\n\n```json\n{\n  \"id\": \"file-abc123\",\n  \"object\": \"file\",\n  \"purpose\": \"batch\",\n  \"created_at\": 1716631200\n}\n```\n\n#### 2. Submit the batch\n\n```http\nPOST /api/openai/v1/batches\nContent-Type: application/json\nAuthorization: Bearer \u003ctoken\u003e\n```\n\n```json\n{\n  \"input_file_id\": \"file-abc123\",\n  \"endpoint\": \"/v1/chat/completions\",\n  \"completion_window\": \"24h\"\n}\n```\n\nResponse `201 Created`:\n\n```json\n{\n  \"id\": \"550e8400-e29b-41d4-a716-446655440001\",\n  \"object\": \"batch\",\n  \"status\": \"validating\",\n  \"input_file_id\": \"file-abc123\",\n  \"output_file_id\": null,\n  \"request_counts\": { \"total\": 2, \"completed\": 0, \"failed\": 0 }\n}\n```\n\n#### 3. Poll until completed\n\n```http\nGET /api/openai/v1/batches/{id}\nAuthorization: Bearer \u003ctoken\u003e\n```\n\nPoll until `status` is `\"completed\"`. Note the `output_file_id` in the response.\n\n#### 4. Download results\n\n```http\nGET /api/openai/v1/files/{output_file_id}/content\nAuthorization: Bearer \u003ctoken\u003e\n```\n\nReturns JSONL, one result per line:\n\n```jsonl\n{\"id\":\"batch_req_abc\",\"custom_id\":\"req-1\",\"response\":{\"status_code\":200,\"body\":{\"id\":\"chatcmpl-123\",\"object\":\"chat.completion\",\"model\":\"llama3.2\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"Hello! How can I help?\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":10,\"completion_tokens\":8,\"total_tokens\":18}}},\"error\":null}\n```\n\n#### Other OpenAI endpoints\n\n```http\nGET  /api/openai/v1/batches                  # list (supports ?limit=\u0026after=)\nPOST /api/openai/v1/batches/{id}/cancel      # cancel\n```\n\n---\n\n### Pointing an existing SDK at this server\n\n**Python (Anthropic SDK):**\n\n```python\nimport anthropic\n\nclient = anthropic.Anthropic(\n    api_key=\"any-value\",           # required by SDK but not validated here\n    base_url=\"http://localhost:8000/api/anthropic\",\n    default_headers={\"Authorization\": \"Bearer \u003ctoken\u003e\"},\n)\n```\n\n**Python (OpenAI SDK):**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"any-value\",\n    base_url=\"http://localhost:8000/api/openai\",\n    default_headers={\"Authorization\": \"Bearer \u003ctoken\u003e\"},\n)\n```\n\n---\n\n## Batch Status Lifecycle\n\n```\npending → processing → completed\n                     → failed\n       → cancelling  → cancelled\n```\n\n| Internal | Anthropic `processing_status` | OpenAI `status` |\n|----------|-------------------------------|-----------------|\n| `pending` | `in_progress` | `validating` |\n| `processing` | `in_progress` | `in_progress` |\n| `completed` | `ended` | `completed` |\n| `failed` | `ended` | `failed` |\n| `cancelling` | `canceling` | `cancelling` |\n| `cancelled` | `ended` | `cancelled` |\n\nBatches expire after 24 hours.\n\n---\n\n## Switching to LM Studio\n\n1. Open LM Studio → start the local server (default port `1234`)\n2. Load a model\n3. Update `.env`:\n\n```env\nINFERENCE_PROVIDER=lmstudio\nINFERENCE_URL=http://localhost:1234\nINFERENCE_MODEL=your-model-name\n```\n\nNo other changes needed.\n\n---\n\n## Concurrency Tuning\n\n`INFERENCE_CONCURRENCY` controls parallel requests per batch chunk.\n\n| Hardware | Value |\n|----------|-------|\n| CPU-only | `1` |\n| GPU with spare VRAM | `3`–`5` |\n\n---\n\n## Postman Collection\n\nImport `Local-Batch-API.postman_collection.json`. Set the `baseUrl` variable to your server URL. The collection auto-saves batch IDs and file IDs between requests so you can run folders top-to-bottom without manually copying values.\n\n---\n\n## Troubleshooting\n\n**Batches stay `pending` forever** — Queue worker not running. Run `php artisan queue:work`.\n\n**`Ollama timeout` in results** — Model is slow or `INFERENCE_TIMEOUT` too low. Raise to `300`.\n\n**Routes return 404** — Routes not registered. Either set `BATCH_API_EXPOSE_ROUTES=true` (no auth) or call `BatchApi::routes()` manually in a middleware group.\n\n**401 Unauthorized on API routes** — Sanctum token missing or invalid. Pass `Authorization: Bearer \u003ctoken\u003e` header.\n\n**`cannot chdir` git error in submodule** — Run `git submodule update --init` in the parent repo.\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigitaldreams%2Flocal-batch-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdigitaldreams%2Flocal-batch-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigitaldreams%2Flocal-batch-api/lists"}