{"id":21223744,"url":"https://github.com/zmh-program/blob-service","last_synced_at":"2025-03-14T15:07:20.553Z","repository":{"id":205016014,"uuid":"711860538","full_name":"zmh-program/blob-service","owner":"zmh-program","description":"📦 Out-Of-The-Box \u0026 Powerful File Parsing Service, support Text/Pdf/Docx/Pptx/Xlsx/Image/Audio parsing, support OCR, support Base64/Local/S3/R2/TG/MinIO storage.","archived":false,"fork":false,"pushed_at":"2024-12-04T01:16:39.000Z","size":66,"stargazers_count":101,"open_issues_count":1,"forks_count":40,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-04T14:25:16.055Z","etag":null,"topics":["blob","fileparser","ocr","storage"],"latest_commit_sha":null,"homepage":"https://blob.chatnio.net","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zmh-program.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-30T10:19:02.000Z","updated_at":"2025-02-24T15:48:04.000Z","dependencies_parsed_at":"2023-11-15T13:28:07.514Z","dependency_job_id":"ac88fbc9-5bfa-4355-8636-053c76e72953","html_url":"https://github.com/zmh-program/blob-service","commit_stats":null,"previous_names":["deeptrain-community/chatnio-blob-service","zmh-program/blob-service"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmh-program%2Fblob-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmh-program%2Fblob-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmh-program%2Fblob-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zmh-program%2Fblob-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zmh-program","download_url":"https://codeload.github.com/zmh-program/blob-service/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243597833,"owners_count":20316844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blob","fileparser","ocr","storage"],"created_at":"2024-11-20T22:53:15.017Z","updated_at":"2025-03-14T15:07:20.516Z","avatar_url":"https://github.com/zmh-program.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \n# 📦 Chat Nio Blob Service\n\n### **🤯 File Service for Chat Nio**\n\n[![Deploy to Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https://github.com/Deeptrain-Community/chatnio-blob-service)\n\n[![Deploy on Zeabur](https://zeabur.com/button.svg)](https://zeabur.com/templates/RWGFOH)\n\n\u003c/div\u003e\n\n## Features\n- ⚡ **Out-of-the-Box**: No External Dependencies Required \u0026 Support Vercel/Render One-Click Deployment\n- ⭐ **Multiple File Types**: Support Text, Pdf, Docx, Excel, Image, Audio etc.\n- 📦 **Multiple Storage Options**: Base64, Local, S3, Cloudflare R2, Min IO, Telegram CDN etc.\n- 🔍 **OCR Support**: Extract Text from Image (Require Paddle OCR API)\n- 🔊 **Audio Support**: Convert Audio to Text (Require Azure Speech to Text Service)\n\n## Supported File Types\n- Text\n- Image (_require vision models_)\n- Audio (_require Azure Speech to Text Service_)\n- Docx (_not support .doc_)\n- Pdf\n- Pptx (_not support .ppt_)\n- Xlsx (_support .xls_)\n\n\n## Deploy by Docker\n\u003e Image: `programzmh/chatnio-blob-service`\n\n```shell\ndocker run -p 8000:8000 programzmh/chatnio-blob-service\n\n# with environment variables\n# docker run -p 8000:8000 -e AZURE_SPEECH_KEY=\"...\" -e AZURE_SPEECH_REGION=\"...\" programzmh/chatnio-blob-service\n\n\n# if you are using `local` storage type, you need to mount volume (/app/static) to the host\n# docker run -p 8000:8000 -v /path/to/static:/app/static programzmh/chatnio-blob-service\n```\n\n\u003e Deploy to [Render.com](https://render.com)\n\u003e \n\u003e [![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://dashboard.render.com/select-image?type=web\u0026image=programzmh%2Fchatnio-blob-service)\n\u003e\n\u003e \n\u003e Select **Web Service** and **Docker** Image, then input the image `programzmh/chatnio-blob-service` and click **Create Web Service**.\n\u003e \u003e ⭐ Render.com Includes Free **750 Hours** of Usage per Month \n\u003e \n\n## Deploy by Source Code\nThe service will be running on `http://localhost:8000`\n## Run\n```shell\ngit clone --branch=main https://github.com/Deeptrain-Community/chatnio-blob-service\ncd chatnio-blob-service\n\npip install -r requirements.txt\nuvicorn main:app\n\n# enable hot reload\n# uvicorn main:app --reload\n```\n\n\n## API\n`POST` `/upload` Upload a file\n```json\n{\n    \"file\": \"[file]\",\n    \"enable_ocr\": false,\n    \"enable_vision\": true,\n    \"save_all\": false\n}\n```\n\n| Parameter       | Type    | Description                                                                          |\n|-----------------|---------|--------------------------------------------------------------------------------------|\n| `file`          | *File   | File to Upload                                                                       |\n| `enable_ocr`    | Boolean | Enable OCR (Default: `false`) \u003cbr/\u003e**should configure OCR config*                    |\n| `enable_vision` | Boolean | Enable Vision (Default: `true`) \u003cbr/\u003e**skip if `enable_ocr` is true*                 |\n| `save_all`      | Boolean | Save All Images (Default: `false`) \u003cbr/\u003e**store all types of files without handling* |\n\n\nResponse\n\n```json\n{\n  \"status\": true,\n  \"type\": \"pdf\",\n  \"content\": \"...\",\n  \"error\": \"\"\n}\n```\n\n| Parameter       | Type     | Description    |\n|-----------------|----------|----------------|\n| `status`        | Boolean  | Request Status |\n| `type`          | String   | File Type      |\n| `content`       | String   | File Data      |\n| `error`         | String   | Error Message  |\n\n## Environment Variables\n\n### `1` 🎨 General Config (Optional)\n\n- `PDF_MAX_IMAGES`: Max Images Extracted from a PDF File (Default: `10`)\n    - **0**: Never Extract Images\n    - **-1**: Extract All Images\n    - **other**: Extract Top N Images\n    - *Tips: The extracted images will be **treated as a normal image** file and directly processed*.\n- `MAX_FILE_SIZE`: Max Uploaded File Size MiB (Default: `-1`, No Limit)\n  - *Tips: Size limit is also depend on the server configuration (e.g. Nginx/Apache Config, Vercel Free Plan Limit **5MB** Body Size)*\n- `CORS_ALLOW_ORIGINS`: CORS Allow Origins (Default: `*`)\n  - e.g.: *http://localhost:3000,https://example.com*\n\n### `2` 🔊 Audio Config (Optional)\n- `AZURE_SPEECH_KEY`: Azure Speech to Text Service Key (Required for Audio Support)\n- `AZURE_SPEECH_REGION`: Azure Speech to Text Service Region (Required for Audio Support)\n\n### `3` 🖼 Storage Config (Optional)\n\u003e [!NOTE]\n\u003e Storage Config Apply to **Image** Files And `Save All` Option Only.\n\n1. ✨ No Storage (Default)\n   - [x] **No Storage Required \u0026 No External Dependencies**\n   - [x] Base64 Encoding/Decoding\n   - [x] Do **Not** Store Anything\n   - [x] Support Serverless Deployment **Without Storage** (e.g. Vercel)\n   - [ ] No Direct URL Access *(Base64 not support models like `gpt-4-all`)*\n\n2. 📁 Local Storage\n   - [ ] **Require Server Environment** (e.g. VPS, Docker)\n   - [x] Support Direct URL Access\n   - [x] Payless Storage Cost\n   - Config:\n     - set env `STORAGE_TYPE` to `local` (e.g. `STORAGE_TYPE=local`)\n     - set env `LOCAL_STORAGE_DOMAIN` to your deployment domain (e.g. `LOCAL_STORAGE_DOMAIN=http://blob-service.onrender.com`)\n     - if you are using Docker, you need to mount volume `/app/static` to the host (e.g. `-v /path/to/static:/app/static`)\n     \n3. 🚀 [AWS S3](https://aws.amazon.com/s3)\n   - [ ] **Payment Storage Cost**\n   - [x] Support Direct URL Access\n   - [x] China Mainland User Friendly\n   - Config:\n     - set env `STORAGE_TYPE` to `s3` (e.g. `STORAGE_TYPE=s3`)\n     - set env `S3_ACCESS_KEY` to your AWS Access Key ID\n     - set env `S3_SECRET_KEY` to your AWS Secret Access Key\n     - set env `S3_BUCKET` to your AWS S3 Bucket Name\n     - set env `S3_REGION` to your AWS S3 Region\n\n4. 🔔 [Cloudflare R2](https://www.cloudflare.com/zh-cn/developer-platform/r2)\n   - [x] **Free Storage Quota ([10GB Storage \u0026 Zero Outbound Cost]((https://developers.cloudflare.com/r2/pricing/)))**\n   - [x] Support Direct URL Access\n   - Config *(S3 Compatible)*:\n     - set env `STORAGE_TYPE` to `s3` (e.g. `STORAGE_TYPE=s3`)\n     - set env `S3_ACCESS_KEY` to your Cloudflare R2 Access Key ID\n     - set env `S3_SECRET_KEY` to your Cloudflare R2 Secret Access Key\n     - set env `S3_BUCKET` to your Cloudflare R2 Bucket Name\n     - set env `S3_DOMAIN` to your Cloudflare R2 Domain Name (e.g. `https://\u003caccount-id\u003e.r2.cloudflarestorage.com`)\n     - set env `S3_DIRECT_URL_DOMAIN` to your Cloudflare R2 Public URL Access Domain Name ([Open Public URL Access](https://developers.cloudflare.com/r2/buckets/public-buckets/), e.g. `https://pub-xxx.r2.dev`)\n\n5. 📦 [Min IO](https://min.io)\n    - [x] **Self Hosted**\n    - [x] Reliable \u0026 Flexible Storage\n    - Config *(S3 Compatible)*:\n      - set env `STORAGE_TYPE` to `s3` (e.g. `STORAGE_TYPE=s3`)\n      - set env `S3_SIGN_VERSION` to `s3v4` (e.g. `S3_SIGN_VERSION=s3v4`)\n      - set env `S3_ACCESS_KEY` to your Min IO Access Key ID\n      - set env `S3_SECRET_KEY` to your Min IO Secret Access Key\n      - set env `S3_BUCKET` to your Min IO Bucket Name\n      - set env `S3_DOMAIN` to your Min IO Domain Name (e.g. `https://oss.example.com`)\n      - *[Optional] If you are using CDN, you can set `S3_DIRECT_URL_DOMAIN` to your Min IO Public URL Access Domain Name (e.g. `https://cdn-hk.example.com`)*\n\n6. ❤ [Telegram CDN](https://github.com/csznet/tgState)\n    - [x] **Free Storage (Rate Limit)**\n    - [x] Support Direct URL Access *(China Mainland User Unfriendly)*\n    - [x] **Limited** File Type \u0026 Format\n    - [x] Config:\n      - set env `STORAGE_TYPE` to `tg` (e.g. `STORAGE_TYPE=tg`)\n      - set env `TG_ENDPOINT` to your TG-STATE Endpoint (e.g. `TG_ENDPOINT=https://tgstate.vercel.app`)\n      - *[Optional] if you are using password authentication, you can set `TG_PASSWORD` to your TG-STATE Password*\n\n    \n### `4` 🔍 OCR Config (Optional)\n\u003e [!NOTE]\n\u003e OCR Support is based on 👉 [PaddleOCR API](https://github.com/cgcel/PaddleOCRFastAPI) (✔ Self Hosted ✔ Open Source)\n\n- `OCR_ENDPOINT` Paddle OCR Endpoint\n    - *e.g.: *http://example.com:8000*\n\n## Common Errors\n- *Cannot Use `Save All` Options Without Storage Config*:\n    - This error occurs when you enable `save_all` option without storage config. You need to set `STORAGE_TYPE` to `local` or other storage type to use this option.\n- *Trying to upload image with Vision disabled. Enable Vision or OCR to process image*:\n    - This error occurs when you disable `enable_vision` and `enable_ocr` at the same time. You need to enable at least one of them to process image files.\n- *.ppt files are not supported, only .pptx files are supported*:\n    - This error occurs when you upload a old version of Office PowerPoint file. You need to convert it to `.pptx` format to process it.\n- *.doc files are not supported, only .docx files are supported*:\n    - This error occurs when you upload a old version of Office Word file. You need to convert it to `.docx` format to process it.\n- *File Size Limit Exceeded*:\n    - This error occurs when you upload a file that exceeds the `MAX_FILE_SIZE` limit. You need to reduce the file size to upload it.\n## Development\n- **~/config.py**: Env Config\n- **~/main.py**: Entry Point\n- **~/utils.py**: Utilities\n- **~/handlers**: File Handlers\n- **~/store**: Storage Handlers\n- **~/static**: Static Files (if using **local** storage)\n\n## Tech Stack\n- Python \u0026 FastAPI\n\n## License\nApache License 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzmh-program%2Fblob-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzmh-program%2Fblob-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzmh-program%2Fblob-service/lists"}