{"id":49069116,"url":"https://github.com/rzem-ai/rzem-ai-inference-engine","last_synced_at":"2026-04-20T06:15:30.020Z","repository":{"id":339186692,"uuid":"1153524945","full_name":"rzem-ai/rzem-ai-inference-engine","owner":"rzem-ai","description":"Text-to-image inference engine with job queue, automatic VRAM management, and LoRA support. Unified API across FLUX, Z-Image, and Qwen-Image model families.","archived":false,"fork":false,"pushed_at":"2026-02-25T23:28:26.000Z","size":317,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-26T01:47:05.327Z","etag":null,"topics":["desktop-application","flux1-dev","flux2-dev","image-generation","qwen-image","z-image"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rzem-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-09T11:58:09.000Z","updated_at":"2026-02-25T23:28:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rzem-ai/rzem-ai-inference-engine","commit_stats":null,"previous_names":["rzem-ai/rzem-ai-inference-engine"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/rzem-ai/rzem-ai-inference-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rzem-ai%2Frzem-ai-inference-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rzem-ai%2Frzem-ai-inference-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rzem-ai%2Frzem-ai-inference-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rzem-ai%2Frzem-ai-inference-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rzem-ai","download_url":"https://codeload.github.com/rzem-ai/rzem-ai-inference-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rzem-ai%2Frzem-ai-inference-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32035391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["desktop-application","flux1-dev","flux2-dev","image-generation","qwen-image","z-image"],"created_at":"2026-04-20T06:15:26.464Z","updated_at":"2026-04-20T06:15:29.927Z","avatar_url":"https://github.com/rzem-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RZEM AI Inference Engine\n\nA Python text-to-image inference engine with job queue, event callbacks, and automatic VRAM management. Supports multiple model families with a unified API that hides architectural differences while exposing full parameter control.\n\n## Supported Models\n\n| Model Family | Transformer | Text Encoder | VAE | Default Steps |\n|---|---|---|---|---|\n| **FLUX.1 Dev** | `FluxTransformer2DModel` | CLIP + T5-XXL | 16-ch AutoencoderKL | 20 |\n| **FLUX.2 Dev** | `Flux2Transformer2DModel` | Qwen3 (multi-layer) | 32-ch AutoencoderKL + BN | 20 |\n| **Z-Image** | `ZImageTransformer2DModel` (S3-DiT) | Qwen3-4B | 16-ch AutoencoderKL | 9 |\n| **Qwen-Image** | `QwenImageTransformer2DModel` (20B MMDiT) | Qwen3 | 16-ch AutoencoderKL | 50 |\n| **FAL.ai Cloud** | Remote (fal-ai endpoints) | N/A | N/A | Endpoint-dependent |\n\nAll local models support LoRA weight patching with automatic format detection (Kohya, Diffusers/PEFT, XLabs, AIToolkit, OneTrainer).\n\nFAL.ai cloud generation delegates to remote endpoints (e.g. `fal-ai/flux/dev`, `fal-ai/flux-pro/v1.1`, `fal-ai/flux-2`) — no local models are loaded.\n\n## Installation\n\n```bash\nuv sync    # or: pip install -e .\n```\n\nRequires Python 3.10+ and PyTorch 2.0+. GPU with 24+ GB VRAM recommended. Apple Silicon M3+ with PyTorch 2.3+ is supported (MPS backend).\n\n## Distribution\n\n### Standalone Executable\n\nBuild a standalone executable that bundles all dependencies (models are still user-provided):\n\n```bash\n# Build server variant (generate + serve commands, ~3-4 GB)\nbash scripts/build_executable.sh server\n\n# Build CLI variant (generate only, ~2.5-3.5 GB)\nbash scripts/build_executable.sh cli\n\n# Build both variants (default)\nbash scripts/build_executable.sh all\n\n# Run the executables\n./dist/rzem-ai-inference-engine-server/rzem-ai-inference-engine-server generate --help\n./dist/rzem-ai-inference-engine-server/rzem-ai-inference-engine-server serve --help\n./dist/rzem-ai-inference-engine-cli/rzem-ai-inference-engine-cli generate --help\n```\n\nSee [packaging/README.md](packaging/README.md) for detailed build instructions, platform requirements, and distribution options.\n\n**Note:** Executables do not include model weights. Models must be downloaded separately via HuggingFace Hub or provided as local paths.\n\n## Usage\n\n### Python API\n\n```python\nfrom rzem_ai_inference_engine import InferenceEngine, JobParams, EventType, TransformerType\n\nengine = InferenceEngine()\n\n# Listen for events\nengine.on(EventType.JOB_PROGRESS, lambda e: print(f\"Step {e.step}/{e.total_steps}\"))\nengine.on(EventType.JOB_COMPLETED, lambda e: e.image.save(\"output.png\"))\n\n# Submit a job\njob_id = engine.submit(JobParams(\n    prompt=\"a cat sitting on a windowsill, golden hour lighting\",\n    transformer_model=\"black-forest-labs/FLUX.1-dev\",\n    transformer_type=TransformerType.FLUX1_DEV,\n    clip_tokenizer=\"black-forest-labs/FLUX.1-dev\",\n    clip_encoder=\"black-forest-labs/FLUX.1-dev\",\n    t5_tokenizer=\"black-forest-labs/FLUX.1-dev\",\n    t5_encoder=\"black-forest-labs/FLUX.1-dev\",\n    vae_model=\"black-forest-labs/FLUX.1-dev\",\n    steps=20,\n    cfg_scale=3.5,\n    width=1024,\n    height=1024,\n    seed=42,\n))\n\n# ... engine processes the job in a background thread ...\n# Call engine.shutdown() when done\n```\n\n### CLI\n\n```bash\n# FLUX.1 Dev — all models from Black Forest Labs\nrzem-ai-inference-engine generate \\\n    --prompt \"a cat sitting on a windowsill, golden hour lighting\" \\\n    --transformer-model black-forest-labs/FLUX.1-dev \\\n    --transformer-type flux1_dev \\\n    --clip-tokenizer black-forest-labs/FLUX.1-dev \\\n    --clip-encoder black-forest-labs/FLUX.1-dev \\\n    --t5-tokenizer black-forest-labs/FLUX.1-dev \\\n    --t5-encoder black-forest-labs/FLUX.1-dev \\\n    --vae-model black-forest-labs/FLUX.1-dev \\\n    --steps 20 --cfg-scale 3.5 \\\n    --width 1024 --height 1024 \\\n    --seed 42 \\\n    --output output.png\n\n# Z-Image Turbo\nrzem-ai-inference-engine generate \\\n    --prompt \"mountain landscape at sunset\" \\\n    --transformer-model Tongyi-MAI/Z-Image-Turbo \\\n    --transformer-type z_image \\\n    --qwen3-tokenizer Qwen/Qwen3-4B \\\n    --qwen3-encoder Qwen/Qwen3-4B \\\n    --vae-model black-forest-labs/FLUX.1-dev \\\n    --steps 9 --cfg-scale 1.0 \\\n    --output output.png\n\n# With LoRAs (format: path:strength)\nrzem-ai-inference-engine generate \\\n    ... \\\n    --lora ./models/anime.safetensors:0.8 \\\n    --lora ./models/detail.safetensors:0.5 \\\n    --output output.png\n```\n\n### Model Paths\n\nModel path arguments accept three formats:\n\n| Format | Example | Behavior |\n|---|---|---|\n| Local path | `./models/flux.safetensors` | Used directly |\n| HuggingFace repo | `black-forest-labs/FLUX.1-dev` | Downloads full repo, auto-resolves subfolders |\n| HuggingFace repo + file | `city96/FLUX.1-dev-gguf/flux1-dev-Q8_0.gguf` | Downloads only the specified file |\n\nMulti-component HF repos (like `black-forest-labs/FLUX.1-dev`) are handled automatically — the engine resolves subfolders like `transformer/`, `text_encoder/`, `vae/`, etc.\n\n## REST API Server\n\nStart the server to accept jobs over HTTP and receive real-time updates via WebSocket:\n\n```bash\n# Listen on all interfaces (enables LAN discovery)\nrzem-ai-inference-engine serve --host 0.0.0.0 --port 8000 --device auto --output-dir ./output\n\n# Or run as a background daemon\nbash scripts/server.sh start --port 8000 --device cuda\nbash scripts/server.sh status\nbash scripts/server.sh stop\n```\n\n| Endpoint | Description |\n|---|---|\n| `POST /jobs` | Submit a generation job |\n| `GET /jobs` | List all jobs |\n| `GET /jobs/{id}` | Get job details |\n| `GET /jobs/{id}/image` | Download generated image |\n| `DELETE /jobs/{id}` | Cancel a job |\n| `GET /models` | List locally cached HuggingFace models |\n| `GET /health` | Health check with queue stats |\n| `WS /ws` | WebSocket — real-time job event broadcasts (JSON) |\n\n### Network Announcement (LAN Discovery)\n\nWhen the server binds to a non-localhost address (e.g. `--host 0.0.0.0`), it automatically announces itself on the local network via **mDNS/DNS-SD** — the same protocol used by AirPlay, Chromecast, and network printers. Client applications can discover running servers without manual configuration.\n\n- **Service type**: `_rzem-ai._tcp.local.`\n- **TXT record**: `version`, `device`, `api=rest`, `ws=/ws`\n- **Disable**: `--no-announce`\n\nWhen bound to `127.0.0.1` (the default), announcement is skipped since the server is only reachable locally.\n\n**Client-side discovery example:**\n\n```python\nfrom zeroconf import ServiceBrowser, ServiceListener, Zeroconf\n\nclass Listener(ServiceListener):\n    def add_service(self, zc: Zeroconf, type_: str, name: str) -\u003e None:\n        info = zc.get_service_info(type_, name)\n        if info:\n            addr = info.parsed_addresses()[0]\n            port = info.port\n            print(f\"Found server at {addr}:{port}\")\n\nzc = Zeroconf()\nbrowser = ServiceBrowser(zc, \"_rzem-ai._tcp.local.\", Listener())\n```\n\n## VRAM Management\n\nThe engine manages a two-tier model cache:\n\n- **VRAM (hot)**: Models actively on the GPU for fast inference\n- **RAM (warm)**: Models offloaded to CPU, moved back to GPU on demand\n\nWhen VRAM is insufficient for the next model, the cache evicts the **smallest unlocked** model first. A 1 GB working memory buffer is reserved for intermediate tensors.\n\nFor FLUX.1 Dev on GPUs with ~32 GB VRAM, the pipeline automatically sequences model loading: text encoders are used then released before the 22 GB transformer is loaded.\n\n### Dtype Selection\n\nAll pipelines resolve dtype automatically via `preferred_dtype(device)`:\n\n| Device | Dtype | Notes |\n|---|---|---|\n| CUDA (Ampere+) | bfloat16 | Native tensor-core support |\n| MPS (M3+) | bfloat16 | Native GPU ALU support (PyTorch 2.3+) |\n| CPU | float32 | No reduced-precision benefit |\n\n### Apple Silicon Notes\n\n- On MPS with unified memory, cache eviction is effectively a no-op — all models share the same physical RAM. Use `--vram-limit` to constrain memory if needed.\n- GGUF quantized models are slower on MPS than full-precision bfloat16 due to unoptimized dequantization kernels. Use full-precision models for best performance.\n\n## Events\n\n| Event | Payload | When |\n|---|---|---|\n| `JOB_QUEUED` | `QueuedEvent` | Job added to queue |\n| `JOB_STARTED` | `StartedEvent` | Processing begins |\n| `JOB_PROGRESS` | `ProgressEvent` | Each denoising step (includes step/total_steps) |\n| `JOB_COMPLETED` | `CompletedEvent` | Success (includes PIL Image and seed) |\n| `JOB_FAILED` | `FailedEvent` | Error (includes message and traceback) |\n| `JOB_CANCELLED` | `CancelledEvent` | Job cancelled before processing |\n| `MODEL_LOADING` | `ModelLoadingEvent` | Model load starting |\n| `MODEL_LOADED` | `ModelLoadedEvent` | Model loaded (includes size and device) |\n| `MODEL_UNLOADED` | `ModelUnloadedEvent` | Model evicted from VRAM |\n\n## Test Scripts\n\n```bash\nbash scripts/test_flux1.sh            # FLUX.1 Dev (BFL repo)\nbash scripts/test_flux1_alt.sh        # FLUX.1 Dev (separate repos)\nbash scripts/test_flux1_gguf.sh       # FLUX.1 Dev (GGUF Q8_0 transformer)\nbash scripts/test_zimage.sh           # Z-Image Turbo\nbash scripts/test_flux1_lora.sh       # FLUX.1 Dev + LoRA (bf16)\nbash scripts/test_flux1_gguf_lora.sh  # FLUX.1 Dev + LoRA (GGUF Q8_0)\n```\n\n## Dependencies\n\n- **PyTorch** \u003e= 2.0 (CUDA recommended, MPS supported on M3+ with PyTorch 2.3+)\n- **diffusers** \u003e= 0.32\n- **transformers** \u003e= 4.40\n- accelerate, safetensors, huggingface-hub\n- pydantic \u003e= 2.0, Pillow, click, einops, sentencepiece, loguru\n- **fal-client** \u003e= 0.5, httpx (FAL.ai cloud generation)\n- **zeroconf** \u003e= 0.131 (mDNS network announcement)\n- fastapi \u003e= 0.110, uvicorn[standard] \u003e= 0.27 (REST API server)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frzem-ai%2Frzem-ai-inference-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frzem-ai%2Frzem-ai-inference-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frzem-ai%2Frzem-ai-inference-engine/lists"}