{"id":30892353,"url":"https://github.com/yaronkoresh/definers","last_synced_at":"2026-04-08T15:00:41.267Z","repository":{"id":312590486,"uuid":"1047984425","full_name":"YaronKoresh/definers","owner":"YaronKoresh","description":"A comprehensive Python toolkit for AI, data processing, media manipulation, and system utilities.","archived":false,"fork":false,"pushed_at":"2026-04-08T13:15:07.000Z","size":2764,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-08T14:30:39.725Z","etag":null,"topics":["artificial-intelligence","cuda","data-science","deep-learning","diffusers","feature-extraction","generative-ai","gpu","gradio","image-generation","machine-learning","multimedia","music-generation","python-library","pytorch","scikit-learn","toolkit","transformers","video-generation","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YaronKoresh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-31T17:02:17.000Z","updated_at":"2026-04-08T12:42:09.000Z","dependencies_parsed_at":"2025-08-31T19:22:13.744Z","dependency_job_id":"2c76adad-445b-469d-ae06-51087634052f","html_url":"https://github.com/YaronKoresh/definers","commit_stats":null,"previous_names":["yaronkoresh/definers"],"tags_count":4,"template":false,"template_full_name":"YaronKoresh/my-template","purl":"pkg:github/YaronKoresh/definers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YaronKoresh%2Fdefiners","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YaronKoresh%2Fdefiners/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YaronKoresh%2Fdefiners/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YaronKoresh%2Fdefiners/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YaronKoresh","download_url":"https://codeload.github.com/YaronKoresh/definers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YaronKoresh%2Fdefiners/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31560476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","cuda","data-science","deep-learning","diffusers","feature-extraction","generative-ai","gpu","gradio","image-generation","machine-learning","multimedia","music-generation","python-library","pytorch","scikit-learn","toolkit","transformers","video-generation","web-scraping"],"created_at":"2025-09-08T19:44:12.747Z","updated_at":"2026-04-08T15:00:41.253Z","avatar_url":"https://github.com/YaronKoresh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Definers\n\nDefiners is a modular Python platform for building AI pipelines, multimodal media workflows, data preparation systems, and safe runtime automation from one codebase.\n\nIt is designed for developers, teams, and companies that need more than a narrow model wrapper or a one-off media script. Definers brings together audio, image, video, machine learning, NLP, web transfer, and platform-aware execution primitives behind a consistent Python-first API and launcher surface.\n\n## Table Of Contents\n\n1. [Positioning](#positioning)\n2. [Why Definers](#why-definers)\n3. [Capability Map](#capability-map)\n4. [Architecture Diagrams](#architecture-diagrams)\n5. [Installation](#installation)\n6. [Quick Start](#quick-start)\n7. [Applications And Launchers](#applications-and-launchers)\n8. [Docker Workflows](#docker-workflows)\n9. [Integrated API Reference](#integrated-api-reference)\n10. [Reliability And Safety](#reliability-and-safety)\n11. [Platform Requirements](#platform-requirements)\n12. [Development Workflow](#development-workflow)\n13. [Troubleshooting](#troubleshooting)\n14. [Adoption Notes For Teams](#adoption-notes-for-teams)\n15. [Contributing](#contributing)\n16. [License](#license)\n\n## Positioning\n\nDefiners is built for multimodal AI systems that need to move from experimentation to repeatable operation without changing toolsets. It gives you one repository and one package surface for:\n\n- media analysis and transformation\n- data preparation and training flows\n- UI launchers and Dockerized applications\n- resilient integrations and downloads\n- safer command execution and platform-aware runtime handling\n\nThe project favors modular adoption. Teams can start with a narrow slice such as `definers.application_data.preparation`, `definers.system`, or one media domain, then expand into broader multimodal workflows when they are ready.\n\n## Why Definers\n\nDefiners is built around a practical view of AI infrastructure: model work, media work, data work, and system work should cooperate instead of living in disconnected libraries.\n\n| Project Value | What It Means In Practice |\n| --- | --- |\n| Unified multimodal surface | Audio, image, video, ML, NLP, web, and launcher flows live in one toolkit |\n| Modular adoption | Optional dependency groups keep installs targeted instead of forcing the full stack |\n| Operational guardrails | Retry, backoff, circuit-breaker, and guarded command execution are part of the public story |\n| Deployment flexibility | The same project supports direct imports, CLI usage, installed app launching, and Docker app folders |\n| Team-friendly growth | Start small, then expand into richer workflows without replacing the surrounding ecosystem |\n\n## Capability Map\n\n| Outcome | Primary Surface | Typical Work |\n| --- | --- | --- |\n| Launch multimodal apps | `definers.presentation.gui_entrypoints`, `definers.presentation.launchers`, CLI commands | Start chat, audio, image, video, translate, train, animation, and FAISS interfaces |\n| Process and analyze audio | `definers.audio` | Feature extraction, mastering, stem separation, transcription, DSP, preview, synthesis |\n| Transform images and video | `definers.image`, `definers.media.video_helpers` | Upscaling, feature extraction, reconstruction, composition, rendering |\n| Prepare datasets and train models | `definers.application_data.preparation`, `definers.ml` | Loading, splitting, vectorization, tokenization, training, inference, export |\n| Orchestrate resilient integrations | `definers.capabilities`, `definers.media.web_transfer` | Retry, circuit breakers, transfer orchestration, downloads |\n| Run tools safely across environments | `definers.system` | Path handling, process execution, filesystem operations, runtime checks |\n\n### Capability Coverage By Domain\n\n| Domain | Core Public Entry Points | Representative Tasks |\n| --- | --- | --- |\n| Audio | `analyze_audio`, `analyze_audio_features`, `audio_preview`, `split_audio`, `remove_silence`, `compact_audio`, `separate_stems`, `separate_stem_layers`, `master`, `autotune_song`, `humanize_vocals` | Analysis, mastering, vocal finishing, cleanup, preview, stem separation, transcription, generation, mixing |\n| Image | `extract_image_features`, `features_to_image`, `save_image`, `get_max_resolution` | Feature extraction, reconstruction, upscaling, export |\n| Video | `features_to_video`, video UI and composition flows | Render pipelines, architect-style composition, media generation |\n| Data | `prepare_data`, `fetch_dataset`, `files_to_dataset`, `create_vectorizer` | Data ingestion, batching, splitting, vectorization |\n| ML | `train`, `fit`, `answer`, `extract_text_features` | Training, inference, prediction, feature conversion |\n| Capabilities | `CircuitBreaker`, `ExponentialBackoffDelay`, `with_retry` | Resilience boundaries for unstable downstreams |\n| Web | `download_file`, `download_and_unzip` | Safe transfer orchestration and download workflows |\n| System | `run`, `run_linux`, `run_windows`, `secure_command` | Guarded command execution and runtime integration |\n\n## Architecture Diagrams\n\n### System Layout\n\n```mermaid\nflowchart TB\n    User[Developer or Team] --\u003e Entry[Python API / CLI / Docker App]\n    Entry --\u003e Presentation[Presentation Layer]\n    Entry --\u003e Runtime[System Layer]\n    Presentation --\u003e Apps[Chat / Audio / Image / Video / Translate / Train / Animation / FAISS]\n    Runtime --\u003e Domains[Audio / Image / Video / ML / Data / Text / Web]\n    Domains --\u003e Resilience[Capabilities]\n    Domains --\u003e Catalogs[Catalogs]\n    Runtime --\u003e Platform[Platform Services]\n    Platform --\u003e Filesystem[Filesystem]\n    Platform --\u003e Processes[Processes]\n    Platform --\u003e Environment[Environment]\n```\n\n### Data And Training Flow\n\n```mermaid\nflowchart LR\n    Source[Local Files or Remote Dataset] --\u003e Prepare[prepare_data]\n    Prepare --\u003e TrainingData[TrainingData]\n    TrainingData --\u003e Train[train or fit]\n    Train --\u003e Model[Trained Artifact or Prediction Flow]\n    Model --\u003e Export[ONNX / Runtime Usage / App Integration]\n```\n\n### Launcher Flow\n\n```mermaid\nsequenceDiagram\n    participant User\n    participant CLI as CLI or Docker App\n    participant Dispatch as cli_dispatch\n    participant Registry as gui_registry\n    participant Chat as definers.presentation.gui_entrypoints.start\n    participant App as Target App\n\n    User-\u003e\u003eCLI: definers start chat\n    CLI-\u003e\u003eDispatch: run_cli(argv, version)\n    Dispatch-\u003e\u003eRegistry: normalize project name\n    Dispatch-\u003e\u003eChat: start(project)\n    Chat-\u003e\u003eApp: launch registered GUI\n    App--\u003e\u003eUser: interactive interface\n```\n\n## Installation\n\n### Installation Matrix\n\n| Goal | Command |\n| --- | --- |\n| Base install | `pip install .` |\n| Audio workflows | `pip install \".[audio]\"` |\n| Image workflows | `pip install \".[image]\"` |\n| Video workflows | `pip install \".[video]\"` |\n| ML workflows | `pip install \".[ml]\"` |\n| NLP workflows | `pip install \".[nlp]\"` |\n| Web and UI workflows | `pip install \".[web]\"` |\n| Full optional stack | `pip install \".[all]\"` |\n| Contributor toolchain | `pip install -e \".[dev]\"` |\n| CUDA extras | `pip install \".[cuda]\" --extra-index-url https://pypi.nvidia.com` |\n\n### Base Install\n\nDefiners currently targets Python 3.10 through 3.12. The optional stacks are not yet supported on Python 3.13.\n\nWhen `definers` is imported, it enables on-demand installation for known optional Python dependencies. If a supported optional module is missing, the runtime installs the required package set and retries automatically.\n\n```bash\npip install .\n```\n\n### Targeted Extras\n\n```bash\npip install \".[audio,ml,web]\"\n```\n\nYou can still preinstall targeted extras for reproducible cold starts, but the runtime doesn't require the entire optional graph up front.\n\nOn Windows, the published extras intentionally omit `stopes` because its dependency chain requires `posix_ipc`, which is not installable there.\n\n`madmom` stays out of the published extras because the reliable install path is a pinned GitHub commit rather than a PyPI-safe requirement string. The runtime installer and CLI installer still support it explicitly:\n\n```bash\ndefiners install madmom --type module\n```\n\n`basic-pitch` also stays out of the published extras because the runtime installer and CLI installer use the pinned fork commit instead of the upstream PyPI release, which conflicts with the published extras. The runtime installer and CLI installer still support it explicitly:\n\n```bash\ndefiners install basic_pitch --type module\n```\n\n`definers install audio` includes both `basic-pitch` and `madmom` through their pinned GitHub commit installs, even though the published extras still leave them out.\n\nFor runtime-managed environments, you can also preinstall optional targets without relying on extras metadata:\n\n```bash\ndefiners install --list\ndefiners install audio\ndefiners install translate --type task\n```\n\n### Development Install\n\n```bash\npip install -e \".[dev]\"\n```\n\n### Windows Helper\n\nWindows contributors can use [scripts/install.bat](scripts/install.bat), which prompts for install groups and includes the extra package index needed for NVIDIA-hosted packages.\n\n### Dependency Group Reference\n\n| Extra | Purpose | Notes |\n| --- | --- | --- |\n| `audio` | Audio analysis, DSP, separation, mastering, transcription, generation | Pulls in heavyweight audio and model dependencies |\n| `image` | Image feature extraction, upscaling, reconstruction | Includes image and computer-vision dependencies |\n| `video` | Video manipulation and rendering flows | Focused on video composition and export |\n| `ml` | Model training, inference, embeddings, export | Includes substantial ML framework dependencies |\n| `nlp` | Translation and language processing | Adds text and inference packages |\n| `web` | Retrieval, scraping, Gradio UI, web utilities | Includes Gradio and scraping-related packages |\nEach Docker entrypoint installs the base `definers` package from Git. Optional Python dependencies are resolved on demand at runtime.\n| `all` | Aggregates the main domain extras | Does not include `dev` or `cuda` |\n| `dev` | Pytest, Ruff, Poe, build, pre-commit, Vulture | Local contributor toolchain |\n\n| `audio` | none at build time |\n| `animation` | none at build time |\n| `chat` | none at build time |\n| `faiss` | none at build time |\n| `image` | none at build time |\n| `train` | none at build time |\n| `translate` | none at build time |\n| `video` | none at build time |\nfrom definers.ml import train\n\ndataset = prepare_data(\n    features=[\"/path/to/features.csv\"],\n    drop=[\"unused_column\"],\n    stratify=\"label\",\n    val_frac=0.1,\n    test_frac=0.1,\n    batch_size=32,\n)\n\nprint(dataset.metadata)\n\nmodel_path = train(\n    features=[\"/path/to/features.csv\"],\n    dataset_label_columns=[\"label\"],\n    order_by=\"shuffle\",\n    stratify=\"label\",\n    val_frac=0.1,\n    test_frac=0.1,\n    batch_size=32,\n)\n\nprint(model_path)\n```\n\n### Train With AutoTrainer\n\n```python\nfrom definers.ml import AutoTrainer\n\ntrainer = AutoTrainer(batch_size=32)\n\nmodel_path = trainer.train_url(\n    \"https://huggingface.co/datasets/owner/dataset\",\n    target=\"label\",\n    save_as=\"answer-model.joblib\",\n)\n\nfile_model_path = trainer.train_files(\n    [\"./features.csv\"],\n    target=[\"./labels.csv\"],\n    save_as=\"offline-model.joblib\",\n)\n\npredictions = trainer.predict([[0.2, 0.4], [0.8, 0.6]])\n\nprint(model_path)\nprint(file_model_path)\nprint(predictions)\n```\n\n### Preview A Training Plan\n\n```python\nfrom definers.application_ml.trainer_plan import render_training_plan_markdown\nfrom definers.ml import AutoTrainer\n\ntrainer = AutoTrainer(batch_size=16, source_type=\"parquet\")\nplan = trainer.training_plan(\n    data=\"owner/dataset\",\n    target=\"label\",\n    label_columns=\"label\",\n    drop=\"unused_column\",\n    order_by=\"shuffle\",\n    select=\"1-200\",\n    stratify=\"label\",\n)\n\nprint(render_training_plan_markdown(plan))\n```\n\n### Launch The ML Studio\n\n```bash\ndefiners train\n```\n\nThe `train` launcher opens a single ML studio surface for:\n\n- training plans, remote/local training, and artifact resume flows\n- saved-model prediction and task-based inference\n- answer runtime execution from the ML facade\n- text feature extraction, reconstruction, summary, and prompt tooling\n- ML health inspection, K-means guidance, RVC checkpoint lookup, and model bootstrap actions\n\n### Inspect ML Health\n\n```python\nfrom definers.ml import get_ml_health_snapshot, ml_health_markdown\n\nsnapshot = get_ml_health_snapshot()\nprint(snapshot.available_prediction_targets)\nprint(ml_health_markdown())\n```\n\n### Extract Audio Features\n\n```python\nfrom definers.audio import analyze_audio_features, extract_audio_features\n\nsummary = analyze_audio_features(\"/path/to/song.wav\")\nvector = extract_audio_features(\"/path/to/song.wav\")\n\nprint(summary)\nprint(None if vector is None else vector.shape)\n```\n\n### Master Audio With Diagnostics\n\n```python\nfrom definers.audio import master\n\noutput_path, report = master(\n    \"./mix.wav\",\n    output_path=\"./mix-mastered.wav\",\n    preset=\"balanced\",\n    report_path=\"./mix-mastered-report.json\",\n)\n\nprint(output_path)\nprint(report.headroom_recovery_mode)\nprint(report.post_spatial_stereo_motion)\nprint(report.post_clamp_true_peak_dbfs)\n```\n\nAvailable mastering presets are `balanced`, `edm`, and `vocal`. When `preset` is omitted, `master()` auto-selects the default preset from the input audio; if you want to override that decision, pass `preset` explicitly.\n\n`balanced` stays neutral, `edm` pushes the loudest and heaviest finish, and `vocal` keeps more openness, motion, and top-end air.\n\nIf you need to customize a preset, prefer the three macro mastering controls instead of low-level thresholds and timings. `SmartMasteringConfig` and the mastering kwargs expose `bass`, `volume`, and `effects`, each on a `0.0` to `1.0` scale. Higher `bass` means more bass weight and less treble emphasis, higher `volume` means a louder and denser master, and higher `effects` means more stereo motion, spatial polish, and dynamic finishing. The detailed compressor, stereo, delivery, and finishing parameters are derived automatically at access time from those three values.\n\n```python\nfrom definers.audio import master\n\noutput_path, report = master(\n    \"./mix.wav\",\n    output_path=\"./mix-mastered.wav\",\n    preset=\"balanced\",\n    effects=0.7,\n)\n```\n\nThe mastering path also adapts its repair intensity automatically for weak or legacy material: when a source is unusually dull, air-starved, almost mono, or closed and bass-loaded, it now rebalances more selectively by opening the upper bands, calming low-end compression pressure, and preserving more perceived air without requiring a separate preset.\n\nFor more aggressive production-style mastering, `master` first runs DEMUCS six-stem separation by default, applies role-specific mastering to drums, bass, vocals, guitar, piano, and the remaining music bed, rebalances those stems with drums intentionally forward and vocals slightly more restrained, saves the mastered stems into a sibling `*_stems` folder next to the final mix, remixes the processed stems, and then applies a final full-mix mastering pass.\n\nThe returned mastering report now exposes stage-aware diagnostics for post-spatial imaging, post-clamp true peak behavior, adaptive headroom recovery, and delivery verification so finishing changes can be inspected instead of guessed.\n\n### Extract Image Features\n\n```python\nfrom definers.image import extract_image_features\n\nfeatures = extract_image_features(\"/path/to/image.png\")\nprint(None if features is None else features.shape)\n```\n\n### Launch An Installed App\n\n```python\nfrom definers.presentation.launchers import launch_installed_project\n\nlaunch_installed_project(\"chat\")\n```\n\n### Run An External Tool Safely\n\n```python\nfrom definers.system import run\n\nrun([\"ffmpeg\", \"-i\", \"input.mp4\", \"output.wav\"])\n```\n\n### Add Retry Around An Unstable Integration\n\n```python\nfrom definers.capabilities import with_retry\n\n@with_retry(max_attempts=3)\nasync def fetch_remote_resource():\n    ...\n```\n\n## Applications And Launchers\n\nDefiners exposes both installed-project launchers and direct CLI dispatch for the same application registry.\n\nGUI launch entry points live directly in `definers.presentation.gui_entrypoints`, while heavyweight media commands are imported from their dedicated presentation services.\n\n### CLI Examples\n\n```bash\npython -m definers --help\ndefiners --version\ndefiners start chat\ndefiners audio\ndefiners translate\ndefiners music-video /path/to/song.wav 1920 1080 30\ndefiners lyric-video /path/to/song.wav /path/to/background.mp4 /path/to/lyrics.txt bottom\n```\n\n### Launchable Projects\n\n| Project | Purpose | Startup Paths |\n| --- | --- | --- |\n| `chat` | Multimodal chat interface | CLI, installed launcher, Docker |\n| `audio` | Mastering, vocal finishing, cleanup, preview, stem separation, and analysis workflows | CLI, installed launcher, Docker |\n| `image` | Image generation and upscaling tools | CLI, installed launcher, Docker |\n| `video` | Video composition and architect workflows | CLI, installed launcher, Docker |\n| `animation` | Chunked image-to-animation workflow | CLI, installed launcher, Docker |\n| `translate` | Translation and caption-oriented interface | CLI, installed launcher, Docker |\n| `train` | Full ML studio for training, prediction, inference, answer runtime, text tooling, health checks, and model bootstrap | CLI, installed launcher, Docker |\n| `faiss` | FAISS-oriented utility surface | CLI, installed launcher, Docker |\n\n### Launcher Contracts\n\n| Surface | Contract |\n| --- | --- |\n| `definers.presentation.cli_dispatch.run_cli(argv, version)` | Parses commands, normalizes app names, and routes to the right launcher or media command |\n| `definers.presentation.launchers.launch_installed_project(project)` | Starts an installed application by project name |\n| `definers.application_shell.command_parser.CliCommandParser.parse_cli_command(...)` | Normalizes incoming requests into typed command routing |\n| `definers.presentation.music_video_service.music_video(...)` and `definers.presentation.lyric_video_service.lyric_video(...)` | Dedicated presentation media services for CLI and app flows |\n\n## Docker Workflows\n\nEach app folder under `docker/` contains a container entrypoint layout with `app.py`, `Dockerfile`, and `docker-compose.yml`.\n\nEach Docker entrypoint installs the base `definers` package from Git. Optional Python dependencies are resolved on demand at runtime.\n\n```bash\ncd docker/chat\ndocker compose up --build\n```\n\nThe same folder shape exists for `audio`, `image`, `video`, `animation`, `translate`, `train`, and `faiss`.\n\n| Docker App | Installed Extras |\n| --- | --- |\n| `audio` | none at build time |\n| `animation` | none at build time |\n| `chat` | none at build time |\n| `faiss` | none at build time |\n| `image` | none at build time |\n| `train` | none at build time |\n| `translate` | none at build time |\n| `video` | none at build time |\n\n| When To Use | Best Fit |\n| --- | --- |\n| Local import-heavy development | Install the package directly |\n| Testing one app in an isolated runtime | Use the relevant Docker folder |\n| Deployment-like app startup | Use Docker and the per-app container layout |\n\n## Integrated API Reference\n\nThis section absorbs the former API specification document so the README can serve as the main technical front door.\n\n### System Architecture Overview\n\nDefiners provides a modular utility toolkit with extension-oriented boundaries for reliability, data handling, media processing, and system orchestration.\n\n### Core Capability Contracts\n\n| API | Contract |\n| --- | --- |\n| `definers.capabilities.CircuitBreaker` | Sync and async operation gate with `CLOSED`, `OPEN`, and `HALF_OPEN` state transitions and runtime snapshots |\n| `definers.capabilities.ExponentialBackoffDelay` | Configurable backoff strategy with base delay, multiplier, max delay, and jitter |\n| `definers.capabilities.with_retry` | Async retry decorator with selective exception boundaries and deterministic re-raise behavior |\n| `definers.media.web_transfer.ResourceRetrievalOrchestrator` | Strategy-driven transfer orchestration for integration-heavy workflows |\n\n### Execution Patterns\n\n| Pattern | Meaning |\n| --- | --- |\n| Resilience composition | Retry and circuit-breaker behavior can be layered to isolate unstable downstreams |\n| Strategy-oriented integrations | Transfer and runtime boundaries are shaped around injectable strategies and protocols |\n| Explicit failure surfaces | Runtime snapshots and typed failures are favored over silent degradation |\n\n### Error Boundary Model\n\n```mermaid\nflowchart TD\n    Call[Integration Call] --\u003e Retry{Retryable Failure?}\n    Retry -- Yes --\u003e Backoff[ExponentialBackoffDelay]\n    Backoff --\u003e Call\n    Retry -- No --\u003e Circuit{Circuit Threshold Reached?}\n    Circuit -- Yes --\u003e Open[Open Circuit]\n    Circuit -- No --\u003e Fail[Raise Failure]\n    Open --\u003e Snapshot[Capture Circuit Snapshot]\n```\n\n### Data Preparation Contracts\n\n| API | Contract |\n| --- | --- |\n| `prepare_data(remote_src=None, features=None, labels=None, url_type=None, revision=None, drop=None, order_by=None, stratify=None, val_frac=0.0, test_frac=0.0, batch_size=1)` | Loads remote or local data, applies optional drops and ordering, performs train/val/test splitting, and returns `TrainingData` |\n| `TrainingData` | Dataclass with `train`, `val`, `test`, and `metadata` |\n\n### Audio Analysis Contracts\n\n| API | Contract |\n| --- | --- |\n| `analyze_audio(audio_path, hop_length=1024, duration=None, offset=0.0)` | Returns dense waveform and frame-domain analysis data including tempo, beat frames, spectral data, RMS bands, and normalization helpers |\n| `analyze_audio_features(audio_path, txt=True)` | Returns either a compact formatted summary such as `C major (120 bpm)` or a `(key, mode, tempo)` tuple |\n| `extract_audio_features(file_path, n_mfcc=20)` | Returns an audio feature vector or `None` on load or extraction failure |\n\n### Image Contracts\n\n| API | Contract |\n| --- | --- |\n| `extract_image_features(image_path)` | Returns a visual feature vector or `None` if extraction fails |\n| `features_to_image(predicted_features, image_shape=(1024, 1024, 3))` | Reconstructs an image-like frame from feature input when shape expectations are met |\n| `get_max_resolution(width, height, mega_pixels=0.25, factor=16)` | Calculates a bounded factored resolution target |\n\n### Public Import Pattern\n\nImport concrete APIs from the implementation module you are using.\n\n```python\nfrom definers.application_data.preparation import prepare_data\nfrom definers.ml import train\nfrom definers.system import run\n```\n\nThe package root is intentionally narrow. It mainly exposes version metadata plus the optional `sox` probe through `definers.sox` and `definers.has_sox()`.\n\n### Catalog Contracts\n\n| Registry | Contract |\n| --- | --- |\n| `definers.catalogs.languages.LANGUAGE_CODES` | Immutable language-code registry |\n| `definers.catalogs.languages.UNESCO_MAPPING` | Immutable UNESCO mapping registry |\n| `definers.catalogs.tasks.TASKS` | Immutable task-to-model registry consumed by constants |\n| `definers.catalogs.references.USER_AGENTS` | Immutable user-agent registry |\n| `definers.catalogs.references.STYLE_CATALOG` | Immutable style registry |\n\nImport catalog registries from their concrete modules, such as `definers.catalogs.languages`, `definers.catalogs.references`, and `definers.catalogs.tasks`, instead of routing through a package-level gateway.\n\n### Web Transfer Facades\n\n| API | Contract |\n| --- | --- |\n| `definers.media.web_transfer.download_file(url, destination)` | Transfer orchestration for file download execution |\n| `definers.media.web_transfer.download_and_unzip(url, extract_to)` | Transfer orchestration for download-and-extract workflows |\n\n## Reliability And Safety\n\nOperational guardrails are a first-class part of the design.\n\n### Reliability Surface\n\n| Concern | Public Surface | Why It Matters |\n| --- | --- | --- |\n| Unstable integrations | `with_retry`, `ExponentialBackoffDelay` | Reduces transient failure noise |\n| Repeated downstream faults | `CircuitBreaker` | Prevents failure amplification and exposes state snapshots |\n| Transfer orchestration | `definers.media.web_transfer` | Centralizes integration-heavy behavior and rejects archive path traversal during unzip |\n| Platform differences | `run_linux`, `run_windows`, `run` | Makes runtime execution explicit |\n\n### Safer Command Execution\n\nAll external command execution flows through `definers.system.run()` and its platform-specific delegates.\n\n- Prefer list-form commands such as `[\"ffmpeg\", \"-i\", \"input.mp4\", \"output.wav\"]`.\n- Avoid free-form shell strings unless shell semantics are required deliberately.\n- Guarded runtime execution rejects unsafe string command patterns such as shell separators in protected paths.\n\n### Regex Hardening\n\nRegex behavior is centralized in `definers.regex_utils` so user-facing pattern work can be constrained instead of compiled ad hoc.\n\n### Optional Dependency Degradation\n\nSome features fail lazily or degrade gracefully when optional dependencies are missing. A prominent example is `sox`, which is probed without forcing `import definers` itself to fail.\n\nThe multimodal answer path now follows the same pattern for media helpers: plain text requests do not require Pillow, `soundfile`, or `librosa` to import successfully, while image and audio helpers are loaded only when matching content paths appear in chat history.\n\nThe `definers.audio` facade now also resolves public exports lazily. Importing a narrow utility such as `value_to_keys` or `audio_preview` no longer pulls the full audio surface into memory up front.\n\n### Guarded Serialized Model Loading\n\nPickle-backed model loads such as `.pkl` and `.joblib` now reject obvious HTML responses, Git LFS pointer files, and known dangerous reducer globals before deserialization starts.\n\nIf a serialized model load is blocked, regenerate the artifact from a trusted source instead of bypassing the guardrail.\n\n## Platform Requirements\n\n### Snapshot\n\n| Area | Current State |\n| --- | --- |\n| Python requirement | `\u003e=3.10` |\n| Check workflow matrix | Python `3.10`, `3.11`, `3.12` |\n| Quality workflow matrix | Python `3.10`, `3.11`, `3.12` |\n| External tools | FFmpeg often required, `sox` sometimes required, CUDA host support optional |\n\n### Practical Prerequisites\n\n- FFmpeg is needed for many audio and video conversion paths.\n- `sox` is optional but used in some audio conversion and loading flows.\n- CUDA-oriented installs require host-level NVIDIA support beyond Python packages alone.\n- Some extras pull direct Git dependencies and heavyweight frameworks, so installation time and environment complexity can rise quickly.\n\n## Development Workflow\n\nDefiners includes a contributor-oriented Python quality pipeline.\n\n### Local Setup\n\n```bash\npip install -e \".[dev]\"\n```\n\n### Main Validation Pipeline\n\n```bash\npoe check\n```\n\n`poe check` runs cleanup, compile verification, dead-code scanning, source sanitization, pre-commit hooks, tests, and a final cleanup pass.\n\n### Focused Commands\n\n```bash\npoe test\npoe coverage\npoe lint\npoe format\npoe build\npoe hook\npoe cli-health\npoe answer-simulations\npoe ml-health\n```\n\n`poe coverage` runs the full test suite with branch coverage and a terminal missing-lines report, then removes the temporary coverage data file so the workspace stays clean.\n\nWhen iterating on multimodal answer behavior, prefer a focused regression pass before the full suite.\n\n```bash\npytest tests/test_application_ml_answer_services.py tests/test_application_ml_answer_history_preparer.py tests/test_answer.py -q\n```\n\nWhen iterating on CLI routing, registry stability, or launcher wiring, use the dedicated health check.\n\n```bash\npoe cli-health\n```\n\nWhen iterating on training flows, AutoTrainer behavior, or workspace bootstrap for AI developers, use the ML-focused health and DX suite.\n\n```bash\npoe ml-health\n```\n\n### CI Validation Story\n\n| Workflow | Purpose |\n| --- | --- |\n| `check.yml` | Pull-request gate across Python `3.10`, `3.11`, and `3.12` with `poe check` |\n| `quality.yml` | Push and manual quality validation including lint, format check, and tests |\n| Additional workflows | Publish, CodeQL, dependency review, stale automation, and repository maintenance |\n\n## Troubleshooting\n\n### `sox` Is Missing\n\n`sox` is optional. Some audio-loading paths return `None` or fail only when the capability is used. Install the binary and place it on `PATH` if your workflow depends on it.\n\n### FFmpeg Commands Fail\n\nInstall FFmpeg and ensure it is available on `PATH`. Definers can help orchestrate FFmpeg-driven workflows, but it cannot supply the binary automatically in every environment.\n\n### CUDA Setup Is Difficult\n\nTreat `cuda` as an advanced path. Start with CPU-oriented or non-CUDA extras first, confirm your workflows, then layer in CUDA once host prerequisites are proven.\n\n### Heavy Extras Install Slowly\n\nInstall only the extras you need. The project is intentionally segmented so narrow adoption does not require the full stack.\n\n### Hugging Face Model URLs Or Sharded Files Fail To Load\n\nPrefer a Hugging Face repo id, a `resolve` URL, or even a copied `blob` URL. Definers now normalizes Hugging Face references through the Hub client instead of treating them as generic web downloads.\n\nIf a model is split into numbered files such as `model-001.safetensors`, `model-002.safetensors`, or `001-model.safetensors`, Definers can discover sibling shards automatically from the shared prefix or suffix around the numeric segment. For Hugging Face repositories it also checks index manifests when they exist.\n\nIf you are using a non-Hugging Face HTTP source, prefer raw artifact URLs rather than HTML pages. Definers rejects obvious HTML and Git LFS pointer downloads before deserialization.\n\nFor `.pkl` and `.joblib` artifacts, Definers also blocks known code-execution reducers during deserialization. Re-export the model from a trusted environment if an older or untrusted artifact is rejected.\n\n### A Shell Command Works Outside `run()` But Not Inside It\n\nPrefer list-form execution. If you need shell syntax, explicitly invoke the shell you intend to use.\n\n## Adoption Notes For Teams\n\nDefiners is easiest to adopt incrementally.\n\n1. Start with one slice such as `definers.application_data.preparation`, `definers.system`, or a single media domain.\n2. Add extras when the team is ready to absorb the relevant runtime and model dependencies.\n3. Standardize on the shared launcher and resilience surfaces as workflows expand.\n\nThat incremental adoption path is one of the project's strongest characteristics: teams can begin with a focused utility layer and grow into a broader multimodal platform without changing ecosystems.\n\n## Contributing\n\nContributor workflow now lives in [CONTRIBUTING.md](CONTRIBUTING.md). Use that guide for Python environment setup, local validation, branch hygiene, and pull request expectations.\n\n## License\n\nDefiners is licensed under the MIT License.\n\n- Allowed: private use, commercial use, modification, redistribution, and deployment.\n- Required: preservation of copyright and license notices.\n- Not provided: warranty, liability coverage, or fitness guarantees.\n\nSee `LICENSE` for the full terms.\n\n## Maintainer\n\nDefiners is owned and maintained by Yaron Koresh. Contributions are welcome through issues and pull requests.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyaronkoresh%2Fdefiners","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyaronkoresh%2Fdefiners","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyaronkoresh%2Fdefiners/lists"}