{"id":44350992,"url":"https://github.com/radlab-dev-group/ml-utils","last_synced_at":"2026-04-22T18:00:39.213Z","repository":{"id":312159722,"uuid":"1046555304","full_name":"radlab-dev-group/ml-utils","owner":"radlab-dev-group","description":"A collection of helper utilities and handlers designed to simplify common machine‑learning workflows.","archived":false,"fork":false,"pushed_at":"2025-12-27T17:34:22.000Z","size":71,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-11T21:32:37.732Z","etag":null,"topics":["geani-prompts","logger","ml","prompt-management","utils","weights-and-biases"],"latest_commit_sha":null,"homepage":"https://github.com/radlab-dev-group/radlab-ml-utils","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/radlab-dev-group.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-28T21:28:50.000Z","updated_at":"2025-12-27T17:34:30.000Z","dependencies_parsed_at":"2025-12-14T21:02:55.989Z","dependency_job_id":null,"html_url":"https://github.com/radlab-dev-group/ml-utils","commit_stats":null,"previous_names":["radlab-dev-group/radlab-ml-utils","radlab-dev-group/ml-utils"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/radlab-dev-group/ml-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radlab-dev-group%2Fml-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radlab-dev-group%2Fml-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radlab-dev-group%2Fml-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radlab-dev-group%2Fml-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/radlab-dev-group","download_url":"https://codeload.github.com/radlab-dev-group/ml-utils/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radlab-dev-group%2Fml-utils/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32148180,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T17:06:48.269Z","status":"ssl_error","status_checked_at":"2026-04-22T17:06:19.037Z","response_time":58,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["geani-prompts","logger","ml","prompt-management","utils","weights-and-biases"],"created_at":"2026-02-11T15:05:58.659Z","updated_at":"2026-04-22T18:00:39.192Z","avatar_url":"https://github.com/radlab-dev-group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# radlab-ml-utils\n\n## 📖 Overview\n\n`radlab-ml-utils` is a collection of helper utilities and handlers designed to simplify  \ncommon machine‑learning workflows. The package includes:\n\n- **OpenAPI handler** – thin client for LLM servers exposing an OpenAPI spec.\n- **Training handler** – utilities for preparing datasets, tokenization and orchestrating training pipelines.\n- **WandB handler** – convenient wrappers around Weights \u0026 Biases for experiment tracking,\n  artifact management and rich logging.\n- **Prompt handler** – loads and manages prompt files (`*.prompt`) from a directory tree,\n  making it easy to reuse and reference prompts programmatically.\n\nThe library is built on Python 3.10 and can be installed via `pip install .` after cloning the repository.\n\n---\n\n## 📂 Project Structure\n\n```\n\nradlab-ml-utils/\n│\n├─ apps/\n│   └─ __init__.py\n│   └─ openapi_test.py          # Example script demonstrating the OpenAPI client\n│\n├─ configs/\n│   └─ ollama_config.json       # Sample OpenAPI configuration file\n│\n├─ rdl_ml_utils/\n│   ├─ handlers/\n│   │   ├─ __init__.py\n│   │   ├─ openapi_handler.py   # Core OpenAPI client implementation\n│   │   ├─ training_handler.py  # Dataset loading \u0026 training helpers\n│   │   ├─ wandb_handler.py     # W\u0026B integration utilities\n│   │   └─ prompt_handler.py    # Prompt loading and lookup utilities\n│   ├─ open_api/\n│   │   ├─ __init__.py\n│   │   ├─ cache_api.py         # Request-response caching utilities for OpenAPI calls\n│   │   └─ queue_manager.py     # Threaded queue/pool for multiple OpenAPI clients\n│   └─ utils/\n│       └─ __init__.py\n│\n├─ .gitignore\n├─ CHANGELOG.md\n├─ LICENSE\n├─ README.md                    # *You are reading it right now*\n├─ requirements.txt\n└─ setup.py\n```\n\n---\n\n## 🛠️ Handlers\n\n### `openapi_handler.py`\n\nThe **OpenAPI client** (`OpenAPIClient`) provides a simple, opinionated interface for interacting with LLM servers that\nfollow the OpenAI‑compatible API schema.\n\n#### Key Features\n\n| Feature                     | Description                                                                                      |\n|-----------------------------|--------------------------------------------------------------------------------------------------|\n| **Flexible initialization** | Pass `base_url` / `model` directly **or** load them from a JSON config file (`open_api_config`). |\n| **Authentication**          | Optional `api_key` is added as a `Bearer` token header when supplied.                            |\n| **Prompt generation**       | `generate(prompt, …)` returns a plain‑text completion.                                           |\n| **Chat completions**        | `chat(messages, …)` works with the standard `[{role, content}]` message format.                  |\n| **System prompt handling**  | A global `system_prompt` can be set at client creation and overridden per call.                  |\n| **Health check**            | `is_available()` performs a quick GET request to verify server reachability.                     |\n| **Context‑manager**         | Use `with OpenAPIClient(...) as client:` for clean entry/exit semantics.                         |\n\n#### Configuration File (`configs/ollama_config.json`)\n\n```json\n{\n  \"base_url\": \"http://localhost:11434\",\n  \"model\": \"MODEL_NAME\",\n  \"api_key\": \"YOUR_API_KEY_IF_NEEDED\",\n  \"system_prompt\": \"You are a helpful AI assistant.\"\n}\n```\n\n#### Example Usage\n\n```python\nfrom rdl_ml_utils.open_api.client import OpenAPIClient\n\n# Load configuration from JSON (recommended for reproducibility)\nwith OpenAPIClient(open_api_config=\"configs/ollama_config.json\") as client:\n    # Verify the server is up\n    if not client.is_available():\n        raise RuntimeError(\"OpenAPI server is not reachable.\")\n\n    # Simple generation\n    answer = client.generate(\n        message=\"Explain logistic regression.\",\n        system_prompt=\"You are a statistics expert.\",\n        max_tokens=512,\n    )\n    print(\"Generation result:\", answer)\n\n    # Chat‑style interaction\n    chat_messages = [\n        {\"role\": \"user\", \"content\": \"What are the biggest challenges in ML today?\"},\n    ]\n    response = client.chat(\n        messages=chat_messages,\n        system_prompt=\"Speak like a senior data scientist.\",\n        max_tokens=256,\n    )\n    print(\"Chat response:\", response)\n```\n\n#### Configuration File (`configs/ollama_config.json`)\n\n```json\n{\n  \"base_url\": \"http://localhost:11434\",\n  \"model\": \"MODEL_NAME\",\n  \"api_key\": \"YOUR_API_KEY_IF_NEEDED\",\n  \"system_prompt\": \"You are a helpful AI assistant.\"\n}\n```\n\n---\n\n### `openapi_queue_manager.py`\n\nThe **OpenAPI queue manager** (`OpenAPIQueue`) provides a thin, thread‑safe wrapper around multiple\n`OpenAPIClient` instances. It loads client configurations from JSON files, creates a pool of worker\nthreads, and processes `generate` and `chat` requests in a first‑in‑first‑out (FIFO) order.\n\n#### Why use it?\n\n| Benefit                          | Explanation                                                                                                                                  |\n|----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|\n| **Parallel client usage**        | Multiple LLM endpoints can be configured and used simultaneously, improving throughput.                                                      |\n| **Automatic request scheduling** | Calls are enqueued and dispatched to the first free client, so you never have to manage locks yourself.                                      |\n| **Simple synchronous API**       | Despite the internal concurrency, the public methods (`generate`, `chat`) block until a result is ready, making integration straightforward. |\n| **Graceful shutdown**            | `close()` cleanly stops all background workers.                                                                                              |\n\n#### Core API\n\n```python\nfrom pathlib import Path\nfrom rdl_ml_utils.open_api.queue_manager import OpenAPIQueue\n\n# Initialise the queue with one or more client config files\nqueue = OpenAPIQueue([\n    Path(\"configs/ollama-config.json\"),\n    Path(\"configs/ollama-config_lab4_1.json\"),\n])\n\n# Generate a completion (handled by the first available client)\nanswer = queue.generate(\n    \"Explain quantum entanglement.\",\n    max_tokens=128,\n)\n\n# Chat‑style interaction\nreply = queue.chat(\n    \"What is the capital of France?\",\n    max_tokens=64,\n)\n\n# When finished, shut down the workers\nqueue.close()\n```\n\nThe class handles all low‑level details (client selection, locking, task queuing)\nso you can focus on the prompts and model logic.\n\n---\n\n### `open_api/cache_api.py`\n\nThe **OpenAPI cache utilities** provide a lightweight caching layer for request/response pairs\nmade via the OpenAPI client stack. This is useful when you want to avoid repeated calls\nfor identical inputs (e.g., during prompt iteration, evaluation, or notebook exploration).\n\n#### Key capabilities\n\n- Request signature hashing based on input content and selected generation parameters.\n- In‑memory cache for fast repeat access and optional persistence to disk to survive process restarts.\n- Transparent use with generation or chat‑style calls to reduce latency and cost.\n- Simple controls to bypass cache for a specific call or to invalidate entries programmatically.\n\n#### Typical usage patterns\n\n- Wrap your OpenAPI calls with a cache lookup, and only call the backend if there is a cache miss.\n- Scope caches by project/run folder so that experiments do not interfere with each other.\n- Periodically clear or prune the cache (e.g., by a max size or time‑to‑live policy) according to your workflow.\n\nExample flow (conceptual):\n\n1) Build a request key from the prompt/messages and key parameters (e.g., model, temperature, max_tokens).\n2) If the key is present, return the stored response.\n3) Otherwise, call the OpenAPI client, store the response under that key, and return it.\n\nThis module is designed to be orthogonal to the queue manager: you can use caching with or without the queue,\nand you can place caching either outside (per request) or inside (per worker) depending on your needs.\n\n---\n\n### `training_handler.py`\n\nThe **Training handler** (`TrainingHandler`) streamlines dataset preparation for transformer‑based models. It:\n\n* Loads JSON‑line datasets using the 🤗 Datasets library.\n* Instantiates a tokenizer from a Hugging‑Face model (e.g., `bert-base-uncased`).\n* Stores useful metadata such as the number of unique labels.\n* Exposes ready‑to‑use `train_dataset` and `eval_dataset` attributes\n  (creation of the actual `DataLoader`s is left to the user, keeping the class framework‑agnostic).\n\n#### Core API\n\n```python\nfrom rdl_ml_utils.handlers.training_handler import TrainingHandler\n\nhandler = TrainingHandler(\n    train_dataset_file_path=\"data/train.jsonl\",\n    eval_dataset_file_path=\"data/valid.jsonl\",\n    base_model=\"distilbert-base-uncased\",\n    train_batch_size=16,\n    workdir=\"./workdir\",\n)\n\n# After initialization:\n#   handler.tokenizer          -\u003e AutoTokenizer instance\n#   handler.train_dataset      -\u003e 🤗 Dataset with training examples\n#   handler.eval_dataset       -\u003e 🤗 Dataset with validation examples\n#   handler.uniq_labels        -\u003e Set of label strings\n```\n\n#### What the class does internally\n\n```python\n# ... existing code ...\n\nself.tokenizer = AutoTokenizer.from_pretrained(\n    self.base_model, use_fast=True\n)\n\ndata = load_dataset(\n    \"json\",\n    cache_dir=\"./cache\",\n    data_files={\n        \"train\": self.train_dataset_file_path,\n        \"validation\": self.eval_dataset_file_path,\n    },\n)\n\nself.train_dataset = data[\"train\"]\nself.eval_dataset = data[\"validation\"]\n\n# ... existing code ...\n```\n\nThe handler is deliberately lightweight: it only prepares raw datasets and tokenizers, leaving model definition,\noptimizer setup and training loops to the user’s own script or training framework (PyTorch, TensorFlow, 🤗 Trainer,\netc.). This makes it easy to plug into existing pipelines while keeping reproducibility (datasets are cached under\n`./cache`).\n\n---\n\n### `wandb_handler.py`\n\nThe **WandB handler** (`WanDBHandler`) centralises all interactions with the Weights \u0026 Biases service, providing a\nhigh‑level API for:\n\n| Action                       | Method                                                                                | Description                                                                                                     |\n|------------------------------|---------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|\n| **Initialize a run**         | `init_wandb`                                                                          | Sets up a W\u0026B run with project name, tags, and a merged configuration dict (run‑specific + training arguments). |\n| **Log scalar metrics**       | `log_metrics`                                                                         | Sends a dictionary of metric name → value pairs, optionally with a step number.                                 |\n| **Store datasets / models**  | `add_dataset`, `add_model`                                                            | Creates a `wandb.Artifact` of type *dataset* or *model* and uploads the supplied directory.                     |\n| **Finish a run**             | `finish_wandb`                                                                        | Calls `wandb.run.finish()` to close the run cleanly.                                                            |\n| **Prepare run metadata**     | `prepare_run_tags`, `prepare_run_name_with_date`, `prepare_simple_run_name_with_date` | Helper functions that add host information, timestamps and model identifiers to run names/tags.                 |\n| **Merge configs**            | `prepare_run_config`                                                                  | Combines a user‑provided dict with attributes of a `training_args` object (e.g., from 🤗 Trainer).              |\n| **Plot confusion matrix**    | `plot_confusion_matrix`                                                               | Uses `wandb.plot.confusion_matrix` to visualise classification performance.                                     |\n| **Log detailed predictions** | `store_prediction_results`                                                            | Creates a `wandb.Table` with raw text, true label, predicted label and optional per‑class probabilities.        |\n\n#### Example Usage\n\n```python\nfrom rdl_ml_utils.handlers.wandb_handler import WanDBHandler\n\n\n# Simple config object (could be a dataclass or Namespace)\nclass WandbConfig:\n    PROJECT_NAME = \"ml-experiments\"\n    PROJECT_TAGS = [\"nlp\", \"classification\"]\n    PREFIX_RUN = \"run_\"\n    BASE_RUN_NAME = \"experiment\"\n\n\nwandb_cfg = WandbConfig()\nrun_cfg = {\"base_model\": \"distilbert-base-uncased\", \"learning_rate\": 3e-5}\ntraining_args = None  # could be an argparse.Namespace with many fields\n\n# Initialise run (name will include timestamp and model name)\nWanDBHandler.init_wandb(\n    wandb_config=wandb_cfg,\n    run_config=run_cfg,\n    training_args=training_args,\n    run_name=None,  # auto‑generated\n)\n\n# Log some metrics during training\nfor epoch in range(3):\n    # ... training logic ...\n    WanDBHandler.log_metrics({\"epoch\": epoch, \"accuracy\": 0.87 + epoch * 0.01})\n\n# After training, store the model artifact\nWanDBHandler.add_model(name=\"distilbert-finetuned\", local_path=\"./workdir/model\")\n\n# Finish the run\nWanDBHandler.finish_wandb()\n```\n\n#### Plotting a Confusion Matrix\n\n```python\n# ground_truth and predictions are list‑like, class_names is a list of label strings\nWanDBHandler.plot_confusion_matrix(\n    ground_truth=y_true,\n    predictions=y_pred,\n    class_names=[\"neg\", \"pos\"],\n    probs=prediction_probs,  # optional probability matrix\n)\n```\n\n#### Storing Detailed Prediction Results\n\n```python\nWanDBHandler.store_prediction_results(\n    texts_str=test_texts,\n    ground_truth=y_true,\n    pred_labels=y_pred,\n    probs=prediction_probs,\n)\n```\n\nAll helper methods automatically add the host name to run tags, ensuring that runs from different machines are easily\ndistinguishable.\n\n---\n\n### `prompt_handler.py`\n\nThe **Prompt handler** (`PromptHandler`) offers a simple way to load, store, and retrieve textual prompts stored as\n`*.prompt` files.  \nPrompts are indexed by a *key* that corresponds to the file’s path **relative to the base directory**, using forward\nslashes and **without the file extension**.\n\n#### Core API\n\n```python\nfrom rdl_ml_utils.handlers.prompt_handler import PromptHandler\n\n# Initialise the handler pointing at a directory that contains *.prompt files\nprompt_dir = \"configs/prompts\"  # any directory you like\nph = PromptHandler(base_dir=prompt_dir)\n\n# List all loaded prompts (key → content)\nall_prompts = ph.list_prompts()\nprint(\"Available prompts:\", list(all_prompts.keys()))\n\n# Retrieve a specific prompt\nkey = \"system/default\"  # corresponds to configs/prompts/system/default.prompt\nprompt_text = ph.get_prompt(key)\nprint(\"Prompt content:\", prompt_text)\n```\n\n#### How It Works\n\n* **Recursive loading** – The handler walks the `base_dir` recursively (`Path.rglob(\"*.prompt\")`).\n* **Key generation** – For each file, the relative path (POSIX style) is taken, the `.prompt` suffix is stripped, and\n  the result becomes the dictionary key.\n* **In‑memory storage** – Prompt contents are kept in a plain Python `dict[str, str]`, making subsequent look‑ups O(1).\n\n#### Typical Use‑Cases\n\n| Scenario                     | How PromptHandler helps                                                                                                                                  |\n|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|\n| **Prompt engineering**       | Keep a library of reusable prompts (system, few‑shot examples, task‑specific templates) in a dedicated folder; retrieve them by logical name at runtime. |\n| **Dynamic prompt selection** | Based on experiment configuration, select the appropriate prompt key (`\"qa/simple\"`, `\"summarization/long\"` etc.) without hard‑coding file paths.        |\n| **Versioned prompts**        | Store multiple versions (`v1.prompt`, `v2.prompt`) in sub‑folders; the key reflects the version (`\"summarization/v1\"`).                                  |\n\n#### Error handling\n\n* `KeyError` is raised if a non‑existent key is requested.\n* `RuntimeError` is raised if a prompt file cannot be read (e.g., permission issues).\n\n---\n\n## 🚀 Getting Started\n\n1. **Clone the repository**\n\n```shell script\ngit clone https://github.com/radlab-dev-group/ml-utils.git\ncd ml-utils\n```\n\n2. **Create a virtual environment and install dependencies**\n\n```shell script\npython -m venv .venv\nsource .venv/bin/activate\npip install -e .\n```\n\n3. **Run the OpenAPI demo**\n\n```shell script\npython apps/openapi_test.py\n```\n\nTip: For higher throughput or multi-endpoint setups, consider using the queue manager described\nin open_api/queue_manager.py. You can also add a local cache layer (see open_api/cache_api.py)\nto avoid recomputing identical requests during experiments.\n\n---\n\n## 📦 Installation\n\n```shell script\npip install git+https://github.com/radlab-dev-group/ml-utils.git\n```\n\nor, after cloning:\n\n```shell script\npip install .\n```\n\n---\n\n## 📜 License\n\nThis project is licensed under the Apache 2.0 License – see the [LICENSE](LICENSE) file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fradlab-dev-group%2Fml-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fradlab-dev-group%2Fml-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fradlab-dev-group%2Fml-utils/lists"}