{"id":31753712,"url":"https://github.com/servicenow/au-harness","last_synced_at":"2025-10-09T17:53:43.169Z","repository":{"id":313867374,"uuid":"1050710586","full_name":"ServiceNow/AU-Harness","owner":"ServiceNow","description":"A comprehensive framework to test audio comprehension of Large Audio Language Models.","archived":false,"fork":false,"pushed_at":"2025-09-29T17:51:13.000Z","size":4467,"stargazers_count":51,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-29T19:39:08.364Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://au-harness.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ServiceNow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-04T20:20:21.000Z","updated_at":"2025-09-25T03:29:04.000Z","dependencies_parsed_at":"2025-09-29T19:37:16.550Z","dependency_job_id":null,"html_url":"https://github.com/ServiceNow/AU-Harness","commit_stats":null,"previous_names":["servicenow/lalmeval","servicenow/lalm-eval","servicenow/au-harness"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ServiceNow/AU-Harness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2FAU-Harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2FAU-Harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2FAU-Harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2FAU-Harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ServiceNow","download_url":"https://codeload.github.com/ServiceNow/AU-Harness/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2FAU-Harness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001804,"owners_count":26083197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-09T17:53:42.079Z","updated_at":"2025-10-09T17:53:43.160Z","avatar_url":"https://github.com/ServiceNow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\" style=\"margin-bottom: 1em;\"\u003e\n\n# AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs\n\n*Comprehensive • Fast • Reproducible*\n\n\u003cimg src=\"assets/images/ver-1-bg.png\" alt=\"AU-Harness Brand Logo\" width=\"200\"\u003e\n\n[![arXiv](https://img.shields.io/badge/arxiv-2509.08031-red)](https://arxiv.org/abs/2509.08031)\n[![Website](https://img.shields.io/badge/🌐_Website-au--harness.github.io-lightgray.svg?color=blue)](https://au-harness.github.io/)\n[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Version](https://img.shields.io/badge/version-0.1.0-orange.svg)](https://github.com/ServiceNow/AU-Harness)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/ServiceNow/AU-Harness/pulls)\n\n**🎯 50+ Datasets • 🚀 380+ Subsets • 📊 9 Metrics • 🔊 21 Audio Tasks**\n\n\u003c/div\u003e\n\n## 📋 Overview\n\nAU-Harness is a standardized, efficient and highly customizable open-source framework for evaluating audio-based language models on Audio-to-Text tasks. Built for researchers and developers, AU-Harness provides a comprehensive suite of tools to benchmark and compare the performance of various audio processing models across a wide range of tasks.\n\n\n## ❓ Why AU-Harness?\n\n1. 🚀 **Blazing Fast**:\n   - Multiple models can be evaluated simultaneously across multiple tasks, datasets and metrics using independent Engines, enabling full parallelization of the evaluation pipeline\n   - Model inference and evaluation is batched, with the only bottleneck being user-set batch size\n   - Dataset Sharding is implemented for linearly scalable inference throughput\n\n\u003cp align='center'\u003e\n  \u003cimg src=\"assets/images/eval_kit_comparison.png\" alt=\"Evaluation Kit Comparison\" width=\"80%\", height=\"auto\"/\u003e\n\u003c/p\u003e\n\n2. 🔧 **Immensely Customizable**:\n   - Dataset and Samples can be customized and filtred by accents, language, length, and more\n   - Models and tasks can be customized by temperature, request parameters, prompts and batch size\n   - Score reporting can be customized through the aggregation parameter\n\n3. 📦 **Super Modular**:\n   - Streamlined evaluation processes allow for better understanding of the codebase\n   - Modularized functions allow for easy extension and customization\n\n4. 🎯 **Wide Task Coverage**:\n   - We support 21 unique tasks over 6 different categories\n   - Over 50 unique datasets, with 380+ unique subsets\n   - 9 different metrics for broader evaluation coverage\n\n\n## 📊 Task Taxonomy \u0026 Structure\n\n\u003cdiv align=\"center\" style=\"margin: 30px 0;\"\u003e\n  \u003cimg src=\"assets/images/taxonomy.png\" alt=\"AU-Harness Task Taxonomy\" style=\"width: 60%; max-width: 600px; border-radius: 16px; box-shadow: 0 10px 30px rgba(0,0,0,0.15);\"\u003e\n\u003c/div\u003e\n\n### 📁 Task Organization\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🗣️ \u003ca href=\"./tasks/speech_recognition/README.md\"\u003e Speech Recognition \u003c/a\u003e\u003c/b\u003e \u003ci\u003e(3 tasks)\u003c/i\u003e\u003c/summary\u003e\n\n- [**asr**](./tasks/speech_recognition/asr/) - Automatic speech recognition\n  - *Datasets: librispeech, voxpopuli, common voice, and more*\n- [**code_switching_asr**](./tasks/speech_recognition/code_switching_asr/) - Transcribe utterances with mixed-language speech.\n- [**long_form_asr**](./tasks/speech_recognition/long_form_asr/) - Transcribe extended audio content\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🎭 \u003ca href=\"./tasks/paralinguistics/README.md\"\u003e Paralinguistics \u003c/a\u003e\u003c/b\u003e \u003ci\u003e(5 tasks)\u003c/i\u003e\u003c/summary\u003e\n\n- [**emotion_recognition**](./tasks/paralinguistics/emotion_recognition/) - Detect emotional states from speech\n- [**accent_recognition**](./tasks/paralinguistics/accent_recognition/) - Identify speaker accents and dialects\n- [**gender_recognition**](./tasks/paralinguistics/gender_recognition/) - Classify speaker gender from voice\n- [**speaker_recognition**](./tasks/paralinguistics/speaker_recognition/) - Identify speaker(s) present in the audio.\n- [**speaker_diarization**](./tasks/paralinguistics/speaker_diarization/) - Segment speech into audio segments attributed to different speakers\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🔊 \u003ca href=\"./tasks/audio_understanding/README.md\"\u003e Audio Understanding \u003c/a\u003e\u003c/b\u003e \u003ci\u003e(2 tasks)\u003c/i\u003e\u003c/summary\u003e\n\n- [**music_understanding**](./tasks/audio_understanding/music_understanding/) - Analyze and understand musical content\n- [**scene_understanding**](./tasks/audio_understanding/scene_understanding/) - Identify and classify audio scenes based on the ambient sound information.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🧠 \u003ca href=\"./tasks/spoken_language_understanding/README.md\"\u003e Spoken Language Understanding \u003c/a\u003e \u003c/b\u003e \u003ci\u003e(5 tasks)\u003c/i\u003e\u003c/summary\u003e\n\n- [**intent_classification**](./tasks/spoken_language_understanding/intent_classification/) - Classify user intents from spoken inputs\n- [**speech_qa**](./tasks/spoken_language_understanding/speech_qa/) - Answer questions based on spoken content\n- [**sqqa**](./tasks/spoken_language_understanding/sqqa/) - Spoken query question-answering with context\n- [**spoken_dialogue_summarization**](./tasks/spoken_language_understanding/spoken_dialogue_summarization/) - Summarize spoken conversations\n- [**translation**](./tasks/spoken_language_understanding/translation/) - Translate given speech into the target language. \n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🧩 \u003ca href=\"./tasks/spoken_language_reasoning/README.md\"\u003e Spoken Language Reasoning \u003c/a\u003e\u003c/b\u003e \u003ci\u003e(4 tasks)\u003c/i\u003e\u003c/summary\u003e\n\n- [**ifeval**](./tasks/spoken_language_reasoning/ifeval/) - Speech Instruction-following capability evaluation\n- [**bfcl**](./tasks/spoken_language_reasoning/bfcl) - Speech Function Calling capability evaluation\n- [**mtbench**](./tasks/spoken_language_reasoning/mtbench/) - Complex multi-turn Instruction-following capability evaluation\n- [**speech_to_sql**](./tasks/spoken_language_reasoning/speech_to_sql/) - Speech-to-Coding capability\n- [**gsm8k**](./tasks/spoken_language_reasoning/gsm8k/) - Grade school math word problems\n\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🔐 \u003ca href=\"./tasks/safety_and_security/README.md\"\u003e Safety and Security \u003c/a\u003e\u003c/b\u003e \u003ci\u003e(2 tasks)\u003c/i\u003e\u003c/summary\u003e\n\n- [**safety**](./tasks/safety_and_security/safety/) - Evaluate model safety and robustness\n- [**spooling**](./tasks/safety_and_security/spoofing/) - Detect synthetic or manipulated audio\n\n\u003c/details\u003e\n\n## 🏗️ Architecture\n\n### General Evaluation Flow\n\n\u003cp align='center'\u003e\n  \u003cimg src=\"assets/images/overview.png\" alt=\"Taxonomy Figure\" width=\"80%\", height=\"auto\"/\u003e\n\u003c/p\u003e\nThe evaluation flow in AU-Harness follows a highly concurrent architecture:\n\n1. **Configuration \u0026 Initialization**: The system parses `config.yaml` to load models, datasets, metrics, and other evaluation parameters.\n\n2. **Engine Assembly**: For each dataset-metric pair, an Engine is created containing:\n   - A dataset\n   - A preprocesser\n   - The specified metric\n   - An appropriate postprocessor\n   - References to all specified models\n\n3. **Concurrent Execution**: \n   - All Engines run simultaneously\n   - Within each Engine, model inference occurs concurrently across all models\n   - After inference completes, the postprocessor transforms model outputs\n   - Evaluation is performed concurrently, with record-level scores logged throughout\n\n4. **Results Aggregation**: The main process awaits completion of all Engines before compiling and reporting final performance metrics.\n\nThis architecture enables efficient scaling with multiple models and datasets while maintaining organized evaluation workflows.\n\n## 🚀 Quick Start\n\nGet up and running in under a minute:\n\n```bash\n# Clone and install\ngit clone https://github.com/ServiceNow/AU-Harness.git\ncd AU-Harness\npip install -r requirements.txt\n\n# Run your first evaluation\ncp sample_config.yaml config.yaml\nbash evaluate.sh\n```\n\nResults will be generated in `run_logs/` with detailed metrics and analysis.\n\n## 💻 Usage\n\nAU-Harness requires setting up a running configuration file (`config.yaml`) to define your evaluation parameters. This file controls which models, datasets, and metrics are used in your evaluation.\n\nTo get started with AU-Harness:\n\n1. Clone this repository\n2. Setup your environment:\n```bash\npython -m venv myEnv\nsource myEnv/bin/activate\npip install -r requirements.txt\n```\n3. Populate your `config.yaml` file based on the example provided in `sample_config.yaml` and instructions below - the given 'config.yaml' already has the mandatory fields\n4. Run the end-to-end evaluation:\n```bash\nbash evaluate.sh\n\n```\n**NOTE:** If you would like to run evaluation with your own customized config, use the command below. Sample customized running configurations are provided in [run_configs](./run_configs/) \n```\nbash evaluate.sh --config /path/to/your/config.yaml\n```\n\n### 🧩 Running Configuration Options\n\nThe `config.yaml` file supports the following customization options. Sample running configurations are available for reference at [sample_config.yaml](./sample_config.yaml).\n\n#### Dataset and Metrics\n```yaml\ndataset_metric:\n  - [\"librispeech_test_other\", \"word_error_rate\"] #evaluate by dataset\n  - [\"emotion_recognition\", \"llm_judge_binary\"] # evaluate by task group\n  - [\"spoken_language_understanding\", \"all\"] # evaluate all metrics of all tasks in group\n```\n\n#### Sampling and Filtering\n```yaml\nfilter:\n  num_samples: 300 # optional - number of samples to run(remove for all)\n  length_filter: [1.0, 30.0] # optional - filters for only audio samples in this length(seconds)\n```\n\n#### Result Aggregation\n```yaml\n# Optional - allows for custom score aggregation at the end. Currently only simple average is supported\n# Follow the format of [x, [y1, y2]] where x is a valid metric, and each y is a valid task or a group (of tasks)\naggregate:\n  - [\"llm_judge_binary\", [\"emotion_recognition\"]]\n  - [\"llm_judge_detailed\", [\"alpaca_audio_test\", \"openhermes_instruction_test\"]]\n  - [\"word_error_rate\", [\"librispeech\"]]\n```\n\n#### Generation parameters override\n```yaml\n# Generation parameters are generally defined for each task in their task configs\n# This can be overriden for specific models and tasks using the following format.\ngeneration_params_override:\n  # Task override - Apply for this task for all models\n  - task: \u003cTASK1\u003e\n    generation_params:\n      temperature: \u003ctemperature\u003e\n      max_gen_tokens: \u003cmax_gen_tokens\u003e\n  # Model override - Apply for this model for all tasks\n  - model: \u003cMODEL1\u003e\n    generation_params:\n      temperature: \u003ctemperature\u003e\n      max_gen_tokens: \u003cmax_gen_tokens\u003e\n  # Model and Task override - Apply for this model and task\n  - model: \u003cMODEL1\u003e\n    task: \u003cTASK1\u003e\n    generation_params:\n      temperature: \u003ctemperature\u003e\n      max_gen_tokens: \u003cmax_gen_tokens\u003e\n```\n\n#### System and User prompt override\n```yaml\n# System prompts and user prompts (high level task instructions) can be overriden from the run config\nprompt_overrides:\n  # User prompt override mandatorily requires a task name because these are generally task specific\n  user_prompt:\n    - task: \u003ctask_name\u003e\n      model: \u003cmodel_name\u003e # (optional)\n      prompt: \u003cprompt_text\u003e\n  # System prompt override mandatorily requires a model name because these are generally model specific\n  system_prompt:\n    - model: \u003cmodel_name\u003e\n      task: \u003ctask_name\u003e # (optional)\n      prompt: \u003cprompt_text\u003e\n```\n\n#### Model Configuration\n```yaml\nmodels:\n  - name: \"gpt-4o-mini-audio-preview-1\" # Mandatory - must be unique\n    inference_type: \"openai\"  # openai(openai), vllm(vllm), or audio transcription(transcription)\n    url: ${ENDPOINT_URL} # Mandatory\n    delay: 100 # Optional\n    retry_attempts: 8 # Optional\n    timeout: 30 # Optional\n    model: \"gpt-4o-mini-audio-preview\" # Mandatory\n    auth_token: ${AUTH_TOKEN} # Mandatory\n    api_version: ${API_VERSION} # Mandatory\n    batch_size: 350 # Mandatory\n    chunk_size: 30  # Optional - Max audio length in seconds\n    \n  - name: \"qwen_2.5_omni\" # Mandatory\n    inference_type: \"vllm\"  # openai, vllm, or audio transcription\n    url: ${ENDPOINT_URL} # Mandatory\n    delay: 100 # Optional\n    retry_attempts: 8 # Optional\n    timeout: 30 # Optional\n    model: \"qwen_2.5_omni\" # Mandatory\n    auth_token: ${AUTH_TOKEN} # Mandatory\n    batch_size: 150 # Mandatory\n    chunk_size: 30  # Optional - Max audio length in seconds\n```\n\n**Note**: Batch-size proportional dataset sharding is implemented when multiple endpoints of the same model are provided. Be sure to have unique 'name' attributes for each unique endpoint, as shown above\n\n##### Inference Types\n\n| Client           | Inference Type                       |\n|------------------|--------------------------------------|\n| \"openai\"         | AsyncAzureOpenAI (Chat Completions)  |\n| \"vllm\"           | AsyncOpenAI (Chat Completions)       |\n| \"transcription\"  | AsyncOpenAI (Transcriptions)         |\n\n#### Judge Configuration\nLLM-Judge setup is required to run any tasks requiring LLM-judge metrics. For specific task-metric pair compatibility, visit [Task Documentation](./tasks/README.md) and [Metric Documentation](./metrics/README.md).\nSample LLM-judge configuration is noted below. We provide [sample run_config](./run_configs/mtbench.yaml) that requires LLM-judge setup accordingly.\n```yaml\njudge_settings:\n  judge_concurrency: 300 # optional - default is 1\n  judge_model: \"gpt-4o-mini\" # mandatory\n  judge_type: \"openai\" # mandatory (vllm or openai)\n  judge_api_version: ${API_VERSION} # optional(needed for openai)\n  judge_api_endpoint: ${API_ENDPOINT} # mandatory\n  judge_api_key: ${API_KEY} # mandatory\n  judge_temperature: 0.1 # optional\n```\n\n### 📝 Task Configuration Options\n#### Adding Datasets\n\nAU-Harness supports adding custom tasks through `task_config` YAML files. These files define the task properties and how they should be processed.\n\n#### Creating a TaskConfig File\n\nCreate a YAML file in the `tasks` directory under the appropriate task groups. Each task should be defined with the following properties, down to the most specific subset:\n\n```yaml\ntask_name: \u003cunique_task_name\u003e\ndataset_path: \u003chuggingface_repo or local_dataset_path\u003e # mandatory\nsubset: \u003csubset\u003e # Optional (recommended)\nsplit: \u003csplit\u003e # mandatory\nlange: \u003clanguage\u003e # mandatory\nmodality: \u003cmodality\u003e # Optional \npreprocessor: \u003cPreprocessorClass\u003e # mandatory\npostprocessor: \u003cPostprocessorClass\u003e # mandatory\naudio_column: \u003caudio_column\u003e # Optional\ntarget_column: \u003ctarget_column\u003e # Optional (recommended)\ninstruction_column: \u003cinstruction_column\u003e # Optional (recommended)\nlong_audio_processing_logic: \u003ctruncate/chunk\u003e # mandatory\n\ngeneration_kwargs:  # mandatory - Additional kwargs to constrain model decoding behaviors\n  temperature: 0.0001 \n  max_completion_tokens: 64\n\nmetrics:\n  - metric: \u003cmetric_name\u003e # mandatory - Metric from the allowed pre-defined metrics\n```\n\n**Important Note:** It is HIGHLY Recommended to add a \"user_prompt\" field tailored specifically to the datasets you are running for the best results, especially for complex tasks.\n\n#### Example\n\nHere's an example task_config for intent classification (SLURP-Intent) datasets:\n\n```yaml\ntask_name: SLURP-intent\ndataset_path: DynamicSuperb/SuperbIC_SLURP-Intent\nsubset: default\nsplit: test\nlanguage: english\npreprocessor: GeneralPreprocessor\npostprocessor: GeneralPostprocessor\naudio_column: audio\ntarget_column: label\ninstruction_column: instruction\nlong_audio_processing_logic: truncate\n\ngeneration_kwargs:\n  temperature: 0.0001\n  max_completion_tokens: 64\n\nmetrics:\n  - metric: llm_judge_binary\n```\n#### Tasks requiring additional setups\nTwo specific datasets require additional customized setups before execution. Follow the provided instructions accordingly:\n- `CallHome` (for ASR and Speaker Diarization Task): Follow the instructions provided in [tasks/speech_recognition/asr/callhome_asr](./tasks/speech_recognition/asr/callhome_asr/README.md)\n- `Speech_to_SQL` (for Speech-to-Coding Task): Follow the instructions provided in [tasks/spoken_language_reasoning](./tasks/spoken_language_reasoning/README.md)\n\n### ⚙️ Customizations\n#### Using Your Dataset\n\nAfter creating the run_config YAML file, you can reference your dataset in the `config.yaml` file:\n\n```yaml\ndataset_metric:\n  - \"[your_dataset_name, metric_name]\" \n```\n\n#### Using Your Own Model\nThe recommended way is to launch [VLLM](https://github.com/vllm-project/vllm) end-points and use the corresponding URLs in the run configs.\n\nIf your model is not yet supported on VLLM, we have an experimental FastAPI based inference server support in the [models/inference_boilerplate/](./models/inference_boilerplate/) directory. You can use this to deploy your own models.\n\n### 📈 Analyzing Results\n\nOnce your run finishes, you can inspect the outputs in a few ways:\n\n- **Full logs**\n  View the complete log at\n  `{created_timestamp}_default.log` (or `{created_timestamp}_{log_file}` where `log_file` is what you set) in the project root.\n\n- **Per-record details**\n  `/run_logs/{created_timestamp}/{task}/{task}_{metric}_{model}.csv`\n\n- **Final aggregated scores**\n  `/run_logs/{created_timestamp}/final_scores.json`\n\nwhere \n  - `task`: name of the task that is run for evaluation\n  - `metric`: pre-defined metric name used for evaluating the given task\n  - `model`: name of the model being evaluated\n  - `created_timestamp`: automatically recorded timestamp used as a unique_ID for each run\n## 📝 Acknowledgement\n\nAU-Harness incorporates some of the design elements and reusable components from ServiceNow's comprehensive internal benchmarking platform, namely CLAE. We'd like to thank the CLAE team for their invaluable feedback and suggestions.\n\n## 📝 Citation\n\nIf you use AU-Harness in your research, please cite our work:\n\n```bibtex\n@article{surapaneni2025auharness,\n  title={AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs},\n  author={Sidharth Surapaneni and Hoang Nguyen and Jash Mehta and Aman Tiwari and Oluwanifemi Bamgbose and Akshay Kalkunte and Sai Rajeswar and Sathwik Tejaswi Madhusudhan},\n  journal={arXiv preprint arXiv:2509.08031},\n  year={2025}\n}\n```\n\n## 📄 License\n\nAU-Harness is licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fservicenow%2Fau-harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fservicenow%2Fau-harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fservicenow%2Fau-harness/lists"}