{"id":17980945,"url":"https://github.com/nvidia/audio-flamingo","last_synced_at":"2025-05-16T04:06:43.780Z","repository":{"id":240960953,"uuid":"803302515","full_name":"NVIDIA/audio-flamingo","owner":"NVIDIA","description":"PyTorch implementation of Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities.","archived":false,"fork":false,"pushed_at":"2025-03-21T23:38:51.000Z","size":3153,"stargazers_count":449,"open_issues_count":9,"forks_count":25,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-12T03:45:28.877Z","etag":null,"topics":["audio-captioning","audio-language-models","audio-question-answering","audio-reasoning","multimodal-large-language-models"],"latest_commit_sha":null,"homepage":"https://research.nvidia.com/labs/adlr/AF2/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-20T13:14:41.000Z","updated_at":"2025-04-11T12:11:16.000Z","dependencies_parsed_at":"2024-05-21T17:29:35.209Z","dependency_job_id":"6f8c056d-235c-4ffd-843b-6c4d69ea4862","html_url":"https://github.com/NVIDIA/audio-flamingo","commit_stats":null,"previous_names":["nvidia/audio-flamingo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Faudio-flamingo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Faudio-flamingo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Faudio-flamingo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Faudio-flamingo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA","download_url":"https://codeload.github.com/NVIDIA/audio-flamingo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514214,"owners_count":21116899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio-captioning","audio-language-models","audio-question-answering","audio-reasoning","multimodal-large-language-models"],"created_at":"2024-10-29T18:07:01.698Z","updated_at":"2025-04-12T03:45:51.249Z","avatar_url":"https://github.com/NVIDIA.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\" style=\"display: flex; justify-content: center; align-items: center; text-align: center;\"\u003e\n  \u003ca href=\"https://github.com/NVIDIA/audio-flamingo\" style=\"margin-right: 20px; text-decoration: none; display: flex; align-items: center;\"\u003e\n    \u003cimg src=\"assets/af_logo.png\" alt=\"Audio Flamingo 2 🔥🚀🔥\" width=\"120\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\" style=\"display: flex; justify-content: center; align-items: center; text-align: center;\"\u003e\n    \u003ch2\u003e\n    Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities\n    \u003c/h2\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\" style=\"display: flex; justify-content: center; margin-top: 10px;\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2503.03983\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2503.03983-AD1C18\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://research.nvidia.com/labs/adlr/AF2/\"\u003e\u003cimg src=\"https://img.shields.io/badge/Demo page-228B22\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/NVIDIA/audio-flamingo\"\u003e\u003cimg src='https://img.shields.io/badge/Github-Audio Flamingo 2-9C276A' style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/NVIDIA/audio-flamingo/stargazers\"\u003e\u003cimg src=\"https://img.shields.io/github/stars/NVIDIA/audio-flamingo.svg?style=social\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\" style=\"display: flex; justify-content: center; margin-top: 10px;\"\u003e\n\u003ca href=\"https://huggingface.co/nvidia/audio-flamingo-2\"\u003e\u003cimg src=\"https://img.shields.io/badge/🤗-Checkpoints (3B)-ED5A22.svg\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/nvidia/audio-flamingo-2-1.5B\"\u003e\u003cimg src=\"https://img.shields.io/badge/🤗-Checkpoints (1.5B)-ED5A22.svg\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/nvidia/audio-flamingo-2-0.5B\"\u003e\u003cimg src=\"https://img.shields.io/badge/🤗-Checkpoints (0.5B)-ED5A22.svg\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\" style=\"display: flex; justify-content: center; margin-top: 10px;\"\u003e\n\u003ca href=\"https://huggingface.co/spaces/nvidia/audio-flamingo-2\"\u003e\u003cimg src=\"https://img.shields.io/badge/🤗-Gradio Demo (3B)-5F9EA0.svg\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/spaces/nvidia/audio-flamingo-2-1.5B\"\u003e\u003cimg src=\"https://img.shields.io/badge/🤗-Gradio Demo (1.5B)-5F9EA0.svg\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/spaces/nvidia/audio-flamingo-2-0.5B\"\u003e\u003cimg src=\"https://img.shields.io/badge/🤗-Gradio Demo (0.5B)-5F9EA0.svg\" style=\"margin-right: 5px;\"\u003e\u003c/a\u003e\n  \n\u003c/div\u003e\n\n## Overview\n\nThis repo contains the PyTorch implementation of [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://arxiv.org/abs/2503.03983). Audio Flamingo 2 achieves the state-of-the-art performance across over 20 benchmarks, with only a 3B parameter small language model. It is improved from our previous [Audio Flamingo](https://arxiv.org/abs/2402.01831). \n\n- We introduce two datasets, AudioSkills for expert audio reasoning, and LongAudio for long audio understanding, to advance the field of audio understanding.\n\n- Audio Flamingo 2 has advanced audio understanding and reasoning capabilities. Especially, Audio Flamingo 2 has expert audio reasoning abilities, and can understand long audio up to 5 minutes.\n\n- Audio Flamingo 2 outperforms larger and proprietary LALMs across 20+ benchmarks, despite being smaller (3B) and trained exclusively on public datasets.\n\n## Main Results\n\nAudio Flamingo 2 outperforms prior SOTA models including GAMA, Audio Flamingo, Qwen-Audio, Qwen2-Audio, LTU, LTU-AS, SALMONN, AudioGPT, Gemini Flash v2 and Gemini Pro v1.5 on a number of understanding and reasoning benchmarks.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg class=\"img-full\" src=\"assets/af2_radar.png\" width=\"300\"\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg class=\"img-full\" src=\"assets/af2_table2.png\" width=\"400\"\u003e\n\u003c/div\u003e\n\n## Audio Flamingo 2 Architecture\n\nAudio Flamingo 2 uses a cross-attention architecture similar to [Audio Flamingo](https://arxiv.org/abs/2402.01831) and [Flamingo](https://arxiv.org/abs/2204.14198). Audio Flamingo 2 can take up to 5 minutes of audio inputs. \n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg class=\"img-full\" src=\"assets/af2_arch.png\" width=\"800\"\u003e\n\u003c/div\u003e\n\n\n## Code Structure\n\n- The folder ```inference/``` contains inference code of Audio Flamingo 2.\n\nThe structure is highly based on the [Open Flamingo](https://github.com/mlfoundations/open_flamingo) repo (commit ```a05dcba```).\n\n\u003c!-- Within each folder, the structure is highly based on the [Open Flamingo](https://github.com/mlfoundations/open_flamingo) repo (commit ```a05dcba```). Each folder is self-contained and we expect no cross dependencies between these folders. --\u003e\n\n## References\n\n\u003c!-- The main training and inferencing code within each folder (```foundation/```, ```chat/```, ```inference/```), including ```train/```, ```src/```, ```data/```, and ```configs/```,  --\u003e\nThe code under ```src/``` and ```configs/``` are modified from [Open Flamingo](https://github.com/mlfoundations/open_flamingo) (commit ```a05dcba```) (MIT license), which borrows from [flamingo-pytorch](https://github.com/lucidrains/flamingo-pytorch) (MIT license), [flamingo-mini](https://github.com/dhansmair/flamingo-mini) (MIT license), and [open_clip](https://github.com/mlfoundations/open_clip) (MIT license). ```src/helpers.py``` also includes self-attention implementations based on [attention-is-all-you-need-pytorch](https://github.com/jadore801120/attention-is-all-you-need-pytorch) (MIT license), which borrows from [OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py) (MIT license). ```my_laion_clap``` is adapted from [LAION-AI/CLAP](https://github.com/LAION-AI/CLAP) (CC0-1.0 license).\n\n## License\n\n- The code in this repo is under MIT license.\n- The checkpoints are for non-commercial use only (see NVIDIA OneWay Noncommercial License). They are also subject to the [Qwen Research license](https://huggingface.co/Qwen/Qwen2.5-3B/blob/main/LICENSE), the [Terms of Use](https://openai.com/policies/terms-of-use) of the data generated by OpenAI, and the original licenses accompanying each training dataset.\n- Notice: Audio Flamingo 2 is built with Qwen-2.5. Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved.\n\n\n## Citation\n\n- Audio Flamingo\n```\n@inproceedings{kong2024audio,\n  title={Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities},\n  author={Kong, Zhifeng and Goel, Arushi and Badlani, Rohan and Ping, Wei and Valle, Rafael and Catanzaro, Bryan},\n  booktitle={International Conference on Machine Learning},\n  pages={25125--25148},\n  year={2024},\n  organization={PMLR}\n}\n```\n\n- Audio Flamingo 2\n```\n@article{ghosh2025audio,\n  title={Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities},\n  author={Ghosh, Sreyan and Kong, Zhifeng and Kumar, Sonal and Sakshi, S and Kim, Jaehyeon and Ping, Wei and Valle, Rafael and Manocha, Dinesh and Catanzaro, Bryan},\n  journal={arXiv preprint arXiv:2503.03983},\n  year={2025}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia%2Faudio-flamingo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvidia%2Faudio-flamingo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia%2Faudio-flamingo/lists"}