{"id":28994003,"url":"https://github.com/magenta/magenta-realtime","last_synced_at":"2025-06-25T03:09:36.045Z","repository":{"id":300290712,"uuid":"1001012484","full_name":"magenta/magenta-realtime","owner":"magenta","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-20T21:00:14.000Z","size":1114,"stargazers_count":23,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-20T22:18:17.730Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/magenta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-12T17:09:21.000Z","updated_at":"2025-06-20T22:17:13.000Z","dependencies_parsed_at":"2025-06-20T22:18:23.551Z","dependency_job_id":"208fee4d-0745-4820-bc08-6bb6b835138a","html_url":"https://github.com/magenta/magenta-realtime","commit_stats":null,"previous_names":["magenta/magenta-realtime"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/magenta/magenta-realtime","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magenta%2Fmagenta-realtime","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magenta%2Fmagenta-realtime/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magenta%2Fmagenta-realtime/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magenta%2Fmagenta-realtime/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/magenta","download_url":"https://codeload.github.com/magenta/magenta-realtime/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/magenta%2Fmagenta-realtime/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261795342,"owners_count":23210621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-25T03:09:35.212Z","updated_at":"2025-06-25T03:09:36.033Z","avatar_url":"https://github.com/magenta.png","language":"Python","funding_links":[],"categories":["Python","Multi-Agent Creative Pipelines","语音合成","6. Generative Media Tools"],"sub_categories":["Design, Drawing \u0026 Web Surfaces","资源传输下载"],"readme":"# Magenta RT: Streaming music generation!\n\nMagenta RealTime is a Python library for streaming music audio generation on\nyour local device. It is the open source / on device companion to\n[MusicFX DJ Mode](https://labs.google/fx/tools/music-fx-dj) and the\n[Lyria RealTime API](https://ai.google.dev/gemini-api/docs/music-generation).\n\nThis is a 👀 sneak preview of the Magenta RT project. We will have\n[more to share](#coming-soon) in the coming weeks including a technical report\nand additional features!\n\nSee our [blog post](https://g.co/magenta/rt) and\n[model card](https://github.com/magenta/magenta-realtime/blob/main/MODEL.md) for\nmore info.\n\n![Animation of chunk-by-chunk generation in Magenta RT](notebooks/diagram.gif)\n\n## Getting started\n\n\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/magenta/magenta-realtime/blob/main/notebooks/Magenta_RT_Demo.ipynb\"\u003e\n  \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open Notebook In Colab\"/\u003e\n\u003c/a\u003e\n\nThe fastest way to get started with Magenta RT is to try our official\n[Colab Demo](https://colab.research.google.com/github/magenta/magenta-realtime/blob/main/notebooks/Magenta_RT_Demo.ipynb)\nwhich runs in real time on freely available TPUs! Here is a quick\n[video walkthrough](https://www.youtube.com/watch?v=SVTuEdeepVs).\n\nIf you have a machine with a TPU or GPU, you may also following the installation\ninstructions below for running locally.\n\n## Local installation\n\nInstall the latest version:\n\n```sh\n# With GPU support:\npip install 'git+https://github.com/magenta/magenta-realtime#egg=magenta_rt[gpu]'\n# With TPU support:\npip install 'git+https://github.com/magenta/magenta-realtime#egg=magenta_rt[tpu]'\n# CPU only\npip install 'git+https://github.com/magenta/magenta-realtime'\n```\n\nOr, clone and install for local editing:\n\n```sh\ngit clone https://github.com/magenta/magenta-realtime.git \u0026\u0026 cd magenta-realtime\npip install -e .[gpu]\n```\n\n## Examples\n\n### Generating audio with Magenta RT\n\nMagenta RT generates audio in short chunks (2s) given a finite amount of past\ncontext (10s). We use crossfading to mitigate boundary artifacts between chunks.\nMore details on our model are coming soon in a technical report!\n\n```py\nfrom magenta_rt import audio, system\nfrom IPython.display import display, Audio\n\nnum_seconds = 10\nmrt = system.MagentaRT()\nstyle = system.embed_style('funk')\n\nchunks = []\nstate = None\nfor i in range(round(num_seconds / mrt.config.chunk_length)):\n  state, chunk = mrt.generate_chunk(state=state, style=style)\n  chunks.append(chunk)\ngenerated = audio.concatenate(crossfade_time=mrt.crossfade_length)\ndisplay(Audio(generated.samples.swapaxes(0, 1), rate=mrt.sample_rate))\n```\n\n### Blending text and audio styles with MusicCoCa\n\nMusicCoCa is a joint embedding model of text and audio styles. Magenta RT is\nconditioned on MusicCoCa embeddings allowing for seamless blending of styles\nusing any number of text and audio prompts.\n\n```py\nfrom magenta_rt import audio, musiccoca\n\nstyle_model = musiccoca.MusicCoCa()\nmy_audio = audio.Waveform.from_file('myjam.mp3')\nweighted_styles = [\n  (2.0, my_audio),\n  (1.0, 'heavy metal'),\n]\nweights = np.array([w for w, _ in weighted_styles])\nstyles = style_model.embed([s for _, s in weighted_styles])\nweights_norm = weights / weights.sum()\nblended = (weights_norm[:, np.newaxis] * styles).mean(axis=0)\n```\n\n### Tokenizing audio with SpectroStream\n\nSpectroStream is a discrete audio codec model operating on high-fidelity music\naudio (stereo, 48kHz). Under the hood, Magenta RT models SpectroStream audio\ntokens using a language model.\n\n```py\nfrom magenta_rt import audio, spectrostream\n\ncodec = spectrostream.SpectroStream()\nmy_audio = audio.Waveform.from_file('jam.mp3')\nmy_tokens = codec.encode(my_audio)\nmy_audio_reconstruction = codec.decode(tokens)\n```\n\n## Running tests\n\nUnit tests:\n\n```sh\npip install -e .[test]\npytest .\n```\n\nIntegration tests:\n\n```sh\npython test/musiccoca_end2end_test.py\npython test/spectrostream_end2end_test.py\npython test/magenta_rt_end2end_test.py\n```\n\n## Coming soon!\n\nThe following is a list of features we have planned for the near future (subject\nto change). Please open an issue if there are features you would like to see, or\nopen a pull request if you would like to contribute!\n\n-   Technical report\n-   Colab for fine tuning\n-   Colab for conditioning on real-time audio input\n\n## Citing this work\n\nA technical report is coming soon. For now, please cite our blog post:\n\n```\n@article{magenta_rt,\n    title={Magenta RealTime},\n    url={https://g.co/magenta/rt},\n    publisher={Google DeepMind},\n    author={Lyria Team},\n    year={2025}\n}\n```\n\n## License and disclaimer\n\nMagenta RealTime is offered under a combination of licenses: the codebase is\nlicensed under\n[Apache 2.0](https://github.com/magenta/magenta-realtime/blob/main/LICENSE),\nand the model weights under\n[Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/legalcode).\n\nIn addition, we specify the following usage terms:\n\nCopyright 2025 Google LLC\n\nUse these materials responsibly and do not generate content, including outputs,\nthat infringe or violate the rights of others, including rights in copyrighted\ncontent.\n\nGoogle claims no rights in outputs you generate using Magenta RealTime. You and\nyour users are solely responsible for outputs and their subsequent uses.\n\nUnless required by applicable law or agreed to in writing, all software and\nmaterials distributed here under the Apache 2.0 or CC-BY licenses are\ndistributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,\neither express or implied. See the licenses for the specific language governing\npermissions and limitations under those licenses. You are solely responsible for\ndetermining the appropriateness of using, reproducing, modifying, performing,\ndisplaying or distributing the software and materials, and any outputs, and\nassume any and all risks associated with your use or distribution of any of the\nsoftware and materials, and any outputs, and your exercise of rights and\npermissions under the licenses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmagenta%2Fmagenta-realtime","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmagenta%2Fmagenta-realtime","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmagenta%2Fmagenta-realtime/lists"}