{"id":25694254,"url":"https://github.com/sinanuozdemir/oreilly-multimodal-ai","last_synced_at":"2025-04-24T07:44:52.154Z","repository":{"id":242703998,"uuid":"771552320","full_name":"sinanuozdemir/oreilly-multimodal-ai","owner":"sinanuozdemir","description":"Learn how multimodal AI merges text, image, and audio for smarter models","archived":false,"fork":false,"pushed_at":"2025-01-21T01:46:10.000Z","size":16923,"stargazers_count":21,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T08:11:12.479Z","etag":null,"topics":["dalle-3","deepgram","diffusion","dreambooth","generative-ai","livekit","llama3","llava","multimodal","multimodal-ai","openai","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://www.oreilly.com/live-events/multimodal-ai-essentials/0642572002285/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sinanuozdemir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-13T14:06:19.000Z","updated_at":"2025-02-23T18:30:44.000Z","dependencies_parsed_at":"2024-06-04T15:14:55.009Z","dependency_job_id":"a8d519a3-abbd-4cd3-b938-0847fd80ec95","html_url":"https://github.com/sinanuozdemir/oreilly-multimodal-ai","commit_stats":null,"previous_names":["sinanuozdemir/oreilly-multimodal-ai"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinanuozdemir%2Foreilly-multimodal-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinanuozdemir%2Foreilly-multimodal-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinanuozdemir%2Foreilly-multimodal-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sinanuozdemir%2Foreilly-multimodal-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sinanuozdemir","download_url":"https://codeload.github.com/sinanuozdemir/oreilly-multimodal-ai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250586628,"owners_count":21454643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dalle-3","deepgram","diffusion","dreambooth","generative-ai","livekit","llama3","llava","multimodal","multimodal-ai","openai","stable-diffusion"],"created_at":"2025-02-24T23:51:56.200Z","updated_at":"2025-04-24T07:44:52.121Z","avatar_url":"https://github.com/sinanuozdemir.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multimodal AI Essentials\n\nWelcome to the \"[Multimodal AI Essentials](https://www.oreilly.com/live-events/multimodal-ai-essentials/0642572002285)\" code repository! In this repo, we will learn how multimodal AI merges text, image, and audio for smarter models.\n\nMuch of the code in these sessions will be featured in the 2nd edition of [my latest book on LLMs](https://a.co/d/ckinlUl):\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://a.co/d/ckinlUl\"\u003e\n        \u003cimg src=\"images/book.jpg\" width=\"150\" alt=\"A Quick Start Guide to LLMs\"\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\nSo if you're itching for more, check it out and please leave a rating/review to tell me what you thought :)\n\nFor even more, check out my [Expert Playlist](https://learning.oreilly.com/playlists/2953f6c7-0e13-49ac-88e2-b951e11388de)!\n\n## Prerequisites\n\n\n- Intermediate - Advanced Python Skills: Comfort with Python is crucial as we'll be using it throughout the course to interact with Hugging Face tools and integrate NLP into practical examples.\n\n- Foundational Machine Learning Knowledge: You should have an understanding of core machine learning principles, as we’ll build upon these concepts when exploring advanced NLP techniques.nologies in dynamic and evolving data environments.\n\n## Installation\n\n1. Clone this repository to your local machine.\n2. Ensure you have set the following api keyes:\n\n- OpenAI key\n\nYou're all set to explore the notebooks!\n\n## Usage - Jupyter Notebooks\n\nThis project contains several Jupyter notebooks each focusing on a specific topic:\n\n1. **[Intro to Multimodality](https://colab.research.google.com/drive/1zYSzDuYFa_cbRlti3scUjfmvradK8Sf4?usp=sharing)**: An introduction to multimodality with CLIP and SHAP-E\n\n\t- **[Whisper](https://colab.research.google.com/drive/1KxLWEEBtgix4zgP52pnxlIoJrZ8sHEYC?usp=sharing)**: An introduction to using Whisper for audio transcription\n\n\t- **[Llava](https://colab.research.google.com/drive/1IwNAz1Ee4YUSRNCU-SOsa7FS8Q2vmpoL?usp=sharing)**: Using an open source mult-turn multimodal engine\n\n\t- [**Multimodal Semantic Search**](https://colab.research.google.com/drive/1aUz0FKQDSAyXyhRyvkkRsSy7S30mpRJc?usp=sharing): Using SigLip model to build an image search system\n\n\n2. **Visual Q/A** - This case study requires you to [download the data from my Dropbox here](https://www.dropbox.com/scl/fo/w6iyfox8gnflvm7g10n47/AB47L7tNEl2Q8eyemZa2GMA?rlkey=v9s8bv6cmjukykpilzimswar0\u0026st=fbulzw4e\u0026dl=0). The code snippets should download them in code if that is easier! Our goal is to emulate the process done by [Llama 3.2-Vision-Instruct](https://colab.research.google.com/drive/1r6Nab2L7rYUBV5e8K8u8EFw98adJu5uh?usp=sharing): one of Meta's latest Llama models that can take in images.\n\t\n\t- Method 1: BERT + ViT -\u003e GPT-2 (Fusion)\n\n\t\t- Constructing and Training our model: [Local](notebooks/constructing_a_vqa_system.ipynb) and notebook in [Colab](https://colab.research.google.com/drive/1zvbruS1DvFrVgXjNouSrrF9-PphKLWWl?usp=sharing)\n\t\t- Using our VQA system: [Local](notebooks/using_our_vqa.ipynb) notebook and [Colab](https://colab.research.google.com/drive/16GOBndQuIBO-UfXdpPte-PXaZS2nsW1H?usp=sharing)\n\t\n\t\t- Method 2: BERT + ViT -\u003e GPT-2 (Fusion)\n\t\t\t- [Train the VQA Model](https://colab.research.google.com/drive/1DSh8_yfubuu5xPVM2BQ-I_eH5rrxLKZU?usp=sharing) and [use it here](https://colab.research.google.com/drive/1AWAk7NTvgTbjktUNB6bmS6T37bgTzRgt?usp=sharing)\n\n3. **Diffusion** - Exploring Diffusion Models and Fine-tuning techniques like Dreambooth\n\n\t- **[Intro to Diffusion (StableDiff + Flux)](https://colab.research.google.com/drive/1EtvcJGs_LdcQfSkbffKWKut8Zk7fccCX?usp=sharing)**: Generating images using diffusion models\n\n\t- **[Dreambooth](https://colab.research.google.com/drive/1tQt1pE6l0MI79W8ZX0MMu0YVmF2I0GB3?usp=sharing)**: Fine-tuning a stable difusion model to make images of yours truly! Ever wonder what I look like blonde? Me neither but AI gave me some ideas of what it would look like.\n\n4. **Texth to Speech** - Fine-tune text to speech models\n\n\t- **[Fine-tuning SpeechT5 to speak Turkish](https://colab.research.google.com/drive/1ZWW2HjvywU48ExX22ZgkZ56MJ50uD4tS?usp=sharing)**: An example of proper SpeechT5 fine-tuning with 150k\u003e high quality audio and transcrption examples\n\t- **[Fine-tuning SpeechT5 to speak like Sinan](https://colab.research.google.com/drive/1CVdyEvBq-7uerFLOmKPolcT3Z8J2rhAm?usp=sharing)**: I grab videos of me from Youtube, extract the audio, run it through OpenAI's whisper to make my own dataset to train the model to sound more like me\n\n5. **Multimodal Applications**\n\n\t- [A Sample Twilio App for Voice Messaging with AI](flask/README.md)\n\n\t- [Livekit + Deepgram + OpenAI for a voice to voice assistant](voice-to-voice/README.md)\n\n\t- [Multimodal Agents](notebooks/Visual%20Agent.ipynb) - Connecting an agent framework with OpenAI's Dall-E 3 diffusion model\n\n6. **Multimodal Evaluation + Ethics**\n\n\t- [Llava-Critic Demo](https://colab.research.google.com/drive/1y0SCzTrjAI0KHp_TxWlkfSCuQYnfbeHM?usp=sharing) - Multimodal LLM (LMM) as a judge\n\t- [Wav2Lip Demo](https://colab.research.google.com/drive/1nlhVZvi_s2jXpG864r3_NohrjyHG5V7j?usp=sharing) - See how modern deepfakes get made. Also find out my favorite movie! If you believe the video you see that is ;) \n\n## Contributing\nPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.\n\n## Book time with me on Intro!\nIf you have questions, I'm available on [Intro](https://intro.co/sinanozdemir) :) \n\n\u003cdiv style=\"text-align: center;\"\u003e\n    \u003ca href=\"https://intro.co/sinanozdemir\"\u003e\n        \u003cimg src=\"images/intro.png\" width=\"300\" alt=\"Book time with me on Intro\"\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinanuozdemir%2Foreilly-multimodal-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsinanuozdemir%2Foreilly-multimodal-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinanuozdemir%2Foreilly-multimodal-ai/lists"}