Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/thaoshibe/awesome-personalized-lmms

A curated list of Awesome Personalized Large Multimodal Models resources
https://github.com/thaoshibe/awesome-personalized-lmms

List: awesome-personalized-lmms

awesome awesome-list large-language-models large-multimodal-models multimodal-models personalization personalized personalized-generation

Last synced: about 1 month ago
JSON representation

A curated list of Awesome Personalized Large Multimodal Models resources

Host: GitHub
URL: https://github.com/thaoshibe/awesome-personalized-lmms
Owner: thaoshibe
Created: 2024-10-18T16:29:26.000Z (3 months ago)
Default Branch: main
Last Pushed: 2024-11-22T20:14:24.000Z (about 1 month ago)
Last Synced: 2024-11-22T21:23:13.584Z (about 1 month ago)
Topics: awesome, awesome-list, large-language-models, large-multimodal-models, multimodal-models, personalization, personalized, personalized-generation
Homepage: https://thaoshibe.github.io/awesome-personalized-lmms/
Size: 1.6 MB
Stars: 1
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

        # Awesome Personalized Large Multimodal Models [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)

📝 A curated list about Personalized Multimodal Models and related resources~ 📚

||

|:-------- |

|**Problem Settings:** Using 3-5 images of a novel concept/subject (e.g., a pet named ``), can we personalize Large Multimodal Models so that:
(1) They retain their original capabilities (e.g.,  Describe a dog) 
while (2) Enabling tailored their capabilities for the novel concept? (e.g., Describe ``)|

### Table of Contents

- [Papers](#papers)

- [Datasets](#datasets)

- [Applications](#applications)

#### 🌱 Contributing

Please feel free to create [pull requests](https://github.com/thaoshibe/awesome-personalized-lmms/pulls) or [an issue](https://github.com/thaoshibe/awesome-personalized-lmms/issues) to add/ correct anything. I really appreciate any help or clarification!

------

|
 * 🙋‍♀️ Personalization has been extensively explored in AI/ML/CV... It's now time for personalizing Large Multimodal Models! 🙋‍♀️*|

|:-------- |

| Over the years, we’ve witnessed the evolution of personalization across various tasks (e.g., object segmentation, image generation). 
 Now, with the rise of Large Multimodal Models (LMMs) -- We have opportunities to personalizing these generalist, large-scale AI systems. 
It’s time to take the leap and bring personalization into the realm of Large Multimodal Models, making them not only powerful but also **user-specific**!|

| *^ Above caption are actually generated by GPT-4o, I feed it the figure and asked it to generate a caption, haha!*|

*(This figure is created by me. If there is anything incorrect, please feel free to correct me! Thank you!)*

-----

### Papers

> *⚠️ Minor Note: The listed works below are specified for settings where users provide 3-5 images, and the system needs to learn about those concepts. There is research on other subtopics (e.g., role-playing, persona, etc.). For these topics, [this repo](https://github.com/HqWu-HITCS/Awesome-Personalized-LLM) might provide better coverage.*

| Title    | Venue    | Year | Input | Output | Link/ Code     |

|:-------- |:--------:|:--------:|:--------:|:--------:|:--------:|

| [paper title] | xx | 2024 | image, text | image, text |  |

|─── Vision Language Model ───  |

| [MC-LLaVA: Multi-Concept Personalized Vision-Language Model](https://arxiv.org/abs/2411.11706) | arXiv | 2024 | image, text | text | [Code](https://github.com/arctanxarc/MC-LLaVA) |

| [Retrieval-Augmented Personalization for Multimodal Large Language Models](https://arxiv.org/html/2410.13360v1) | arXiv | 2024 | image, text | text | [Page](https://hoar012.github.io/RAP-Project/), [Code](https://github.com/Hoar012/RAP-MLLM)|

| [Yo'LLaVA: Your Personalized Language and Vision Assistant](https://arxiv.org/abs/2406.09400) | NeurIPS | 2024 | image, text | text | [Page](https://thaoshibe.github.io/YoLLaVA), [Code](https://github.com/WisconsinAIVision/YoLLaVA)|

| [MyVLM: Personalizing VLMs for user-specific queries](https://arxiv.org/abs/2403.14599) | ECCV | 2024 | image, text | text | [Page](https://snap-research.github.io/MyVLM/), [Code](https://github.com/snap-research/MyVLM)|

|─── Large Language Models ───  |

| [Personalized Large Language Models](https://arxiv.org/abs/2402.09269) | ICDMw | 2024 | text | text | |

| [LaMP: When Large Language Models Meet Personalization](https://aclanthology.org/2024.acl-long.399/) | ACL | 2024 | text | text | [Page](https://lamp-benchmark.github.io/), [Code](https://github.com/LaMP-Benchmark/LaMP)|

| [Learning to Predict Persona Information forDialogue Personalization without Explicit Persona Description](https://arxiv.org/abs/2111.15093) | ACL | 2023 | text | text | |

| [Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge](https://arxiv.org/abs/2112.08619) | AAAI | 2022 | text | text | [Code](https://github.com/ncsoft/FoCus)|

| [A Personalized Dialogue Generator with Implicit User Persona Detection](https://arxiv.org/abs/2204.07372) | COLING | 2022 | text | text | |

| [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://arxiv.org/abs/1801.07243) | ACL | 2018 | text | text | |

### Datasets

| Name    | Year | # Concepts | Link | Notes |

|---------|------|------------|------|-------|

| MC-LLaVA | 2024 | -- | [GitHub](https://github.com/arctanxarc/MC-LLaVA) | with [MC-LLaVA](https://arxiv.org/abs/2411.11706), multiple concepts |

| Yo'LLaVA | 2024 | 40 | [GitHub](https://github.com/WisconsinAIVision/YoLLaVA?tab=readme-ov-file#yollava-dataset) | with [Yo'LLaVA](https://thaoshibe.github.io/YoLLaVA/), single concept |

| MyVLM   | 2024 | 29 | [GitHub](https://github.com/snap-research/MyVLM?tab=readme-ov-file#dataset--pretrained-concept-heads) | with [MyVLM](https://snap-research.github.io/MyVLM/), single concept |

### Applications

- [Memory and new controls for ChatGPT](https://openai.com/index/memory-and-new-controls-for-chatgpt/)

---

⣶⣶⣶⣶⣶⣖⣒⡄⠀⣶⡖⠲⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣤⠠⡄⠀⠀⠀⠀

⠙⠛⣿⣿⣿⡟⠛⠃⢀⣿⣿⣆⣦⣴⠂⠤⠀⠀⠀⣠⣤⣴⣆⠠⢄⠀⠀⠀⣤⡤⢤⣤⣤⠤⢄⠀⠀⢻⣿⣦⡇⢀⣤⢤⠀

⠀⢀⣿⣿⣿⡇⠀⠀⢸⣿⣿⣿⠛⣿⣷⣄⡇⠀⣼⣿⣿⡟⢿⣷⡄⣣⠀⢘⣿⣿⣿⠿⣿⣧⣈⡆⠀⢹⣿⣿⣷⣾⣧⣴⠀

⠀⢰⣿⣿⣿⠀⠀⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⠙⠛⣻⣧⣾⣿⣿⡷⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⢸⣿⣿⣿⣿⣿⡇⠀

⠀⢸⣿⣿⣿⠀⠀⠀⢸⣿⣿⡿⠀⣿⣿⣿⠃⠀⣰⣾⣿⡿⣿⣿⣿⣟⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⢸⣿⣿⣿⣿⡏⢇⠀

⠀⣼⣿⣿⣿⠀⠀⠀⣸⣿⣿⣟⢠⣿⣿⣿⠀⠀⣿⣿⡟⣇⣾⣿⣿⣯⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⢼⣿⣿⣿⣿⣷⡈⡀

⠀⠻⠿⠿⠟⠀⠀⠀⠻⠿⠿⠏⠸⣿⣿⣿⠀⠀⢿⣿⣿⣿⣿⣿⣿⡇⠀⢸⣿⣿⣿⠀⣿⣿⣿⡇⠀⣿⣿⣿⡟⢻⣿⣧⣇

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠀⠀⠉⠉⠀⠀⠀⠉⠉⠁⠀⠉⠉⠉⠀⠀⠘⠙⠋⠁⠈⠋⠛⠉

⠀⠀⠀⠀⠀⠀⢀⣠⣤⡀⠀⢀⣀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣤⡤⠠⡄⠀⠀⠀⠀⠀⠀⠀

⠀⠀⠀⠀⠀⠀⢹⣿⣄⠱⣠⣿⣧⣴⠀⠀⣠⣤⣤⣀⣀⡀⠀⠀⢀⣤⠤⡀⢀⣠⡤⢄⠀⠈⣿⣿⣦⡇⠀⠀⠀⠀⠀⠀⠀

⠀⠀⠀⠀⠀⠀⠈⢿⣿⣷⣿⣿⣿⡏⠀⣾⣿⣿⣿⣶⣄⡉⡄⠀⣿⣿⣤⣝⢸⣿⣦⣼⠀⠀⣿⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀

⠀⠀⠀⠀⠀⠀⠀⠀⢿⣿⣿⣿⠏⠀⠐⣿⣿⣿⠉⣿⣿⣷⡇⠀⣽⣿⣿⣯⢸⣿⣿⣿⠀⠀⢹⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀

⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⠀⠀⢠⣿⣿⣿⠀⣿⣿⣿⡇⠀⣻⣿⣿⡷⢸⣿⣿⣿⠀⠀⢸⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀

⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⠀⠀⠀⢿⣿⣿⣄⣿⣿⣿⠇⠀⢹⣿⣿⣿⣸⣿⣿⣿⠀⠀⢠⣽⣧⡄⠀⠀⠀⠀⠀⠀⠀

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠛⠛⠋⠀⠀⠀⠈⠛⠛⠛⠛⠛⠉⠀⠀⠈⠛⠛⠛⠋⠛⠛⠋⠀⠀⠈⠛⠛⠁⠀⠀⠀⠀⠀⠀⠀

*And good luck with your research! 🤗✨*