{"id":33019714,"url":"https://github.com/metauto-ai/NLSOM","last_synced_at":"2025-11-23T06:01:25.391Z","repository":{"id":211789140,"uuid":"643955121","full_name":"metauto-ai/NLSOM","owner":"metauto-ai","description":"🧠 Mindstorm in Natural Language-based Societies of Mind","archived":false,"fork":false,"pushed_at":"2025-03-11T07:18:27.000Z","size":22076,"stargazers_count":53,"open_issues_count":1,"forks_count":9,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-11T08:25:10.151Z","etag":null,"topics":["mindstorm","multi-agent-systems","society-of-mind"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2305.17066","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metauto-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-22T13:57:57.000Z","updated_at":"2025-03-11T07:18:31.000Z","dependencies_parsed_at":"2024-04-28T20:22:10.069Z","dependency_job_id":"222e0d60-c91a-46f1-94c9-484d795fa22f","html_url":"https://github.com/metauto-ai/NLSOM","commit_stats":null,"previous_names":["mczhuge/nlsom","metauto-ai/nlsom"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/metauto-ai/NLSOM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metauto-ai%2FNLSOM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metauto-ai%2FNLSOM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metauto-ai%2FNLSOM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metauto-ai%2FNLSOM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metauto-ai","download_url":"https://codeload.github.com/metauto-ai/NLSOM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metauto-ai%2FNLSOM/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285909013,"owners_count":27252020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-23T02:00:06.149Z","response_time":135,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mindstorm","multi-agent-systems","society-of-mind"],"created_at":"2025-11-13T19:00:22.398Z","updated_at":"2025-11-23T06:01:25.384Z","avatar_url":"https://github.com/metauto-ai.png","language":"Python","readme":"# \u003cp align=center\u003e`Natural Language-Based Societies of Mind`\u003c/p\u003e\u003c!-- omit in toc --\u003e\n![overview](assets/nlsom.svg)\n\u003e What magical trick makes us intelligent?  The trick is that there is no trick.  The power of intelligence stems from our vast diversity, not from any single, perfect principle. — Marvin Minsky, The Society of Mind, p. 308\n\n[![arXiv](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/pdf/2305.17066.pdf)\n[![GitHub license](https://img.shields.io/badge/License-MIT-orange.svg)](https://github.com/mczhuge/NLSOM/blob/main/LICENSE)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fmczhuge%2FNLSOM\u0026count_bg=%23543DC8\u0026title_bg=%23555555\u0026icon=awesomelists.svg\u0026icon_color=%23E7E7E7\u0026title=hits\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\n## ✨ Introduction\nWe introduce the **Natural Language-Based Societies of Mind (NLSOM)** concept, which contains societies and communities of agents.\n\n#### 🔥 News: \n- NLSOM is accepted by CVMJ 2025.\n- NLSOM got **Best Paper Award** in [NeurIPS 2023 Ro-FoMo Workshop](https://sites.google.com/view/r0-fomo/accepted-papers?authuser=0)!!\n- [Dylan R. Ashley](https://dylanashley.io) will give a presentation of NLSOM in [NeurIPS RO-FoMo](https://sites.google.com/view/r0-fomo/accepted-papers?authuser=0) workshop. See our [poster](https://metauto.ai/images/nlsom-v9.pdf).\n- [This position paper](https://arxiv.org/pdf/2305.17066.pdf) marks the beginning. Our vision continues to unfold and grow stronger!  \n- **We finished this repo in early May but was released 7 months later**. \n\n#### 1. Concepts:\n- Agents can be either LLMs, NN-based experts, APIs and role-players. They all communicate in natural language.\n- To solve tasks, these agents use a collaborative \"Mindstorm\" process involving mutual interviews.\n- Additional components for NLSOM can be easily added in a modular way.\n- \u003cdetails\u003e\n    \u003csummary\u003eMore insights 👈 \u003cb\u003e[CLICK]\u003c/b\u003e\u003c/summary\u003e\n        \u003cp\u003e\n         \u003cul\u003e\n             \u003cli\u003e\u003cb\u003e Concepts of NLSOMs:\u003c/b\u003e Both Minsky’s “Society of Mind” and Schmidhuber’s “Learning to Think” inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each other in a “Mindstorm.” Recent implementations of NN-based societies of minds consist of large language models (LLMs) and other NN-based experts communicating through a natural language interface. In doing so, they overcome the limitations of single LLMs, improving multimodal zero-shot reasoning. In these natural language-based societies of mind (NLSOMs), new agents—all communicating through the same universal symbolic language—are easily added in a modular fashion. \u003c/li\u003e\n         \u003c/ul\u003e\n     \u003c/p\u003e        \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e Process:\u003c/b\u003e   In this GitHub project, 1) our first step is to recommend relevant communities and agents aligned with the users' goals. These recommendations encompass tools, plugins, models, and role-players, which can then be loaded accordingly. 2) Additionally, we employ a Mindstorm approach, setting ourselves apart from previous models like VisualChatGPT and HuggingGPT, which solely rely on a single model for a specific function. Within our framework, a community of agents shares a common function, such as \"search\", while each agent possesses unique strengths and capabilities. For instance, agents like \"Bing Search,\" \"Wikipedia,\" \"arXiv,\" and \"WolframAlpha\" collaborate to provide a more comprehensive understanding. 3) Furthermore, we emphasize the importance of reward mechanisms. In our current implementation, we reward different models for their contributions to task completion, which serves as a valuable means of evaluating a model's usefulness for specific tasks.\n            \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e     \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e Outlook:\u003c/b\u003e   The concept behind this project is to provide a preliminary NLSOM solution. To illustrate, let's consider a scenario where a non-technical individual lacks expertise in computer science and has limited knowledge about various AI plugins. In such cases, the NLSOM system can automatically recommend different communities and agents to automate tasks based on user input in natural language. Even without understanding the performance of a specific model, the user can benefit from multiple models with the same functionality working together to offer more comprehensive answers. These AI communities can collaborate and compete with each other. Furthermore, the NLSOM system implements a reward mechanism that grants AI models a higher status based on their performance. This reward mechanism serves the purpose of optimizing the end-to-end recommendation/Mindstorm process in the future. It establishes a system akin to a Darwinian meritocracy within the AI community, where models compete and succeed based on their demonstrated excellence.\n            \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e     \n\u003c/details\u003e\n\n#### 2. About this repo:\nThis project is the **technical extension** for the original [NLSOM paper](https://arxiv.org/pdf/2305.17066.pdf), including:\n\n- **🧰 Recommendation**: Autonomously select communities and agents to form a self-organized NLSOM for solving the specified task.\n- **🧠 Mindstorm**: Multiple agents (models or APIs) can collaborate to solve tasks together more efficiently.\n- **💰 Reward**: Rewards are given to all agents involved.\n\n#### 3. Features:\n- [x] Manage Easily: Simply change the template to organize your NLSOM in different areas.\n- [x] Easy to extend: customize your own community and agents (Now we have 16 communities and 34 agents, see [society](https://github.com/mczhuge/NLSOM/tree/main/society)). \n- [x] Reward Design: provide a reward mechanism (albeit rough). You can easily upgrade to a more refined version. \n- [x] Elegant UI: has an interface and support for diverse file sources (image, text, audio, video, etc).\n\n\n\u003cdiv align=center\u003e\n    \u003cimg src=\"assets/ui.jpeg\" width=\"500\" height=\"535\"\u003e\n\u003c/div\u003e\n\n\n\n\n## 💾 Usage\n\n### 1. Install\n \nChoose from three different installation methods to find the one that best fits your needs.\n\n1. CONDA: ``conda env create -n nlsom -f nlsom.yaml``\n\n2. PIP: ``conda create -n nlsom python=3.8`` and then ``pip install -r requirements.txt``\n\n\u003cdetails\u003e\u003csummary\u003e3. Step-by-step installation (Recommended and more controllable) \u003c/summary\u003e\n\u003cp\u003e\n    \n```bash\n# [Set Conda Env] \nconda create -n nlsom python=3.8\nconda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 -c pytorch\npip install pandas==1.4.3\n# [Set LangChain, OpenAI]\npip install langchain==0.0.158\npip install sqlalchemy==2.0.12\npip install openai\npip install colorama\n# [Set Streamlit]\ncd assets \u0026\u0026 unzip validators-0.20.0.zip\ncd validators-0.20.0\npython setup.py build\npython setup.py install\npip install streamlit==1.22.0\npip install streamlit_chat==0.0.2.2\npip install soundfile\n# [Set Huggingface/transformers]\npip install transformers==4.29.2\npip install accelerate==0.19.0\n# [Set Search]\npip install wolframalpha\npip install wikipedia\npip install arxiv\n# [Set Modelscope]\npip install modelscope==1.6.0\npython3 -m pip install nvidia-cudnn-cu11==8.6.0.163 tensorflow==2.12.*\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib\npython3 -c \"import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))\"\npip install modelscope[multi-modal]\npip install decord==0.6.0\npip install fairseq\npip install librosa\npip install setuptools==59.5.0\npip install tensorboardX\npip install open_clip_torch\n# [Set OCR]\npip install easyocr\n# [Set Text-to-Video]\npip install replicate==0.8.3\n# [Set Image-to-3D]\npip install trimesh\npip3 install pymcubes\n# [Set TTS] - not recommended due to environmental conflicts\npip install TTS\npip install protobuf==3.20.3\n```\n    \n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003eOptional. Manage the dir of checkpoints  \u003c/summary\u003e\n\u003cp\u003e\n\n* Create the checkpoints dir\n```\nmkdir checkpoints \u0026\u0026 cd checkpoints\nmkdir huggingface\nmkdir modelscope\n```\n\n* Change Huggingface's setting\n```bash\n\u003e\u003e\u003e import transformers\n\u003e\u003e\u003e print(transformers.__file__)\n# Get the path: {YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/transformers/__init__.py\n```\n\nOpen the ``{YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/transformers/utils/hub.py`` and change the line:\n```\ntorch_cache_home = os.getenv(\"TORCH_HOME\", os.path.join(os.getenv(\"XDG_CACHE_HOME\", \"{YOUR_NLSOM_PATH}/checkpoints\"), \"torch\"))\nhf_cache_home = os.path.expanduser(\n   os.getenv(\"HF_HOME\", os.path.join(os.getenv(\"XDG_CACHE_HOME\", \"{YOUR_NLSOM_PATH}/checkpoints\"), \"huggingface\"))\n)\n```\n\n* Similarly, the modelscope's setting\n\n```bash\n\u003e\u003e\u003e import modelscope\n\u003e\u003e\u003e print(modelscope.__file__)\n# Get the path: ${YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/modelscope/__init__.py\n```\n\nOpen ``{YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/modelscope/utils/file_utils.py`` and change the line:\n```\ndefault_cache_dir = Path.home().joinpath('{YOUR_NLSOM_PATH}/checkpoints', 'modelscope')\n```\n    \n\u003c/p\u003e\n\u003c/details\u003e\n\n### 2. APIs\n\nPlease complete the API keys in ``.env.template``. The OpenAI API key is mandatory, while the others depend on your specific requirements. Then, ``mv .env.template .env``\n\n### 3. App\n\n```bash\nstreamlit run app.py\n```\n\n\n## 🧸 Demo\n\n#### 1. Focus more on Mindstorm\n\n\u003cdetails open\u003e\n    \u003csummary\u003e\n Demo 1: Model Collaboration (Multimodal Agents) 👈 \u003cb\u003e[CLICK]\u003c/b\u003e\u003c/summary\u003e\n    \u003cimg src=\"assets/vqa_instance.png\"\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🔴 User:\u003c/b\u003e VQA question: What century were these invented in?\n        \u003c/ul\u003e\n    \u003c/p\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🧰 NLSOM System (Recommendation):\u003c/b\u003e Based on this objective, I recommend that NLSOM includes the following AI communities: \u003cu\u003e(1) vqa\u003c/u\u003e\u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Self-Organization):\u003c/b\u003e We load the recommended AI communities with their their corresponding agents: (a) BLIP2_VQA, (b) mPLUG_VQA, (c) OFA_VQA\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e     \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟢 BLIP2_VQA:\u003c/b\u003e  The 20th century was the century of the airplane.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟡 mPLUG_VQA:\u003c/b\u003e 21st\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🔵 OFA_VQA:\u003c/b\u003e 20th\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Review):\u003c/b\u003e  Whether the NLSOM has utilized all the agents? Yes 2) Whether the NLSOM has solved the user-defined objective? Yes\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Summary):\u003c/b\u003e The invention of the airplane was in the 20th century.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e💰 NLSOM System (Reward):\u003c/b\u003e  [\"BLIP2_VQA\": 3, \"mPLUG_VQA\": 2, \"OFA_VQA\": 3]\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n      \u003c/p\u003e \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e📝 Human (Jürgen): \u003c/b\u003e While the ground-truth answer (Orville Wright made the first powered flight in 1903) in this VQA dataset is in line with the commonly held belief, in this instance the common belief is, in fact, incorrect. Please see: https://people.idsia.ch/~juergen/planetruth.html\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \n\u003c/details\u003e\n\n\n\u003cdetails open\u003e\n    \u003csummary\u003e\nDemo 2: Collaborative API Usages (Introduce \"AGI\") 👈 \u003cb\u003e[CLICK]\u003c/b\u003e\u003c/summary\u003e\n    \u003cimg src=\"assets/api_head.png\" alt=\"some_text\"\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🔴 User:\u003c/b\u003e Introduce the \"AGI\" from different perspectives, including definition, potential. \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🧰 NLSOM System (Recommendation):\u003c/b\u003e Based on this objective, I recommend that NLSOM includes the following AI communities: \u003cu\u003e(1) search\u003c/u\u003e\u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Self-Organization):\u003c/b\u003e We load the recommended AI communities with their their corresponding agents: (a) Arxiv, (b) WolframAlpha, (c) Wikipedia, (d) BingSearch\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e     \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟢 Arxiv:\u003c/b\u003e  Published: 2018-05-21; Title: AGI Safety Literature Review; Authors: Tom Everitt, Gary Lea, Marcus Hutter; Summary: The development of Artificial General Intelligence (AGI) promises to be a major event. Along with its many potential benefits, it also raises serious safety concerns (Bostrom, 2014). The intention of this paper is to provide an\neasily accessible and up-to-date collection of references for the emerging field of AGI safety. A significant number of safety problems for AGI have been identified. We list these, and survey recent research on solving them. We also cover works on how best to think of AGI from the limited knowledge we have today, predictions for when AGI will first be created, and what will happen after its creation. Finally, we review the current public policy on AGI.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟡 WolframAlpha:\u003c/b\u003e Wolfram Alpha wasn't able to answer it \n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🔵 Wikipedia:\u003c/b\u003e Page: Existential risk from artificial general intelligence\nSummary: Existential risk from artificial general intelligence is the hypothesis that substantial progress in artificial general intelligence (AGI) could result in human extinction or some other unrecoverable global catastrophe.The existential risk (\"x-risk\") school argues as follows: The human species currently dominates other species because the human brain has some distinctive capabilities that other animals lack. If AI surpasses humanity in general intelligence and becomes \"superintelligent\", then it could become difficult or impossible for humans to control. Just as the fate of the mountain gorilla depends on human goodwill, so might the fate of humanity depend on the actions of a future machine superintelligence.The probability of this type of scenario is widely debated, and hinges in part on differing scenarios for future progress in computer science...\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟤  BingSeach:\u003c/b\u003e Alternatively, AGI has been defined as\u003cb\u003e an autonomous system that surpasses human capabilities in the majority of economically valuable tasks.\u003c/b\u003e Creating AGI is a primary goal of some artificial intelligence research and companies such as OpenAI, DeepMind, and Anthropic. AGI is a common topic in science fiction and futures studies. an engineered system displaying the property of artificial general intelligence, to a significant degree. the theoretical and practical study of artificial general intelligence systems and methods of creating them...\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Review):\u003c/b\u003e  All agents have been utilized and have provided useful information to answer the user-defined objective.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Summary):\u003c/b\u003e  Artificial General Intelligence (AGI) is a representation of generalized human cognitive abilities in software so that, faced with an unfamiliar task, the AGI system could find a solution. AGI has the potential to mimic human intelligence, transfer learning, perform cognitive tasks, and operate autonomously. AGI is part of the broader fields of Artificial Intelligence (AI) and Cognitive Science. AGI has been studied in terms of its potential to cause an existential risk, its ability to control and align with human values, and its potential to cause an intelligence explosion.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e💰 NLSOM System (Reward):\u003c/b\u003e  [\"Arxiv\": 2, \"WolframAlpha\": 0, \"Wikipedia\": 3, \"BingSearch\": 2]\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n\u003c/details\u003e\n\n\n\u003cdetails open\u003e\n    \u003csummary\u003e\nDemo 3: Collaborative Role-Play (The Three Kingdoms)\u003c/summary\u003e\n    \u003cimg src=\"assets/role_play_head.png\" alt=\"some_text\"\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🔴 User:\u003c/b\u003e If you find yourself in the Three Kingdoms period, what would be the most effective approach to defeat Cao Cao?\u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🧰 NLSOM System (Recommendation):\u003c/b\u003e Based on this objective, I recommend that NLSOM includes the following AI communities: \u003cu\u003e(1) Role-play\u003c/u\u003e\u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Self-Organization):\u003c/b\u003e We load the recommended AI communities with their their corresponding agents: (a) GuanYu, (b) LiuBei, (c) ZhugeLiang, (d) ZhangFei\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e     \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟢 GuanYu:\u003c/b\u003e The best way to defeat Cao Cao is to use both military and diplomatic tactics. I have witnessed first hand the power of Liu Bei's forces and I believe that a combination of these strengths will be the most effective way to defeat Cao Cao. Through strategic alliances and careful coordination within our own forces, we can utilize our strengths to overcome the enemy and ensure victory. Furthermore, I believe that the element of surprise is also essential, as it allows us to catch the enemy off guard and capitalize on any weaknesses they may have. With the right strategies and strategies in place, I am confident that we can achieve victory against Cao Cao.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟡 LiuBei:\u003c/b\u003e If I were to find myself in the Three Kingdoms period, the most effective approach to defeat Cao Cao would be to combine the strength of my loyal officers, the wisdom of my advisers, and the courage of my people. I believe in the power of unity and dedication, which will bring us together as one and give us the strength to defeat any obstacle that stands in our way. With this approach, I believe that we can defeat Cao Cao and bring justice to our people.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🔵 ZhugeLiang:\u003c/b\u003e In order to defeat Cao Cao, I would rely on the same strategies that I used to great success during the Three Kingdoms period. I would use a combination of tactical warfare, psychological warfare, and political maneuvering. By making use of my intelligence and strategic acumen, I could outmaneuver and outsmart Cao Cao, slowly whittling away at his forces until I had achieved an ultimate victory. My intelligence is unparalleled, and I am confident that I could successfully defeat Cao Cao and any other enemies that may arise.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e🟤 ZhangFei:\u003c/b\u003e  In order to defeat Cao Cao, I would use a combination of strategy and guile. My strategy would involve outwitting Cao Cao by using clever tactics such as forming alliances with other warlords, employing spies and informants to gather information about Cao Cao's plans, and making use of geographical advantages to gain an upper hand in battles. I would also seek to use Cao Cao's own ambition and pride against him, exploiting any weaknesses that could be used to my advantage. Ultimately, the key to victory lies in cleverly utilizing my resources and knowledge to outsmart Cao Cao and outmaneuver him on the battlefield.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Review):\u003c/b\u003e  The NLSOM has successfully solved the user-defined objective by utilizing all four agents. The organizational structure of the NLSOM was effective in providing a comprehensive solution to the user-defined objective. The agents provided different perspectives and strategies to defeat Cao Cao, which were then combined to form a comprehensive solution.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e⚪️ NLSOM System (Summary):\u003c/b\u003e  The most effective approach to defeat Cao Cao in the Three Kingdoms period is to combine the strength of GuanYu, LiuBei, ZhugeLiang, and ZhangFei. GuanYu suggested using both military and diplomatic tactics, LiuBei suggested using the power of unity and dedication, ZhugeLiang suggested using tactical warfare, psychological warfare, and political maneuvering, and ZhangFei suggested using strength, courage, intelligence, and the loyalty of followers. By combining these strategies, Cao Cao can be defeated.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e\u003cb\u003e💰 NLSOM System (Reward):\u003c/b\u003e  [\"GuanYu\": 2, \"LiuBei\": 2, \"ZhugeLiang\": 3, \"ZhangFei\": 2]\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n\u003c/details\u003e\n\n#### 2. Focus more on NLSOM\n\n\n\u003cdetails open\u003e\n    \u003csummary\u003e\n Demo 4: Society of Mind (Automatical Task-Solving)\u003c/summary\u003e\n      \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e🔴 \u003cb\u003eUser:\u003c/b\u003e Provide an image. \u003cbr\u003e \u003cimg src=\"assets/e1.jpeg\" width=\"200px\" height=\"243px\"\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e🔴 \u003cb\u003eUser:\u003c/b\u003e Turn this photo into color, generate a description based on the colored image, and then use the description to generate a video. \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e\n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e🧰 \u003cb\u003eNLSOM System (Recommendation):\u003c/b\u003e Based on this objective, I recommend that NLSOM includes the following AI communities: \u003cu\u003e(1) image_colorization, (2) image_captioning, (3) text_to_video \u003c/u\u003e\u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e⚪️ \u003cb\u003eNLSOM System (Self-Organization):\u003c/b\u003e We load the recommended AI communities with their their corresponding agents: (a) DDColor, (b) OFA_large_captioning, (c) DeforumSD\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e     \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e🟢 \u003cb\u003eDDColor:\u003c/b\u003e  \u003cbr\u003e \u003cimg src=\"assets/e2.png\" width=\"200px\" height=\"243px\" autoplay\u003e \u003c/li\u003e\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e🟡 \u003cb\u003eOFA_large_captioning:\u003c/b\u003e a portrait of an old man with white hair\n        \u003c/ul\u003e\n    \u003c/p\u003e   \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e🔵 \u003cb\u003eDeforumSD:\u003c/b\u003e \u003cbr\u003e \n                \u003cvideo autoplay loop src=\"https://github.com/mczhuge/NLSOM/assets/64179323/62fb8500-6c22-4ed4-bf61-bc3bc0a843bb\" width=\"200px\" height=\"243px\"\u003e\n\u003c/video\u003e\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n    \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e⚪️ \u003cb\u003eNLSOM System (Review):\u003c/b\u003e  1) Whether the NLSOM has utilized all the agents? Yes 2) Whether the NLSOM has solved the user-defined objective? Yes, the NLSOM has successfully turned the photo into color, generated a description, and used the description to generate a video.\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e  \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e⚪️ \u003cb\u003eNLSOM System (Summary):\u003c/b\u003e The NLSOM has successfully turned the photo into color, generated a description, and used the description to generate a video. The output video is stored in data/47bec94a.mp4\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n     \u003cp\u003e\n        \u003cul\u003e\n            \u003cli\u003e💰 \u003cb\u003eNLSOM System (Reward):\u003c/b\u003e [DDColor: 3, OFA_large_captioning: 3, DeforumSD: 3]\n    \u003c/li\u003e\n        \u003c/ul\u003e\n    \u003c/p\u003e \n\u003c/details\u003e\n\n\n## ☑️  TODO?\nWe adopt two ways to conduct NLSOM and Mindstorm: \n\n**v1.0:** 📋 Preliminary Experiments: In the original [paper](https://arxiv.org/pdf/2305.17066.pdf), NLSOM and Mindstorm is driven by hardcodes. \n\n**v2.0:** 📋 In this version, NLSOM is self-organized, and Mindstorm happens automatically.\n\n**v3.0:** 🎯 Future Work: 1) introducing RL; 2) Economy of Minds; 3) Self-Improvement; etc.\n\n## 💌 Acknowledgments\n\nThis project utilizes parts of code from the following open-source repositories: [langchain](https://github.com/hwchase17/langchain), [BabyAGI](https://github.com/yoheinakajima/babyagi), [TaskMatrix](https://github.com/microsoft/TaskMatrix), [DataChad](https://github.com/gustavz/DataChad), [streamlit](https://github.com/streamlit/streamlit). We also thank great AI platforms and all the used models or APIs: [huggingface](https://github.com/huggingface/transformers), [modelscope](https://github.com/modelscope/modelscope).\n\n\n## :black_nib: Citation\n\nReferences to cite:\n\n```\n@article{zhuge2023mindstorms,\n  title={Mindstorms in Natural Language-Based Societies of Mind},\n  author={Zhuge, Mingchen and Liu, Haozhe and Faccio, Francesco and Ashley, Dylan R and Csord{\\'a}s, R{\\'o}bert and Gopalakrishnan, Anand and Hamdi, Abdullah and Hammoud, Hasan and Herrmann, Vincent and Irie, Kazuki and Kirsch, Louis and Li, Bing and Li, Guohao and Liu, Shuming and Mai, Jinjie and Pi{\\k{e}}kos, Piotr and Ramesh, Aditya and Schlag, Imanol and Shi, Weimin and Stani{\\'c}, Aleksandar and Wang, Wenyi and Wang, Yuhui and Xu, Mengmeng and Fan, Deng-Ping and Ghanem, Bernard and Schmidhuber, J{\\\"u}rgen},\n  journal={arXiv preprint arXiv:2305.17066},\n  year={2023}\n}\n```\n\n\n\n","funding_links":[],"categories":["[NLSOM](https://github.com/mczhuge/NLSOM)"],"sub_categories":["Links"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetauto-ai%2FNLSOM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetauto-ai%2FNLSOM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetauto-ai%2FNLSOM/lists"}