{"id":28676575,"url":"https://github.com/zjunlp/machinesom","last_synced_at":"2025-06-13T23:05:13.803Z","repository":{"id":180995189,"uuid":"663732093","full_name":"zjunlp/MachineSoM","owner":"zjunlp","description":"[ACL 2024]  Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View","archived":false,"fork":false,"pushed_at":"2024-05-16T12:58:40.000Z","size":26561,"stargazers_count":74,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-05-16T14:00:30.359Z","etag":null,"topics":["agent","agent-learning","agents","artificial-intelligence","large-language-models","llm","machine-learning","natural-language-processing","society-of-mind","theory-of-mind"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-08T01:18:02.000Z","updated_at":"2024-05-16T12:58:43.000Z","dependencies_parsed_at":"2023-10-01T15:24:38.276Z","dependency_job_id":"48620943-30ba-43e4-9f37-ea58dd4a44bf","html_url":"https://github.com/zjunlp/MachineSoM","commit_stats":null,"previous_names":["zjunlp/machinesom"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/MachineSoM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMachineSoM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMachineSoM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMachineSoM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMachineSoM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/MachineSoM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FMachineSoM/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259732770,"owners_count":22903087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","agent-learning","agents","artificial-intelligence","large-language-models","llm","machine-learning","natural-language-processing","society-of-mind","theory-of-mind"],"created_at":"2025-06-13T23:05:12.999Z","updated_at":"2025-06-13T23:05:13.792Z","avatar_url":"https://github.com/zjunlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n![case](figs/gif.gif)\n\n\n# 🧩 MachineSoM\n\nCode for the paper \"Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View\".\n\n![settings](figs/setting.jpg)\n\n\n- The Society of Mind (SoM): the emergence of intelligence from collaborative and communicative computational modules, enabling humans to collaborate and complete complex tasks effectively\n- Societies of LLM agents with different **traits**: **easy-going and overconfident**\n- Collaboration Processes: **debate and self-reflection**\n- Interaction Strategies: when to interact, interact with whom\n\n## *📬 News!*\n\n- **[2023.05.16]** The paper \"Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View\" is **accepted by ACL 2024 main conference.** \n- **[2024.03.11]** The paper \"Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View\" is **accepted by Workshop on LLM Agents, ICLR 2024.** \n- **[2023.10.03]** The paper \"Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View\" is released.\n- **[2023.07.13] MachineSoM code is released!**\n\n## *🎉 Quick Links*\n\n- [🛠️ Requirements \u0026 Dataset](#1)\n- [🚴 How to run](#2)\n\n\n\u003ch2 id=\"1\"\u003e🛠️ Requirements \u0026 Dataset\u003c/h2\u003e\n\nConfigure the environment using the following command:\n\n```bash\nconda create -n masom python=3.9\npip install -r requirements.txt\n```\n\nThe data we sampled and used for the experiment is in the folder `eval_data`. You can download the raw datasets for *[MMLU](https://huggingface.co/datasets/cais/mmlu)*, *[Math](https://github.com/google/BIG-bench/blob/761845c22056c885429efd2cfcec345ae00c1de7/bigbench/benchmark_tasks/chess_state_tracking/synthetic_short/task.json)* and *[Chess Move Validity](https://github.com/hendrycks/math)* separately. \n\n\n\u003ch2 id=\"2\"\u003e🚴 How to run\u003c/h2\u003e\n\nHere is a brief overview of each file in the folder `src`:\n\n```python\n# Core Code\n|- api.py               # API for storing the experiments\n|- dataloader.py        # Load datasets for experiments\n|- evaluate.py          # Evaluate the results of the experiment\n|- prompt.py            # Stores all the prompts involved in the experiment.\n|- utils.py             # The management center of the agents.\n|- run_main.py          # Simulate a society of agents using different collaborative strategies to solve problems (main experiment)\n|- run_agent.py         # Exploring the Impact of Agent Quantity\n|- run_strategy.py      # All agents can adopt different thinking patterns for collaboration\n\n# Other Code\n|- run_main.sh          # Main experiment running script\n|- run_agent.sh         # The running script of the experiment on the number of agents\n|- run_strategy.sh      # The running script of the experiment on the strategies\n|- run_turn.sh          # The running script of the experiment on the number of collaboration rounds\n|- conformity_and_consistency.py    # Process and draw figures such as Figure 6 and Figure 7 in the paper\n|- draw                 # Include the drawing code for all the figures in the paper\n    |- draw_10_agent.py\n    |- draw_conformity\n    |- ...\n```\n\n1. Edit `src/api.py` to add your api-key.\n\n   ```python\n   openai_api = {\n       \"replicate\":[\n           \"api_1\", \"api_2\", \n       ],\n       \"dashscope\": [\n           \"api_1\"\n       ],\n       \"openai\": [\n           \"api_1\"\n       ],\n       \"anyscale\":[\n           \"api_1\", \"api_2\", \"api_3\"\n       ]\n   }\n   ```\n\n   Our coding framework offers compatibility with a variety of inference services across multiple platforms, such as **Replicate**, **OpenAI**, **Dashscope**, and **Anyscale**. Specifically, **Dashscope** facilitates the deployment of the `Qwen` model, whereas **OpenAI** provides support for the `GPT` model integration.\n\n\n2. Execute the scripts `run_main.sh`, `run_agent.sh`, `run_turn.sh`, and `run_strategy.sh` housed in the `src` directory. These scripts are designed to initiate a variety of experiments: the main experiment (corresponding to Table 2 in the paper), variations in agent numbers (Figure 3 in the paper), differing collaboration round counts (Figure 4 in the paper), and trials involving alternative collaboration strategies (Figure 5 in the paper). You can adjust the parameters within the scripts to accommodate different experimental settings.\n\n    All the data in the paper is available for download on [Google Drive](https://drive.google.com/file/d/1KefQzwfINdTZI-vM_eYJis0npc32gBJt/view?usp=sharing).\n\n\n3. Execute the `evaluate.py` in the `src` directory. \n\n    a. For the main experiment results, you can execute the following command:\n    ```bash\n    python evaluate.py main_table --experiment_type gpt-1106-main --dataset mmlu\n    ```\n    This code will be output in LaTeX code format. The argument `--experiment` should be the name of a folder. To replicate the results presented in the paper, after downloading and uncompressing it into the `src` root directory, rename the `upload` to `results`. At this point, the available options for the `--experiment` parameter are `gpt-1106-main`, `llama13-main`, `llama70-main`, `qwen-main`, and `mixtral-main`. The optional values for argument `--dataset` are `mmlu`, `math`, and `chess`.\n\n    b. For the siginificant test, you can execute the following commands:\n    ```bash\n    python evaluate.py anova --types main --dataset chess --experiment_type \"['llama13-main','gpt-1106-main']\"\n    python evaluate.py anova --types turn --dataset chess  --experiment_type \"['llama13-turn-4','llama70-turn-4']\"\n    python evaluate.py anova --types 10-turn --dataset chess --experiment_type \"['gpt-1106-turn-10', 'qwen-turn-10', 'mixtral-turn-10']\"\n    python evaluate.py anova --types agent --dataset chess --experiment_type \"['llama13-main','llama70-main']\"\n    python evaluate.py anova --types 10-agent --dataset chess --experiment_type \"['gpt-1106-main','qwen-main']\"\n    python evaluate.py anova --types strategy --dataset chess --experiment_type \"['gpt-1106-main','qwen-main']\"\n    ```\n    You can change the `--dataset` and `--experiment_type` to get the original result (e.g., Table 6 in the paper).\n\n    c. We also provide code for drawing figures in the paper. For the vast majority of figures, the following code can be executed:\n    ```bash\n    python evaluate.py draw --types distribute --experiment_type gpt-1106-main\n    python evaluate.py draw --types agent --experiment_type llama13-main\n    python evaluate.py draw --types turn --experiment_type llama70-main\n    python evaluate.py draw --types strategy --experiment_type gpt-1106-main\n    python evaluate.py draw --types 10-agent --experiment_type gpt-1106-main\n    python evaluate.py draw --types 10-turn --experiment_type gpt-1106-main --dataset chess\n    python evaluate.py draw --types radar --experiment_type gpt-1106-main\n    python evaluate.py draw --types 10-agent-consistent --experiment_type gpt-1106-main\n    python evaluate.py draw --types word --experiment_type gpt-1106-main\n    ```\n\n    d. It should be noted that in order to obtain the results of Figures 6 and 7, you need to execute the following code:\n    ```bash\n    python conformity_and_consistency.py --experiment_type 'mixtral-main' --type 'consistent'\n    python conformity_and_consistency.py --experiment_type 'mixtral-main' --type 'conformity'\n    ```\n\n    \u003e There are stylistic differences between the figures in the paper and those generated by the code, such as legend, color, layout, etc. But the data has not changed at all.\n\n## How to Cite 📝\n📋 Thank you very much for your interest in our work. If you use or extend our work, please cite the following paper:\n\n```bibtex\n@inproceedings{ACL2024_MachineSoM,\n  author       = {Jintian Zhang and\n                  Xin Xu and\n                  Ningyu Zhang and\n                  Ruibo Liu and\n                  Bryan Hooi and\n                  Shumin Deng},\n  title        = {Exploring Collaboration Mechanisms for {LLM} Agents: {A} Social Psychology View},\n  booktitle    = {{ACL} {(1)}},\n  pages        = {14544--14607},\n  publisher    = {Association for Computational Linguistics},\n  year         = {2024},\n  url          = {https://aclanthology.org/2024.acl-long.782/}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fmachinesom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Fmachinesom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fmachinesom/lists"}