{"id":20779440,"url":"https://github.com/freedomintelligence/mamo","last_synced_at":"2025-04-09T15:52:23.903Z","repository":{"id":241101629,"uuid":"803978685","full_name":"FreedomIntelligence/Mamo","owner":"FreedomIntelligence","description":"Mamo: a Mathematical Modeling Benchmark with Solvers","archived":false,"fork":false,"pushed_at":"2024-06-12T12:39:39.000Z","size":507,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-23T18:11:16.010Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-sa-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-21T18:13:47.000Z","updated_at":"2025-03-16T09:29:38.000Z","dependencies_parsed_at":"2025-01-18T08:32:44.567Z","dependency_job_id":"6327f146-f54e-44b8-b47e-0d94c315769a","html_url":"https://github.com/FreedomIntelligence/Mamo","commit_stats":null,"previous_names":["freedomintelligence/mamo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMamo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMamo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMamo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FMamo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/Mamo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248064323,"owners_count":21041855,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:27:56.443Z","updated_at":"2025-04-09T15:52:23.883Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cspan style=\"color:blue\"\u003e\u003cb\u003eMamo\u003c/b\u003e\u003c/span\u003e: a \u003cspan style=\"color:blue\"\u003e\u003cb\u003eMa\u003c/b\u003e\u003c/span\u003ethematical \u003cspan style=\"color:blue\"\u003e\u003cb\u003eMo\u003c/b\u003e\u003c/span\u003edeling Benchmark with Solvers\n\n\u003c!-- \u003cp align=\"center\"\u003e\n  \u003ca 📃 href=\"https://arxiv.org/abs/2405.13144\"\u003eOur Paper\u003c/a\u003e\n\u003c/p\u003e --\u003e\n\n\n## Table of Contents\n- [Introduction](#introduction)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Data Description](#data-description)\n  - [Running Benchmark](#running-benchmark)\n\u003c!-- - [Results](#results)\n- [Citation](#citation) --\u003e\n\n## Introduction\nWellcome to Mamo! This is a benchmark for mathematical modeling. \n\n\n\n## Installation\n\n### Prerequisites\n\nBefore you begin, ensure you have met the following requirements:\n\n- You have installed [Git](https://git-scm.com/).\n- You have a [GitHub](https://github.com/) account.\n- You have access to the terminal (Command Prompt on Windows, Terminal on macOS and Linux).\n- If you want to run Optimization parts, please make sure you have an optimization solver that can read .lp file installed (Here we use COPT, you could use the solver you want and do some small change in the codes)\n\n### Clone the Repository\nTo get started, clone the repository and install the necessary dependencies:\n```bash\ncd ~/Projects\ngit clone https://github.com/FreedomIntelligence/Mamo.git\ncd Mamo\n```\n\n- For the close source models, we call the APIs, the requirements are:\n```bash\npip install -r requirements.txt\n```\n\n- For open source models, we use VLLM for deployment and inference.\n```bash\npip install vllm\n```\n\nReplace the ~/Projects with the directionary for Mamo\n\n\n## Usage\n  ### data-description\n  - The data consists of two parts: ODE and Optimization, while optimziation part consists of Easy_LP and Complex_LP, see in Data folder (mamo_ode.jsonl, mamo_complex_lp.jsonl, mamo_easy_lp.jsonl)\n  - Each data is a json object, with \n  ```json\n  {\n      \"id\": id, \n      \"Question\": the question, \n      \"Answer\": the answer, \n      \"Category\": ode/optimization, \n      \"Type\": first_order_equation/.../easy_lp/complex_lp\n  }\n  ```\n  ### Running Benchmark\n\n  To run the benchmark(for close source model calling API), follow these steps:\n\n  1. **Navigate to the Script Directory**:\n  First, go to the appropriate script directory. Depending on your needs, use one of the following commands:\n  ```bash\n  cd scripts/scripts_optimization/mamo_script_optimization\n  or \n  cd scripts/scripts_ode/mamo_script_ode\n  ```\n\n  2. **Fill in the Required Paths**\n    Ensure all necessary path variables are correctly set in your environment or script files.\n\n  3. **Execute the Scripts in Bash**:\n  Run the following scripts in order. Choose the appropriate set of scripts based on whether you are working with ODE or optimization:\n\n  - For ODE\n  ```bash\n  bash 1.prepare_query_ode.sh\n  bash 2.run_model_ode.sh\n  bash 3.run_code_comp_ode.sh\n  bash 4.fix_error_ode.sh\n  bash 5.rerun_code_comp_ode.sh\n  ```\n\n  - For Optimization:\n  ```bash\n  bash 1.prepare_query_optimization.sh\n  bash 2.run_model_optimization.sh\n  bash 3.run_code_comp_optimization.sh\n  bash 4.fix_error_optimization.sh\n  bash 5.rerun_code_comp_optimization.sh\n  ```\n\n  To run the benchmark(for open source model), follow these steps:\n\n  1. **Navigate to the Script Directory**:\n  First, go to the appropriate script directory. Depending on your needs, use one of the following commands:\n  ```bash\n  cd scripts/scripts_optimization/mamo_script_optimization\n  or \n  cd scripts/scripts_ode/mamo_script_ode\n  ```\n\n  2. **Fill in the Required Paths**\n    Ensure all necessary path variables are correctly set in your environment or script files.\n\n  3. **Execute the Scripts in Bash**:\n  Run the following scripts in order. Choose the appropriate set of scripts based on whether you are working with ODE or optimization:\n\n  - For ODE\n  ```bash\n  bash 1.prepare_query_ode.sh\n  bash 2.run_model_ode_open.sh\n  bash 3.run_code_comp_ode.sh\n  bash 4.fix_error_ode_open.sh\n  bash 5.rerun_code_comp_ode.sh\n  ```\n\n  - For Optimization:\n  ```bash\n  bash 1.prepare_query_optimization.sh\n  bash 2.run_model_optimization_open.sh\n  bash 3.run_code_comp_optimization.sh\n  bash 4.fix_error_optimization_open.sh\n  bash 5.rerun_code_comp_optimization.sh\n  ```\n\n  ### Few-shot Experiment\n\n  To run the few shots experiment, here is the instruction:\n\n  1. **Navigate to the Script Directory**:\n  Select the target file based on your needs and use one of\n  ```bash\n  cd scripts/scripts_optimization/few_shot_scripts_ode\n  cd scripts/scripts_ode/few_shot_scripts_optimization\n  ```\n  2. **Fill in the Required Paths**:\n  Make sure to update file paths and adjust environment-specific parameters in the scripts in your local environment.\n\n  3. **Select data from datasets**:\n  Extract samples from the corresponding data sets (`mamo_ode.jsonl`, `mamo_complex_lp.jsonl`, `mamo_easy_lp.jsonl`) to form the data sets (`ode_select.jsonl`, `lp_select.jsonl`) of few shots.\n\n  4. **Update the appropriate prompts**:\n  Choose the prompts with the number of shots you need from data folders `ode_few_shot_prompt` and `lp_few_shot_prompt` to update the `prompt_template`\" in the `1.prepare_query_ode.py` and `1.prepare_query_optimization.py` respectively.\n\n  5. **Execute the Scripts in Bash**:\n  Adopt the scripts sequentially depending on your needs as listed below.\n  - For ODE\n  ```bash\n  bash 1.prepare_query_ode.sh\n  bash 2.run_model_ode.sh\n  bash 3.run_code_comp_ode.sh\n  bash fix_error_ode.sh\n  ```\n\n  - For Optimization:\n  ```bash\n  bash 1.prepare_query_optimization.sh\n  bash 2.run_model_optimization.sh\n  bash 3.run_code_comp_optimization.sh\n  call remove_quotes.bat\n  ```\n\n\n\n## Result\n\n\n ## Citation\n\nIf you use this work, please cite it as follows:\n\n```bibtex\n@misc{huang2024mamo,\n      title={Mamo: a Mathematical Modeling Benchmark with Solvers}, \n      author={Xuhan Huang and Qingning Shen and Yan Hu and Anningzhe Gao and Benyou Wang},\n      year={2024},\n      eprint={2405.13144},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fmamo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fmamo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fmamo/lists"}