{"id":20779453,"url":"https://github.com/freedomintelligence/ovm","last_synced_at":"2025-04-30T19:43:01.974Z","repository":{"id":208810636,"uuid":"719423092","full_name":"FreedomIntelligence/OVM","owner":"FreedomIntelligence","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-02T06:08:26.000Z","size":4483,"stargazers_count":65,"open_issues_count":0,"forks_count":6,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-30T19:22:22.419Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-11-16T06:08:18.000Z","updated_at":"2025-02-22T02:04:35.000Z","dependencies_parsed_at":"2023-12-15T08:42:00.452Z","dependency_job_id":null,"html_url":"https://github.com/FreedomIntelligence/OVM","commit_stats":null,"previous_names":["freedomintelligence/ovm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FOVM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FOVM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FOVM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FOVM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/OVM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251771958,"owners_count":21641341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:28:00.446Z","updated_at":"2025-04-30T19:43:01.945Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning\n\n\nCode, metrics, and models for the paper [Outcome-supervised Verifiers for Planning in Mathematical Reasoning](https://arxiv.org/pdf/2311.09724.pdf)\n\n\nThe key technical implementations (`utils/sampling.py`):\n\n1. **Value-guided beam search**: step-level beam search guided by a value model\n\n2. **Allow batch generation with caculator using cache** (2-3 times faster than a naive implementation)\n\n\n## Models and Data\n\n| Model                | Dataset      | Link                     |\n|----------------------|--------------|--------------------------|\n| OVM-Llama2-7B        | GSM8K        | [parameters](https://huggingface.co/FreedomIntelligence/OVM-llama2-7b)  |\n| OVM-Mistral-7B       | GSM8K        | [parameters](https://huggingface.co/FreedomIntelligence/OVM-Mistral-7b) |\n\nSee the training data of our value models (generated by the generators) in [dataset](https://huggingface.co/datasets/FreedomIntelligence/OVM-dataset)\n\nSee the training data for Process Reward Models in GSM8K in [dataset](https://huggingface.co/datasets/FreedomIntelligence/OVM-process)\n\n## Somethings for code\n\n\n1. Directories\n- `configs`: for model training with `accelerate`\n- `data`: benchmark, and generator created data for training the value model\n- `eval_results`: metrics and responses\n    - `generator`: generator-only (greedy, self-consistency, or pass@k)\n    - `verifier`: ORM accuracy\n    - `generator_with_verifier`: guided beam search, i.e. OVM and PRM\n- `scripts`: scripts for training and inference\n- `utils`: functions and classes\n\n\n2. target_set\n- GSM8K: there are `train` and `test`, which corresponds to training set and test set respectively\n- Game of 24: there are `train` and `mid`\n    - `train`: the first 900 problems\n    - `mid`: problems index 901-1000\n\n3. scripts for GSM8K and Game of 24 are similar. For simplicity, we only take GSM8K as the example below. You can simply run the same pipeline in Game of 24 by replacing `gsm8k` with `game24`\n\n\n\n## Training\n\n### Train the generator\n\nTraining data for generator:\n- GSM8K: `data/gsm8k/train.jsonl`, from [OpenAI GSM8K](https://github.com/openai/grade-school-math/blob/master/grade_school_math/data/train.jsonl)\n- Game of 24: `data/game24/train.jsonl`, the first 900 problems in `data/game24/24.csv` (from [ToT](https://github.com/princeton-nlp/tree-of-thought-llm/blob/master/src/tot/data/24/24.csv)) with enumerated solutions\n\nTo run the script `train_generator.sh` (under `scripts/gsm8k` or `scripts/game24`), you should first set `WANDB_API_KEY`, `WANDB_ENTITY`, `model_name_or_path`, `save_dir`. The generator is named by `save_generator_id`\n\n```bash\ncd OVM\nbash scripts/gsm8k/train_generator.sh\n```\n\n\n### Train the OVM\n\n#### Generation\n\nFirst use the generator `generator_id` to generate `n_solutions` for each question in the training set,\n```bash\ncd OVM\nbash scripts/gsm8k/generate.sh\n```\nYou should first config the path of your generator checkpoint `model_name_or_path`, and set `--target_set train`\n\nThe output will be saved to `data/gsm8k/model_generation/`\n\n\n#### Training\n\nTrain OVM using `train_verifier.sh`. First set `WANDB_API_KEY`, `WANDB_ENTITY`, `save_dir`, and `checkpoint_dir` (the path of generator checkpoint). The verifier is named with `save_verifier_id`\n```bash\ncd OVM\nbash scripts/gsm8k/train_verifier.sh\n```\n\n\n\n## Inference\n\n### Value-Guided Beam Search\n\nConfig your generator checkpoint path `model_name_or_path` and verifier checkpoint path `verifier_model_name_or_path` in `eval_step_beam.sh`\n```bash\ncd OVM\nbash scripts/gsm8k/eval_step_beam.sh\n```\n\n(when `dedup_mode=1`, it will prioritize linguistically different candidates, which means when the sorted candidates are ['a', 'a', 'b', 'b', 'c'] it will select ['a', 'b', 'c'] rather than ['a', 'a', 'b'] if n_beam=3)\n\nThe output will be saved to `eval_results/gsm8k/generator_with_verifier/test` \n(or `eval_results/game24/generator_with_verifier/mid`)\n\n\n### Vanilla Sampling with ORM\n\n1. First sample the data: config the generator checkpoint `model_name_or_path`, and set `--target_set test`\n    ```bash\n    cd OVM\n    bash scripts/gsm8k/generate.sh\n    ```\n\n2. Then call ORM to score and rerank the samples: config the verifier checkpoint `verifier_model_name_or_path`\n    ```bash\n    cd OVM\n    bash scripts/gsm8k/eval_with_verifier.sh\n    ```\n\nThe output will be saved to `eval_results/gsm8k/generator_with_verifier/test` \n\n\n\n### Greedy\n\nConfig your generator checkpoint path `model_name_or_path`\n```bash\ncd OVM\nbash scripts/gsm8k/greedy_eval.sh\n```\nThe output will be saved to `eval_results/gsm8k/generator/test` \n\n\n\n\n\n## Citation\n```\n@misc{yu2023outcomesupervised,\n      title={Outcome-supervised Verifiers for Planning in Mathematical Reasoning}, \n      author={Fei Yu and Anningzhe Gao and Benyou Wang},\n      year={2023},\n      eprint={2311.09724},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}\n```\n\n## Star History\n\n\u003ca href=\"https://star-history.com/#FreedomIntelligence/OVM\u0026Date\"\u003e\n  \u003cpicture\u003e\n    \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=FreedomIntelligence/OVM\u0026type=Date\u0026theme=dark\" /\u003e\n    \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=FreedomIntelligence/OVM\u0026type=Date\" /\u003e\n    \u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=FreedomIntelligence/OVM\u0026type=Date\" /\u003e\n  \u003c/picture\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fovm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fovm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fovm/lists"}