{"id":51406452,"url":"https://github.com/declare-lab/grail","last_synced_at":"2026-07-04T11:30:27.528Z","repository":{"id":362423761,"uuid":"1258173937","full_name":"declare-lab/grail","owner":"declare-lab","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-04T05:28:56.000Z","size":636,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-04T07:16:15.733Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/declare-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-03T10:37:33.000Z","updated_at":"2026-06-04T05:29:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/declare-lab/grail","commit_stats":null,"previous_names":["declare-lab/grail"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/declare-lab/grail","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fgrail","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fgrail/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fgrail/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fgrail/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/declare-lab","download_url":"https://codeload.github.com/declare-lab/grail/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fgrail/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35120680,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-04T11:30:26.821Z","updated_at":"2026-07-04T11:30:27.512Z","avatar_url":"https://github.com/declare-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards\n\nOfficial codebase for the paper [**\"GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards\"**](http://arxiv.org/abs/2606.04889), authored by *Tej Deep Pala, Vernon Toh, and Soujanya Poria* at the **DeCLaRe Lab, Nanyang Technological University**.\n\n## 📌 Abstract\n\nReinforcement learning with verifiable rewards (e.g., GRPO) is a standard paradigm to enhance mathematical and logical reasoning in Large Language Models (LLMs). However, standard GRPO broadcasts a uniform sequence-level advantage scalar to all tokens in a rollout. This uniform credit assignment dilutes the gradient signal: flawed reasoning steps, intermediate derivation errors, and filler words receive the same advantage signal as pivotal, logically sound steps.\n\nTo solve this, we introduce **Gradient-Reweighted Advantage (GRAIL)**, an intrinsic, token-wise advantage reweighting method. GRAIL uses **gradient-activation saliency** to assign larger advantage signal to tokens that are locally sensitive to the final answer.\n\nAcross five model architectures (Qwen3-4B/8B, OctoThinker-3B/8B-Short, R1-Distill-Llama-8B) evaluated on six mathematical reasoning benchmarks (MATH500, AIME24, AMC23, MinervaMATH, CollegeMATH, OlympiadBench), **GRAIL consistently outperforms standard GRPO**, yielding an average absolute accuracy boost of **3.60%** and Pass@3 improvement of **3.05%** without requiring any external process-level supervision (PRMs).\n\n## 📁 Repository Structure\n\n```\ngrail/\n├── requirements.txt            # System dependencies\n├── README.md                   # Project documentation\n├── src/\n│   ├── train/\n│   │   ├── grpo.py             # Main trainer launcher\n│   │   ├── grpo.sh             # Training execution bash script\n│   │   ├── grail_trainer.py    # Customized GrailTrainer implementing loss logic\n│   │   ├── rewards.py          # Math outcome-based reward function\n│   │   ├── training_configs/   # Directory for YAML configuration files\n│   │   │   ├── deepspeed_zero2.yaml\n│   │   │   ├── qwen3_grpo.yaml\n│   │   │   ├── qwen3_grail.yaml\n│   │   │   └── qwen3_oar_g.yaml\n│   │   └── utils/              # Arguments, logs, and datasets utilities\n│   └── eval/\n│       ├── run_eval.py          # Main benchmarking execution script\n│       ├── run_eval.sh          # Serving \u0026 evaluation pipeline orchestration\n│       ├── start_vllm.sh        # Serving policy models via vLLM\n│       ├── eval_results_base.py # Combines results across datasets\n│       ├── compute_checkpoint_stats.py  # Standalone token-level saliency stats\n│       ├── token_analysis.sh   # Visualizes checkpoint saliency dynamics\n│       ├── aggregate_and_plot.py  # Plotting script for positional analysis\n│       ├── convert_to_excel.py # Generates unified XLSX result reports\n│       ├── compile_res.sh      # Aggregates evaluation results\n│       └── eval_data/          # Benchmark dataset JSONL files (AIME24, MATH, etc.)\n```\n\n---\n\n## ⚙️ Installation \u0026 Setup\n\nSet up a Python environment (Conda recommended) and install dependencies:\n\n```bash\nconda create -n grail python=3.11 -y\nconda activate grail\npip install -r requirements.txt\npip install flash-attn --no-build-isolation\n```\n\n## 🚀 Training Instructions\n\nTrain model configurations using DeepSpeed ZeRO-2 and the accelerate launcher.\n\n### Running GRAIL Training:\nModify variables (e.g. `WANDB_API_KEY`, `HF_TOKEN`) inside [grpo.sh](./src/train/grpo.sh) and launch:\n\n```bash\ncd src/train\nbash grpo.sh\n```\n\nBy default, training leverages the config [qwen3_grail.yaml](./src/train/training_configs/qwen3_grail.yaml) which includes the following key parameters:\n\n```yaml\n# GRAIL Parameters\nuse_grail: true\ngrail_std: 0.5                  # Spread scaling factor (\\sigma_w)\ngrail_mean: 1.0                 # Baseline neutral weight (w_mean)\ngrail_w_min: 0.5                # Minimum weight bounds (w_min)\ngrail_w_max: 5.0                # Maximum weight bounds (w_max)\ngrail_leaf_source: \"embeddings\" # Saliency leaf source\ngrail_rollout_symmetry: \"wrong\" # Apply reweighting on: \"all\", \"correct\", or \"wrong\"\n```\n\n## 📊 Serving \u0026 Benchmark Evaluation\n\nEvaluations are computed over 6 reasoning suites (`aime24`, `math`, `college_math`, `minerva_math`, `olympiadbench`, `amc23`) using vLLM for high-throughput serving.\n\n### 1. Launch the vLLM Server\nSetup your model path inside [start_vllm.sh](./src/eval/start_vllm.sh) and run:\n```bash\ncd src/eval\nbash start_vllm.sh\n```\n\n### 2. Run Benchmarks\nRun evaluations by targeting the served port (configured in [run_eval.sh](./src/eval/run_eval.sh)):\n```bash\ncd src/eval\nbash run_eval.sh\n```\n\n### 3. Generate Reports\nCompile JSONL evaluation results into a single Excel sheet using [compile_res.sh](./src/eval/compile_res.sh):\n```bash\nbash compile_res.sh\n```\n\n## 📈 Saliency Diagnostics \u0026 Token Analysis\n\nTo evaluate how token-level saliency weights change over training checkpoints, execute the token analysis suite:\n\n```bash\ncd src/eval\nbash token_analysis.sh\n```\n\nThis pipeline:\n1. Runs [compute_checkpoint_stats.py](./src/eval/compute_checkpoint_stats.py) to extract token gradients and weights post-hoc.\n2. Runs [aggregate_and_plot.py](./src/eval/aggregate_and_plot.py) to generate plots showing the U-shape distribution of weights across normalized reasoning spans.\n\n```bibtex\n@misc{pala2026grail,\n      title={GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards},\n      author={Tej Deep Pala and Vernon Toh and Soujanya Poria},\n      year={2026},\n      eprint={2606.04889},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2606.04889},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fgrail","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeclare-lab%2Fgrail","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fgrail/lists"}