{"id":28381123,"url":"https://github.com/humancompatibleai/evaluating-rewards","last_synced_at":"2025-08-07T18:16:03.012Z","repository":{"id":35447510,"uuid":"205413758","full_name":"HumanCompatibleAI/evaluating-rewards","owner":"HumanCompatibleAI","description":"Library to compare and evaluate reward functions","archived":false,"fork":false,"pushed_at":"2023-10-23T06:44:52.000Z","size":611,"stargazers_count":67,"open_issues_count":6,"forks_count":8,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-06-25T01:44:02.490Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2006.13900","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HumanCompatibleAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-08-30T15:56:26.000Z","updated_at":"2025-05-12T18:44:21.000Z","dependencies_parsed_at":"2025-06-25T01:38:46.711Z","dependency_job_id":"8b706143-5127-46af-9d76-3787a4c261a1","html_url":"https://github.com/HumanCompatibleAI/evaluating-rewards","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HumanCompatibleAI/evaluating-rewards","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanCompatibleAI%2Fevaluating-rewards","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanCompatibleAI%2Fevaluating-rewards/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanCompatibleAI%2Fevaluating-rewards/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanCompatibleAI%2Fevaluating-rewards/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HumanCompatibleAI","download_url":"https://codeload.github.com/HumanCompatibleAI/evaluating-rewards/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanCompatibleAI%2Fevaluating-rewards/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266775383,"owners_count":23982273,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-30T03:38:30.352Z","updated_at":"2025-07-24T01:05:35.344Z","avatar_url":"https://github.com/HumanCompatibleAI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/HumanCompatibleAI/evaluating-rewards.svg?style=svg)](https://circleci.com/gh/HumanCompatibleAI/evaluating_rewards)\n[![codecov](https://codecov.io/gh/HumanCompatibleAI/evaluating-rewards/branch/master/graph/badge.svg)](https://codecov.io/gh/HumanCompatibleAI/evaluating_rewards)\n\n# Evaluating Rewards\n\n`evaluating_rewards` is a library to compare and evaluate reward functions. The accompanying paper,\n[Quantifying Differences in Reward Functions](https://arxiv.org/abs/2006.13900), describes the\nmethods implemented in this repository.\n\n## Getting Started\n\n### Installation\n\nTo install `evaluating_rewards`, clone the repository and run:\n\n```\npip install evaluating_rewards/\n```\n\nTo install in developer mode so that edits will be immediately available:\n\n```\npip install -e evaluating_rewards/\n```\n\nThe package is compatible with Python 3.6 and upwards. There is no support for\nPython 2.\n\n### Computing EPIC Distance\n\nThe `evaluating_rewards.analysis.dissimilarity_heatmaps.plot_epic_heatmap` script provides\na convenient front-end to generate heatmaps of EPIC distance between reward models.\nFor example, to replicate Figure 2(a) from the paper, simply run:\n\n```bash\npython -m evaluating_rewards.analysis.dissimilarity_heatmaps.plot_epic_heatmap with point_mass paper\n# If you want higher precision results like the paper (warning: will take several hours), use:\n# python -m evaluating_rewards.analysis.dissimilarity_heatmaps.plot_epic_heatmap with point_mass paper high_precision \n```\n\n`plot_epic_heatmap` uses the `evaluating_rewards.epic_sample` module to compute the EPIC distance.\nYou may wish to use this module directly, for example when integrating EPIC into an existing\nevaluation suite. Check out this [notebook](examples/epic_demo.ipynb) for an example of how to use\n`epic_sample`.\n\n## Technical Structure\n\n`evaluating_rewards` consists of:\n\n-   the main package, containing some reward learning algorithms and distance methods. In particular:\n    + `epic_sample.py` contains the implementation of EPIC.\n    + `rewards.py` and `comparisons.py` define deep reward models and\n        associated comparison methods, and `tabular.py` the equivalent in a\n        tabular (finite-state) setting.\n    + `serialize.py` loads reward models, both from this package and other projects.\n-   `envs`, a sub-package defining some simple environments and associated\n    hard-coded reward models and (in some cases) policies.\n-   `experiments`, a sub-package defining helper methods supporting experiments\n    we have performed.\n\n## Obtaining Reward Models\n\nTo use `evaluating_rewards`, you will need to have some reward models to\ncompare. We provide native support to load reward models output by\n[imitation](https://github.com/humancompatibleai/imitation), an open-source\nimplementation of AIRL and GAIL. It is also simple to add new formats to\n`serialize.py`.\n\n## Reproducing the Paper\n\nNote in addition to the Python requirements, you will need to install\n[GNU Parallel](https://www.gnu.org/software/parallel/) to run these bash scripts, for example with\n`sudo apt-get parallel`.\n\nBy default results will be saved to `$HOME/output`; set the `EVAL_OUTPUT_ROOT` environment variable\nto override this.\n\n### Distance Between Hand-Designed Reward Functions\n\nTo replicate the results in section 6.1 \"Comparing hand-designed reward functions\", run:\n\n```bash\n./runners/comparison/hardcoded_npec.sh\n./runners/comparison/hardcoded_figs.sh\n```\n\nThe first script, `hardcoded_npec.sh`, computes the NPEC distance between the hand-designed\nreward functions. This is relatively slow since it requires training a deep network.\n\nThe second script, `hardcoded_figs`, plots the EPIC, NPEC and ERC heatmaps. For NPEC, the\ndistances precomputed by the first script are used. For EPIC and ERC they are computed on\ndemand since the distances are relatively cheap to compute.\n\nThe distances in gridworlds are produced by a separate script, exploiting the tabular nature of\nthe problem. Run:\n\n```bash\n# NPEC\npython -m evaluating_rewards.analysis.dissimilarity_heatmaps.plot_gridworld_heatmap with paper\n# EPIC\npython -m evaluating_rewards.analysis.dissimilarity_heatmaps.plot_gridworld_heatmap with paper kind=fully_connected_random_canonical_direct\n```\n\n### Distance of Learned Reward Model to Ground Truth\n\nTo replicate the results in section 6.2 \"Predicting policy performance from reward distance\" and\nsection 6.3 \"Sensitivity of reward distance to visitation state distribution\", you should run:\n\n```bash\n./runners/transfer_point_maze.sh\npython -m evaluating_rewards.scripts.pipeline.combined_distances with point_maze_learned_good high_precision\n```\n\nThe first script, `transfer_point_maze.sh`, 0) trains a synthetic expert policy using RL on the\nground-truth reward; 1) trains reward models via IRL on demonstrations from this expert, and\nvia preference comparison and regression with labels from the ground-truth reward; 2a) computes\nNPEC distance of these rewards from the ground-truth; 2b) trains policies using RL on the learned\nreward models; 3) evaluates the resulting policies.\n\nThe second script, `combined_distances`, generates a table and also computes the EPIC and ERC distances\nof the learned reward models from the ground truth.\n\n## License\n\nThis library is licensed under the terms of the Apache license. See\n[LICENSE](LICENSE) for more information.\n\nDeepMind holds the copyright to all work prior to October 4th, during the\nlead author's (Adam Gleave) internship at DeepMind. Subsequent modifications\nconducted at UC Berkeley are copyright of the author.\nDisclaimer: This is not an officially supported Google or DeepMind product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumancompatibleai%2Fevaluating-rewards","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhumancompatibleai%2Fevaluating-rewards","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumancompatibleai%2Fevaluating-rewards/lists"}