{"id":26664011,"url":"https://github.com/social-ai-uoft/ad-paper","last_synced_at":"2026-05-01T01:31:52.761Z","repository":{"id":260516322,"uuid":"881494969","full_name":"social-ai-uoft/ad-paper","owner":"social-ai-uoft","description":" [NeurIPS 2024] Temporal-Difference Learning Using Distributed Error Signals","archived":false,"fork":false,"pushed_at":"2024-11-08T17:21:22.000Z","size":730,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T15:50:41.374Z","etag":null,"topics":["artificial-dopamine","machine-learning","neurips-2024","paper","reinforcement-learning","temporal-differencing-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2411.03604","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/social-ai-uoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-31T17:26:22.000Z","updated_at":"2024-12-13T01:23:18.000Z","dependencies_parsed_at":"2024-10-31T19:26:16.695Z","dependency_job_id":"09331545-d2bb-4e57-ad9b-db8b6a244af3","html_url":"https://github.com/social-ai-uoft/ad-paper","commit_stats":null,"previous_names":["social-ai-uoft/ad-paper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/social-ai-uoft/ad-paper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/social-ai-uoft%2Fad-paper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/social-ai-uoft%2Fad-paper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/social-ai-uoft%2Fad-paper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/social-ai-uoft%2Fad-paper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/social-ai-uoft","download_url":"https://codeload.github.com/social-ai-uoft/ad-paper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/social-ai-uoft%2Fad-paper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32482460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-dopamine","machine-learning","neurips-2024","paper","reinforcement-learning","temporal-differencing-learning"],"created_at":"2025-03-25T15:43:48.525Z","updated_at":"2026-05-01T01:31:52.742Z","avatar_url":"https://github.com/social-ai-uoft.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Temporal-Difference Learning Using Distributed Error Signals [[paper](https://arxiv.org/abs/2411.03604)]\n\nJonas Guan, Shon Eduard Verch, Claas Voelcker, Ethan C. Jackson, Nicolas Papernot, William A. Cunningham\n\n## Abstract\n\nA computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc) to update synaptic weights. Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.\n\n## Usage\n\n### Environment Setup\n\nInstall the Poetry package manager and then run `poetry install` in the root directory of this repository. By default, JAX and PyTorch will be installed with CPU only support. To install GPU support, run `poetry add jax --extras \"cuda11\"` manually. You'll need to change the CUDA version to match your system if you're using a different version.\n\nIf using Visual Studio Code, we recommend installing the Python extension and setting the Python interpreter to the one managed by Poetry (which can be found by running `poetry env info` in the terminal).\n\n### Running an Experiment\n\nWe provide a minimal training script in `scripts/train.py`. For example, to train an AD-DQN model on the `CartPole-v1 environment`, run:\n\n```bash\npoetry run python scripts/train.py CartPole-v1 ad_dqn\n```\n\nThis script supports training a model with the standard AD-DQN algorithm and AD-QRDQN, its quantile regression variant.\n\nWe use Weights and Biases (WandB) to log results, so you'll need to sign up for an account and run `poetry run wandb login` to authenticate. Once you've done that, you can view the results on the WandB dashboard by running `poetry run wandb ui` or by visiting the project page in your browser. Results are logged locally in the `runs` directory.\n\nSee the script for more details, including how to change hyperparameters.\n\n## Results\n\nWe empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation. The figures below show the results of our experiments on MinAtar tasks, the DeepMind Control Suite, and Gymnasium classic control tasks. See our [paper](https://arxiv.org/abs/2411.03604) for more details.\n\n### MinAtar \u0026 DeepMind Control Suite\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./figures/results_addqn_minatar_dmc.png\" width=\"80%\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./figures/results_adqrdqn_dmc.png\" width=\"80%\"\u003e\n\u003c/p\u003e\n\n### Classic Control Tasks\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./figures/results_addqn_classic_control.png\" width=\"80%\"\u003e\n\u003c/p\u003e\n\n\n## Citation\n\nIf you find our work useful or relevant to your research, please kindly cite our paper:\n\n```bibtex\n@article{guan2024temporal,\n    title={Temporal-Difference Learning Using Distributed Error Signals},\n    author={Guan, Jonas and Verch, Shon and Voelcker, Claas A and Jackson, Ethan C and Papernot, Nicolas and Cunningham, William},\n    journal={Advances in Neural Information Processing Systems},\n    volume={38},\n    year={2024},\n    url={https://arxiv.org/abs/2411.03604}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsocial-ai-uoft%2Fad-paper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsocial-ai-uoft%2Fad-paper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsocial-ai-uoft%2Fad-paper/lists"}