{"id":17990895,"url":"https://github.com/wellecks/symbolic_generalization","last_synced_at":"2025-03-25T23:31:45.306Z","repository":{"id":46044803,"uuid":"411386727","full_name":"wellecks/symbolic_generalization","owner":"wellecks","description":"Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics (AAAI 2022)","archived":false,"fork":false,"pushed_at":"2022-02-24T21:27:34.000Z","size":18,"stargazers_count":14,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-20T22:35:52.996Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wellecks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-28T17:56:59.000Z","updated_at":"2024-06-02T14:39:01.000Z","dependencies_parsed_at":"2022-09-14T13:10:30.779Z","dependency_job_id":null,"html_url":"https://github.com/wellecks/symbolic_generalization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellecks%2Fsymbolic_generalization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellecks%2Fsymbolic_generalization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellecks%2Fsymbolic_generalization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellecks%2Fsymbolic_generalization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wellecks","download_url":"https://codeload.github.com/wellecks/symbolic_generalization/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245561787,"owners_count":20635818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T19:19:37.391Z","updated_at":"2025-03-25T23:31:40.274Z","avatar_url":"https://github.com/wellecks.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics\n\n[Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics](https://arxiv.org/pdf/2109.13986.pdf)\\\nSean Welleck, Peter West, Jize Cao, Yejin Choi\\\nAAAI 2022\n\nThis repo contains code for **automated** (SAGGA) and **rule-based** (robustness, compositionality) failure discovery and verification.\n\nPlease cite our work if you found the resources in this repository useful:\n```\n@inproceedings{welleck2022symbolic,\n    title={Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics},\n    author={Sean Welleck and Peter West and Jize Cao and Yejin Choi},\n    booktitle={AAAI},\n    year={2022},\n    url={https://arxiv.org/pdf/2109.13986.pdf}\n}\n```\n\n## Setup\n\n#### Library code\n\nWe use utilities from the implementation of  [[Lample \\\u0026 Charton](https://arxiv.org/abs/1912.01412)].\n1. Clone their [[repo](https://github.com/facebookresearch/SymbolicMathematics/tree/4596d070e1a9a1c2239c923d7d68fda577c8c007)]. The repo filepath is then provided as a `--symbolic-math-repo-path` command line argument in the scripts below.\n\nThen setup our code by running:\n```bash\npython setup.py develop\n```\n\n#### Data and models\n\nWe use pretrained models and data from [[Lample \\\u0026 Charton](https://arxiv.org/abs/1912.01412)]. \\\nFrom their [[repo](https://github.com/facebookresearch/SymbolicMathematics/tree/4596d070e1a9a1c2239c923d7d68fda577c8c007)]:\n1. Download and unpack the `FWD + BWD + IBP` model [[link](https://dl.fbaipublicfiles.com/SymbolicMathematics/models/fwd_bwd_ibp.pth)].\\\nThe model path is provided as a `--model-path` command line argument in the scripts below.\n\n\n\nFor experiments involving validation primitives:\n\n2. Download the Integration `FWD`, `BWD`, `IBP` datasets. \n\n## SAGGA\n#### Robustness\n```bash\npython admath/genetic.py --basis polynomial_robustness --mutation-params robustness\n\npython admath/genetic.py --basis trig_robustness --mutation-params robustness \n```\n\n#### Robustness - explicit distance\n```bash\npython admath/genetic.py --fitness-type target_problems --basis target --mutation-params distance\n```\n\n\n#### General / exploits\n```bash\npython admath/genetic.py --basis polynomial --fitness-type length_penalty\n\npython admath/genetic.py --basis trig --ops trig --fitness-type trig_length_penalty\n```\n\n\n#### Target length\n```bash\npython admath/genetic.py --fitness-type target_length --target-length 10 --tau 0.05 --seed-size 50 --generation-size 300 --archive-size-terminate 5000\n\npython admath/genetic.py --fitness-type target_length --target-length 20 --tau 0.05 --seed-size 50 --generation-size 300 --archive-size-terminate 5000\n\npython admath/genetic.py --fitness-type target_length --target-length 40 --tau 0.05 --seed-size 50 --generation-size 300 --archive-size-terminate 5000\n```\n\n\n## Simple Primitives\n\n#### Robustness\nGenerate\n```bash\npython admath/robustness.py --output-dir ./output\n\n==\u003e Wrote to ./output/simple_robustness_all.json\n```\n\nCheck and verify\n```bash\npython admath/verify.py \\\n--generations ./output/simple_robustness_all_generations.json \\ \n--method sympy \\\n--early-stop \\    \n--output-dir ./output\n\n==\u003e Wrote to ./output/simple_robustness_all_verified_sympy_sagemath.json\n```\n\n#### Compositionality\nCompositionality uses verified functions from the Robustness experiment via the `--generations-file` parameter.\n```bash\npython admath/compositionality.py \\\n--generations-file ./output/simple_robustness_all_verified_sympy.json\n\n==\u003e Wrote to ./output/simple_comp_all.json\n```\n\nThen do the \"check and verify\" step above.\n\n\n## Validation Primitives - Robustness and Compositionality\n\nSee `notebooks/`.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwellecks%2Fsymbolic_generalization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwellecks%2Fsymbolic_generalization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwellecks%2Fsymbolic_generalization/lists"}