{"id":33123217,"url":"https://github.com/cedricrupb/nbfbaselines","last_synced_at":"2025-11-19T23:02:20.021Z","repository":{"id":91841046,"uuid":"508287595","full_name":"cedricrupb/nbfbaselines","owner":"cedricrupb","description":"Neural baselines for finding and fixing single token bugs in Python","archived":false,"fork":false,"pushed_at":"2023-11-23T09:42:48.000Z","size":374,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-08T07:32:14.884Z","etag":null,"topics":["bugs","neural-networks","repair"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cedricrupb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-06-28T12:21:39.000Z","updated_at":"2023-11-16T03:31:53.000Z","dependencies_parsed_at":"2023-04-01T06:06:12.783Z","dependency_job_id":"3644ada1-1c8b-4259-8670-bb0938edf328","html_url":"https://github.com/cedricrupb/nbfbaselines","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/cedricrupb/nbfbaselines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedricrupb%2Fnbfbaselines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedricrupb%2Fnbfbaselines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedricrupb%2Fnbfbaselines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedricrupb%2Fnbfbaselines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cedricrupb","download_url":"https://codeload.github.com/cedricrupb/nbfbaselines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cedricrupb%2Fnbfbaselines/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285342137,"owners_count":27155385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-19T02:00:05.673Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugs","neural-networks","repair"],"created_at":"2025-11-15T05:00:43.636Z","updated_at":"2025-11-19T23:02:20.011Z","avatar_url":"https://github.com/cedricrupb.png","language":"Python","funding_links":[],"categories":["Data Sets and Benchmarks"],"sub_categories":["Bug Fix"],"readme":"# NBFBaselines\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cedricrupb/nbfbaselines/blob/main/demo.ipynb) \n[[**PAPER**](https://github.com/cedricrupb/nbfbaselines/blob/main/paper/ase23-preprint.pdf) | [**ARTIFACT**](https://zenodo.org/records/7900059)]\n\u003e Neural baselines for finding and fixing single token bugs in Python\n\nSoftware bugs can interrupt our workflow and finding them can often be very time-consuming. Especially smaller bugs are often hard to spot since they affect only a few lines of code out of thousands. For example, recognizing that a variable was not renamed after refactoring or that a loop will run in an out-of-bounds error because of the comparison operator is often difficult.\n\nNBFBaselines collects baseline algorithm for learning to localize and repair software bugs. These algorihtms are designed to assist a developer in finding and fixing simple software bugs. They learn from millionth of bugs and their fixes to detect and repair new bugs in unseen code. Currently, bug finders often support a wide range of simple bugs such as:\n* **Variable Misuses:** The developer uses a variable although another was meant\n```python\ndef compare(x1, x2):\n    s1 = str(x1)\n    s2 = str(x1) # Bug: x2 instead of x1\n    return s1 == s2\n```\n* **Binary Operator Bugs:** The wrong binary operator was used\n```python\ndef add_one(L):\n    i = 0\n    while i \u003c= len(L): \n        L[i] = L[i] + 1\n        i += 1\n```\n* **Unary Operator Bugs:** A unary operator was used mistakenly or was forgotten\n```python\nif namespace: # Bug: not namespace instead of namespace\n    self.namespacesFilter = [ \"prymatex\", \"user\" ] \nelse:\n    self.namespacesFilter = namespace.split()\n```\n* **Wrong Literal Bugs:** The wrong literal was used\n```python\ndef add_one(L):\n    i = 0\n    while i \u003c len(L): \n        L[i] = L[i] + 1\n        i += 2 # Bug: 1 instead of 2\n```\n\n# RealiT: Training on mutants and real bugs\nRealiT is a neural bug localization and repair model trained on mutants and real bug fixes obtained from open source Python projects. Due to this combined training RealiT achieves a high performance in the localization and repair of real bugs. RealiT currently support the localization and repair of the four types of single statement bugs discussed before. You can try RealiT yourself by following our quick start guide.\n\n## Quick start\nYou can try `RealiT` with only a few lines of code:\n```python\nfrom nbfbaselines import NBFModel\n\nrealit_model = NBFModel.from_pretrained(\"realit\")\n\nrealit_model(\"\"\"\n\ndef f(x, y):\n    return x + x\n\n\"\"\")\n\n# Output:\n# [{'text': 'def f ( x , y ) :\\n    return x + y\\n    ',\n#  'before': 'x', 'after': 'y',\n#  'prob': 0.981337789216316}]\n\n```\n`RealiT` is mostly trained on function implementation written in Python 3. Therefore, it will work best if provided with a complete function implementation. It will likely also work on partial code snippets (e.g. partial function implementation or statement outside of a function).\n\nYou can also test `RealiT` in your browser without cloning this project. For this, open the following [Colab link](https://colab.research.google.com/github/cedricrupb/nbfbaselines/blob/main/demo.ipynb).\n\n\n## Models\n\nWe trained and publish several baseline models for neural bug detection. All models were evaluated on [PyPIBugs](https://github.com/microsoft/neurips21-self-supervised-bug-detection-and-repair) and on the test portion of [ETH Py150k](https://www.sri.inf.ethz.ch/py150). We used PyPIBugs to estimate the localization (Loc), repair (Rep) and Joint localization and repair performance (Joint). ETH Py150k was used to estimate the false positive rate (FPR).\n\n**RealiT models:**\n| model_id | FPR | Joint | Loc | Rep | description |\n|----------|-----|-------|-----|-----|-------------|\n| `realit5x-noreal` | 25.2 | 21.4 | 25.8 | 59.9 | RealiT without fine-tuning on real bugs and 5x mutants injected during pre-training |\n| `realit-noreal` | 30.0 | 24.9 | 30.4 | 65.6 | RealiT without fine-tuning on real bugs |\n| `realit` | 22.0 | 36.7 | 41.9 | 73.5 | Base RealiT model (Transformer) trained on 100x mutants and fine-tuned on real bugs. |\n| `csnrealit` | 16.5 | 38.7 | 42.7 | 76.7 | RealiT further pre-trained on the Python corpus of CodeSearchNet. |\n\nYou can try these models easily yourself:\n```python\nfrom nbfbaselines import NBFModel\n\nnbf_model = NBFModel.from_pretrained([model_id])\n\n```\n\n## Evaluation\nThe models were evaluated with the following Python script:\n```bash\n$ python run_eval.py [model_id] [test_file] [output_file] [options...]\n```\nFor `model_id`, you can use any of the model ids specified before.\nThe format of the test file is described below. The script accepts further options:\n| option | description | default |\n|--------|-------------|---------|\n| `--topk` | Activates beam search over the topk most likely repairs | 1 |\n| `--cpu` | The script tries to run the code on an Nvidia GPU if availale. This forces the script to run on CPU. | False |\n| `--max_length` | The maximal number of tokens to be accepted by the model. If an example has more token, we assume that no bug is detected. Tune this if the model exceeds the available memory.| 5000 |\n| `--ignore_error` | The script stops if the evaluation runs into an error (e.g. because the code contains a syntax error). This ignores the error and continues the evaluation. | False |\n\n**Test file format:**\nAs a test\nfile, we expect a file in json lines format where each JSON object has the following\nformat:\n```json\n{\n    \"input_text\": [buggy_code_or_correct_code],\n    \"error_marker\": [sl, sc, el, ec],\n    \"repair\": [repair_text]\n}\n```\nNote that `input_text` is required. The fields `error_marker` and `repair` are required if the code contains a bug. `error_marker` specifies the start line (sl), start char in that line (sc), end line (el) and the end char in that line (ec). The `repair` can be any string. The JSON object can contain further meta data. An example for a test_file is given in `data/real_bugs_dummy.jsonl`.\n\n## Status of Graph Models\nThe graph models will be published soon. Until then, please refer to our [artifact](https://zenodo.org/records/7900059) containing the complete implementation of the graph models.\n\n## Project info\n\nCedric Richter - [@cedrichter](https://twitter.com/cedrichter) - cedric.richter@uol.de\n\nDistributed under the MIT license. See `LICENSE` for more information.\n\nIf you find this project or our artifact useful, please cite our work: \n```bibtex\n@inproceedings{richter2023how,\n  title={How to Train Your Neural Bug Detector: Artificial vs Real Bugs},\n  author={Cedric Richter, Heike Wehrheim},\n  booktitle={ASE},\n  year={2023}\n}\n```\n\nThis project is currently incomplete and mainly designed to showcase `RealiT`. However, we plan to release the full code base including further pre-trained models, training script and evaluation code.\n\nFor more information on `RealiT` specifically, please check our [preprint](https://arxiv.org/abs/2207.00301).\n\nFeel free to open an issue if anything unexpected happens.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcedricrupb%2Fnbfbaselines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcedricrupb%2Fnbfbaselines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcedricrupb%2Fnbfbaselines/lists"}