{"id":24854383,"url":"https://github.com/abhiram-kandiyana/error-explainer","last_synced_at":"2025-03-26T14:25:47.403Z","repository":{"id":259622024,"uuid":"868261206","full_name":"Abhiram-kandiyana/Error-Explainer","owner":"Abhiram-kandiyana","description":"Explanation of Programming Errors using Open-source LLMs","archived":false,"fork":false,"pushed_at":"2024-10-08T12:52:52.000Z","size":969,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-31T15:42:01.032Z","etag":null,"topics":["code-llama","few-shot-prompting","finetuning-llms","low-rank-adaptation","text-alignment"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Abhiram-kandiyana.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-05T22:35:01.000Z","updated_at":"2024-10-08T12:52:55.000Z","dependencies_parsed_at":"2024-10-26T23:44:36.588Z","dependency_job_id":"a6b6784b-ff6e-47f2-aac7-47386dc6283f","html_url":"https://github.com/Abhiram-kandiyana/Error-Explainer","commit_stats":null,"previous_names":["abhiram-kandiyana/error-explainer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Abhiram-kandiyana%2FError-Explainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Abhiram-kandiyana%2FError-Explainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Abhiram-kandiyana%2FError-Explainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Abhiram-kandiyana%2FError-Explainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Abhiram-kandiyana","download_url":"https://codeload.github.com/Abhiram-kandiyana/Error-Explainer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245669346,"owners_count":20653132,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-llama","few-shot-prompting","finetuning-llms","low-rank-adaptation","text-alignment"],"created_at":"2025-01-31T15:37:10.878Z","updated_at":"2025-03-26T14:25:47.377Z","avatar_url":"https://github.com/Abhiram-kandiyana.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Error-Explainer\nThe efficiency of debugging is critically impacted by the quality of error log messages. The compiler\nmessages are mostly vague and sometimes misleading which leaves the programmer clueless about the\nsource of error. Most beginners spend hours trying to find and fix a bug due to this issue. This project explores\nthe application of large language models (LLMs) for enhancing debugging productivity through clearer and\nmore actionable error explanations. Utilizing the pretrained Code-LLaMA model, known for its adeptness in\nprogramming and coding tasks, we investigate two primary approaches to improve explanation quality: finetuning and prompting strategies.\n\nKeywords: LLMs, few-shot prompting, Text Alignment, Code-Llama, Low Rank Adaptation (LoRA)\n\n\u003c!---\n## Why not ChatGPT?\nYes, you can simply use ChatGPT but:\n1. It is close-sourced and as an engineer it becomes hard to understand, improve or evaluate it's performance\n2. It is slow. GPT-4o  \n--\u003e\n\n# Overview\nOur study introduces a custom error and alignment dataset tailored to refine the baseline capabilities of the Code-LLaMA model, aiming to produce more relevant and accurate\nerror explanations. We assess the effectiveness of each strategy using Perplexity (PPL), BERT-score, and\ncomprehensive human evaluation metrics. These metrics evaluate the clarity, relevance, and actionability\nof the explanations, contributing to an empirical understanding of how LLMs can be optimized to support\nsoftware developers in debugging tasks. Our findings not only shed light on the comparative strengths and\nlimitations of fine-tuning versus prompting within the context of error explanation but also propose a\nframework using few prompt strategies for further enhancement of automated debugging assistance tools\nin software development environments.\n\n# Results\n\n## Test Results\n| Strategy                       | BERT-score | Perplexity (PPL) | Human-Eval |\n|---------------------------------|------------|------------------|------------|\n| Zero-shot   | 0.82       | 9.6              | 28.67      |\n| Fine-Tuning                     | 0.78       | 9.4              | 40.33      |\n| Random-Prompting 4-shot         | 0.88       | 4.9              | 32.67      |\n| \u003cmark\u003eSame-Class-Prompting 4-shot\u003c/mark\u003e | \u003cmark\u003e0.89\u003c/mark\u003e   | 4        | \u003cmark\u003e46.67\u003c/mark\u003e  |\n| Manual-Prompting 4-shot         | 0.39       | \u003cmark\u003e2.7\u003c/mark\u003e              | 18.33      |\n\n## Human Evaluation Results\n| Examiner  | Zero-shot | Finetuned | Random-Prompting 4-shot | Same-class-Prompting 4-shot | Manual-Prompting 4-shot | GPT-4 (baseline) |\n|-----------|----------------------|-----------|-------------|-------------|-------------|-------|\n| 1         | 21                   | 44        | 31          | 46          | 17          | 51    |\n| 2 | 28                   | 34        | 35          | 47          | 17          | 49    |\n| 3         | 37                   | 43        | 32          | 47          | 21          | 53    |\n| **Average** | 28.67                | 40.33     | 32.67       | \u003cmark\u003e46.67 \u003c/mark\u003e      | 18.33       | \u003cmark\u003e51\u003c/mark\u003e   |\n\n**NOTE**: To read more about the results, metrics and methodology, please go to \"./project report.pdf\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhiram-kandiyana%2Ferror-explainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhiram-kandiyana%2Ferror-explainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhiram-kandiyana%2Ferror-explainer/lists"}