https://github.com/mrigankpawagi/guardrails
VSCode extension to help developers set up guardrails around their functions, by helping them disambiguate purpose statements.
https://github.com/mrigankpawagi/guardrails
llms python specifications vscode-extension
Last synced: about 1 year ago
JSON representation
VSCode extension to help developers set up guardrails around their functions, by helping them disambiguate purpose statements.
- Host: GitHub
- URL: https://github.com/mrigankpawagi/guardrails
- Owner: mrigankpawagi
- License: mit
- Created: 2023-07-07T10:56:03.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2025-01-11T17:49:19.000Z (over 1 year ago)
- Last Synced: 2025-04-12T01:14:00.323Z (about 1 year ago)
- Topics: llms, python, specifications, vscode-extension
- Language: Python
- Homepage: https://marketplace.visualstudio.com/items?itemName=MrigankPawagi.guardrails
- Size: 11.2 MB
- Stars: 5
- Watchers: 1
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
# GuardRails: Automated Suggestions for Clarifying Ambiguous Purpose Statements
VSCode extension to help developers write complete and unambiguous docstrings for their Python functions. Install it now from the [Visual Studio Code Marketplace](https://marketplace.visualstudio.com/items?itemName=MrigankPawagi.guardrails).
> [!IMPORTANT]
> Our paper was presented at the 6th Annual COMPUTE Conference by ACM India. Check it out [here](https://dl.acm.org/doi/10.1145/3627217.3627234) (or [here](https://arxiv.org/abs/2312.08189) on arXiv).
> [!WARNING]
> Due to recent updates in the Copilot's VSCode extension, functionality critical for GuardRails is no longer supported. Please hang on while we update GuardRails. If you are interested in contributing, check out [this](https://github.com/mrigankpawagi/GuardRails/pull/6).
## Requirements
You need Python 3 installed on your system to use GuardRails.
__Note:__
GuardRails is fully tested only on Windows and currently provides experimental support for Linux.
### Installing Python Dependencies
```bash
$ python -m pip install hypothesis hypothesis[cli] hypothesis[ghostwriter] black
$ python -m pip install git+https://github.com/mrigankpawagi/mutpy.git#egg=mutpy
$ python -m pip install func-timeout
```
## A Motivating Example
```python
def first_nonzero(nums: list[float]) -> float:
""" Return the first non-zero value in nums.
>>> first_nonzero([0.0 , 3.7 , 0.0])
3.7
"""
```
This is an example of an incomplete function definition in Python. It has a purpose statement (called the _docstring_) which contains a functional example (called a _doctest_). The purpose statement is ambiguous because it does not specify the intended behavior on lists with no non-zero elements.
## Contributions
### Heuristic
We propose a novel heuristic that suggests such inputs using Large Language Models (LLMs). Using these suggestions, the programmer may choose to clarify the purpose statement (e.g., by providing a _functional example_ that specifies the intended behavior on such an input). Check out our paper for more details on our heuristic.
### Open Dataset
To assess the quality of inputs suggested by our heuristic, and to facilitate future research, we create an open dataset of purpose statements with known ambiguities. For each function, we provide four variants of prompts.
- **S** (Signature)
- **SP** (**S** + Purpose statement)
- **SP1** (**SP** + 1 functional example)
- **SPx** (**SP** + More functional examples)
Our dataset is publicly available [here](https://docs.google.com/spreadsheets/d/e/2PACX-1vRwmXlP8V6gbXtB1oQ5IUXfbRjW3eoCYKcbm-zN4uXphd_AK4Wj0CZzVmeXW4XvF2_scszdCD89CFpV/pubhtml?gid=0&single=true). We use this dataset to compare GuardRails with test cases generated by GitHub Copilot Chat. More details on this are available in our paper.