https://github.com/ur-whitelab/nlcc-data

Benchmark dataset and evaluation for large language models that generate code
https://github.com/ur-whitelab/nlcc-data

Last synced: about 1 month ago
JSON representation

Benchmark dataset and evaluation for large language models that generate code

• Host: GitHub
• URL: https://github.com/ur-whitelab/nlcc-data
• Owner: ur-whitelab
• Created: 2021-08-22T05:18:47.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2022-12-07T05:04:26.000Z (about 2 years ago)
• Last Synced: 2024-04-15T15:11:14.188Z (9 months ago)
• Language: Jupyter Notebook
• Homepage: https://ur-whitelab.github.io/nlcc-data/
• Size: 71.1 MB
• Stars: 7
• Watchers: 5
• Forks: 1
• Open Issues: 40
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# nlcc-data

## Contribution instructions

- First, make sure you're in the main branch with `git checkout main`, and update with `git pull origin`

- Then, make a branch with `git checkout -b `

- After preparing your prompt/test following the example in smoke, push with `git push origin `

- Go to the repo URL and open a PR from pull requests tab

- You will see a report with the result (did it pass), responses, and prompts after combining with context

## Categories of Prompts

- Topic: cheminf, spectroscopy, qm, md, bio, genchem, numerics, other, stats, thermo, vis

- Eval: code, human

- Special: disabled

### Special variables

Files can be accessed by the test scripts relative to the example `yml` file directory using the `_FILE_DIR_` variable. For example (from `dipole`):

```

import numpy as np

import os

coordinates = np.loadtxt(os.path.join(_FILE_DIR_,"water.xyz"),usecols=(1,2,3))

```

### Papermill

```sh

papermill -f output.yml human_eval_template.ipynb test.ipynb

jupyter-nbconvert --to=html --no-input test.ipynb

```