https://github.com/ise-uiuc/nnsmith

Automatic DNN generation for fuzzing and more
https://github.com/ise-uiuc/nnsmith

compiler deep-learning fuzzing machine-learning pytorch tensorflow

Last synced: about 22 hours ago
JSON representation

Automatic DNN generation for fuzzing and more

• Host: GitHub
• URL: https://github.com/ise-uiuc/nnsmith
• Owner: ise-uiuc
• License: apache-2.0
• Created: 2021-10-30T00:33:45.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2024-03-19T07:37:10.000Z (9 months ago)
• Last Synced: 2024-12-17T19:07:16.329Z (8 days ago)
• Topics: compiler, deep-learning, fuzzing, machine-learning, pytorch, tensorflow
• Language: Python
• Homepage: https://pypi.org/project/nnsmith/
• Size: 1 MB
• Stars: 120
• Watchers: 7
• Forks: 27
• Open Issues: 6
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        


    



# NNSmith

[![](https://github.com/ise-uiuc/nnsmith/actions/workflows/ci.yaml/badge.svg)](https://github.com/ise-uiuc/nnsmith/actions/workflows/ci.yaml)

[![](https://img.shields.io/pypi/v/nnsmith?color=g)](https://pypi.org/project/nnsmith/)

[![](https://static.pepy.tech/badge/nnsmith)](https://pepy.tech/project/nnsmith)

[![](https://img.shields.io/pypi/l/nnsmith)](https://github.com/ise-uiuc/nnsmith/blob/main/LICENSE)

🌟NNSmith🌟 is a random DNN generator and a fuzzing infrastructure, primarily designed for automatically validating deep-learning frameworks and compilers.

## Support Table



| Models | [`tvm`](https://github.com/apache/tvm) | [`pt2`](https://pytorch.org/get-started/pytorch-2.0/) | [`torchjit`](https://pytorch.org/docs/stable/jit.html) | [`tensorrt`](https://github.com/NVIDIA/TensorRT) | [`onnxruntime`](https://github.com/microsoft/onnxruntime) | [`xla`](https://www.tensorflow.org/xla) | [`tflite`](https://www.tensorflow.org/lite) |

| ------------ | ------------------------------------ | ----------------------------------------------- | ---------------------------------------------- | ----------------------------------------- | ------------------------------------- | ----------------------------------------------------- | ------------ |

| ONNX         | ✅                                    |                                                |                                               | ✅ | ✅ |                                                       |  |

| PyTorch |                                      | ✅📈 | ✅📈 |                                          |                                      |                                         |                                             |

| TensorFlow |                                      |                                                       |                                                        |                                           |                                       | ✅                                                    | ✅ |

✅: Supported; 📈: Supports gradient check;



## Quick Start

**Install latest code (GitHub HEAD):**

```shell

pip install pip --upgrade

pip install "nnsmith[torch,onnx] @ git+https://github.com/ise-uiuc/nnsmith@main" --upgrade

# [optional] add more front- and back-ends such as [tensorflow] and [tvm,onnxruntime,...] in "[...]"

```

Install latest stable release  [click]



```shell

pip install "nnsmith[torch,onnx]" --upgrade

```



Install latest pre-release  [click]



```shell

pip install "nnsmith[torch,onnx]" --upgrade --pre

```



Setting up graphviz for debugging [click]



Graphviz provides `dot` for visualizing graphs in nice pictures. But it needs to be installed via the following methods:

```shell

sudo apt-get install graphviz graphviz-dev      # Linux

brew install graphviz                           # MacOS

conda install --channel conda-forge pygraphviz  # Conda

choco install graphviz                          # Windows

pip install pygraphviz  # Final step.

```

Also see [pygraphviz install guidance](https://pygraphviz.github.io/documentation/stable/install.html).



```shell

# Generate a random model in "nnsmith_outputs/*"

nnsmith.model_gen model.type=onnx debug.viz=true

```

## Learning More

- 🐛 [**Uncovered bugs**](doc/bugs.md).

- 📚 [**Documentation**](doc/): [CLI](doc/cli.md), [concept](doc/concept.md), [logging](doc/log-and-err.md), and [known issues](doc/known-issues.md).

- 🤗 [**Contributing to NNSmith**](doc/CONTRIBUTING.md)

- 📝 We use [hydra](https://hydra.cc/) to manage configurations. See `nnsmith/config/main.yaml`.

## Papers

 📜 NeuRI: Diversifying DNN Generation via Inductive Rule Inference  [click :: citation]



```bibtex

@article{liu2023neuri,

  title = {NeuRI: Diversifying DNN Generation via Inductive Rule Inference},

  author = {Liu, Jiawei and Peng, Jinjun and Wang, Yuyao and Zhang, Lingming},

  journal = {arXiv preprint arXiv:2302.02261},

  year = {2023},

}

```





    

    

    

    



 📜 NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers  [click :: citation]



```bibtex

@inproceedings{liu2023nnsmith,

  title={Nnsmith: Generating diverse and valid test cases for deep learning compilers},

  author={Liu, Jiawei and Lin, Jinkun and Ruffy, Fabian and Tan, Cheng and Li, Jinyang and Panda, Aurojit and Zhang, Lingming},

  booktitle={Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2},

  pages={530--543},

  year={2023}

}

```