https://github.com/carpedm20/ENAS-pytorch

PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing"
https://github.com/carpedm20/ENAS-pytorch

google-brain neural-architecture-search pytorch

Last synced: 2 months ago
JSON representation

PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing"

• Host: GitHub
• URL: https://github.com/carpedm20/ENAS-pytorch
• Owner: carpedm20
• License: apache-2.0
• Created: 2018-02-15T04:54:37.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2023-07-06T21:33:33.000Z (over 1 year ago)
• Last Synced: 2024-11-05T03:42:31.457Z (2 months ago)
• Topics: google-brain, neural-architecture-search, pytorch
• Language: Python
• Homepage:
• Size: 12.9 MB
• Stars: 2,703
• Watchers: 108
• Forks: 494
• Open Issues: 40
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-production-machine-learning - ENAS-PyTorch - pytorch.svg?style=social) - Efficient Neural Architecture Search (ENAS) in PyTorch based [on this paper](https://arxiv.org/abs/1802.03268). (Neural Architecture Search)
• Awesome-AIML-Data-Ops - ENAS-PyTorch - pytorch.svg?style=social) - Efficient Neural Architecture Search (ENAS) in PyTorch based [on this paper](https://arxiv.org/abs/1802.03268). (Neural Architecture Search)
• awesome-production-machine-learning - ENAS-PyTorch - pytorch.svg?style=social) - Efficient Neural Architecture Search (ENAS) in PyTorch based [on this paper](https://arxiv.org/abs/1802.03268). (AutoML)
• awesome-AutoML-and-Lightweight-Models - carpedm20/ENAS-pytorch
• awesome-python-machine-learning-resources - GitHub - 84% open · ⏱️ 16.06.2020): (超参数优化和AutoML)
• StarryDivineSky - carpedm20/ENAS-pytorch - pytorch 是一个基于 PyTorch 的实现，用于高效的神经网络架构搜索(ENAS)，该项目通过参数共享来减少 NAS 的计算需求（GPU小时）1000 倍。该项目在 Penn Treebank 语言建模方面取得了最先进的结果。ENAS 通过一个控制器 LSTM 来决定使用何种激活函数以及连接哪些节点，从而发现 RNN 细胞。该项目还包括用于发现 CNN 架构的代码，以及用于生成GIF 图像的代码，以展示生成的样本。 (神经网络结构搜索_Neural_Architecture_Search)

README

        
# Efficient Neural Architecture Search (ENAS) in PyTorch

PyTorch implementation of [Efficient Neural Architecture Search via Parameters Sharing](https://arxiv.org/abs/1802.03268).



**ENAS** reduce the computational requirement (GPU-hours) of [Neural Architecture Search](https://arxiv.org/abs/1611.01578) (**NAS**) by 1000x via parameter sharing between models that are subgraphs within a large computational graph. SOTA on `Penn Treebank` language modeling.

**\*\*[Caveat] Use official code from the authors: [link](https://github.com/melodyguan/enas)\*\***

## Prerequisites

- Python 3.6+

- [PyTorch==0.3.1](https://pytorch.org/get-started/previous-versions/)

- tqdm, scipy, imageio, graphviz, tensorboardX

## Usage

Install prerequisites with:

    conda install graphviz

    pip install -r requirements.txt

To train **ENAS** to discover a recurrent cell for RNN:

    python main.py --network_type rnn --dataset ptb --controller_optim adam --controller_lr 0.00035 \

                   --shared_optim sgd --shared_lr 20.0 --entropy_coeff 0.0001

    python main.py --network_type rnn --dataset wikitext

To train **ENAS** to discover CNN architecture (in progress):

    python main.py --network_type cnn --dataset cifar --controller_optim momentum --controller_lr_cosine=True \

                   --controller_lr_max 0.05 --controller_lr_min 0.0001 --entropy_coeff 0.1

or you can use your own dataset by placing images like:

    data

    ├── YOUR_TEXT_DATASET

    │   ├── test.txt

    │   ├── train.txt

    │   └── valid.txt

    ├── YOUR_IMAGE_DATASET

    │   ├── test

    │   │   ├── xxx.jpg (name doesn't matter)

    │   │   ├── yyy.jpg (name doesn't matter)

    │   │   └── ...

    │   ├── train

    │   │   ├── xxx.jpg

    │   │   └── ...

    │   └── valid

    │       ├── xxx.jpg

    │       └── ...

    ├── image.py

    └── text.py

To generate `gif` image of generated samples:

    python generate_gif.py --model_name=ptb_2018-02-15_11-20-02 --output=sample.gif

More configurations can be found [here](config.py).

## Results

Efficient Neural Architecture Search (**ENAS**) is composed of two sets of learnable parameters, controller LSTM *θ* and the shared parameters *ω*. These two parameters are alternatively trained and only trained controller is used to derive novel architectures.

### 1. Discovering Recurrent Cells

![rnn](./assets/rnn.png)

Controller LSTM decide 1) what activation function to use and 2) which previous node to connect.

The RNN cell **ENAS** discovered for `Penn Treebank` and `WikiText-2` dataset:

 

Best discovered ENAS cell for `Penn Treebank` at epoch 27:



You can see the details of training (e.g. `reward`, `entropy`, `loss`) with:

    tensorboard --logdir=logs --port=6006

### 2. Discovering Convolutional Neural Networks

![cnn](./assets/cnn.png)

Controller LSTM samples 1) what computation operation to use and 2) which previous node to connect.

The CNN network **ENAS** discovered for `CIFAR-10` dataset:

(in progress)

### 3. Designing Convolutional Cells

(in progress)

## Reference

- [Neural Architecture Search with Reinforcement Learning](https://arxiv.org/abs/1611.01578)

- [Neural Optimizer Search with Reinforcement Learning](https://arxiv.org/abs/1709.07417)

## Author

Taehoon Kim / [@carpedm20](http://carpedm20.github.io/)