{"id":13640746,"url":"https://github.com/MINGUKKANG/ENAS-Tensorflow","last_synced_at":"2025-04-20T07:31:05.452Z","repository":{"id":44541449,"uuid":"134113666","full_name":"mingukkang/ENAS-Tensorflow","owner":"mingukkang","description":"Efficient Neural Architecture search via parameter sharing(ENAS) micro search Tensorflow code for windows user","archived":false,"fork":false,"pushed_at":"2018-07-19T14:16:46.000Z","size":29685,"stargazers_count":113,"open_issues_count":6,"forks_count":33,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-08-03T01:17:38.603Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mingukkang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-20T02:31:29.000Z","updated_at":"2024-03-22T05:01:13.000Z","dependencies_parsed_at":"2022-07-16T06:00:44.068Z","dependency_job_id":null,"html_url":"https://github.com/mingukkang/ENAS-Tensorflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingukkang%2FENAS-Tensorflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingukkang%2FENAS-Tensorflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingukkang%2FENAS-Tensorflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingukkang%2FENAS-Tensorflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mingukkang","download_url":"https://codeload.github.com/mingukkang/ENAS-Tensorflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223821905,"owners_count":17208756,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:14.060Z","updated_at":"2024-11-09T11:30:21.898Z","avatar_url":"https://github.com/mingukkang.png","language":"Python","funding_links":[],"categories":["Neural Architecture Search","AutoML"],"sub_categories":[],"readme":"## ENAS-Tensorflow\n\nI will explain the code of Efficient Neural Architecture Search(ENAS), especially case of micro search.\n\nUnlike the author's code, This code can work in a windows 10 enviroment and you can use png files as datasets.\n\nAlso you can apply data augmentation using \"n_aug_img\" which is explained below. \n\n## Enviroment\n- OS: Window 10(Ubuntu 16.04 is possible)\n\n- Graphic Card /RAM : 1080TI /32G\n\n- Python 3.5\n\n- Tensorflow-gpu version:  1.4.0rc2 \n\n- OpenCV 3.4.1\n\n\n## How to run\n\n**\u003cbr/\u003eAt first, you should unpack the attached data as shown below.**\n\n![사진1](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/unpack.PNG)\n\n**\u003cbr/\u003e Next, You should change the code below to suit your situation.**\n\n```\n\u003cmain_controller_child_trainer.py and main_child_trainer.py\u003e\n\nDEFINE_string(\"output_dir\", \"./output\" , \"\")\nDEFINE_string(\"train_data_dir\", \"./data/train\", \"\")\nDEFINE_string(\"val_data_dir\", \"./data/valid\", \"\")\nDEFINE_string(\"test_data_dir\", \"./data/test\", \"\")\nDEFINE_integer(\"channel\",1, \"MNIST: 1, Cifar10: 3\")\nDEFINE_integer(\"img_size\", 32, \"enlarge image size\")\nDEFINE_integer(\"n_aug_img\",1 , \"if 2: num_img: 55000 -\u003e aug_img: 110000, elif 1: False\")\n```\nIt is recommended to set \"n_aug_img\" = 1 to find the child network, and to use 2 ~ 4 to train the found child network.\n\n**\u003cbr/\u003eThen, You can train Controller of ENAS with the following short code:**\n```\npython main_controller_child_trainer.py\n```\n**\u003cbr/\u003eAfter finishing,   you can train the child network with the following code:**\n\n```\nCase of MNIST \n\npython main_child_trainer.py --child_fixed_arc \"1 2 1 3 0 1 0 4 1 1 1 1 0 1 0 1 1 0 0 1 0 1 0 4 1 0 2 0 0 3 1 1 0 0 0 0 4 1 1 0\"\n```\n\n```\nCase of Cifar 10\n\npython main_child_trainer.py --child_fixed_arc \"1 0 1 1 1 1 0 0 1 1 0 0 0 3 0 3 1 3 1 1 1 1 0 3 0 3 0 3 1 3 0 1 1 3 0 2 0 3 1 0\"\n```\n\n```\nCase of Welding Defects\n\npython main_child_trainer.py --child_fixed_arc \"1 0 0 1 0 0 1 1 2 2 1 1 1 1 1 2 1 0 0 0 0 0 0 3 2 2 1 0 2 0 2 3 0 3 4 0 1 0 3 2\"\n```\n\nThe string in the above code like \"1 2 1 3 0 1 ~ \" is the result of main_controller_child_trainer.py\n\nThe first 20 numbers are for the architecture for convolution layers, and the rest are for pooling layers.\n\n## Result\n\n### 1. ENAS cells discoved in the micro search space\n\nAfter training \u003cmain_controller_child_trainer.py\u003e, we got the following child_arc_seq and visualized it as shown below.\n\n#### MNIST\n\n```\n\"1 2 1 3 0 1 0 4 1 1 1 1 0 1 0 1 1 0 0 1 0 1 0 4 1 0 2 0 0 3 1 1 0 0 0 0 4 1 1 0\"\n```\n\n\u003cbr/\u003e![사진2](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/MNIST_convCell.png)\n\n\u003cbr/\u003e![사진3](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/MNIST_Reduction_cell.png)\n\n#### CIFAR 10\n\n```\n\"1 0 1 1 1 1 0 0 1 1 0 0 0 3 0 3 1 3 1 1 1 1 0 3 0 3 0 3 1 3 0 1 1 3 0 2 0 3 1 0\"\n```\n\n\u003cbr/\u003e![사진2](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/CIFAR10_Convolution_cell.png)\n\n\u003cbr/\u003e![사진3](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/CIFAR10_Reduction_cell.png)\n\n#### Welding Defects\n\n```\n\"1 0 0 1 0 0 1 1 2 2 1 1 1 1 1 2 1 0 0 0 0 0 0 3 2 2 1 0 2 0 2 3 0 3 4 0 1 0 3 2\"\n```\n\n\u003cbr/\u003e![사진2](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Welding_Defects_Convolutional_Cell.png)\n\n\u003cbr/\u003e![사진3](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Welding_Defects_Reduction_Cell.png)\n\n### 2. Final structure of the child network\n\n#### MNIST\n\u003cbr/\u003e![사진4](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/MNIST_final_network.png)\n\n#### CIFAR 10\n\u003cbr/\u003e![사진4](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/CIFAR10_final_network.png)\n\n#### Welding Defects\n\u003cbr/\u003e![사진4](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Welding_final_network.png)\n\n### 3. Test Accuracy\n\n```\nMNIST\nTest Accuracy : 99.77%\n```\n\n```\nCIFAR 10\nTest Accuracy : \n```\n\n```\nWelding Defects\nTest Accuracy : 100.00% \n```\n\n### 4. Graphs\n\n\u003ctable align='center'\u003e\n\u003ctr align='center'\u003e\n\u003ctd\u003e Controller Validation Accuracy(reward) \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src = 'images/Controller_reward_graph.png' height = '300px'\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ctable align='center'\u003e\n\u003ctr align='center'\u003e\n\u003ctd\u003e ChildNetwork Loss ＆ Test Accuracy for MNIST Dataset\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src = 'images/MNIST_child_network_graph.png' height = '300px'\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ctable align='center'\u003e\n\u003ctr align='center'\u003e\n\u003ctd\u003e ChildNetwork Loss ＆ Test Accuracy for Welding Defects Dataset \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src = 'images/Welding_Child_network_graph.png' height = '300px'\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Explained\n\n### 1. Controller\n\nFirst, we will build the sampler as shown in the picture below.\n\n\u003cbr/\u003e![사진5](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Controller_init.png)\n\n\u003cbr/\u003eThen we will make controller using sampler's output \"next_c_1, next_h_1\".\n\n\u003cbr/\u003e![사진6](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Controller.PNG)\n\n\u003cbr/\u003e After getting the \"next_c_5, next_h_5\", you must do the following to renew \"Anchors,   Anchors_w_1\".\n\n\u003cbr/\u003e![사진7](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Anchors_appen.PNG)\n\n### 2. Controller_Loss\n\nTo enable the Controller to make better networks, ENAS uses REINFORCE with a moving average baseline to reduce variance.\n\n```python\n\u003cmicro_controller.py\u003e\n\nfor all index:\n    curr_log_prob = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=index)\n    log_prob += curr_log_prob\n    curr_ent = tf.stop_gradient(tf.nn.softmax_cross_entropy_with_logits(\n    logits=logits, labels=tf.nn.softmax(logits)))\n    entropy += curr_ent\n\nfor all op_id:\n    curr_log_prob = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=op_id)\n    log_prob += curr_log_prob\n    curr_ent = tf.stop_gradient(tf.nn.softmax_cross_entropy_with_logits(\n    logits=logits, labels=tf.nn.softmax(logits)))\n    entropy += curr_ent\n\narc_seq_1, entropy_1, log_prob_1, c, h = self._build_sampler(use_bias=True) # for convolution cell\narc_seq_2, entropy_2, log_prob_2, _, _ = self._build_sampler(prev_c=c, prev_h=h) # for reduction cell \nself.sample_entropy = entropy_1 + entropy_2\nself.sample_log_prob = log_prob_1 + log_prob_2    \n```\n\n```python\n\u003cmicro_controller.py\u003e\n\n    self.valid_acc = (tf.to_float(child_model.valid_shuffle_acc) /\n                      tf.to_float(child_model.batch_size))\n    self.reward = self.valid_acc \n\n    if self.entropy_weight is not None:\n      self.reward += self.entropy_weight * self.sample_entropy\n\n    self.sample_log_prob = tf.reduce_sum(self.sample_log_prob)\n    self.baseline = tf.Variable(0.0, dtype=tf.float32, trainable=False)\n    baseline_update = tf.assign_sub(\n      self.baseline, (1 - self.bl_dec) * (self.baseline - self.reward))\n\n    with tf.control_dependencies([baseline_update]):\n      self.reward = tf.identity(self.reward)\n\n    self.loss = self.sample_log_prob * (self.reward - self.baseline)\n```\n\n### 3. Child Network \n\n(1) Schematic of Child Network\n\n\u003cbr/\u003e![사진8](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Schematic_child_network.png)\n\n(2) _enas_layers\n\n```python\n\u003cmicro_child.py\u003e\n\ndef _enas_layers(self, layer_id, prev_layers, arc, out_filters):\n    '''\n    prev_layers : previous two layers. ex) layers[●,●]\n    ●'s shape = [None, H, W, C]\n    arc: \"0 1 0 1 0 3 0 0 2 2 0 2 1 0 0 1 1 3 0 1 1 1 0 1 0 1 2 1 0 0 0 0 0 0 1 3 1 1 0 1\"\n    out = [self._enas_conv(x, curr_cell, prev_cell, 3, out_filters), \n           self._enas_conv(x, curr_cell, prev_cell, 5, out_filters),\n           avg_pool,\n           max_pool, \n           x]\n    '''\n    \n    retrun output # calculated by arc, np.shape(output) = [None, H, W, out_filters]\n                  # if child_fixed_arc is not None, np.shape(output) = [None, H, W, n*out_filters]\n                  # where n is the number of not being used nodes in the coonv cell or Reduction cell.\n```\n\n(3) factorized_reduction\n\n```python\n\u003cmicro_child.py\u003e\n\ndef factorized_reduction(self, x, out_filters, strides = 2, is_training = True):\n    '''\n    x : x is last previous layer's output.\n    out_filters: 2*(previous layer's channel)\n    '''\n    \n    stride_spec = self._get_strides(stride)  # [1,2,2,1]\n    \n    # Skip path 1\n    path1 = tf.nn.avg_pool(x, [1, 1, 1, 1], stride_spec, \"VALID\", data_format=self.data_format)  \n\n    with tf.variable_scope(\"path1_conv\"):\n        inp_c = self._get_C(path1)\n        w = create_weight(\"w\", [1, 1, inp_c, out_filters // 2])  \n        path1 = tf.nn.conv2d(path1, w, [1, 1, 1, 1], \"VALID\", data_format=self.data_format)  \n\n        # Skip path 2\n        # First pad with 0\"s on the right and bottom, then shift the filter to\n        # include those 0\"s that were added.\n    if self.data_format == \"NHWC\":\n        pad_arr = [[0, 0], [0, 1], [0, 1], [0, 0]]\n        path2 = tf.pad(x, pad_arr)[:, 1:, 1:, :]\n        concat_axis = 3\n    else:\n        pad_arr = [[0, 0], [0, 0], [0, 1], [0, 1]]\n        path2 = tf.pad(x, pad_arr)[:, :, 1:, 1:]\n        concat_axis = 1\n\n    path2 = tf.nn.avg_pool(path2, [1, 1, 1, 1], stride_spec, \"VALID\", data_format=self.data_format)\n    with tf.variable_scope(\"path2_conv\"):\n        inp_c = self._get_C(path2)\n        w = create_weight(\"w\", [1, 1, inp_c, out_filters // 2])\n        path2 = tf.nn.conv2d(path2, w, [1, 1, 1, 1], \"VALID\", data_format=self.data_format)\n\n    # Concat and apply BN\n    final_path = tf.concat(values=[path1, path2], axis=concat_axis)\n    final_path = batch_norm(final_path, is_training, data_format=self.data_format)\n\n    return final_path\n```\n\n(4) _maybe_calibrate_size\n\n```python\n\u003cmicro_child.py\u003e\n\ndef _maybe_calibrate_size(self, layers, out_filters, is_training): \n    \"\"\"Makes sure layers[0] and layers[1] have the same shapes.\"\"\"\n    hw = [self._get_HW(layer) for layer in layers]  \n    c = [self._get_C(layer) for layer in layers]  \n\n    with tf.variable_scope(\"calibrate\"):\n        x = layers[0]  \n        if hw[0] != hw[1]:  \n            assert hw[0] == 2 * hw[1]  \n            with tf.variable_scope(\"pool_x\"):\n                x = tf.nn.relu(x)\n                x = self._factorized_reduction(x, out_filters, 2, is_training)\n        elif c[0] != out_filters:  \n            with tf.variable_scope(\"pool_x\"):\n                w = create_weight(\"w\", [1, 1, c[0], out_filters])\n                x = tf.nn.relu(x)\n                x = tf.nn.conv2d(x, w, [1, 1, 1, 1], \"SAME\", data_format=self.data_format)\n                x = batch_norm(x, is_training, data_format=self.data_format)  \n\n        y = layers[1]  \n        if c[1] != out_filters:  \n            with tf.variable_scope(\"pool_y\"):\n                w = create_weight(\"w\", [1, 1, c[1], out_filters])\n                y = tf.nn.relu(y)\n                y = tf.nn.conv2d(y, w, [1, 1, 1, 1], \"SAME\", data_format=self.data_format)\n                y = batch_norm(y, is_training, data_format=self.data_format)\n    return [x, y]\n```\n\n(5) Others\n\nYou can see more details of the child network in \u003cmicro_child.py\u003e\n\n### 4. Summary of learning mechanism\n\n\u003cmain_child_controller_trainer.py\u003e\n```\n1. Train the Child Network during 1 Epoch. (Momentum optimization)\n※ 1 Epoch = (Total data size / batch size) times parameters update.\n\n2. Train the controller 'FLAGS.controller_train_steps x FLAGS.controller_num_aggregate' times. (Adam Optimization)\n\n3. Repeat \"1\", \"2\" as many as we want.(160 Epochs)\n\n4. Choose the child network architecture with the highest validation accuracy.\n```\n\n\u003cmain_child_trainer.py\u003e\n```\n1. Train the child Network which is selected above as many as we want. (Momentum optimization, 660 Epochs)\n```\n\n## Augmentation\n\n### 1. Code\n\n```python\ndef aug(image, idx):\n    augmentation_dic = {0: enlarge(image, 1.2),\n                        1: rotation(image),\n                        2: random_bright_contrast(image),\n                        3: gaussian_noise(image),\n                        4: Flip(image)}\n\n    image = augmentation_dic[idx]\n    return image\n```\n\nFunction enlarge, rotation, random_bright_contrast and Flip are writen using cv2.\n\nIn the case of MNIST Data, I do not apply flip! you can check more details in \u003cdata_utils.py\u003e\n\n### 2. Images\n\n## Graphs\n\n\n#### MNIST\n![사진9](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/MNIST_AUG.png)\n\n#### CIFAR10\n![사진9](https://github.com/MINGUKKANG/ENAS-Tensorflow/blob/master/images/Cifar10_AUG.png)\n\n#### Welding Defects\n\u003ctable align='center'\u003e\n\u003ctr align='center'\u003e\n\u003ctd\u003e Welding OK \u003c/td\u003e\n\u003ctd\u003e Welding NG \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\u003cimg src = 'images/Welding_OK.jpg' height = '250px'\u003e\n\u003ctd\u003e\u003cimg src = 'images/Welding_NG.jpg' height = '250px'\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## References\n**Paper: https://arxiv.org/abs/1802.03268**\n\n**Autors' implementation: https://github.com/melodyguan/enas**\n\n**Data Pipeline: https://github.com/MINGUKKANG/MNIST-Tensorflow-Code**\n\n## License\nAll rights related to this code are reserved to the author of ENAS\n\n(Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMINGUKKANG%2FENAS-Tensorflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMINGUKKANG%2FENAS-Tensorflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMINGUKKANG%2FENAS-Tensorflow/lists"}