{"id":13713189,"url":"https://github.com/taki0112/Tensorflow2-Cookbook","last_synced_at":"2025-05-06T23:30:47.349Z","repository":{"id":98366735,"uuid":"236712750","full_name":"taki0112/Tensorflow2-Cookbook","owner":"taki0112","description":"Simple Tensorflow 2.x Cookbook for easy-to-use","archived":false,"fork":false,"pushed_at":"2020-05-19T08:48:54.000Z","size":2860,"stargazers_count":264,"open_issues_count":1,"forks_count":51,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-03-31T10:12:07.823Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taki0112.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-28T10:48:25.000Z","updated_at":"2024-05-14T03:05:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"48d469a7-1545-42e8-a737-309cf8c31a05","html_url":"https://github.com/taki0112/Tensorflow2-Cookbook","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taki0112%2FTensorflow2-Cookbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taki0112%2FTensorflow2-Cookbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taki0112%2FTensorflow2-Cookbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taki0112%2FTensorflow2-Cookbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taki0112","download_url":"https://codeload.github.com/taki0112/Tensorflow2-Cookbook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252787180,"owners_count":21804211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T23:01:29.210Z","updated_at":"2025-05-06T23:30:45.134Z","avatar_url":"https://github.com/taki0112.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/black_teaser.png\" width=800px\u003e\n\u003c/div\u003e\n\n# [Tensorflow1 Cookbook](https://github.com/taki0112/Tensorflow-Cookbook)\n\n\n## Contributions\nIn now, this repo provides **`general architectures`** and **`functions`** that are useful for the GAN and classification.\n\nI will continue to add useful things to other areas.\n\nAlso, your pull requests and issues are always welcome.\n\nAnd tell me what you want to implement on the issue. I'll implement it.\n\n## Functional vs Sequential\n## Functional API [[Template code]](./template/functional)\n### Pros\n* More **fast** than Sequential\n* More **easy** to create a flexible model architecture\n  * Easy to use some layer operaions like `concatenate`, `add` , ... \n\n### Cons\n* **Define** `tf.keras.layers.Input` \n  * You have to know the `shape of input tensor` \n* **Define** `tf.keras.Model` \n  * You have to create the `Model` constructor at the end \n\n## Sequential API [[Template code]](./template/sequential)\n### Pros\n* **Simple** to use\n  * Similar to Pytorch style\n\n### Cons\n* **Hard** to create a flexible model architecture\n\n## Example code\n* [CycleGAN](./example_cyclegan)\n\n---\n\n## How to use\n## 1. Import\n### Funtional API\n* `ops_functional.py`  \n  * Functional API operations\n  * from ops_functional import *\n\n### Sequential API\n* `ops_sequential.py` \n  * Sequential API operations\n  * from ops_sequential import *\n\n### Common\n* `utils.py` \n  * image processing + something useful functions (e.g. automatic_gpu_usage)\n    * `automatic_gpu_usage` : Automatically manage gpu memory\n    * `multiple_gpu_usage` : You can set gpu memory limit\n  * from utils import *\n\n## 2. Network template\n### Functional API\n```python\nfrom ops_functional import *\nfrom utils import *\n\nautomatic_gpu_usage() # for efficient gpu use\n\ninput_shape = [img_height, img_width, img_ch]\ninputs = tf.keras.layers.Input(input_shape, name='input')\n\n# architecture\nx = conv(inputs, channels=64, kernel=3, stride=2, pad=1, pad_type='reflect', use_bias=False, sn=False, name='conv')\nx = batch_norm(x, name='batch_norm')\nx = relu(x)\n\nx = global_avg_pooling(x)\nx = fully_connected(x, units=10, sn=False, name='fc')\n\nmodel = tf.keras.Model(inputs, s, name='model')\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.01)\n```\n\n### Sequential API\n```python\nfrom ops_sequential import *\nfrom utils import *\n\nautomatic_gpu_usage() # for efficient gpu use\n\nmodel = []\n\nmodel += [Conv(channels=64, kernel=3, stride=2, pad=1, pad_type='reflect', use_bias=False, sn=False, name='conv')]\nmodel += [BatchNorm(name)]\nmodel += [Relu()]\n\nmodel += [Global_Avg_Pooling()]\nmodel += [FullyConnected(units=10, sn=False, name='fc')]\n\nmodel = Sequential(model, name='model')\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.01)\n```\n\n## 3. Data pipeline\n```python\nimg_class = Image_data(img_height, img_width, img_ch, dataset_path, augment_flag)\nimg_class.preprocess()\n\nimg_slice = tf.data.Dataset.from_tensor_slices(img_class.dataset)\ngpu_device = '/gpu:0'\nimg_slice = img_slice. \\\n                apply(shuffle_and_repeat(dataset_num)). \\\n                apply(map_and_batch(img_class.image_processing, self.batch_size,\n                                    num_parallel_batches=AUTOTUNE,\n                                    drop_remainder=True)). \\\n                apply(prefetch_to_device(gpu_device, AUTOTUNE))\n\ndataset_iter = iter(img_slice)\n```\n\n## 4. Restore\n```python\nckpt = tf.train.Checkpoint(model=model, optimizer=optimizer)\nmanager = tf.train.CheckpointManager(ckpt, checkpoint_dir, max_to_keep=2)\nstart_iteration = 0\n\nif manager.latest_checkpoint:\n  ckpt.restore(manager.latest_checkpoint)\n  start_iteration = int(manager.latest_checkpoint.split('-')[-1])\n  print('Latest checkpoint restored!!')\nelse:\n  print('Not restoring from saved checkpoint')\n```\n\n## 5-1. Train\n```python\ndef train_step(img):\n  with tf.GradientTape() as tape:\n    logit = model(img)\n    \n    # calculate loss\n    \"\"\"\n    if classification\n    your_loss = cross_entropy_loss(logit, label)\n    \"\"\"\n    \n    loss = your_loss + regularization_loss(model)\n  \n  train_variable = model.trainable_variables\n  gradient = tape.gradient(loss, train_variable)\n  optimizer.apply_gradients(zip(gradient, train_variable))\n  \n  return loss\n\ndef train():\n  # setup tensorboard\n  summary_writer = tf.summary.create_file_writer(log_dir)\n  \n  for idx in range(start_iteration, total_iteration):\n    img = next(dataset_iter)\n    \n    \n    # update network\n    loss = train_step(img)\n    \n    # save to tensorboard\n    with summary_writer.as_default():\n      tf.summary.scalar('loss', loss, step=idx)\n    \n    # save ckpt\n    manager.save(checkpoint_number=idx + 1)\n  \n  # save model for final step\n  manager.save(checkpoint_number=total_iteration)\n```\n\n## 5-2. Multi-GPUs train [[Template code]](./template/sequential_multi_gpu)\n```python\nstrategy = tf.distribute.MirroredStrategy()\nNUM_GPUS = strategy.num_replicas_in_sync\n\ntotal_iteration = iteration // NUM_GPUS\n\nwith strategy.scope():\n  # copy \u0026 paste\n  # 2. Network template\n  # 3. Data pipeline\n  # 4. Restore\n\ndef train_step(img):\n  \"\"\" SAME \"\"\"\n  \ndef distribute_train_step(img):\n  with strategy.scope():\n    loss = strategy.experimental_run_v2(train_step, args=(img))\n    \n    loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, loss, axis=None)\n    \n    return loss\n\ndef train():\n  # setup tensorboard\n  summary_writer = tf.summary.create_file_writer(log_dir)\n  \n  for idx in range(start_iteration, total_iteration):\n    img = next(dataset_iter)\n    \n    # update network\n    loss = distribute_train_step(img)\n    \n    \"\"\"\n    SAME\n    \"\"\"\n```\n\n\n\n---\n\n## Weight\n```python\nweight_initializer = tf.initializers.RandomNormal(mean=0.0, stddev=0.02)\nweight_regularizer = tf.keras.regularizers.l2(0.0001)\nweight_regularizer_fully = tf.keras.regularizers.l2(0.0001)\n```\n\n### Initialization\n* `Xavier` : tf.initializers.GlorotUniform() or tf.initializers.GlorotNormal()\n* `He` : tf.initializers.VarianceScaling()\n* `Normal` : tf.initializers.RandomNormal(mean=0.0, stddev=0.02)\n* `Truncated normal` : tf.initializers.TruncatedNormal(mean=0.0, stddev=0.02)\n* `Orthogonal` : tf.initializers.Orthogonal0.02)\n\n### Regularization\n* `l2_decay` : tf.keras.regularizers.l2(0.0001)\n* `orthogonal_regularizer` : orthogonal_regularizer(0.0001) # orthogonal_regularizer_fully(0.0001)\n\n---\n\n## Option\n* `padding='SAME'` \n  * pad = ceil[ (kernel - stride) / 2 ]\n* `pad_type` \n  * 'zero' or 'reflect'\n* `sn` \n  * use spectral normalization of not\n\n---\n\n## Examples of Functional API\n## Recurrent\n```python\nx = various_rnn(x, n_hidden=128, n_layer=2, dropout_rate=0.5, training=True, bidirectional=True, rnn_type='lstm', name='rnn')\n```\n### LSTM\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://stanford.edu/~shervine/teaching/cs-230/illustrations/lstm.png\" width = '300px'\u003e\n\u003c/div\u003e\n\n### GRU\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://stanford.edu/~shervine/teaching/cs-230/illustrations/gru.png\" width = '300px'\u003e\n\u003c/div\u003e\n\n### Bidirectional\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://stanford.edu/~shervine/teaching/cs-230/illustrations/bidirectional-rnn.png\" width = '300px'\u003e\n\u003c/div\u003e\n\n### Deep (n_layer \u003e 1)\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://stanford.edu/~shervine/teaching/cs-230/illustrations/deep-rnn.png\" width = '300px'\u003e\n\u003c/div\u003e\n\n\n## Convolution\n### basic conv\n```python\nx = conv(x, channels=64, kernel=3, stride=2, pad=1, pad_type='reflect', use_bias=True, sn=True, name='conv')\n```\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/vdumoulin/conv_arithmetic/raw/master/gif/padding_strides.gif\" width = '300px'\u003e\n\u003c/div\u003e\n\n### partial conv (NVIDIA [Partial Convolution](https://github.com/NVIDIA/partialconv))\n```python\nx = partial_conv(x, channels=64, kernel=3, stride=2, use_bias=True, padding='SAME', sn=True, name='partial_conv')\n```\n\n![p_conv](https://github.com/taki0112/partial_conv-Tensorflow/raw/master/assets/partial_conv.png)\n![p_result](https://github.com/taki0112/partial_conv-Tensorflow/raw/master/assets/classification.png)\n\n### dilated conv\n```python\nx = dilate_conv(x, channels=64, kernel=3, rate=2, use_bias=True, padding='VALID', sn=True, name='dilate_conv')\n```\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/vdumoulin/conv_arithmetic/raw/master/gif/dilation.gif\" width = '300px'\u003e\n\u003c/div\u003e\n\n---\n\n## Deconvolution\n### basic deconv\n```python\nx = deconv(x, channels=64, kernel=3, stride=1, padding='SAME', use_bias=True, sn=True, name='deconv')\n```\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/vdumoulin/conv_arithmetic/raw/master/gif/padding_strides_transposed.gif\" width = '300px'\u003e\n\u003c/div\u003e\n\n---\n\n## Fully-connected\n```python\nx = fully_connected(x, units=64, use_bias=True, sn=True, snamecope='fully_connected')\n```\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://stanford.edu/~shervine/teaching/cs-230/illustrations/fully-connected.png\"\u003e\n\u003c/div\u003e\n\n---\n\n## Pixel shuffle\n```python\nx = conv_pixel_shuffle_down(x, scale_factor=2, use_bias=True, sn=True, name='pixel_shuffle_down')\nx = conv_pixel_shuffle_up(x, scale_factor=2, use_bias=True, sn=True, name='pixel_shuffle_up')\n```\n* `down` ===\u003e [height, width] -\u003e [**height // scale_factor, width // scale_factor**]\n* `up` ===\u003e [height, width] -\u003e [**height \\* scale_factor, width \\* scale_factor**]\n\n![pixel_shuffle](./assets/pixel_shuffle.png)\n\n\n---\n\n## Block\n### residual block\n```python\nx = resblock(x, channels=64, is_training=is_training, use_bias=True, sn=True, name='residual_block')\nx = resblock_down(x, channels=64, is_training=is_training, use_bias=True, sn=True, name='residual_block_down')\nx = resblock_up(x, channels=64, is_training=is_training, use_bias=True, sn=True, name='residual_block_up')\n```\n* `down` ===\u003e [height, width] -\u003e [**height // 2, width // 2**]\n* `up` ===\u003e [height, width] -\u003e [**height \\* 2, width \\* 2**]\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://cdn-images-1.medium.com/max/1600/1*FqmD91PvbH7NKCnQWFJxvg.png\"\u003e\n\u003c/div\u003e\n\n### dense block\n```python\nx = denseblock(x, channels=64, n_db=6, is_training=is_training, use_bias=True, sn=True, name='denseblock')\n```\n* `n_db` ===\u003e The number of dense-block\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/taki0112/Densenet-Tensorflow/raw/master/assests/Denseblock.JPG\" height = '400px'\u003e\n\u003c/div\u003e\n\n### residual-dense block\n```python\nx = res_denseblock(x, channels=64, n_rdb=20, n_rdb_conv=6, is_training=is_training, use_bias=True, sn=True, name='res_denseblock')\n```\n* `n_rdb` ===\u003e The number of RDB\n* `n_rdb_conv` ===\u003e per RDB conv layer\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=./assets/compare.png height = '400px'\u003e\n  \u003cimg src=./assets/rdn.png height = '350px' width='800px'\u003e\n  \u003cimg src=./assets/rdb.png height = '250px' width='650px'\u003e\n\u003c/div\u003e\n\n### attention block\n```python\nx = self_attention(x, use_bias=True, sn=True, name='self_attention')\nx = self_attention_with_pooling(x, use_bias=True, sn=True, name='self_attention_version_2')\n\nx = squeeze_excitation(x, ratio=16, use_bias=True, sn=True, name='squeeze_excitation')\n\nx = convolution_block_attention(x, ratio=16, use_bias=True, sn=True, name='convolution_block_attention')\n\nx = global_context_block(x, use_bias=True, sn=True, name='gc_block')\n\nx = srm_block(x, use_bias=False, is_training=is_training, name='srm_block')\n```\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/taki0112/Self-Attention-GAN-Tensorflow/raw/master/assests/framework.PNG\"\u003e\n\u003c/div\u003e\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/hujie-frank/SENet/blob/master/figures/SE-Inception-module.jpg\" width=\"420\"\u003e\n  \u003cimg src=\"https://github.com/hujie-frank/SENet/blob/master/figures/SE-ResNet-module.jpg\"  width=\"420\"\u003e\n\u003c/div\u003e\n\n---\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://bloglunit.files.wordpress.com/2018/08/screen-shot-2018-08-22-at-8-42-27-pm.png?w=2800\"\u003e\n  \u003cimg src=\"https://bloglunit.files.wordpress.com/2018/08/screen-shot-2018-08-22-at-8-47-09-pm.png?w=2800\"\u003e\n\u003c/div\u003e\n\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=./assets/gcb.png\u003e\n\u003c/div\u003e\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=./assets/srm.png height='350' width='500'\u003e\n\u003c/div\u003e\n\n---\n\n## Normalization\n```python\nx = batch_norm(x, training=training, name='batch_norm')\nx = layer_norm(x, name='layer_norm')\nx = instance_norm(x, name='instance_norm')\nx = group_norm(x, groups=32, name='group_norm')\n\nx = pixel_norm(x)\n\nx = batch_instance_norm(x, name='batch_instance_norm')\nx = layer_instance_norm(x, name='layer_instance_norm')\nx = switch_norm(x, scope='switch_norm')\n\nx = condition_batch_norm(x, z, training=training, name='condition_batch_norm'):\n\nx = adaptive_instance_norm(x, gamma, beta)\nx = adaptive_layer_instance_norm(x, gamma, beta, smoothing=True, name='adaLIN')\n\n```\n* See [this](https://github.com/taki0112/BigGAN-Tensorflow) for how to use `condition_batch_norm`\n* See [this](https://github.com/taki0112/MUNIT-Tensorflow) for how to use `adaptive_instance_norm`\n* See [this](https://github.com/taki0112/UGATIT) for how to use `adaptive_layer_instance_norm` \u0026 `layer_instance_norm`\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/taki0112/Group_Normalization-Tensorflow/raw/master/assests/norm.png\"\u003e\n\u003c/div\u003e\n\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/taki0112/Switchable_Normalization-Tensorflow/raw/master/assests/teaser.png\"\u003e\n\u003c/div\u003e\n\n---\n\n## Activation\n```python\nx = relu(x)\nx = lrelu(x, alpha=0.2)\nx = tanh(x)\nx = sigmoid(x)\nx = swish(x)\nx = elu(x)\n```\n\n---\n\n## Pooling \u0026 Resize\n```python\nx = nearest_up_sample(x, scale_factor=2)\nx = bilinear_up_sample(x, scale_factor=2)\nx = nearest_down_sample(x, scale_factor=2)\nx = bilinear_down_sample(x, scale_factor=2)\n\nx = max_pooling(x, pool_size=2)\nx = avg_pooling(x, pool_size=2)\n\nx = global_max_pooling(x)\nx = global_avg_pooling(x)\n\nx = flatten(x)\nx = hw_flatten(x)\n```\n\n---\n\n## Loss\n### classification loss\n```python\nloss, accuracy = classification_loss(logit, label)\n\nloss = dice_loss(n_classes=10, logit, label)\n```\n\n### regularization loss\n```python\nmodel_reg_loss = regularization_loss(model)\n```\n* If you want to use `regularizer`, then you should write it\n\n### pixel loss\n```python\nloss = L1_loss(x, y)\nloss = L2_loss(x, y)\nloss = huber_loss(x, y)\nloss = histogram_loss(x, y)\n\nloss = gram_style_loss(x, y)\n\nloss = color_consistency_loss(x, y)\n```\n* `histogram_loss` means the difference in the color distribution of the image pixel values.\n* `gram_style_loss` means the difference between the styles using gram matrix.\n* `color_consistency_loss` means the color difference between the generated image and the input image.\n\n### gan loss\n```python\nd_loss = discriminator_loss(Ra=True, gan_type='wgan-gp', real_logit=real_logit, fake_logit=fake_logit)\ng_loss = generator_loss(Ra=True, gan_type='wgan-gp', real_logit=real_logit, fake_logit=fake_logit)\n```\n* `Ra`\n  * use [relativistic gan](https://arxiv.org/pdf/1807.00734.pdf) or not\n* `loss_func`\n  * gan\n  * lsgan\n  * hinge\n  * wgan-gp\n  * dragan\n  * [realness](https://github.com/taki0112/RealnessGAN-Tensorflow)\n  * [sphere](https://github.com/taki0112/SphereGAN-Tensorflow)\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=./assets/relativistic.png\u003e\n\u003c/div\u003e\n\n### [vdb loss](https://arxiv.org/abs/1810.00821)\n```python\nd_bottleneck_loss = vdb_loss(real_mu, real_logvar, i_c) + vdb_loss(fake_mu, fake_logvar, i_c)\n```\n\n### kl-divergence (z ~ N(0, 1))\n```python\nloss = kl_loss(mean, logvar)\n```\n\n---\n\n## Author\n[Junho Kim](http://bit.ly/jhkim_ai)\n","funding_links":[],"categories":["Machine Learning Tutorials"],"sub_categories":["Data Management"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaki0112%2FTensorflow2-Cookbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaki0112%2FTensorflow2-Cookbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaki0112%2FTensorflow2-Cookbook/lists"}