{"id":13688625,"url":"https://github.com/wagamamaz/tensorlayer-tricks","last_synced_at":"2025-04-09T09:10:59.903Z","repository":{"id":72877648,"uuid":"82088131","full_name":"wagamamaz/tensorlayer-tricks","owner":"wagamamaz","description":"How to use TensorLayer","archived":false,"fork":false,"pushed_at":"2018-09-17T12:03:44.000Z","size":58,"stargazers_count":346,"open_issues_count":0,"forks_count":62,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-04-02T08:06:58.807Z","etag":null,"topics":["computer-vision","data-science","deep-learning","keras","lasagne","machine-learning","natural-language-processing","neural-network","neural-networks","nlp","reinforcement-learning","tensorboard","tensorflow","tensorflow-experiments","tensorflow-framework","tensorflow-library","tensorflow-models","tensorflow-tutorials","tensorlayer","tflearn"],"latest_commit_sha":null,"homepage":"tensorlayer.org","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wagamamaz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-02-15T17:39:55.000Z","updated_at":"2025-02-06T01:13:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"538dd551-70f9-4f1c-b462-6964574dcbfc","html_url":"https://github.com/wagamamaz/tensorlayer-tricks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wagamamaz%2Ftensorlayer-tricks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wagamamaz%2Ftensorlayer-tricks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wagamamaz%2Ftensorlayer-tricks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wagamamaz%2Ftensorlayer-tricks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wagamamaz","download_url":"https://codeload.github.com/wagamamaz/tensorlayer-tricks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248008630,"owners_count":21032556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","data-science","deep-learning","keras","lasagne","machine-learning","natural-language-processing","neural-network","neural-networks","nlp","reinforcement-learning","tensorboard","tensorflow","tensorflow-experiments","tensorflow-framework","tensorflow-library","tensorflow-models","tensorflow-tutorials","tensorlayer","tflearn"],"created_at":"2024-08-02T15:01:17.901Z","updated_at":"2025-04-09T09:10:59.886Z","avatar_url":"https://github.com/wagamamaz.png","language":null,"readme":"# How to use TensorLayer\n\nWhile research in Deep Learning continues to improve the world, we use a bunch of tricks to implement algorithms with TensorLayer day to day.\n\nHere are a summary of the tricks to use TensorLayer.\nIf you find a trick that is particularly useful in practice, please open a Pull Request to add it to the document. If we find it to be reasonable and verified, we will merge it in.\n\n- 🇨🇳 [《深度学习：一起玩转TensorLayer》](https://item.jd.com/12286942.html)已上架。\n\n## 1. Installation\n * To keep your TL version and edit the source code easily, you can download the whole repository by excuting `git clone https://github.com/zsdonghao/tensorlayer.git` in your terminal, then copy the `tensorlayer` folder into your project \n * As TL is growing very fast, if you want to use `pip` install, we suggest you to install the master version \n * For NLP application, you will need to install [NLTK and NLTK data](http://www.nltk.org/install.html)\n\n## 2. Interaction between TF and TL\n * TF to TL : use [InputLayer](https://tensorlayer.readthedocs.io/en/latest/modules/layers.html#input-layers)\n * TL to TF : use [network.outputs](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#understand-basic-layer)\n * Other methods [issues7](https://github.com/tensorlayer/tensorlayer/issues/7), multiple inputs [issues31](https://github.com/tensorlayer/tensorlayer/issues/31)\n\n## 3. Training/Testing switching\n * Use [network.all_drop](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#understand-basic-layer) to control the training/testing phase (for [DropoutLayer](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#dropout-layer) only) see [this example](https://github.com/tensorlayer/tensorlayer/blob/master/examples/basic_tutorials/tutorial_mlp_dropout1.py) and [Understand Basic layer](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#understand-basic-layer)\n * Alternatively, set `is_fix` to `True` in [DropoutLayer](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#dropout-layer), and build different graphs for training/testing by reusing the parameters. You can also set different `batch_size` and noise probability for different graphs. This method is the best when you use [GaussianNoiseLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#tensorlayer.layers.GaussianNoiseLayer), [BatchNormLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#normalization-layers) and etc. Here is an example:\n\n```python\ndef mlp(x, is_train=True, reuse=False):\n    with tf.variable_scope(\"MLP\", reuse=reuse):\n      net = InputLayer(x, name='in')\n      net = DropoutLayer(net, 0.8, True, is_train, name='drop1')\n      net = DenseLayer(net, n_units=800, act=tf.nn.relu, name='dense1')\n      net = DropoutLayer(net, 0.8, True, is_train, name='drop2')\n      net = DenseLayer(net, n_units=800, act=tf.nn.relu, name='dense2')\n      net = DropoutLayer(net, 0.8, True, is_train, name='drop3')\n      net = DenseLayer(net, n_units=10, act=tf.identity, name='out')\n      logits = net.outputs\n      net.outputs = tf.nn.sigmoid(net.outputs)\n      return net, logits\nx = tf.placeholder(tf.float32, shape=[None, 784], name='x')\ny_ = tf.placeholder(tf.int64, shape=[None, ], name='y_')\nnet_train, logits = mlp(x, is_train=True, reuse=False)\nnet_test, _ = mlp(x, is_train=False, reuse=True)\ncost = tl.cost.cross_entropy(logits, y_, name='cost')\n```\n\nMore in [here](https://github.com/tensorlayer/tensorlayer/blob/master/examples/basic_tutorials/tutorial_mlp_dropout2.py).\n\n## 4. Get variables and outputs\n * Use [tl.layers.get_variables_with_name](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#get-variables-with-name) instead of using [net.all_params](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#understanding-the-basic-layer)\n```python\ntrain_vars = tl.layers.get_variables_with_name('MLP', True, True)\ntrain_op = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(cost, var_list=train_vars)\n```\n * This method can also be used to freeze some layers during training, just simply don't get some variables\n * Other methods [issues17](https://github.com/zsdonghao/tensorlayer/issues/17), [issues26](https://github.com/zsdonghao/tensorlayer/issues/26), [FQA](http://tensorlayer.readthedocs.io/en/latest/user/more.html#exclude-some-layers-from-training)\n * Use [tl.layers.get_layers_with_name](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#name-scope-and-sharing-parameters) to get list of activation outputs from a network.\n```python\nlayers = tl.layers.get_layers_with_name(network, \"MLP\", True)\n```\n * This method usually be used for activation regularization.\n  \n## 5. Data augmentation for large dataset\nIf your dataset is large, data loading and data augmentation will become the bottomneck and slow down the training.\nTo speed up the data processing you can:\n\n* Use TFRecord or TF DatasetAPI, see [cifar10 examples](https://github.com/tensorlayer/tensorlayer/tree/master/examples/basic_tutorials)\n\n## 6. Data augmentation for small dataset\nIf your data size is small enough to feed into the memory of your machine, and data augmentation is simple. To debug easily, you can:\n\n* Use [tl.iterate.minibatches](http://tensorlayer.readthedocs.io/en/latest/modules/iterate.html#tensorlayer.iterate.minibatches) to shuffle and return the examples and labels by the given batchsize.\n* Use [tl.prepro.threading_data](http://tensorlayer.readthedocs.io/en/latest/modules/prepro.html#tensorlayer.prepro.threading_data) to read a batch of data at the beginning of every step, the performance is slow but good for small dataset.\n* For time-series data, use [tl.iterate.seq_minibatches, tl.iterate.seq_minibatches2, tl.iterate.ptb_iterator and etc](http://tensorlayer.readthedocs.io/en/latest/modules/iterate.html#time-series)\n\n## 7. Pre-trained CNN and Resnet\n* Pre-trained CNN\n * Many applications make need pre-trained CNN model\n * TL provides pre-trained VGG16, VGG19, MobileNet, SqueezeNet and etc : [tl.models](https://tensorlayer.readthedocs.io/en/stable/modules/models.html#)\n * [tl.layers.SlimNetsLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#external-libraries-layers) allows you to use all [Tf-Slim pre-trained models](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models) and [tensorlayer/pretrained-models](https://github.com/tensorlayer/pretrained-models)\n* Resnet\n * Implemented by \"for\" loop [issues85](https://github.com/zsdonghao/tensorlayer/issues/85)\n * Other methods [by @ritchieng](https://github.com/ritchieng/wideresnet-tensorlayer)\n\n## 8. Using `tl.models`\n\n* Use pretrained VGG16 for ImageNet classification\n```python\nx = tf.placeholder(tf.float32, [None, 224, 224, 3])\n# get the whole model\nvgg = tl.models.VGG16(x)\n# restore pre-trained VGG parameters\nsess = tf.InteractiveSession()\nvgg.restore_params(sess)\n# use for inferencing\nprobs = tf.nn.softmax(vgg.outputs)\n```\n\n* Extract features with VGG16 and retrain a classifier with 100 classes\n```python\nx = tf.placeholder(tf.float32, [None, 224, 224, 3])\n# get VGG without the last layer\nvgg = tl.models.VGG16(x, end_with='fc2_relu')\n# add one more layer\nnet = tl.layers.DenseLayer(vgg, 100, name='out')\n# initialize all parameters\nsess = tf.InteractiveSession()\ntl.layers.initialize_global_variables(sess)\n# restore pre-trained VGG parameters\nvgg.restore_params(sess)\n# train your own classifier (only update the last layer)\ntrain_params = tl.layers.get_variables_with_name('out')\n```\n\n* Reuse model\n\n```python\nx1 = tf.placeholder(tf.float32, [None, 224, 224, 3])\nx2 = tf.placeholder(tf.float32, [None, 224, 224, 3])\n# get VGG without the last layer\nvgg1 = tl.models.VGG16(x1, end_with='fc2_relu')\n# reuse the parameters of vgg1 with different input\nvgg2 = tl.models.VGG16(x2, end_with='fc2_relu', reuse=True)\n# restore pre-trained VGG parameters (as they share parameters, we don’t need to restore vgg2)\nsess = tf.InteractiveSession()\nvgg1.restore_params(sess)\n```\n\n## 9. Customized layer\n* 1. [Write a TL layer directly](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#customizing-layers)\n* 2. Use [LambdaLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#lambda-layers), it can also accept functions with new variables. With this layer you can connect all third party TF libraries and your customized function to TL. Here is an example of using Keras and TL together.\n\n```python\nimport tensorflow as tf\nimport tensorlayer as tl\nfrom keras.layers import *\nfrom tensorlayer.layers import *\ndef my_fn(x):\n    x = Dropout(0.8)(x)\n    x = Dense(800, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    x = Dense(800, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    logits = Dense(10, activation='linear')(x)\n    return logits\n\nnetwork = InputLayer(x, name='input')\nnetwork = LambdaLayer(network, my_fn, name='keras')\n...\n```\n\n## 10. Sentences tokenization\n * Use [tl.nlp.process_sentence](https://tensorlayer.readthedocs.io/en/stable/modules/nlp.html#tensorlayer.nlp.process_sentence) to tokenize the sentences, [NLTK and NLTK data](http://www.nltk.org/install.html) is required\n \n```python\n\u003e\u003e\u003e captions = [\"one two , three\", \"four five five\"] # 2个 句 子 \n\u003e\u003e\u003e processed_capts = []\n\u003e\u003e\u003e for c in captions:\n\u003e\u003e\u003e    c = tl.nlp.process_sentence(c, start_word=\"\u003cS\u003e\", end_word=\"\u003c/S\u003e\")\n\u003e\u003e\u003e    processed_capts.append(c)\n\u003e\u003e\u003e print(processed_capts)\n... [['\u003cS\u003e', 'one', 'two', ',', 'three', '\u003c/S\u003e'],\n... ['\u003cS\u003e', 'four', 'five', 'five', '\u003c/S\u003e']]\n```\n \n * Then use [tl.nlp.create_vocab](https://tensorlayer.readthedocs.io/en/stable/modules/nlp.html#tensorlayer.nlp.create_vocab) to create a vocabulary and save as txt file (it will return a [tl.nlp.SimpleVocabulary object](https://tensorlayer.readthedocs.io/en/stable/modules/nlp.html#tensorlayer.nlp.SimpleVocabulary) for word to id only)\n\n```python\n\u003e\u003e\u003e tl.nlp.create_vocab(processed_capts, word_counts_output_file='vocab.txt', min_word_count=1)\n... [TL] Creating vocabulary.\n... Total words: 8\n... Words in vocabulary: 8\n... Wrote vocabulary file: vocab.txt\n```\n \n * Finally use [tl.nlp.Vocabulary](https://tensorlayer.readthedocs.io/en/stable/modules/nlp.html#vocabulary-class) to create a vocabulary object from the txt vocabulary file created by `tl.nlp.create_vocab`\n \n```python\n\u003e\u003e\u003e vocab = tl.nlp.Vocabulary('vocab.txt', start_word=\"\u003cS\u003e\", end_word=\"\u003c/S\u003e\", unk_word=\"\u003cUNK\u003e\")\n... INFO:tensorflow:Initializing vocabulary from file: vocab.txt\n... [TL] Vocabulary from vocab.txt : \u003cS\u003e \u003c/S\u003e \u003cUNK\u003e\n... vocabulary with 10 words (includes start_word, end_word, unk_word)\n...   start_id: 2\n...   end_id: 3\n...   unk_id: 9\n...   pad_id: 0\n```\n\nThen you can map word to ID or vice verse as follow:\n```python\n\u003e\u003e\u003e vocab.id_to_word(2)\n... 'one'\n\u003e\u003e\u003e vocab.word_to_id('one')\n... 2\n\u003e\u003e\u003e vocab.id_to_word(100)\n... '\u003cUNK\u003e'\n\u003e\u003e\u003e vocab.word_to_id('hahahaha')\n... 9\n```\n \n * More pre-processing functions for sentences in [tl.prepro](https://tensorlayer.readthedocs.io/en/stable/modules/prepro.html#sequence) and [tl.nlp](https://tensorlayer.readthedocs.io/en/stable/modules/nlp.html)\n\n## 11. Dynamic RNN and sequence length\n * Apply zero padding on a batch of tokenized sentences as follow:\n```python\n\u003e\u003e\u003e sequences = [[1,1,1,1,1],[2,2,2],[3,3]]\n\u003e\u003e\u003e sequences = tl.prepro.pad_sequences(sequences, maxlen=None, \n...         dtype='int32', padding='post', truncating='pre', value=0.)\n... [[1 1 1 1 1]\n...  [2 2 2 0 0]\n...  [3 3 0 0 0]]\n```\n\n * Use [tl.layers.retrieve_seq_length_op2](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#tensorlayer.layers.retrieve_seq_length_op2) to automatically compute the sequence length from placeholder, and feed it to the `sequence_length` of [DynamicRNNLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#dynamic-rnn-layer)\n\n```python\n\u003e\u003e\u003e data = [[1,2,0,0,0], [1,2,3,0,0], [1,2,6,1,0]]\n\u003e\u003e\u003e o = tl.layers.retrieve_seq_length_op2(data)\n\u003e\u003e\u003e sess = tf.InteractiveSession()\n\u003e\u003e\u003e tl.layers.initialize_global_variables(sess)\n\u003e\u003e\u003e print(o.eval())\n... [2 3 4]\n```\n\n * Other methods [issues18](https://github.com/tensorlayer/tensorlayer/issues/18)\n\n## 12. Save models\n\n- 1. [tl.files.save_npz](https://tensorlayer.readthedocs.io/en/stable/modules/files.html#save-network-into-list-npz) save all model parameters (weights) into a a list of array, restore using `tl.files.load_and_assign_npz`\n- 2. [tl.files.save_npz_dict](https://tensorlayer.readthedocs.io/en/stable/modules/files.html#save-network-into-dict-npz) save all model  parameters (weights) into a dictionary of array, key is the parameter name, restore  using `tl.files.load_and_assign_npz_dict`\n- 3. [tl.files.save_ckpt](https://tensorlayer.readthedocs.io/en/stable/modules/files.html#save-network-into-ckpt) save  all model parameters (weights) into TensorFlow ckpt file, restore using `tl.files.load_ckpt`.\n\n## 13. Compatibility with other TF wrappers\nTL can interact with other TF wrappers, which means if you find some codes or models implemented by other wrappers, you can just use it !\n * Other TensorFlow layer implementations can be connected into TensorLayer via [LambdaLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#lambda-layers), see example [here](https://github.com/tensorlayer/tensorlayer/tree/master/examples/keras_tfslim))\n * TF-Slim to TL: [SlimNetsLayer](https://tensorlayer.readthedocs.io/en/stable/modules/layers.html#tensorlayer.layers.SlimNetsLayer) (you can use all Google's pre-trained convolutional models with this layer !!!)\n \n## 14. Others\n * `BatchNormLayer`'s `decay` default is 0.9, set to 0.999 for large dataset.\n * Matplotlib issue arise when importing TensorLayer [issues](https://github.com/tensorlayer/tensorlayer/issues/79), see [FQA](https://tensorlayer.readthedocs.io/en/stable/user/faq.html#visualization)\n \n## Useful links\n * [Awesome-TensorLayer](https://github.com/tensorlayer/awesome-tensorlayer) for all examples\n * TL official sites: [Docs](https://tensorlayer.readthedocs.io), [中文文档](https://tensorlayercn.readthedocs.io), [Github](https://github.com/tensorlayer/tensorlayer)\n * [Learning Deep Learning with TF and TL ](https://github.com/wagamamaz/tensorflow-tutorial)\n * Follow [zsdonghao](https://github.com/zsdonghao) for further examples\n\n## Author\n - Zhang Rui\n - Hao Dong\n","funding_links":[],"categories":["Others"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwagamamaz%2Ftensorlayer-tricks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwagamamaz%2Ftensorlayer-tricks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwagamamaz%2Ftensorlayer-tricks/lists"}