https://github.com/voyrox/tensorflow-notes

https://github.com/voyrox/tensorflow-notes

Last synced: 11 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/voyrox/tensorflow-notes
• Owner: Voyrox
• Created: 2024-05-25T06:03:06.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2024-05-25T06:39:04.000Z (about 2 years ago)
• Last Synced: 2025-03-09T11:39:46.704Z (over 1 year ago)
• Size: 13.7 KB
• Stars: 4
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Understanding Neural Network Layers in TensorFlow.js

TensorFlow.js (tfjs) provides a comprehensive set of neural network layers for building and training models in . Each layer serves a specific purpose in the network architecture, from simple linear transformations to complex activation functions. Here is an in-depth explanation of various neural network layers available in tfjs along with code examples to demonstrate their usage.

## Table of Contents

- [Understanding Neural Network Layers in TensorFlow.js](#understanding-neural-network-layers-in-tensorflowjs)

  - [Table of Contents](#table-of-contents)

  - [Dense Layer](#dense-layer)

    - [Description](#description)

    - [Key Parameters](#key-parameters)

    - [Example](#example)

  - [Activation Layer](#activation-layer)

    - [Descirption](#descirption)

    - [Key Parameters](#key-parameters-1)

    - [Activation Functions](#activation-functions)

      - [ELU (Exponential Linear Unit)](#elu-exponential-linear-unit)

      - [Exponential](#exponential)

      - [GELU (Gaussian Error Linear Unit)](#gelu-gaussian-error-linear-unit)

      - [Hard Sigmoid](#hard-sigmoid)

      - [Hard SiLU (Hard Swish)](#hard-silu-hard-swish)

      - [Leaky ReLU](#leaky-relu)

      - [Linear](#linear)

      - [Log Softmax](#log-softmax)

      - [Mish](#mish)

      - [ReLU (Rectified Linear Unit)](#relu-rectified-linear-unit)

      - [ReLU6](#relu6)

      - [SELU (Scaled Exponential Linear Unit)](#selu-scaled-exponential-linear-unit)

      - [Sigmoid](#sigmoid)

      - [SiLU (Swish)](#silu-swish)

      - [Softmax](#softmax)

      - [Softplus](#softplus)

      - [Softsign](#softsign)

      - [Tanh (Hyperbolic Tangent)](#tanh-hyperbolic-tangent)

    - [Example](#example-1)

  - [Dropout Layer](#dropout-layer)

    - [Description](#description-1)

  - [Key Parameters](#key-parameters-2)

    - [Example](#example-2)

  - [Conv2D Layer](#conv2d-layer)

    - [Description](#description-2)

    - [Key Parameters](#key-parameters-3)

    - [Example](#example-3)

  - [MaxPooling2D Layer](#maxpooling2d-layer)

    - [Description](#description-3)

    - [Key Parameters](#key-parameters-4)

    - [Example](#example-4)

  - [Flatten Layer](#flatten-layer)

    - [Description](#description-4)

    - [Example](#example-5)

  - [LSTM Layer](#lstm-layer)

    - [Description](#description-5)

    - [Key Parameters](#key-parameters-5)

    - [Example](#example-6)

  - [GRU Layer](#gru-layer)

    - [Description](#description-6)

    - [Key Parameters](#key-parameters-6)

    - [Example](#example-7)

  - [BatchNormalization Layer](#batchnormalization-layer)

    - [Description](#description-7)

    - [Example](#example-8)

    - [Example Model](#example-model)

  - [Model Compilation](#model-compilation)

    - [Optimizers](#optimizers)

      - [Adam](#adam)

      - [sgd](#sgd)

      - [rmsprop](#rmsprop)

      - [categoricalCrossentropy](#categoricalcrossentropy)

      - [binaryCrossentropy](#binarycrossentropy)

      - [meanSquaredError](#meansquarederror)

      - [accuracy](#accuracy)

      - [precision](#precision)

      - [recall](#recall)

    - [Output of Predictions](#output-of-predictions)

## Dense Layer

### Description

The `Dense` layer is a fully connected layer, meaning each neuron in the layer is connected to every neuron in the previous layer. This layer is often used in the output layer of a model.

### Key Parameters

- `units`: Number of neurons in the layer.

- `activation`: Activation function to use. If you don't specify anything, no activation is applied.

### Example

```js

const tf = require('@tensorflow/tfjs');

// Define a simple model with one dense layer

const model = tf.sequential();

model.add(tf.layers.dense({

  units: 5,

  inputShape: [10],

  activation: 'relu'

}));

// Print the model summary

model.summary();

```

## Activation Layer

### Descirption

The Activation layer applies an activation function to an output. Activation functions introduce non-linearities into the model, allowing it to learn more complex patterns.

### Key Parameters

- `activation`: The activation function to apply (e.g., `relu`, `sigmoid`, `softmax`, etc.).

### Activation Functions

Activation functions are crucial in neural networks as they introduce non-linearities, allowing the network to learn complex patterns. Below is a list of activation functions provided by Keras:

- **deserialize(...)**: Return a Keras activation function via its config.

- **elu(...)**: Exponential Linear Unit.

- **exponential(...)**: Exponential activation function.

- **gelu(...)**: Gaussian error linear unit (GELU) activation function.

- **get(...)**: Retrieve a Keras activation function via an identifier.

- **hard_sigmoid(...)**: Hard sigmoid activation function.

- **hard_silu(...)**: Hard SiLU activation function, also known as Hard Swish.

- **hard_swish(...)**: Hard SiLU activation function, also known as Hard Swish.

- **leaky_relu(...)**: Leaky relu activation function.

- **linear(...)**: Linear activation function (pass-through).

- **log_softmax(...)**: Log-Softmax activation function.

- **mish(...)**: Mish activation function.

- **relu(...)**: Applies the rectified linear unit activation function.

- **relu6(...)**: Relu6 activation function.

- **selu(...)**: Scaled Exponential Linear Unit (SELU).

- **serialize(...)**: Serialize an activation function to its config.

- **sigmoid(...)**: Sigmoid activation function.

- **silu(...)**: Swish (or Silu) activation function.

- **softmax(...)**: Softmax converts a vector of values to a probability distribution.

- **softplus(...)**: Softplus activation function.

- **softsign(...)**: Softsign activation function.

- **swish(...)**: Swish (or Silu) activation function.

- **tanh(...)**: Hyperbolic tangent activation function.

Here are detailed descriptions of three commonly used activation functions:

#### ELU (Exponential Linear Unit)

Defined as:

 

$$ f(x) = \begin{cases} 

x & \text{if } x > 0 \\

\alpha (e^x - 1) & \text{if } x \leq 0 

\end{cases} $$

- **Pros**: Helps with the vanishing gradient problem and can produce negative outputs which shift the mean of the activations closer to zero.

- **Cons**: More computationally expensive than ReLU.

#### Exponential

Defined as:

 

$$ f(x) = e^x $$

- **Pros**: Can be useful in specific applications where exponential growth is needed.

- **Cons**: Can lead to very large values, causing numerical instability.

#### GELU (Gaussian Error Linear Unit)

Defined as:

 

$$ f(x) = x \cdot \Phi(x) $$

where \( \Phi(x) \) is the cumulative distribution function of the standard normal distribution.

- **Pros**: Smooth approximation, often outperforms ReLU and ELU in practice.

- **Cons**: Computationally more complex than ReLU and ELU.

#### Hard Sigmoid

Defined as:

 

$$ f(x) = \max(0, \min(1, 0.2x + 0.5)) $$

- **Pros**: Computationally simpler than the standard sigmoid.

- **Cons**: Not differentiable everywhere.

#### Hard SiLU (Hard Swish)

Defined as:

 

$$ f(x) = x \cdot \max(0, \min(1, 0.2x + 0.5)) $$

- **Pros**: Provides a non-linearity similar to Swish but computationally cheaper.

- **Cons**: Not differentiable everywhere.

#### Leaky ReLU

Defined as:

 

$$ f(x) = \begin{cases} 

x & \text{if } x > 0 \\

\alpha x & \text{if } x \leq 0 

\end{cases} $$

- **Pros**: Allows a small, non-zero gradient when the unit is inactive, preventing dead neurons.

- **Cons**: The slope of the negative part must be set manually.

#### Linear

Defined as:

 

$$ f(x) = x $$

- **Pros**: Simple and computationally efficient.

- **Cons**: Does not introduce any non-linearity, limiting the network's capacity to learn complex patterns.

#### Log Softmax

$$ f(x_i) = \log\left(\frac{e^{x_i}}{\sum_{j} e^{x_j}}\right) $$

- **Pros**: Converts logits into log-probabilities, useful for numerical stability in classification tasks.

- **Cons**: Computationally expensive for a large number of classes.

#### Mish

Defined as:

 

$$ f(x) = x \cdot \tanh(\ln(1 + e^x)) $$

- **Pros**: Smooth and non-monotonic, can lead to better performance than ReLU.

- **Cons**: More computationally expensive than ReLU.

#### ReLU (Rectified Linear Unit)

Defined as:

 

$$ f(x) = \max(0, x) $$

- **Pros**: Simple and effective, helps mitigate the vanishing gradient problem.

- **Cons**: Can lead to dead neurons if a neuron always outputs a negative value.

#### ReLU6

Defined as:

 

$$ f(x) = \min(\max(0, x), 6) $$

- **Pros**: Bounded activation to avoid overflow, commonly used in mobile and embedded vision applications.

- **Cons**: Can still lead to dead neurons.

#### SELU (Scaled Exponential Linear Unit)

Defined as:

 

$$ f(x) = \lambda \begin{cases} 

x & \text{if } x > 0 \\

\alpha (e^x - 1) & \text{if } x \leq 0 

\end{cases} $$

- **Pros**: Self-normalizing properties, can lead to faster convergence.

- **Cons**: Requires careful initialization and a specific architecture.

#### Sigmoid

Defined as:

 

$$ f(x) = \frac{1}{1 + e^{-x}} $$

- **Pros**: Outputs values between 0 and 1, useful for binary classification problems.

- **Cons**: Can cause the vanishing gradient problem and has slow convergence.

#### SiLU (Swish)

Defined as:

 

$$ f(x) = x \cdot \sigma(x) $$

where \( \sigma(x) \) is the sigmoid function.

- **Pros**: Smooth and non-monotonic, often outperforms ReLU in practice.

- **Cons**: More computationally expensive than ReLU.

#### Softmax

$$ f(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}} $$

- **Pros**: Converts logits into probabilities, making it useful for multi-class classification problems.

- **Cons**: Computationally expensive for a large number of classes.

#### Softplus

Defined as:

 

$$ f(x) = \ln(1 + e^x) $$

- **Pros**: Smooth and differentiable, helps mitigate the vanishing gradient problem.

- **Cons**: More computationally expensive than ReLU.

#### Softsign

Defined as:

 

$$ f(x) = \frac{x}{1 + |x|} $$

- **Pros**: Smooth and computationally efficient.

- **Cons**: Can lead to slow convergence.

#### Tanh (Hyperbolic Tangent)

Defined as:

 

$$ f(x) = \tanh(x) $$

- **Pros**: Outputs values between -1 and 1, useful for zero-centered data.

- **Cons**: Can cause the vanishing gradient problem.

### Example

```js

// Add an activation layer to the model

model.add(tf.layers.activation({activation: 'relu'}));

```

## Dropout Layer

### Description

The Dropout layer randomly sets a fraction of input units to 0 at each update during training time, which helps prevent overfitting.

## Key Parameters

- `rate`: Fraction of the input units to drop (a float between 0 and 1).

### Example

```js

// Add a dropout layer with 50% dropout rate

model.add(tf.layers.dropout({rate: 0.5}));

```

## Conv2D Layer

### Description

The Conv2D layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs. It's used in Convolutional Neural Networks (CNNs) for image processing.

### Key Parameters

- `filters`: Number of output filters.

- `kernelSize`: Size of the convolution kernel.

- `strides`: Strides of the convolution.

- `activation`: Activation function to use.

### Example

```js

// Add a 2D convolutional layer

model.add(tf.layers.conv2d({

  inputShape: [28, 28, 1],

  filters: 32,

  kernelSize: 3,

  activation: 'relu'

}));

```

## MaxPooling2D Layer

### Description

The MaxPooling2D layer downsamples the input along its spatial dimensions (height and width) by taking the maximum value over an input window (of size defined by poolSize).

### Key Parameters

- `poolSize`: Size of the max pooling windows.

- `strides`: Strides of the pooling operation.

### Example

```js

// Add a max pooling layer

model.add(tf.layers.maxPooling2d({poolSize: [2, 2]}));

```

## Flatten Layer

### Description

The Flatten layer flattens the input, i.e., it converts a multi-dimensional input into a one-dimensional tensor. This is often used to transition from convolutional layers to dense layers.

### Example

```js

// Add a flatten layer

model.add(tf.layers.flatten());

```

## LSTM Layer

### Description

The LSTM layer is a type of recurrent neural network (RNN) layer that is useful for sequence prediction problems.

### Key Parameters

- `units`: Number of LSTM units.

- `activation`: Activation function to use.

### Example

```js

// Add an LSTM layer

model.add(tf.layers.lstm({

  units: 50,

  inputShape: [10, 20]

}));

```

## GRU Layer

### Description

The GRU (Gated Recurrent Unit) layer is another type of RNN layer similar to LSTM but with fewer parameters.

### Key Parameters

- `units`: Number of GRU units.

- `activation`: Activation function to use.

### Example

```js

// Add a GRU layer

model.add(tf.layers.gru({

  units: 50,

  inputShape: [10, 20]

}));

```

## BatchNormalization Layer

### Description

The BatchNormalization layer normalizes the activations of the previous layer at each batch, i.e., applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1.

### Example

```js

// Add a batch normalization layer

model.add(tf.layers.batchNormalization());

```

### Example Model

Here is a complete example that combines several of these layers to build a simple model for image classification:

```js

const model = tf.sequential();

model.add(tf.layers.conv2d({

  inputShape: [28, 28, 1],

  filters: 32,

  kernelSize: 3,

  activation: 'relu'

}));

model.add(tf.layers.maxPooling2d({poolSize: [2, 2]}));

model.add(tf.layers.flatten());

model.add(tf.layers.dense({units: 128, activation: 'relu'}));

model.add(tf.layers.dropout({rate: 0.5}));

model.add(tf.layers.dense({units: 10, activation: 'softmax'}));

model.compile({

  optimizer: 'adam',

  loss: 'categoricalCrossentropy',

  metrics: ['accuracy']

});

// Print the model summary

model.summary();

```

This example defines a simple convolutional neural network (CNN) for classifying images. It includes convolutional layers, max pooling, dropout for regularization, and dense layers with a final softmax activation for classification.

## Model Compilation

The `model.compile` method configures the model for training. It specifies the optimizer, loss function, and metrics used to evaluate the model. Below are different configurations you can use with `model.compile`.

### Optimizers

#### Adam

```js

model.compile(optimizer='adam', loss='categoricalCrossentropy', metrics=['accuracy'])

```

- **Pros:** Adaptive learning rate, works well on most problems, faster convergence.

- **Cons:** Can be computationally expensive, might overfit on small datasets.

#### sgd

```js

model.compile(optimizer='sgd', loss='categoricalCrossentropy', metrics=['accuracy'])

```

- **Pros:** Simple and easy to implement, often performs well on large datasets.

- **Cons:** Requires careful tuning of learning rate, slower convergence compared to adaptive methods.

#### rmsprop

```js

model.compile(optimizer='rmsprop', loss='categoricalCrossentropy', metrics=['accuracy'])

```

- **Pros:** Adapts learning rate based on average of recent magnitudes, good for recurrent neural networks.

- **Cons:** Learning rate needs to be carefully tuned, might not perform well on simple tasks.

Loss Functions

#### categoricalCrossentropy

```js

model.compile(optimizer='adam', loss='categoricalCrossentropy', metrics=['accuracy'])

```

- **Pros:** Suitable for multi-class classification problems.

- **Cons:** Can be numerically unstable if not implemented properly.

#### binaryCrossentropy

```js

model.compile(optimizer='adam', loss='binaryCrossentropy', metrics=['accuracy'])

```

- **Pros:** Suitable for binary classification problems.

- **Cons:** Not suitable for multi-class classification problems.

#### meanSquaredError

```js

model.compile(optimizer='adam', loss='meanSquaredError', metrics=['accuracy'])

```

- **Pros:** Suitable for regression problems.

- **Cons:** Not suitable for classification problems.

Metrics

#### accuracy

```js

model.compile(optimizer='adam', loss='categoricalCrossentropy', metrics=['accuracy'])

```

- **Pros:** Easy to interpret, widely used in classification problems.

- **Cons:** Can be misleading if the dataset is imbalanced.

#### precision

```js

model.compile(optimizer='adam', loss='categoricalCrossentropy', metrics=['precision'])

```

- **Pros:** Useful for classification problems with imbalanced classes.

- **Cons:** Can be misleading if the dataset has many classes.

#### recall

```js

model.compile(optimizer='adam', loss='categoricalCrossentropy', metrics=['recall'])

```

- **Pros:** Useful for classification problems with imbalanced classes.

- **Cons:** Can be misleading if the dataset has many classes.

### Output of Predictions

When making predictions using a model compiled with `categoricalCrossentropy` loss, the output will be in the form of probabilities for each class. For example, if you have a model predicting three classes, the output might look like this:

```js

let predictions = model.predict(x)

console.log(predictions)

// Output: [[0.1, 0.7, 0.2], [0.3, 0.4, 0.3], ...]

```

Each value represents the probability of the corresponding class.

In contrast, if you use `binaryCrossentropy` for binary classification, the output will be a single probability for each input, representing the likelihood of the positive class:

```js

let predictions = model.predict(x)

console.log(predictions)

// Output: [0.8, 0.3, ...]

```

For regression problems using meanSquaredError, the output will be the actual predicted values:

```js

let predictions = model.predict(x)

console.log(predictions)

// Output: [3.4, 2.1, ...]

```

Understanding the different configurations of `model.compile` can help you tailor your model to better suit your specific problem and dataset.