Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/cosmic-cortex/neural-networks-from-scratch
An implementation of convolutional networks in NumPy!
https://github.com/cosmic-cortex/neural-networks-from-scratch
Last synced: about 5 hours ago
JSON representation
An implementation of convolutional networks in NumPy!
- Host: GitHub
- URL: https://github.com/cosmic-cortex/neural-networks-from-scratch
- Owner: cosmic-cortex
- License: mit
- Created: 2019-03-07T11:24:41.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-04-19T13:58:33.000Z (over 3 years ago)
- Last Synced: 2024-01-06T14:46:59.941Z (9 months ago)
- Language: Python
- Homepage:
- Size: 48.8 KB
- Stars: 75
- Watchers: 4
- Forks: 15
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# neural-networks-from-scratch
# Contents
- [Quickstart](#quickstart)
- [A simple example CNN](#CNN-example)
- [The `Net` object](#net)
- [Layers](#layers)
- [`Linear`](#linear)
- [`Conv2D`](#conv2d)
- [`MaxPool2D`](#maxpool2d)
- [`BatchNorm2D`](#batchnorm2d)
- [`Flatten`](#flatten)
- [Losses](#losses)
- [`CrossEntropyLoss`](#crossentropyloss)
- [`MeanSquareLoss`](#meansquareloss)
- [Activations](#activations)
# Quickstart
## Installation
To run the examples, creating a virtual environment is recommended.
```bash
virtualenv neural-networks-from scratch
```
When a virtual environment is in place, all requirements can be installed with pip.
```bash
source neural-networks-from-scratch/bin/activate
pip install -r requirements.txt
```
## A simple example CNN
A simple convolutional network for image classification can be found in `CNN_custom_dataset.py`. To try it on your own dataset, you should prepare your images in the following format:
```bash
images_folder
|-- class_01
|-- 001.png
|-- ...
|-- class_02
|-- 001.png
|-- ...
|-- ...
```
Its required argument is
- `--dataset`: path to the dataset,
while the optional arguments are
- `--epochs`: number of epochs,
- `--batch_size`: size of the training batch,
- `--lr`: learning rate.
## The `Net` object
To define a neural network, the `nn.net.Net` object can be used. Its parameters are
* `layers`: a list of layers from `nn.layers`, for example `[Linear(2, 4), ReLU(), Linear(4, 2)]`,
* `loss`: a loss function from `nn.losses`, for example `CrossEntropyLoss` or `MeanSquareLoss`.
If you would like to train the model with data `X` and label `y`, you should
1) perform the forward pass, during which local gradients are calculated,
2) calculate the loss,
3) perform the backward pass, where global gradients with respect to the variables and layer parameters are calculated,
4) update the weights.
In code, this looks like the following:
```python3
out = net(X)
loss = net.loss(out, y)
net.backward()
net.update_weights(lr)
```
# Layers
The currently implemented layers can be found in `nn.layers`. Each layer is a callable object, where calling performs the forward pass and calculates local gradients. The most important methods are:
- `.forward(X)`: performs the forward pass for X. Instead calling `forward` directly, the layer object should be called directly, which calculates and caches local gradients.
- `.backward(dY)`: performs the backward pass, where `dY` is the gradient propagated backwards from the consequtive layer.
- `.local_grad(X)`: calculates the local gradient of the input.
The input to the layers should always be a `numpy.ndarray` of shape `(n_batch, ...)`. For the 2D layers for images, the input should have shape `(n_batch, n_channels, n_height, n_width)`.
## `Linear`
A simple fully connected layer.
Parameters:
- `in_dim`: integer, dimensions of the input.
- `out_dim`: integer, dimensions of the output.
Usage:
- input: `numpy.ndarray` of shape `(N, in_dim)`.
- output: `numpy.ndarray` of shape `(N, out_dim)`.
## `Conv2D`
2D convolutional layer. Parameters:
- `in_channels`: integer, number of channels in the input image.
- `out_channels`: integer, number of filters to be learned.
- `kernel_size`: integer or tuple, the size of the filter to be learned. Defaults to 3.
- `stride`: integer, stride of the convolution. Defaults to 1.
- `padding`: integer, number of zeros to be added to each edge of the images. Defaults to 0.
Usage:
- input: `numpy.ndarray` of shape `(N, C_in, H_in, W_in)`.
- output: `numpy.ndarray` of shape `(N, C_out, H_out, W_out)`.
## `MaxPool2D`
2D max pooling layer. Parameters:
- `kernel_size`: integer or tuple, size of the pooling window. Defaults to 2.
Usage:
- input: `numpy.ndarray` of shape `(N, C, H, W)`.
- output: `numpy.ndarray` of shape `(N, C, H//KH, W//KW)` with kernel size `(KH, KW)`.
## `BatchNorm2D`
2D batch normalization layer. Parameters:
- `n_channels`: integer, number of channels.
- `epsilon`: epsilon parameter for BatchNorm, defaults to 1e-5.
Usage:
- input: `numpy.ndarray` of shape `(N, C, H, W)`.
- output: `numpy.ndarray` of shape `(N, C, H, W)`.
## `Flatten`
A simple layer which flattens the outputs of a 2D layer for images.
Usage:
- input: `numpy.ndarray` of shape `(N, C, H, W)`.
- output: `numpy.ndarray` of shape `(N, C*H*W)`.
# Losses
The implemented loss functions are located in `nn.losses`. As Layers, they are callable objects, with predictions and targets as input.
## `CrossEntropyLoss`
Cross-entropy loss. Usage:
- input: `numpy.ndarray` of shape `(N, D)` containing the class scores for each element in the batch.
- output: `float`.
## `MeanSquareLoss`
Mean square loss. Usage:
- input: `numpy.ndarray` of shape `(N, D)`.
- output: `numpy.ndarray` of shape `(N, D)`.
# Activations
The activation layers for the network can be found in `nn.activations`. They are functions, applying the specified activation function elementwisely on a `numpy.ndarray`. Currently, the following activation functions are implemented:
- ReLU
- Leaky ReLU
- Sigmoid