https://github.com/thevilledev/zig-nn
A lightweight neural network library implemented in Zig
https://github.com/thevilledev/zig-nn
learning neural-networks zig
Last synced: over 1 year ago
JSON representation
A lightweight neural network library implemented in Zig
- Host: GitHub
- URL: https://github.com/thevilledev/zig-nn
- Owner: thevilledev
- License: mit
- Created: 2025-03-13T19:51:10.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-13T20:15:11.000Z (over 1 year ago)
- Last Synced: 2025-03-13T20:37:04.556Z (over 1 year ago)
- Topics: learning, neural-networks, zig
- Language: Zig
- Homepage:
- Size: 0 Bytes
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Neural Network in Zig
[](https://github.com/thevilledev/zig-nn)
A minimalistic neural network implementation in Zig for learning purposes. This project focuses on understanding the fundamentals of neural networks including hidden layers, activation functions, and basic training processes.
## Features
- Matrix operations from scratch
- Common activation functions (Sigmoid, ReLU, Tanh)
- Advanced activation functions (Swish, GLU, SwiGLU)
- Basic feed-forward neural network architecture
- Support for gated architectures used in modern transformer models
## Testing
The project includes a comprehensive test suite to verify the functionality of all components. The build system is configured to run tests for each module separately, making it easy to identify which component has issues.
### Running Tests
```bash
# Run all tests
zig build test
# Run tests for specific components
zig build test-matrix # Run matrix operation tests
zig build test-activation # Run activation function tests
zig build test-layer # Run neural network layer tests
zig build test-network # Run full network tests
```
## Advanced Activation Functions
This library implements several advanced activation functions used in modern neural networks:
### Swish
Swish is an activation function introduced by Google Brain that often outperforms ReLU:
```
swish(x) = x * sigmoid(β * x)
```
Where β is typically set to 1.0.
### GLU (Gated Linear Unit)
GLU was introduced in "Language Modeling with Gated Convolutional Networks" and is used in many transformer architectures:
```
GLU(x, W, V, b, c) = (x·W + b) ⊗ σ(x·V + c)
```
Where ⊗ is element-wise multiplication and σ is the sigmoid function.
### SwiGLU (Swish Gated Linear Unit)
SwiGLU is a variant of GLU that uses the Swish activation function instead of sigmoid:
```
SwiGLU(x, W, V, b, c) = (x·W + b) ⊗ swish(x·V + c)
```
These gated activation functions are particularly useful in transformer architectures and have been shown to improve performance in many natural language processing tasks.
## Learning Goals
This project serves as a learning exercise for:
- Understanding neural network fundamentals
- Implementing mathematical operations in Zig
- Working with Zig's memory management and error handling
## License
MIT License