https://github.com/imaddabbura/tiny-pytorch
Tiny Deep Learning framework similar to PyTorch.
https://github.com/imaddabbura/tiny-pytorch
autograd deep-learning deep-learning-framework deep-learning-system gpu machine-learning neural-network python pytorch pytorch-implementation tensor
Last synced: 7 months ago
JSON representation
Tiny Deep Learning framework similar to PyTorch.
- Host: GitHub
- URL: https://github.com/imaddabbura/tiny-pytorch
- Owner: ImadDabbura
- License: apache-2.0
- Created: 2022-12-15T16:41:39.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-07-29T20:11:58.000Z (8 months ago)
- Last Synced: 2025-08-30T12:07:20.217Z (7 months ago)
- Topics: autograd, deep-learning, deep-learning-framework, deep-learning-system, gpu, machine-learning, neural-network, python, pytorch, pytorch-implementation, tensor
- Language: Python
- Homepage: https://imaddabbura.github.io/tiny-pytorch/
- Size: 1.28 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Tiny-PyTorch 🧠
## **Unravel the magic of modern deep learning by building a PyTorch-like framework from the ground up.**
[](https://www.python.org/downloads/)
[](https://pypi.org/project/tiny-pytorch/)
[](https://pypi.org/project/tiny-pytorch/)
[](https://codecov.io/gh/imaddabbura/tiny-pytorch)
[](https://github.com/psf/black)
Tiny-PyTorch is an educational deep learning framework built entirely in Python. It demystifies the core machinery of libraries like PyTorch by providing a clean, focused, and from-scratch implementation of the essential components.
---
## Philosophy: Understanding by Building
The best way to truly understand how complex systems work is to build them yourself. Tiny-PyTorch is born from this philosophy. While production frameworks like PyTorch and TensorFlow provide powerful, high-level abstractions, their internal complexity can be a barrier to learning.
This project strips away those abstractions, allowing you to:
- **See the Core Logic:** Grasp the fundamental algorithms and data structures that power deep learning, from the `Tensor` object to the backpropagation process.
- **Connect Theory to Code:** Bridge the gap between the mathematical concepts of deep learning and their concrete implementation.
- **Become a Better Practitioner:** Use high-level frameworks more effectively by understanding their internal mechanics, performance trade-offs, and potential pitfalls.
---
## ✨ Core Features
- **Dynamic Computation Graph:** Tensors track their history, allowing for flexible model architectures.
- **Reverse-Mode Automatic Differentiation:** An efficient gradient calculation engine (`autograd`) built from scratch.
- **Extensible `nn.Module` System:** A familiar API for building complex neural network layers and models.
- **Standard Optimizers:** Implementations of `SGD` and `Adam` to handle parameter updates.
- **Hardware Acceleration:** A pluggable backend system supporting `NumPy`, custom `CPU` (C++), and `CUDA` (GPU) operations.
- **Data Loading Utilities:** `Dataset` and `DataLoader` classes for efficient data pipelines.
---
## 🏗️ Project Architecture
The framework is built in a bottom-up fashion, where each layer of abstraction relies on the one below it. This mirrors the logical structure of major deep learning libraries.
```mermaid
graph TD
subgraph "High-Level API"
C[nn.Module] --> D[Optimizers]
B[Tensors & Autograd] --> C
end
subgraph "Low-Level Engine"
A[NDArray] --> B
subgraph "Backends"
direction LR
np[NumPy]
cpu[CPU]
gpu[CUDA]
end
Backend --> A
end
style C fill:#f9f,stroke:#333,stroke-width:2px
style B fill:#ccf,stroke:#333,stroke-width:2px
style A fill:#cfc,stroke:#333,stroke-width:2px
```
1. **Backends (NumPy, CPU, CUDA):** Perform the actual mathematical computations on flat arrays of data.
2. **NDArray:** A generic, strided N-dimensional array class that provides a unified interface over different backends.
3. **Tensor & Autograd:** The heart of the framework. A `Tensor` wraps an `NDArray` and builds a dynamic computation graph. The `autograd` engine traverses this graph to perform reverse-mode automatic differentiation.
4. **High-Level API (`nn`, `optimizer`):** Provides the familiar modules, layers, and optimization algorithms for building and training neural networks.
---
## 🚀 Quick Start
To install Tiny-PyTorch, you have two main options:
### Install from PyPI (using pip)
You can install the latest stable version directly from PyPI using pip:
```bash
pip install tiny-pytorch
```
### Install from Source (GitHub Repository)
To get the very latest development version or if you plan to contribute, you can install from the GitHub repository:
1. **Clone the repository:**
```bash
git clone https://github.com/your-username/tiny-pytorch.git
```
2. **Navigate to the project directory:**
```bash
cd tiny-pytorch
```
3. **Install in editable mode:** This allows you to make changes to the source code and have them reflected without reinstalling.
```bash
pip install -e .
```
Here's a simple example of defining a model and running a forward/backward pass.
```python
import tiny_pytorch as tp
import tiny_pytorch.nn as nn
# 1. Define a simple model
class SimpleNet(nn.Module):
def __init__(self, in_features, out_features):
self.fc1 = nn.Linear(in_features, 64)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(64, out_features)
def forward(self, x):
x = self.fc1(x)
x = self.relu(x)
return self.fc2(x)
# 2. Initialize model, optimizer, and loss function
model = SimpleNet(in_features=10, out_features=1)
optimizer = tp.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.MSELoss()
# 3. Create dummy data
x_train = tp.randn(32, 10, requires_grad=True)
y_true = tp.randn(32, 1)
# 4. Perform a single training step
optimizer.zero_grad() # Reset gradients
y_pred = model(x_train) # Forward pass
loss = loss_fn(y_pred, y_true) # Compute loss
loss.backward() # Backward pass (autograd)
optimizer.step() # Update weights
print(f"Loss: {loss.item():.4f}")
```
---
## 🗺️ Roadmap
The project is developed in two main phases. Our current progress is tracked below.
- **Phase I: Core Framework (NumPy Backend)**
- [x] `Tensor`: The main multi-dimensional array with autograd support.
- [x] `Op`: The base class for all tensor operations.
- [x] `Automatic Differentiation`: Reverse-mode autograd engine.
- [x] `init`: Parameter initialization functions (`kaiming`, `xavier`, etc.).
- [x] `nn`: Core neural network layers (`Linear`, `ReLU`, `BatchNorm`, `Conv2d`).
- [x] `optimizer`: `SGD` and `Adam` optimizers.
- [x] `data`: `Dataset` and `DataLoader` for data handling.
- **Phase II: Hardware Acceleration & Advanced Models**
- [x] `NDArray`: Generic, strided N-dimensional array.
- [x] NumPy Backend
- [x] CPU Backend (C++)
- [x] CUDA Backend (GPU)
- [x] Advanced CNN operations (e.g., `padding`, `dilation`).
- [x] ResNet implementation.
- [x] RNN and LSTM layers.
- [x] A simple Language Model.
---
## 📚 Documentation
The official documentation, including detailed API references and tutorials, is hosted at:
**[https://imaddabbura.github.io/tiny-pytorch/](https://imaddabbura.github.io/tiny-pytorch/)**
---
## ⚠️ Limitations
As an educational project, Tiny-PyTorch has some intentional simplifications:
- **Explicit Broadcasting:** Broadcasting for element-wise operations must be done manually if tensor shapes do not match.
- **Single Data Type:** `NDArray` only supports the `float32` `dtype`.
- **Contiguous Memory:** Operations on the underlying 1D array require a call to `compact()` to ensure the data is in a contiguous memory block.
- **Limited Reductions:** Reduction operations (e.g., `sum`, `max`) can only be performed on a single axis or all axes at once.
---
## License
Tiny-PyTorch is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.