https://github.com/andrewdarnall/deeplearning4all
Pet project to fall in love with C++ again
https://github.com/andrewdarnall/deeplearning4all
cpp deep-learning
Last synced: 9 months ago
JSON representation
Pet project to fall in love with C++ again
- Host: GitHub
- URL: https://github.com/andrewdarnall/deeplearning4all
- Owner: AndrewDarnall
- License: mit
- Created: 2025-07-13T20:29:58.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-07-13T20:37:19.000Z (12 months ago)
- Last Synced: 2025-07-13T22:22:16.902Z (12 months ago)
- Topics: cpp, deep-learning
- Homepage:
- Size: 3.91 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Project: Deep Learning 4 All
The goal of the project is that of creating a GPU library, hardware-agnostic in order to further democratize Deep Learning and Artificial Intelligence allowing for ***anyone*** to leverage whatever parallel compute
they possess, be it an integrated GPU in a CPU, or an older AMD GPU that is not included in the ROCm family.
Then this library will also wrap up with the PyTorch (and perhaps TensorFlow some day) Python libraries in order to fully leverage the power of *parallel computation* with ***no additional cost** and ***without vendor locking***
## Main Roadmap
🛠️ What you could do next (if serious)
Here’s how you might start:
🔹 Phase 1: Proof of Concept
• Implement a Tensor class in C++ that wraps OpenCL buffers
• Build a simple add, matmul, conv2d kernel
• Create a PyTorch C++ Extension that routes a few ops to your OpenCL backend
🔹 Phase 2: Add device/runtime manager
• Implement device registration, memory allocator, simple scheduler
• Support both CPU + GPU compute fallbacks
• Explore OpenCL peer-to-peer support or zero-copy
🔹 Phase 3: Distributed runtime
• Design message passing over network (gRPC or MPI)
• Split model and gradients
• Implement async parameter sync (ala DDP, Horovod)
---
## Prerequisite Roadmap
🧠 1. Math & Theoretical Foundations
Key Topics:
• Linear algebra (matrices, tensors, eigenvalues)
• Multivariate calculus
• Optimization (SGD, Adam, backpropagation)
• Numerical stability (log-sum-exp, float16 underflow/overflow)
Exercises:
✅ Implement forward and backward pass for:
• Matrix multiplication
• Convolution (no libraries!)
• Sigmoid, ReLU, Softmax, CrossEntropy
🛠 Mini-Project:
Write a simple autodiff engine (like tinygrad or micrograd) in Python from scratch.
⸻
🔧 2. Systems Programming
Key Topics:
• Pointers, memory allocation (heap vs. stack)
• Multi-threading and concurrency
• File I/O, networking sockets
• CMake, build systems
• Shared libraries (e.g., .so vs .dll)
• SIMD/vectorization (SSE, AVX)
Exercises:
✅ Write:
• A memory allocator in C
• A thread pool
• A TCP server/client
• A shared object that exports a C function to Python using ctypes or cffi
🛠 Mini-Project:
A small C++ matrix library with SIMD optimizations and multithreaded matmul.
⸻
⚙️ 3. GPU Programming (OpenCL & CUDA)
Key Topics:
• GPU memory hierarchy (global, shared, local, constant)
• Thread blocks / warps / execution model
• Kernel launches, synchronization
• Buffer transfers (host ⇄ device)
• Profiling & performance tuning
Exercises:
✅ Write in OpenCL or CUDA:
• Vector add
• Matmul
• 2D convolution
• Max pooling
🛠 Mini-Project:
Implement ResNet-18 inference on OpenCL using hand-written kernels.
⸻
🧱 4. Deep Learning Engine Internals
Key Topics:
• Tensor data structures
• Autograd graph
• Static vs. dynamic graph execution (e.g., TF vs PyTorch)
• Memory-efficient backprop
• Operator fusion, graph optimization
Exercises:
✅ Implement:
• A graph representation for ops
• Forward/backward execution
• Intermediate result reuse & buffer pooling
🛠 Mini-Project:
Write a mini DL framework in C++ with OpenCL backend support.
⸻
📦 5. Interfacing with PyTorch
Key Topics:
• PyTorch’s ATen (tensor library)
• Custom C++ extensions
• Binding C++ to Python with PyBind11
• Registering custom backends (device hooks, dispatchers)
Exercises:
✅ Use:
• torch::Tensor in C++
• Write a C++ op and call it from Python
🛠 Mini-Project:
Add a custom backend to PyTorch that routes add, matmul, and conv to OpenCL.
⸻
📡 6. Distributed Training
Key Topics:
• Gradient averaging (all-reduce, ring-reduce)
• Parameter servers
• Communication libraries (MPI, gRPC)
• Fault tolerance
Exercises:
✅ Implement:
• Data parallel SGD
• Simple parameter server over gRPC
• AllReduce with socket comms
🛠 Mini-Project:
Train MNIST using multiple processes over networked GPUs.
⸻
🧰 7. Tooling / DevOps / Testing
Key Topics:
• CMake, Bazel
• CI/CD pipelines (GitHub Actions, GitLab)
• Unit testing (GoogleTest, Catch2)
• Profiling (valgrind, perf, nvprof, clinfo)
• Containerization (Docker)
Exercises:
✅ Build:
• A Dockerized C++ inference service
• A GitHub Actions pipeline to test C++ OpenCL kernels
🛠 Mini-Project:
CI pipeline for building, testing, and benchmarking your OpenCL DL backend.
---
Yours truly, @TheComputerScientist
---