{"id":20166033,"url":"https://github.com/pabannier/nanograd","last_synced_at":"2025-04-10T01:20:27.259Z","repository":{"id":57444816,"uuid":"323687644","full_name":"PABannier/nanograd","owner":"PABannier","description":"A lightweight deep learning framework","archived":false,"fork":false,"pushed_at":"2021-02-28T22:07:47.000Z","size":2510,"stargazers_count":32,"open_issues_count":3,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T03:01:41.177Z","etag":null,"topics":["convolutions","deep-learning","neural-networks"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PABannier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-22T17:11:42.000Z","updated_at":"2024-06-01T12:07:03.000Z","dependencies_parsed_at":"2022-09-26T17:30:34.821Z","dependency_job_id":null,"html_url":"https://github.com/PABannier/nanograd","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PABannier%2Fnanograd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PABannier%2Fnanograd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PABannier%2Fnanograd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PABannier%2Fnanograd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PABannier","download_url":"https://codeload.github.com/PABannier/nanograd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248138098,"owners_count":21053810,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutions","deep-learning","neural-networks"],"created_at":"2024-11-14T00:42:26.164Z","updated_at":"2025-04-10T01:20:27.229Z","avatar_url":"https://github.com/PABannier.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  nanograd\n  \u003cbr\u003e\n\u003c/h1\u003e\n\n\u003ch4 align=\"center\"\u003eA lightweight deep learning framework.\u003c/h4\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/badge.svg\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#description\"\u003eDescription\u003c/a\u003e •\n  \u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e •\n  \u003ca href=\"#todo\"\u003eTODO\u003c/a\u003e •\n  \u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\n\u003c/p\u003e\n\n\n## Description\n\nAfter verification, nanograd is not a city in Russia...\n\nHowever, it is a PyTorch-like lightweight deep learning framework. Use it to implement any DL algo you want with little boilerplate code.\n\nEssentially, Nanograd is a continuously updated project. The goal is to implement as many features as possible while using as few abstraction layers as possible (only Numpy functions are allowed). Any contribution to the repo is welcome.\n\nThe library has a built-in auto-differentiation engine that dynamically builds a computational graph. The framework is built with basic features to train neural nets: basic ops, layers, weight initializers, optimizers and loss functions. Additional tools are developed to visualize your network: computational\ngraph visualizers or activation map visualizers (SOON!).\n\nThe repo will be updated regularly with new features and examples. \n\nInspired from \u003ca href=\"https://github.com/geohot/tinygrad\"\u003egeohot's tinygrad\u003c/a\u003e.\n\n\n## Features\n\n- PyTorch-like autodifferentiation engine (dynamically constructed computational graph)\n- Weight initialization: Glorot uniform, Glorot normal, Kaiming uniform, Kaiming normal\n- Activations: ReLU, Sigmoid, tanh, Swish, ELU, LeakyReLU\n- Convolutions: Conv1d, Conv2d, MaxPool2d, AvgPool2d\n- Layers: Linear, BatchNorm1d, BatchNorm2d, Flatten, Dropout\n- Optimizers: SGD, Adam, AdamW\n- Loss: CrossEntropyLoss, Mean squared error\n- Computational graph visualizer (see example)\n\n### A quick side-by-side comparison between PyTorch and Nanograd for tensor computations\n\n#### Basic tensor calculations\n\n**PyTorch**\n\n```python\na = torch.empty((30, 30, 2))\n         .normal_(mean=3, std=4)\nb = torch.empty((30, 30, 1))\n         .normal_(mean=10, std=2)\n\na.requires_grad = True\nb.requires_grad = True\n\nc = a + b\nd = c.relu()\ne = c.sigmoid()\nf = d * e\n\nf.sum().backward()\n\nprint(a.grad)\nprint(b.grad)\n```\n\n**Nanograd**\n\n```python\na = Tensor.normal(3, 4, (30, 30, 2), requires_grad=True)\nb = Tensor.normal(10, 2, (30, 30, 1), requires_grad=True)\n\nc = a + b\nd = c.relu()\ne = c.sigmoid()\nf = d * e\n\nf.backward()\n\nprint(a.grad)\nprint(b.grad)\n```\n\n\n### Training a CNN on MNIST\n\n```python\n\n# Model, loss \u0026 optim\nmodel = CNN()\nloss_function = CrossEntropyLoss()\noptim = SGD(model.parameters(), lr=0.01, momentum=0)\n\n# Training loop\nBS = 128\nlosses, accuracies = [], []\nSTEPS = 1000\n\nfor i in tqdm(range(STEPS), total=STEPS):\n  samp = np.random.randint(0, X_train.shape[0], size=(BS))\n  X = tensor.Tensor(X_train[samp])\n  Y = tensor.Tensor(Y_train[samp])\n\n  optim.zero_grad()\n\n  out = model(X)\n\n  cat = out.data.argmax(1)\n  accuracy = (cat == Y.data).mean()\n\n  loss = loss_function(out, Y)\n  loss.backward()\n\n  optim.step()\n\n  loss, accuracy = float(loss.data), float(accuracy)\n  losses.append(loss)\n  accuracies.append(accuracy)\n\nY_test_preds = model(tensor.Tensor(X_test)).data.argmax(1)\nprint((Y_test == Y_test_preds).mean())\n\n```\n\n\n### Visualizing a computational graph \n\nVisualizing a computational graph has never been that easy. Just call **plot_forward** and **plot_backward**.\n\n```python\nf.plot_forward()\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/forward_graph.png\"\u003e\n\u003c/p\u003e\n\n\n```python\nf.plot_backward()\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/backward_graph.png\"\u003e\n\u003c/p\u003e\n\n## TODO\n\n- Solve batchnorm issues\n- Add GRU, LSTM cells\n- Code example with EfficientNet-B0, CIFAR-10, MNIST\n- Code a transformer with Nanograd and train it on GPU\n\n\n## License\n\nMIT\n\n---\n\n\u003e GitHub [@PABannier](https://github.com/PABannier) \u0026nbsp;\u0026middot;\u0026nbsp;\n\u003e Twitter [@el_PA_B](https://twitter.com/el_PA_B)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpabannier%2Fnanograd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpabannier%2Fnanograd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpabannier%2Fnanograd/lists"}