{"id":20388181,"url":"https://github.com/samrere/pytortto","last_synced_at":"2025-09-04T12:43:53.561Z","repository":{"id":53938882,"uuid":"508578499","full_name":"samrere/pytortto","owner":"samrere","description":"deep learning from scratch. uses numpy/cupy, trains in GPU, follows pytorch API","archived":false,"fork":false,"pushed_at":"2023-09-05T19:59:36.000Z","size":9832,"stargazers_count":18,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-18T00:36:41.409Z","etag":null,"topics":["autograd","cupy","deep-learning","gpu","machine-learning","machine-learning-from-scratch","neural-network","neural-networks-from-scratch","numpy","numpy-neural-network"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/samrere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-29T06:49:02.000Z","updated_at":"2024-08-03T06:25:32.000Z","dependencies_parsed_at":"2022-08-13T05:10:15.694Z","dependency_job_id":null,"html_url":"https://github.com/samrere/pytortto","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrere%2Fpytortto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrere%2Fpytortto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrere%2Fpytortto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samrere%2Fpytortto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/samrere","download_url":"https://codeload.github.com/samrere/pytortto/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224552189,"owners_count":17330251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autograd","cupy","deep-learning","gpu","machine-learning","machine-learning-from-scratch","neural-network","neural-networks-from-scratch","numpy","numpy-neural-network"],"created_at":"2024-11-15T03:07:46.433Z","updated_at":"2024-11-15T03:07:47.247Z","avatar_url":"https://github.com/samrere.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyTortto\n- [Intro](#Intro)\n- [Installation](#Installation)\n- [Prerequisites](#Prerequisites)\n- [Quick start](#Quick-start)\n- [Examples](#Examples)\n  - [Resnet](#Resnet)\n  - [UNet](#UNet)\n  - [Vision Transformer](#Vision-Transformer)\n  - [DCGAN](#DCGAN)\n  - [YOLOv1](#YOLOv1)\n\n\n## Intro\nThis is a pytorch style machine learning framework implemented entirely in numpy, with GPU acceleration from cupy.\n\nSimilar to the pokemon \"ditto\", pytortto works exactly like pytorch, although inferior in terms of speed. The purpose of this project is to understand how deep learning algorithms and frameworks like pytorch work under the hood. Max effort was given to correctness, calculation efficiency (like simpler Jacobian in logsoftmax, efficient implementation of convolution etc.), numerical stability (log-sum-exp used in logsigmoid, BCEWithLogitsLoss etc.), and memory efficiency (implementation of caching, view etc.).  \n\nWhen computed in GPU, Tortto is around 1.5(vision transformers) ~ 3(CNNs) times slower than pytorch. It also achieves the same complexity as pytorch, which means tortto can be used to train relatively larger models such as `resnet101` and vision transformer `ViT-B/16` with the same speed ratio.\n\nTortto implements reverse mode automatic differentiation and supports dynamic computation graph like pytorch.\n\n## Installation\n```python\npip install tortto\n```\n## New in version 1.3\n* Computation graph now uses operations as nodes, instead of using tensors.\n* Supports in-place operations.\n\n## Prerequisites\n**numpy** (required): Only use its basic functions.  \n\n**scipy** (optional): Can be installed to improve efficiency in some functions:  \n* `scipy.special.erf` used in `nn.GELU`. approximation of erf is used when scipy is not installed.\n* `scipy.sparse` used in `nn.Embedding` when setting `sparse=True`. Can't use sparse if neither scipy or cupy is installed.   \n\n**cupy** (optional): Compute Numpy functions in GPU. Use `.cuda()`/`.cpu()` to send tensors or modules between CPU and GPU.  \n\n**pytorch** (optional): Only use its dataset, dataloder and transforms, required in actual training to load and preprocess data. In each iteration, torch tensors (i.e. data, label) will be converted to tortto tensors by `tortto_tensor = tortto.tensor(torch_tensor.numpy()).cuda()`. All computations after that is done in tortto. See examples below.\n\n## Quick start\npytortto uses the same syntax as pytorch. Here shows the forward and backward propagation of some functions.\n\n```python\nimport tortto as tt\n\nx = tt.tensor([[1.1, 2.0, 3.6], [4.7, 3.14, 2.718]], requires_grad=True)\ny = tt.tensor([[2., 3.99], [8.4, 1.5], [2.5, 7.8]], requires_grad=True)\noutput = tt.tanh(1/(tt.cat([tt.sin((tt.exp(x ** 1.4) / 3.1 ** tt.log(x)) @ y), y]).mean()))\noutput.backward()\n\nprint(output)\n# tensor(0.3489, grad_fn=\u003ctanhBackward\u003e)\nprint(x.grad)\n# [[ -0.0276   0.6477 -11.827 ]\n#  [ 27.2554  -5.3062   1.0978]]\nprint(y.grad)\n# [[-10.8005   8.2755]\n#  [ -0.3336   0.2187]\n#  [  0.8972  -0.9684]]\n```\n\nin-place operations:\n```python\nimport tortto as tt\n\nx0 = tt.tensor([[-0.5722, -1.3110, -1.4200, -0.3545],\n                [ 0.0945, -1.2108, -0.1059,  0.8041],\n                [-0.5110,  1.4361, -1.1575, -0.7639]], requires_grad=True)\nx = x0 + 0 # make it non-leaf for in-place operations\nx[:, 2:] = tt.sign(x[:, 2:]) * tt.sqrt(tt.abs_(x[:, 2:]))\nx.backward(x)\n\nprint(x._version)\n# 2\n\nprint(x0.grad)\n# [[-0.5722 -1.3110 -0.5000 -0.5000]\n#  [ 0.0945 -1.2108 -0.5000  0.5000]\n#  [-0.5110  1.4361 -0.5000 -0.5000]]\n```\n\ntortto also implements common Modules. Let's try `Conv2d` and compare its result and speed with pytorch:  \n[conv2d comparison in GPU](https://github.com/samrere/pytortto/blob/main/examples/conv2d_result_speed_comparison.ipynb)  \n\n### [quick start: reverse prediction](https://github.com/samrere/pytortto/blob/main/examples/transformers/reverse_prediction.ipynb)\nNext, Let's train a transformer encoder in 40 seconds, inspired from [UvA Deep Learning Tutorials](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial6/Transformers_and_MHAttention.html):  \nThe goal is to train a simple transformer encoder that can reverse the input sequence. i.e. if the input is `4,1,9,7,5,2,2`, the correct output would be `2,2,5,7,9,1,4`.  \n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://github.com/samrere/pytortto/blob/main/img/sequence_reverse.png?raw=True\" width=\"250\"\u003e\n\u003c/p\u003e\n\n\n## Examples\n**All examples trained on a Tesla T4 (16GB memory) GPU**  \n**When comparing speed with pytorch, `torch.backends.cudnn.benchmark` is set to `False` for a fair comparison**\n## Resnet  \n* Trained on CIFAR10\n* Each epoch = 45k train + 5k validation \n* Tested on full 10k test samples\n\nmodel | test acc. | #.filters | n | epochs trained | speed (min/epoch) | pytorch speed (min/epoch) | speed comparison\n--- | --- | --- | --- | --- | --- | --- | --- |\n[small-preact_resnet110](https://github.com/samrere/pytortto/tree/main/examples/resnet/small_preact_resnet_110) | **94.08%**| (16,32,64) | basicBlock:(18,18,18) | 180 | 3.65 | 1.2 | 3.65/1.2=3.0\n[preact_resnet18](https://github.com/samrere/pytortto/blob/main/examples/resnet/preact_resnet18) | **94.65%**| (64,128,256,512) | basicBlock:(2,2,2,2) | 180 | 2.0 | 0.75 | 2.0/0.75=2.7\n[standard_resnet50 (finetune)](https://github.com/samrere/pytortto/blob/main/examples/resnet/resnet50_finetune) | **96.31%**| (64,128,256,512) | bottleNeck:(3,4,6,3) | 15 | 20.65 | 8.82 | 20.65/8.82=2.3\n[preact_resnet101](https://github.com/samrere/pytortto/blob/main/examples/resnet/preact_resnet101) | **94.78%**| (64,128,256,512) | bottleNeck:(3,4,23,3) | 200 | 9.2 | 4.3 | 9.2/4.3=2.1\n* `small_preact_resnet110`, `preact_resnet18` and `preact_resnet101` are trained from scratch. Kernel size of the first conv layer is 3, because CIFAR-10 images are 32x32 in size.\n* `standard_resnet50` fintunes the full pretrained model. Kernel size of the first conv layer is 7. CIFAR-10 images are resized to 224x224 before feeding into the model.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/samrere/pytortto/blob/main/img/cifar10.png?raw=True\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## UNet\n* Trained on the carvana dataset\n* GPU memory: 1.5GB\n* Training time: 7 mins\n\nmodel | image size | #.filters | batchsize | train/val/test | epochs\n--- | --- | --- | --- | --- | --- |\n[UNet](https://github.com/samrere/pytortto/tree/main/examples/unet) | resized to 3x64x64 | 32,64,128,256,512,256,128,64,32 | 32 | 3658/646/784 | 20 \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/samrere/pytortto/blob/main/img/carvana.png?raw=True\" width=\"600\"\u003e\n\u003c/p\u003e\n\n## Vision Transformer  \n* Trained on CIFAR-10  \n* Finetune: Each epoch = 4500 train + 500 validation  \n* Tested on full 10k test samples\n\nmodel | test acc. | layers | Hidden size | MLP size | Heads | epochs trained | speed (min/epoch) | pytorch speed (min/epoch) | speed comparison\n--- | --- | --- | --- | --- | --- | --- | ---| ---| --- |\n[ViT-B/16 (finetune)](https://github.com/samrere/pytortto/tree/main/examples/transformers/vision_transformer_finetune) | **97.42%**| 12 | 768 | 3072 | 12 | 15 | 4.2 | 2.7 | 4.2/2.7=1.56\n* `ViT-B/16` Finetunes the full pretrained model. CIFAR-10 images are resized to 224x224 before feeding into the model.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/samrere/pytortto/blob/main/img/attn.png?raw=True\" width=\"600\"\u003e\n\u003c/p\u003e\n\n## DCGAN\n* Dataset adapted from https://github.com/bchao1/Anime-Face-Dataset\n* Images resized to 3x64x64\n\nmodel | epochs trained | speed (min/epoch) | pytorch speed (min/epoch) | speed comparison\n--- | --- | ---| ---| --- |\n[DCGAN](https://github.com/samrere/pytortto/tree/main/examples/gan/dcgan) | 100 | 6.27 | 2.38 | 6.27/2.38=2.6\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/samrere/pytortto/blob/main/img/fake.png?raw=True\" width=\"450\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/samrere/pytortto/blob/main/img/transition.png?raw=True\" width=\"800\"\u003e\n\u003c/p\u003e\n\n## YOLOv1\n[YOLO v1](https://github.com/samrere/pytortto/tree/main/examples/yolo) is work in progress...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamrere%2Fpytortto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamrere%2Fpytortto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamrere%2Fpytortto/lists"}