{"id":27062585,"url":"https://github.com/biegehydra/nanogptdotnet","last_synced_at":"2025-04-05T15:17:50.982Z","repository":{"id":206600334,"uuid":"709635681","full_name":"biegehydra/NanoGptDotnet","owner":"biegehydra","description":"A miniature large language model (LLM) that generates shakespeare like text written in C#. Project meant to help dotnet developers get introduced to torch and AI/LLM's. Code filled with comments to help you learn.","archived":false,"fork":false,"pushed_at":"2024-03-15T02:05:08.000Z","size":483,"stargazers_count":25,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-03-15T03:38:51.020Z","etag":null,"topics":["ai","csharp","dotnet","gpt","llm","torch","torchsharp"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/biegehydra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-25T05:02:04.000Z","updated_at":"2024-03-12T17:50:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"edbb5a7c-ea4f-4de3-a999-e429036f1cc8","html_url":"https://github.com/biegehydra/NanoGptDotnet","commit_stats":null,"previous_names":["biegehydra/nanogptdotnet"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biegehydra%2FNanoGptDotnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biegehydra%2FNanoGptDotnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biegehydra%2FNanoGptDotnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biegehydra%2FNanoGptDotnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/biegehydra","download_url":"https://codeload.github.com/biegehydra/NanoGptDotnet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247353719,"owners_count":20925329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","csharp","dotnet","gpt","llm","torch","torchsharp"],"created_at":"2025-04-05T15:17:50.516Z","updated_at":"2025-04-05T15:17:50.971Z","avatar_url":"https://github.com/biegehydra.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"## NanoGptDotnet\nTranslated from python code written by Andrej Karpathy https://www.youtube.com/watch?v=kCc8FmEb1nY\n\n### Running on CUDA\nThe project is currently setup to run on a CPU. To run on CUDA go to [Shared.csporj](https://github.com/biegehydra/NanoGptDotnet/blob/master/src/Shared/Shared.csproj) and uncomments the line for the CUDA package and delete the line for the cpu package.\n\n### Contributing\n\nFeel free to change comments, add comments, or fix the code in the master branch if there are bugs. If you'd like to improve the model with techniques not used in the video, create a seperate branch for that.\n\n### Debugging\n\nThe built in `.ToString()` method is not good for tensors, so if your trying to print a tensor while debugging, I recommend using extension method `.ToFormattedString()` that I wrote.\n\n### NanoGpt\nI was able to achieve similar results to Kaparthy. I had to use my CPU which took half a day to run. I got down to a training loss of around 0.67 and a validation loss of around 1.5. I'm not sure why my train loss and validation loss had a large spread than Kaparthy. Here's an example output of that model.\n\n```\nICHARD III:\nBut look on my father's fault, but not most good.\n\nQUEEN ELIZABETH:\nNor cheer will I thee speak; but thou hast a word, till now.\n\nKING RICHARD III:\nThen, good kind her heart; the devil\nBut calls me sometimes of blood for yourselves.\nWhat service must I do? Or else thou art moved at\nHer princess, having entleman of fair princely\nWhich I have power to dispose offender:\nIf I cannot discharge him, but action\nThat now respected, whatsoever more, title,\nConstants to the world's name; and if not they see\nThe truth, crying but his within the fairest cover:\nIf the wise burning fools that they lose\nThe feedering steel gaoler them in sex the clouds,\nYou must have pair'd for sufferance.\n\nCOMINIUS:\nNo remedy;\nThat they have dropp'd from war upon you.\n\nMENENIUS:\nThe gods keep you on!\nThere's no more!\n\nCORIOLANUS:\nNo, ay, but to good with them all. You, the first\nNo grave for my poor general than you,--\nWhether for the poor good star!\n\nVOLUMNIA:\nNot that you should not leave about the ship,--\nWhich you save every word of your eyes that,--\nBeing suffer'd with the sweets, I do notthing\nMake the oseemable of strength, your enemy\nIs that ever goad to be so,--think my meaning,\nNothing by that himself so shall not share\nAbove again him, which owe have little,\nNot raged to know the cheek to the purpose,\nNot summer showing: post it, sir, increaseth,\nThat cames to practise one would move you do,\nBut by that you need not, my lord.\n\nMARCIUS:\nBe not according:\nThe young Rome are sent forth, more strength on you,\nBut queen overta'en.\n\nMessenger:\nMy lord,\nI knees like  a Capitol\n```\n\n### Bigram Model\n\nI was able to achieve the same, maybe slightly better results with the BigramModel. A loss of around 2.365 compared to Kaparthys 2.48.\n\n```\nstep 0: train loss 4.5227, val loss 4.5216\nstep 300: train loss 2.7147, val loss 2.7128\nstep 600: train loss 2.4478, val loss 2.4512\nstep 900: train loss 2.4031, val loss 2.4163\nstep 1200: train loss 2.3846, val loss 2.3902\nstep 1500: train loss 2.3717, val loss 2.3847\nstep 1800: train loss 2.3749, val loss 2.3902\nstep 2100: train loss 2.3649, val loss 2.3808\nstep 2400: train loss 2.3697, val loss 2.3756\nstep 2700: train loss 2.3642, val loss 2.3853\n\nFRe s\nI mu ofr sthenck f hethefos.\nWhtre IO fofel t abar alin A pr llf two s havere r makimy t o ndem'dineer.\n\nThisite;\nHOw st hoffelan moner yonourfis aind, cinlelle IUCour gotefoulerd kl;\nMI\nI:\nYoureitl e.\nTE:\nCoagarutenow.\nWerd RAnes eeco nd gowhiderdw t, ty gin nde iot.\nA: gre r, tak led.\nIOMIswofte.\nste; wixputinte\nSeile t thrand.\n\n's ack'd pprs f thechuce tes Ye an t'ers neawas the l I wnd treme;\nAs'de asinge f seenthoured arisout cer\nHar s, ro h, t mend Mexy aran spo\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiegehydra%2Fnanogptdotnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbiegehydra%2Fnanogptdotnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiegehydra%2Fnanogptdotnet/lists"}