{"id":13472105,"url":"https://github.com/karpathy/LLM101n","last_synced_at":"2025-03-26T15:31:20.005Z","repository":{"id":245479830,"uuid":"806315903","full_name":"karpathy/LLM101n","owner":"karpathy","description":"LLM101n: Let's build a Storyteller","archived":true,"fork":false,"pushed_at":"2024-08-01T01:20:33.000Z","size":275,"stargazers_count":32941,"open_issues_count":19,"forks_count":1800,"subscribers_count":3087,"default_branch":"master","last_synced_at":"2025-03-26T05:02:51.258Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karpathy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-27T00:23:38.000Z","updated_at":"2025-03-26T03:00:25.000Z","dependencies_parsed_at":"2024-10-30T02:51:41.055Z","dependency_job_id":null,"html_url":"https://github.com/karpathy/LLM101n","commit_stats":null,"previous_names":["karpathy/llm101n"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karpathy%2FLLM101n","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karpathy%2FLLM101n/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karpathy%2FLLM101n/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karpathy%2FLLM101n/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karpathy","download_url":"https://codeload.github.com/karpathy/LLM101n/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245681290,"owners_count":20655165,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:00:51.936Z","updated_at":"2025-03-26T15:31:19.667Z","avatar_url":"https://github.com/karpathy.png","language":null,"funding_links":[],"categories":["Others","A01_文本生成_文本对话","Trending LLM Projects","课程 Course","Summary","NLP","LLM教程与课程 (2025版)"],"sub_categories":["大语言对话模型及数据","3. Pretraining","LLM 评估与数据"],"readme":"# LLM101n: Let's build a Storyteller\n\n---\n\n**!!! NOTE: this course does not yet exist. It is current being developed by [Eureka Labs](https://eurekalabs.ai). Until it is ready I am archiving this repo !!!**\n\n---\n\n![LLM101n header image](llm101n.jpg)\n\n\u003e  What I cannot create, I do not understand. -Richard Feynman\n\nIn this course we will build a Storyteller AI Large Language Model (LLM). Hand in hand, you'll be able to create, refine and illustrate little [stories](https://huggingface.co/datasets/roneneldan/TinyStories) with the AI. We are going to build everything end-to-end from basics to a functioning web app similar to ChatGPT, from scratch in Python, C and CUDA, and with minimal computer science prerequisites. By the end you should have a relatively deep understanding of AI, LLMs, and deep learning more generally.\n\n**Syllabus**\n\n- Chapter 01 **Bigram Language Model** (language modeling)\n- Chapter 02 **Micrograd** (machine learning, backpropagation)\n- Chapter 03 **N-gram model** (multi-layer perceptron, matmul, gelu)\n- Chapter 04 **Attention** (attention, softmax, positional encoder)\n- Chapter 05 **Transformer** (transformer, residual, layernorm, GPT-2)\n- Chapter 06 **Tokenization** (minBPE, byte pair encoding)\n- Chapter 07 **Optimization** (initialization, optimization, AdamW)\n- Chapter 08 **Need for Speed I: Device** (device, CPU, GPU, ...)\n- Chapter 09 **Need for Speed II: Precision** (mixed precision training, fp16, bf16, fp8, ...)\n- Chapter 10 **Need for Speed III: Distributed** (distributed optimization, DDP, ZeRO)\n- Chapter 11 **Datasets** (datasets, data loading, synthetic data generation)\n- Chapter 12 **Inference I: kv-cache** (kv-cache)\n- Chapter 13 **Inference II: Quantization** (quantization)\n- Chapter 14 **Finetuning I: SFT** (supervised finetuning SFT, PEFT, LoRA, chat)\n- Chapter 15 **Finetuning II: RL** (reinforcement learning, RLHF, PPO, DPO)\n- Chapter 16 **Deployment** (API, web app)\n- Chapter 17 **Multimodal** (VQVAE, diffusion transformer)\n\n**Appendix**\n\nFurther topics to work into the progression above:\n\n- Programming languages: Assembly, C, Python\n- Data types: Integer, Float, String (ASCII, Unicode, UTF-8)\n- Tensor: shapes, views, strides, contiguous, ...\n- Deep Learning frameworks: PyTorch, JAX\n- Neural Net Architecture: GPT (1,2,3,4), Llama (RoPE, RMSNorm, GQA), MoE, ...\n- Multimodal: Images, Audio, Video, VQVAE, VQGAN, diffusion\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarpathy%2FLLM101n","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarpathy%2FLLM101n","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarpathy%2FLLM101n/lists"}