{"id":30285657,"url":"https://github.com/kjpou1/llm-zero-to-trained","last_synced_at":"2026-05-19T10:04:57.712Z","repository":{"id":304615691,"uuid":"1019314364","full_name":"kjpou1/llm-zero-to-trained","owner":"kjpou1","description":"Building a Large Language Model from scratch for deep understanding — inspired by Sebastian Raschka’s book, implemented entirely by hand.","archived":false,"fork":false,"pushed_at":"2025-07-14T09:02:15.000Z","size":158,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-14T09:41:42.227Z","etag":null,"topics":["attention","deep-learning","educational","from-scratch","from-scratch-in-python","llm","machine-learning","natural-language-processing","python","tokenizer","torch","training-loop","transformers","uv-pm"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kjpou1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-14T06:24:51.000Z","updated_at":"2025-07-14T09:02:18.000Z","dependencies_parsed_at":"2025-07-14T09:41:46.879Z","dependency_job_id":"58df7660-a91d-4196-9638-3063154a0d3a","html_url":"https://github.com/kjpou1/llm-zero-to-trained","commit_stats":null,"previous_names":["kjpou1/llm-zero-to-trained"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kjpou1/llm-zero-to-trained","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kjpou1%2Fllm-zero-to-trained","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kjpou1%2Fllm-zero-to-trained/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kjpou1%2Fllm-zero-to-trained/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kjpou1%2Fllm-zero-to-trained/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kjpou1","download_url":"https://codeload.github.com/kjpou1/llm-zero-to-trained/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kjpou1%2Fllm-zero-to-trained/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270763461,"owners_count":24641026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","deep-learning","educational","from-scratch","from-scratch-in-python","llm","machine-learning","natural-language-processing","python","tokenizer","torch","training-loop","transformers","uv-pm"],"created_at":"2025-08-16T20:07:47.184Z","updated_at":"2026-05-19T10:04:57.665Z","avatar_url":"https://github.com/kjpou1.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **LLM: Zero to Trained**\n\n\u003e A fully manual, from-scratch implementation of a Large Language Model (LLM), guided by the book [**Build a Large Language Model from Scratch**](https://github.com/rasbt/LLMs-from-scratch) by Sebastian Raschka.\n\nUnlike simply running code from the [reference repo](https://github.com/rasbt/LLMs-from-scratch.git), this project is about **understanding and re-implementing** every major component — tokenizer, model, data loader, training loop — by hand.\n\n---\n- [**LLM: Zero to Trained**](#llm-zero-to-trained)\n  - [🎯 Objective](#-objective)\n  - [📁 Project Structure](#-project-structure)\n  - [🧠 Learning Sources](#-learning-sources)\n  - [🗒️ Progress Log](#️-progress-log)\n  - [🔧 Getting Started](#-getting-started)\n  - [🧰 Environment Setup (with uv)](#-environment-setup-with-uv)\n  - [🚀 CLI: Start Building](#-cli-start-building)\n  - [📚 References \\\u0026 Inspirations](#-references--inspirations)\n  - [� Internal Documentation](#-internal-documentation)\n  - [📜 License](#-license)\n\n---\n\n## 🎯 Objective\n\nTo develop a working LLM from first principles by:\n\n* Writing each component from scratch in Python\n* Following the structure and logic from Raschka’s book\n* Validating ideas through notebooks and experiments\n* Building modular, reusable, CLI-driven code\n\n---\n\n## 📁 Project Structure\n\n```\nllm-zero-to-trained/\n├── src/llmscratch/        ← Modular CLI-driven Python package\n│   ├── config/            ← Config loader with .env, CLI, YAML support\n│   ├── models/            ← Core dataclasses and SingletonMeta\n│   ├── runtime/           ← CLI argument parsing and dispatch\n│   ├── launch_host.py     ← Entry point for all commands (e.g., preprocess)\n│   └── host.py            ← Command execution coordinator\n├── notebooks/             ← Book-aligned exploration notebooks\n├── configs/               ← YAML configs for datasets, vocab, etc.\n├── datasets/              ← Raw and processed tokenized data\n├── pyproject.toml         ← Project metadata and CLI definition\n├── README.md              ← This file\n└── PROGRESS.md            ← Running log of milestones\n```\n\n---\n\n## 🧠 Learning Sources\n\n* 📘 *Build a Large Language Model from Scratch* – Sebastian Raschka (2024)\n* 💻 [LLMs-from-scratch GitHub Repo](https://github.com/rasbt/LLMs-from-scratch)\n* 🧠 Karpathy’s [minGPT](https://github.com/karpathy/minGPT) and [nanoGPT](https://github.com/karpathy/nanoGPT) (inspirational, but not reused)\n\n---\n\n## 🗒️ Progress Log\n\nSee [PROGRESS.md](./PROGRESS.md) for completed milestones, model checkpoints, and active development notes.\n\n---\n\n## 🔧 Getting Started\n\n```bash\ngit clone https://github.com/kjpou1/llm-zero-to-trained.git\ncd llm-zero-to-trained\n```\n\n---\n\n## 🧰 Environment Setup (with [uv](https://docs.astral.sh/uv/getting-started/installation/))\n\n```bash\nuv venv\nsource .venv/bin/activate        # macOS/Linux\n# OR\n.venv\\Scripts\\activate           # Windows\n\nuv pip install --editable .\nuv sync\n```\n\n\u003e 🧪 This enables the `llmscratch` CLI from anywhere and installs all dependencies with reproducible locking via `uv`.\n\n---\n\n## 🚀 CLI: Start Building\n\nUse the CLI to run modular LLM pipelines:\n\n```bash\nllmscratch preprocess --config configs/data_config.yaml\n```\n\nMore commands like `train`, `sample`, and `evaluate` will follow as the project evolves.\n\n---\n\n## 📚 References \u0026 Inspirations\n\nWhile the architecture is influenced by great projects, all code is original and written from scratch:\n\n* [Raschka’s LLM Book](https://leanpub.com/llms-from-scratch)\n* [minGPT](https://github.com/karpathy/minGPT)\n* [nanoGPT](https://github.com/karpathy/nanoGPT)\n* [Hugging Face Transformers](https://github.com/huggingface/transformers)\n\n---\n\n## 📄 Internal Documentation\n\nThis project includes implementation-focused documentation aligned with academic papers and architectural design:\n\n* [`bpe_implementation.md`](docs/tokenizers/bpe_implementation.md) — Byte Pair Encoding (BPE) training process, aligned with Sennrich et al. (2015)\n\n---\n\n## 📜 License\n\nThis project is MIT licensed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkjpou1%2Fllm-zero-to-trained","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkjpou1%2Fllm-zero-to-trained","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkjpou1%2Fllm-zero-to-trained/lists"}