{"id":49878965,"url":"https://github.com/paytonison/tater-tot","last_synced_at":"2026-05-15T13:13:52.595Z","repository":{"id":355581994,"uuid":"1227806179","full_name":"paytonison/tater-tot","owner":"paytonison","description":"Small educational C++20 character-level language model with a tiny automatic-differentiation engine, training/generation CLIs, and checkpointing.","archived":false,"fork":false,"pushed_at":"2026-05-13T04:15:56.000Z","size":2087,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-13T05:32:47.731Z","etag":null,"topics":["autodiff","character-level-model","cmake","cpp","cpp20","educational","from-scratch","language-model","machine-learning","neural-network"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paytonison.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-03T07:26:13.000Z","updated_at":"2026-05-13T04:16:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/paytonison/tater-tot","commit_stats":null,"previous_names":["paytonison/tater-tot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paytonison/tater-tot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paytonison%2Ftater-tot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paytonison%2Ftater-tot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paytonison%2Ftater-tot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paytonison%2Ftater-tot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paytonison","download_url":"https://codeload.github.com/paytonison/tater-tot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paytonison%2Ftater-tot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33067699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autodiff","character-level-model","cmake","cpp","cpp20","educational","from-scratch","language-model","machine-learning","neural-network"],"created_at":"2026-05-15T13:13:52.158Z","updated_at":"2026-05-15T13:13:52.585Z","avatar_url":"https://github.com/paytonison.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tater Tot\n\nA tiny character-level decoder-only Transformer language model implemented from\nscratch in C++ with a parallel standard-library Python port.\n\nTater Tot is a compact GPT-style language model built to show the core mechanics\nof modern autoregressive Transformers without relying on PyTorch, TensorFlow,\nONNX Runtime, CUDA, or another machine learning framework. It is intentionally\nsmall, educational, and architecture-focused: the point is to make the pieces of\na Transformer visible in ordinary C++20 and plain Python.\n\nThe project includes:\n\n- a custom `Tensor` type and reverse-mode autodiff engine\n- a character-level data pipeline\n- learned token and positional embeddings\n- causal multi-head self-attention\n- pre-layernorm Transformer blocks\n- GELU feed-forward layers\n- residual connections\n- an Adam optimizer\n- binary checkpoint save/load\n- autoregressive text generation\n\n## Agent-Assisted One-Shot Development\n\nTater Tot began as a human-originated project idea: a small, from-scratch language-model implementation intended to make transformer internals concrete, inspectable, and experimentally comparable to a conventional Python baseline.\n\nThe initial version of the project was generated in a single Codex pass from high-level project instructions, and the later transformer upgrade was likewise produced as a one-shot architectural expansion.\n\nThe goal is not to present the code as hand-authored line by line. Instead, this repository documents a workflow: human concept formation, specification, agentic implementation, human review, debugging, training, evaluation, and iterative refinement.\n\nThis makes Tater Tot both a small language-model implementation and a case study in how coding agents can compress the distance between concept and working artifact. The human role shifts away from raw syntax production and toward architecture, constraint-setting, supervision, interpretation, and experimental design.\n\n## Architecture\n\nTater Tot is a character-level autoregressive decoder-only Transformer. Raw text\nis encoded as character IDs, embedded, passed through causal Transformer blocks,\nand projected back to vocabulary logits.\n\n```text\nRaw text\n  -\u003e character vocabulary\n  -\u003e token IDs\n  -\u003e token embeddings\n  -\u003e positional embeddings\n  -\u003e Transformer blocks\n       -\u003e pre-LayerNorm\n       -\u003e causal multi-head self-attention\n       -\u003e residual add\n       -\u003e pre-LayerNorm\n       -\u003e MLP/feed-forward with GELU\n       -\u003e residual add\n  -\u003e final LayerNorm\n  -\u003e LM head\n  -\u003e logits over character vocabulary\n  -\u003e sampling\n  -\u003e generated text\n```\n\nThe model keeps the sequence axis alive logically as\n`[batch, context, embed]`. Internally, the C++ tensor code stores that activation\nas `[batch * context, embed]` when convenient, so each row is still a specific\ntoken position.\n\nAttention mixes information across token positions. The causal mask prevents a\nposition from attending to future characters, so position `t` can only use\npositions `\u003c= t`. After attention, the feed-forward network transforms each\ntoken vector independently. Residual connections preserve information across\nsub-layers and make optimization easier, while LayerNorm stabilizes the scale of\nactivations. The final LM head maps each final hidden vector to next-character\nscores over the vocabulary.\n\n## What Changed\n\nThe original prototype was a character-level MLP language model. It embedded the\ncharacters in a context window, flattened the whole context into one vector, and\npredicted a single next character.\n\nOld model:\n\n```text\ntoken embeddings + position embeddings\n  -\u003e flatten context\n  -\u003e MLP\n  -\u003e next-character logits\n```\n\nThe current model keeps token positions separate, applies causal self-attention,\nand predicts the next character at every position in the context window.\n\nNew model:\n\n```text\ntoken embeddings + position embeddings\n  -\u003e causal Transformer blocks\n  -\u003e LM head\n  -\u003e next-character logits at every position\n```\n\nThis is now a real tiny decoder-only Transformer, not a flattened-context MLP.\n\n## Training Objective\n\nTraining uses next-character prediction. For each sampled text window, the input\ncharacters are shifted by one position to form the targets:\n\n```text\nInput:  N i e t z s c h\nTarget: i e t z s c h e\n```\n\nThe batching code trains every position in the context window. For a batch with\n`batch_size` rows and `context` characters per row, the model returns logits\nshaped approximately:\n\n```text\n[batch_size * context, vocab_size]\n```\n\n`softmax_cross_entropy` consumes those logits together with\n`batch_size * context` target IDs.\n\n## Generation\n\nGeneration is autoregressive:\n\n1. Encode the prompt into character IDs.\n2. Keep the most recent `context` IDs, left-padding with ID `0` if needed.\n3. Run the model over that context.\n4. Take only the final position's logits.\n5. Sample one character ID.\n6. Append the sampled character.\n7. Repeat until the requested number of characters has been generated.\n\nSampling currently supports temperature and top-k filtering. There is no top-p,\nbeam search, or repetition penalty.\n\n## Project Structure\n\n```text\nCMakeLists.txt\n  CMake build configuration for the library, training binary, generation binary,\n  and test binary.\n\ninclude/tater/tensor.hpp\nsrc/tensor.cpp\n  Tensor object, shapes, reverse-mode autodiff, broadcasting, matrix multiply,\n  embeddings, GELU, LayerNorm, causal self-attention, softmax cross-entropy,\n  gradient zeroing, and gradient clipping.\n\ninclude/tater/model.hpp\nsrc/model.cpp\n  ModelConfig, TinyCharModel, Transformer block parameters, initialization,\n  forward pass, named parameter access, sampling, text generation, and loss\n  estimation.\n\ninclude/tater/data.hpp\nsrc/data.cpp\n  Character vocabulary construction, encoding/decoding, text loading,\n  train/validation splitting, and full-position batch creation.\n\ninclude/tater/optimizer.hpp\nsrc/optimizer.cpp\n  Adam optimizer implementation.\n\ninclude/tater/checkpoint.hpp\nsrc/checkpoint.cpp\n  Checkpoint save/load for Transformer config, vocabulary, and named parameter\n  tensors.\n\nexamples/train.cpp\n  `tater_train` CLI entry point and training loop.\n\nexamples/generate.cpp\n  `tater_generate` CLI entry point for loading a checkpoint and sampling text.\n\ntests/test_main.cpp\n  Unit tests, gradient checks, Transformer shape checks, checkpoint round-trip,\n  tiny overfit smoke test, and generation smoke test.\n\npython/tater/*.py\npython/tater_train.py\npython/tater_generate.py\ntests/test_python.py\n  A standard-library Python copy of the C++ implementation. It mirrors the same\n  Tensor/autodiff engine, byte-level data pipeline, Transformer model, Adam\n  optimizer, V2 binary checkpoint format, training CLI, generation CLI, and\n  Python smoke tests. It has no dependency on PyTorch, NumPy, or any other\n  third-party package.\n```\n\n## Build\n\nThe project uses CMake and the C++20 standard library.\n\n```sh\ncmake -S . -B build\ncmake --build build\nctest --test-dir build --output-on-failure\n```\n\nThe build creates:\n\n- `build/tater_train`\n- `build/tater_generate`\n- `build/tater_tests`\n\nThe Python port does not need a build step:\n\n```sh\npython3 tests/test_python.py\n```\n\n## Train\n\nPut a plain text file at `data/input.txt`, or pass another text path with\n`--data`.\n\n```sh\n./build/tater_train \\\n  --data data/input.txt \\\n  --steps 1000 \\\n  --context 64 \\\n  --embed 64 \\\n  --layers 2 \\\n  --heads 4 \\\n  --hidden 256 \\\n  --batch 16 \\\n  --lr 0.003 \\\n  --clip 1.0 \\\n  --print-every 50 \\\n  --sample-every 200 \\\n  --eval-batches 4 \\\n  --checkpoint checkpoints/model.bin \\\n  --seed 1337\n```\n\nThe equivalent Python entry point accepts the same training options:\n\n```sh\npython3 python/tater_train.py \\\n  --data data/input.txt \\\n  --steps 1000 \\\n  --context 64 \\\n  --embed 64 \\\n  --layers 2 \\\n  --heads 4 \\\n  --hidden 256 \\\n  --batch 16 \\\n  --lr 0.003 \\\n  --clip 1.0 \\\n  --print-every 50 \\\n  --sample-every 200 \\\n  --eval-batches 4 \\\n  --checkpoint checkpoints/model.bin \\\n  --seed 1337\n```\n\nTraining prints mini-batch loss, an estimated validation loss, global gradient\nnorm, and periodic samples. The checkpoint is written at the end of training.\n\nImportant model constraints:\n\n- `vocab_size`, `context`, `embed`, `hidden`, `layers`, and `heads` must be\n  non-zero.\n- `embed` must be divisible by `heads`.\n- `--hidden` defaults to `embed * 4` in the training CLI.\n\n## Generate\n\nAfter training, generate text from a checkpoint:\n\n```sh\n./build/tater_generate \\\n  --checkpoint checkpoints/model.bin \\\n  --prompt \"Once upon a time\" \\\n  --tokens 300 \\\n  --temperature 0.9 \\\n  --top-k 20 \\\n  --seed 1337\n```\n\nThe Python generator can load the same V2 checkpoint format:\n\n```sh\npython3 python/tater_generate.py \\\n  --checkpoint checkpoints/model.bin \\\n  --prompt \"Once upon a time\" \\\n  --tokens 300 \\\n  --temperature 0.9 \\\n  --top-k 20 \\\n  --seed 1337\n```\n\nGeneration prints the prompt followed by sampled characters. `temperature`\ncontrols randomness. `top-k` limits sampling to the most likely `k` characters\nwhen greater than zero.\n\n## Checkpointing\n\nCheckpoints store:\n\n1. Magic string: `TATER_TOT_CHECKPOINT_V2_TRANSFORMER`\n2. Transformer config: vocabulary size, context length, embedding size, hidden\n   size, layer count, and attention head count\n3. Vocabulary characters\n4. Named parameter tensors in deterministic model order\n\nOlder MLP checkpoints are not compatible with the Transformer checkpoint format.\nThe loader rejects V1 MLP checkpoints with a clear error rather than silently\nloading mismatched parameters.\n\n## Tests\n\nRun the full local test binary through CTest:\n\n```sh\nctest --test-dir build --output-on-failure\n```\n\nThe current test suite covers:\n\n- tensor shape handling and basic math outputs\n- finite-difference gradient checks for core ops, GELU, LayerNorm, and\n  softmax cross-entropy\n- Transformer logits shape for `[batch * context, vocab_size]`\n- finite gradients for every named model parameter after backpropagation\n- full-position batch target creation\n- checkpoint save/load round trip\n- tiny repeated-text overfit smoke test\n- generation smoke test\n\n## Benchmarks\n\nThe `benchmarks/` directory contains a reproducible C++ vs. pure-Python\ncomparison harness. It builds a shared corpus from `data/pg1998.txt` and\n`data/pg52124.txt` by default, runs both implementations with the same prompt,\nmodel settings, optimization settings, generation settings, and seed list, then\nwrites a machine-readable CSV plus generated samples.\n\n```sh\npython3 benchmarks/run_cpp_vs_python.py\n```\n\nFor 50 trials per implementation:\n\n```sh\npython3 benchmarks/run_cpp_vs_python.py --trials 50\n```\n\nFor a fast smoke check:\n\n```sh\npython3 benchmarks/run_cpp_vs_python.py \\\n  --seeds 1 \\\n  --steps 1 \\\n  --eval-batches 1 \\\n  --context 8 \\\n  --embed 8 \\\n  --layers 1 \\\n  --heads 2 \\\n  --hidden 16 \\\n  --batch 2 \\\n  --sample-length 32\n```\n\nSee `benchmarks/README.md` for the exact default settings, output files, and\ncaveats. Benchmark outputs, including the per-run CSV and summary CSV, are\ngenerated artifacts and are ignored by git.\n\nThe system used to run the benchmark tests was an Apple M1 Max. All benchmark\nruns were performed on the CPU only, without GPU, Neural Engine, or other\naccelerator backends.\n\n### Current Findings\n\nThe completed 50-trial benchmark used seeds `1` through `50` with the default\nbenchmark settings. In those runs, the C++ implementation is currently\noutperforming the pure-Python baseline in wall-clock speed:\n\n```text\nimplementation  avg train sec  avg generate sec  avg train loss  avg val loss\ncpp                    1.340021          0.657506        3.465902      3.433631\npython                 7.237891          3.289465        3.450698      3.422890\n```\n\nThat is about `5.4x` faster for training and `5.0x` faster for generation in\nthis benchmark. The early C++ samples also appeared more sentence-shaped in the\nobserved runs: C++ produced non-zero punctuation in `3` of `50` samples, while\nthe Python baseline produced none in those same measured runs. This is a\nqualitative signal, not a proof of better language modeling. The Python run had\nslightly lower average train and validation loss, and model-quality conclusions\nneed repeated seed-controlled benchmarks at meaningful training lengths.\n\nC++ is a strong fit for this project's AI goals because the core model is meant\nto be transparent and inspectable rather than hidden behind a framework. The C++\npath has low abstraction overhead, faster training and generation loops in the\nmeasured benchmark, explicit control over the tensor/autodiff/model internals,\nand a layout that makes memory, ownership, and performance costs easier to\nreason about. That makes it useful as the transparent experimental model core.\n\nThe current C++ implementation is not more compact by raw source size. Counting\nthe comparable implementation and CLI source sets, `include/tater`, `src`, and\n`examples` contain `12` files, `2,275` total lines, and `87,978` bytes. The\nPython baseline in `python/tater`, `python/tater_train.py`, and\n`python/tater_generate.py` contains `8` files, `1,768` total lines, and `57,616`\nbytes.\n\n## Limitations\n\n- The model is tiny and character-level.\n- Character handling is byte-oriented, not Unicode code point tokenization.\n- There is no BPE, WordPiece, or sentencepiece-style tokenizer.\n- Training examples are plain text files loaded into memory.\n- There is no GPU acceleration, CUDA backend, BLAS integration, SIMD kernel, or\n  fused attention implementation.\n- There is no dropout.\n- There is no top-p sampling, beam search, repetition penalty, or advanced\n  decoding strategy.\n- Checkpoints do not store optimizer state.\n- This is an educational and portfolio implementation, not production inference\n  infrastructure and not comparable to industrial LLMs.\n\n## Roadmap\n\nReasonable next improvements:\n\n- cleaner CLI help and configuration files\n- more focused gradchecks, especially for causal attention\n- tokenizer experiments beyond character IDs\n- optional weight tying between token embeddings and the LM head\n- dropout in attention and feed-forward layers\n- learning-rate scheduling\n- perplexity and richer validation reporting\n- faster kernels for matrix multiplication and attention\n- optional CMake cleanup and install targets\n- richer sampling controls such as top-p and repetition penalties\n- richer benchmark summaries and visualizations\n\nLonger-term experimental roadmap:\n\n- **C++ Tater Tot baseline:** continue developing the current from-scratch C++ implementation as the primary systems-level reference model. The purpose is to keep the Transformer machinery visible, inspectable, and close to the hardware.\n\n- **Python Tater Tot baseline:** build a matching Python version using the same dataset, model shape, tokenizer assumptions, training schedule, and generation settings. This provides a conventional high-level implementation to compare against the C++ version.\n\n- **Custom performance benchmark:** create a repeatable benchmark suite that compares the C++ and Python implementations across:\n  - training speed\n  - inference speed\n  - memory use\n  - tokens or characters generated per second\n  - validation loss / perplexity\n  - qualitative generation samples from matched prompts\n\n  The goal is not merely to ask which language is faster in the abstract, but to measure which implementation produces better practical results under comparable constraints.\n\n- **Handmade MoE experiment:** prototype a small mixture-of-experts system using existing Gemma-family models as external expert modules. Instead of training one monolithic model from scratch, the system would stitch together multiple specialized Gemma instances and route tasks between them.\n\n- **Mediator model:** add a central mediator model that sits between the user/task and the expert models. The mediator would decide which expert or combination of experts should handle a request, merge their outputs, resolve disagreements, and produce the final response.\n\n- **Lexical / definition expert:** explore a dedicated expert for words, definitions, morphology, etymology, and sense disambiguation. This connects to the broader idea of using dictionary-like knowledge not as a literal embedding matrix, but as a specialized semantic subsystem.\n\n- **MoE evaluation harness:** design tests that compare the handmade MoE against single-model baselines on:\n  - factual consistency\n  - definition accuracy\n  - reasoning quality\n  - code generation\n  - stylistic control\n  - routing accuracy\n  - disagreement resolution\n\nThis roadmap turns Tater Tot from a single tiny Transformer into a staged research artifact: first a C++ implementation, then a Python comparison target, then a controlled benchmark, and finally a handmade MoE architecture where multiple existing models act as experts coordinated by a mediator.\n\n## License\n\nThe original Tater Tot source code and project documentation are licensed under\nthe MIT License. See [LICENSE](LICENSE).\n\nTraining corpora under `data/` are input texts and retain their original source\nterms. Verify those terms before redistributing the corpus files or derived\ndatasets.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaytonison%2Ftater-tot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaytonison%2Ftater-tot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaytonison%2Ftater-tot/lists"}