https://github.com/cattolatte/zenith-nlp-framework
Zenith is a clean, from-scratch library for generative NLP, focused on decoder-only transformer language models, causal language modeling, and modern text generation. Built on PyTorch tensor primitives, it emphasizes readability, reproducibility, and production-inspired software engineering.
https://github.com/cattolatte/zenith-nlp-framework
bert deep-learning docker fastapi gpt hydra lora mlflow mlops natural-language-processing nlp peft pytorch transformer
Last synced: 1 day ago
JSON representation
Zenith is a clean, from-scratch library for generative NLP, focused on decoder-only transformer language models, causal language modeling, and modern text generation. Built on PyTorch tensor primitives, it emphasizes readability, reproducibility, and production-inspired software engineering.
- Host: GitHub
- URL: https://github.com/cattolatte/zenith-nlp-framework
- Owner: cattolatte
- License: mit
- Created: 2025-10-14T08:21:14.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2026-07-05T13:41:53.000Z (1 day ago)
- Last Synced: 2026-07-05T15:17:10.176Z (1 day ago)
- Topics: bert, deep-learning, docker, fastapi, gpt, hydra, lora, mlflow, mlops, natural-language-processing, nlp, peft, pytorch, transformer
- Language: Python
- Homepage:
- Size: 218 KB
- Stars: 2
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Zenith
### A from-scratch generative NLP library — decoder-only language models & text generation
[](https://github.com/cattolatte/zenith-nlp-framework/releases)
[](https://www.python.org)
[](https://pytorch.org)
[](LICENSE)
[](#project-status)
---
Zenith is a clean, from-scratch library for **generative NLP**: decoder-only
transformer language models, causal-LM training, and **text generation** — built
on PyTorch tensor primitives. The architecture is hand-written (causal
self-attention, pre-norm blocks, weight-tied embeddings) and readable end to end;
PyTorch supplies only autograd, containers and optimizers.
Zenith is a standalone project. It is also the **generative counterpart** to
[Polaris](https://github.com/cattolatte/Polaris), a from-scratch engine focused on
*understanding* text (transformer encoders, classification). Polaris encodes;
Zenith generates. The two are complementary but independent — Zenith can
*optionally* reuse Polaris' tokenizers (`pip install zenith-nlp[polaris]`), but it
ships its own and does not depend on Polaris.
## What's here
- **Decoder-only transformer** (`DecoderLM`) — causal self-attention, pre-norm
blocks, tied embeddings, written from scratch.
- **Tokenizers** — a dependency-free byte-level tokenizer (`ByteTokenizer`) and a
from-scratch, trainable byte-level BPE (`BPETokenizer`), both lossless.
- **Text generation** (`Generator`) — greedy, temperature, top-k, nucleus (top-p),
repetition penalty, and beam search, with a KV-cache and streaming.
- **Causal-LM training** (`CausalLMTrainer`) — warmup/cosine schedule, gradient
clipping, best-checkpoint saving, per-epoch samples, MLflow tracking, on-disk
run records, and a deterministic mode.
- **Efficient fine-tuning & scaling** — LoRA adapters (`zenith.peft`), gradient
accumulation, mixed precision (AMP), and `torchrun`-native distributed (DDP)
training — all opt-in.
- **Evaluation** — held-out `perplexity` / `evaluate`, and a `zenith eval` command.
- **Serving** — a FastAPI service (`POST /generate`, SSE `POST /generate/stream`),
a `zenith serve` command, and an interactive `zenith chat` REPL.
- **Hydra-configured** runs and sweeps; a small `zenith` CLI.
See [BENCHMARKS.md](BENCHMARKS.md) for the evaluation methodology and
[docs/modules.md](docs/modules.md) for a module overview. On the roadmap:
QLoRA/FSDP for larger-scale training, and sweep-result aggregation.
## Install
```bash
git clone https://github.com/cattolatte/zenith-nlp-framework.git
cd zenith-nlp-framework
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]" # torch, hydra, omegaconf, mlflow, typer
```
## Usage
Train a language model on the bundled corpus (or point `data.corpus_path` at your
own text):
```bash
python -m zenith.cli.train # defaults
python -m zenith.cli.train training.epochs=50 model.embed_dim=384
python -m zenith.cli.train tokenizer=bpe # from-scratch BPE
python -m zenith.cli.train peft=lora # LoRA fine-tuning
python -m zenith.cli.train training.amp=true training.grad_accum_steps=4
python -m zenith.cli.train -m training.learning_rate=1e-3,3e-4,1e-4 # sweep
torchrun --nproc_per_node=4 -m zenith.cli.train # multi-GPU (DDP)
```
Evaluate held-out perplexity:
```bash
zenith eval -m zenith-lm.pt -c data/tiny_corpus.txt
```
Generate text from a trained checkpoint:
```python
from zenith import load_pretrained
gen = load_pretrained("zenith-lm.pt")
print(gen.generate("Once upon a time", max_new_tokens=200, temperature=0.8))
```
Or from the CLI:
```bash
zenith generate -m zenith-lm.pt "Once upon a time" --temperature 0.8
zenith chat -m zenith-lm.pt # interactive REPL, streams as it generates
```
Serve it over HTTP (blocking + streaming):
```bash
zenith serve -m zenith-lm.pt # POST /generate, POST /generate/stream (SSE)
curl -s localhost:8000/generate -d '{"prompt":"Once","max_new_tokens":100}'
```
## Architecture
```text
src/zenith/
├── models/ # decoder-only transformer (from scratch)
├── tokenizers/ # byte-level tokenizer
├── data/ # causal-LM datasets & corpus helpers
├── generation/ # sampling / decoding (+ streaming)
├── training/ # causal-LM training loop
├── peft/ # LoRA adapters
├── distributed/ # DDP helpers
├── tracking/ # optional MLflow experiment tracking
├── experiments/ # environment capture & on-disk run records
├── serving/ # FastAPI generation service (+ SSE streaming)
├── cli/ # Hydra train entrypoint + `zenith` CLI (serve, chat, …)
└── checkpoint.py # self-describing save / load
```
## Project status
Zenith is under **active development**, mid-way through a redesign from an early,
general NLP framework into the focused generative library above. Phase 1 (the
generative core: model, tokenizer, training, generation) is in place; decoding
strategies, PEFT, distributed training and serving follow. Interfaces may change
until the first tagged release.
## License
MIT — see [LICENSE](LICENSE).
by K Satya Sai Nischal