{"id":35324576,"url":"https://github.com/cattolatte/zenith-nlp-framework","last_synced_at":"2026-07-06T03:00:18.159Z","repository":{"id":318736268,"uuid":"1075966778","full_name":"cattolatte/zenith-nlp-framework","owner":"cattolatte","description":"Zenith is a clean, from-scratch library for generative NLP, focused on decoder-only transformer language models, causal language modeling, and modern text generation. Built on PyTorch tensor primitives, it emphasizes readability, reproducibility, and production-inspired software engineering.","archived":false,"fork":false,"pushed_at":"2026-07-05T13:41:53.000Z","size":223,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-05T15:17:10.176Z","etag":null,"topics":["bert","deep-learning","docker","fastapi","gpt","hydra","lora","mlflow","mlops","natural-language-processing","nlp","peft","pytorch","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cattolatte.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-14T08:21:14.000Z","updated_at":"2026-07-05T14:06:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"1d4f23fb-549a-47e9-98b7-3b8bcf6d6400","html_url":"https://github.com/cattolatte/zenith-nlp-framework","commit_stats":null,"previous_names":["cattolatte/zenith-nlp-framework"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/cattolatte/zenith-nlp-framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cattolatte%2Fzenith-nlp-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cattolatte%2Fzenith-nlp-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cattolatte%2Fzenith-nlp-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cattolatte%2Fzenith-nlp-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cattolatte","download_url":"https://codeload.github.com/cattolatte/zenith-nlp-framework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cattolatte%2Fzenith-nlp-framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35176125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","deep-learning","docker","fastapi","gpt","hydra","lora","mlflow","mlops","natural-language-processing","nlp","peft","pytorch","transformer"],"created_at":"2025-12-31T01:07:36.366Z","updated_at":"2026-07-06T03:00:18.153Z","avatar_url":"https://github.com/cattolatte.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Zenith\n\n### A from-scratch generative NLP library — decoder-only language models \u0026 text generation\n\n[![Release](https://img.shields.io/github/v/release/cattolatte/zenith-nlp-framework?label=release\u0026color=6E56CF)](https://github.com/cattolatte/zenith-nlp-framework/releases)\n[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org)\n[![PyTorch](https://img.shields.io/badge/PyTorch-2.0%2B-ee4c2c.svg?logo=pytorch\u0026logoColor=white)](https://pytorch.org)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![Status](https://img.shields.io/badge/status-active%20development-orange.svg)](#project-status)\n\n\u003c/div\u003e\n\n---\n\nZenith is a clean, from-scratch library for **generative NLP**: decoder-only\ntransformer language models, causal-LM training, and **text generation** — built\non PyTorch tensor primitives. The architecture is hand-written (causal\nself-attention, pre-norm blocks, weight-tied embeddings) and readable end to end;\nPyTorch supplies only autograd, containers and optimizers.\n\nZenith is a standalone project. It is also the **generative counterpart** to\n[Polaris](https://github.com/cattolatte/Polaris), a from-scratch engine focused on\n*understanding* text (transformer encoders, classification). Polaris encodes;\nZenith generates. The two are complementary but independent — Zenith can\n*optionally* reuse Polaris' tokenizers (`pip install zenith-nlp[polaris]`), but it\nships its own and does not depend on Polaris.\n\n## What's here\n\n- **Decoder-only transformer** (`DecoderLM`) — causal self-attention, pre-norm\n  blocks, tied embeddings, written from scratch.\n- **Tokenizers** — a dependency-free byte-level tokenizer (`ByteTokenizer`) and a\n  from-scratch, trainable byte-level BPE (`BPETokenizer`), both lossless.\n- **Text generation** (`Generator`) — greedy, temperature, top-k, nucleus (top-p),\n  repetition penalty, and beam search, with a KV-cache and streaming.\n- **Causal-LM training** (`CausalLMTrainer`) — warmup/cosine schedule, gradient\n  clipping, best-checkpoint saving, per-epoch samples, MLflow tracking, on-disk\n  run records, and a deterministic mode.\n- **Efficient fine-tuning \u0026 scaling** — LoRA adapters (`zenith.peft`), gradient\n  accumulation, mixed precision (AMP), and `torchrun`-native distributed (DDP)\n  training — all opt-in.\n- **Evaluation** — held-out `perplexity` / `evaluate`, and a `zenith eval` command.\n- **Serving** — a FastAPI service (`POST /generate`, SSE `POST /generate/stream`),\n  a `zenith serve` command, and an interactive `zenith chat` REPL.\n- **Hydra-configured** runs and sweeps; a small `zenith` CLI.\n\nSee [BENCHMARKS.md](BENCHMARKS.md) for the evaluation methodology and\n[docs/modules.md](docs/modules.md) for a module overview. On the roadmap:\nQLoRA/FSDP for larger-scale training, and sweep-result aggregation.\n\n## Install\n\n```bash\ngit clone https://github.com/cattolatte/zenith-nlp-framework.git\ncd zenith-nlp-framework\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -e \".[all]\"      # torch, hydra, omegaconf, mlflow, typer\n```\n\n## Usage\n\nTrain a language model on the bundled corpus (or point `data.corpus_path` at your\nown text):\n\n```bash\npython -m zenith.cli.train                                  # defaults\npython -m zenith.cli.train training.epochs=50 model.embed_dim=384\npython -m zenith.cli.train tokenizer=bpe                    # from-scratch BPE\npython -m zenith.cli.train peft=lora                        # LoRA fine-tuning\npython -m zenith.cli.train training.amp=true training.grad_accum_steps=4\npython -m zenith.cli.train -m training.learning_rate=1e-3,3e-4,1e-4   # sweep\ntorchrun --nproc_per_node=4 -m zenith.cli.train             # multi-GPU (DDP)\n```\n\nEvaluate held-out perplexity:\n\n```bash\nzenith eval -m zenith-lm.pt -c data/tiny_corpus.txt\n```\n\nGenerate text from a trained checkpoint:\n\n```python\nfrom zenith import load_pretrained\n\ngen = load_pretrained(\"zenith-lm.pt\")\nprint(gen.generate(\"Once upon a time\", max_new_tokens=200, temperature=0.8))\n```\n\nOr from the CLI:\n\n```bash\nzenith generate -m zenith-lm.pt \"Once upon a time\" --temperature 0.8\nzenith chat -m zenith-lm.pt          # interactive REPL, streams as it generates\n```\n\nServe it over HTTP (blocking + streaming):\n\n```bash\nzenith serve -m zenith-lm.pt         # POST /generate, POST /generate/stream (SSE)\ncurl -s localhost:8000/generate -d '{\"prompt\":\"Once\",\"max_new_tokens\":100}'\n```\n\n## Architecture\n\n```text\nsrc/zenith/\n├── models/          # decoder-only transformer (from scratch)\n├── tokenizers/      # byte-level tokenizer\n├── data/            # causal-LM datasets \u0026 corpus helpers\n├── generation/      # sampling / decoding (+ streaming)\n├── training/        # causal-LM training loop\n├── peft/            # LoRA adapters\n├── distributed/     # DDP helpers\n├── tracking/        # optional MLflow experiment tracking\n├── experiments/     # environment capture \u0026 on-disk run records\n├── serving/         # FastAPI generation service (+ SSE streaming)\n├── cli/             # Hydra train entrypoint + `zenith` CLI (serve, chat, …)\n└── checkpoint.py    # self-describing save / load\n```\n\n## Project status\n\nZenith is under **active development**, mid-way through a redesign from an early,\ngeneral NLP framework into the focused generative library above. Phase 1 (the\ngenerative core: model, tokenizer, training, generation) is in place; decoding\nstrategies, PEFT, distributed training and serving follow. Interfaces may change\nuntil the first tagged release.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n\u003cdiv align=\"center\"\u003e\nby K Satya Sai Nischal\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcattolatte%2Fzenith-nlp-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcattolatte%2Fzenith-nlp-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcattolatte%2Fzenith-nlp-framework/lists"}