{"id":36236738,"url":"https://github.com/m4n5ter/tokstream","last_synced_at":"2026-01-11T06:01:16.469Z","repository":{"id":329561703,"uuid":"1120027099","full_name":"M4n5ter/tokstream","owner":"M4n5ter","description":"A token streaming simulator powered by Hugging Face tokenizers. It downloads a tokenizer from HF Hub and generates tokens at a target rate, with live stats for target vs actual throughput.","archived":false,"fork":false,"pushed_at":"2025-12-20T10:41:17.000Z","size":1033,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-22T15:51:05.547Z","etag":null,"topics":["simulator","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/M4n5ter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-20T10:32:22.000Z","updated_at":"2025-12-20T10:41:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/M4n5ter/tokstream","commit_stats":null,"previous_names":["m4n5ter/tokstream"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/M4n5ter/tokstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/M4n5ter%2Ftokstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/M4n5ter%2Ftokstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/M4n5ter%2Ftokstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/M4n5ter%2Ftokstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/M4n5ter","download_url":"https://codeload.github.com/M4n5ter/tokstream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/M4n5ter%2Ftokstream/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28293188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T04:44:51.577Z","status":"ssl_error","status_checked_at":"2026-01-11T04:44:44.232Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["simulator","tokenizer"],"created_at":"2026-01-11T06:01:15.824Z","updated_at":"2026-01-11T06:01:16.462Z","avatar_url":"https://github.com/M4n5ter.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tokstream\n\n[中文](README_ZH.md) | English\n\nA token streaming simulator powered by Hugging Face tokenizers. It downloads a tokenizer from HF Hub and generates tokens at a target rate, with live stats for target vs actual throughput.\n\n## Highlights\n\n- Rust CLI with high‑precision pacing (sleep + spin)\n- Web demo (WASM) and npx executable\n- Random English / Chinese generation and text replay\n- Configurable filtering strategy\n- Target vs actual tokens/sec stats\n- Workspace layout with reusable core\n\n## Project Layout\n\n```\n.\n├── crates\n│   ├── tokstream-core   # tokenizer engine\n│   ├── tokstream-cli    # Rust CLI\n│   └── tokstream-wasm   # wasm-bindgen bindings\n├── npm                  # npx CLI + web demo\n├── bin                  # npm bin entry\n├── Cargo.toml           # workspace\n├── justfile\n├── package.json\n├── README.md\n└── README_ZH.md\n```\n\n## Rust CLI\n\n### Quick Start\n\n```bash\ncargo run -p tokstream-cli -- --model gpt2 --mode english --rate 8\ncargo run -p tokstream-cli -- --model gpt2 --mode chinese --rate 8\ncargo run -p tokstream-cli -- --model gpt2 --mode text --text \"Hello\" --repeat 3\n```\n\n### Install from crates.io\n\n```bash\ncargo install tokstream-cli\n# or\ncargo binstall tokstream-cli\n```\n\nNotes:\n- The binary name is `tokstream` after installation.\n- `cargo binstall` will compile from source unless you provide prebuilt release assets and set `repository` in the crate metadata.\n\n### Model \u0026 Auth\n\n- `--model \u003cid\u003e` HF Hub model id (default: `gpt2`)\n- `--revision \u003crev\u003e` HF revision (default: `main`)\n- `--hf-token \u003ctoken\u003e` access token for private models\n\n### Modes\n\n- `--mode \u003cenglish|chinese|text\u003e`\n- `--text \u003ctext\u003e` text mode input\n- `--text-file \u003cpath\u003e` text mode input from file\n- `--loop-text` loop text forever\n- `--repeat \u003cn\u003e` repeat text n times\n\n### Rate Control\n\n- `--rate \u003cn\u003e` target tokens/sec\n- `--rate-min \u003cn\u003e` min rate for random range\n- `--rate-max \u003cn\u003e` max rate for random range\n- `--rate-sample-interval \u003cn\u003e` sampling interval for rate range (seconds, default: 1)\n- `--batch \u003cn\u003e` tokens emitted per batch\n- `--max-tokens \u003cn\u003e` stop after n tokens\n\n### Pacing \u0026 Throughput\n\n- `--pace \u003cstrict|sleep\u003e` pacing mode (default: `strict`)\n- `--spin-threshold-us \u003cn\u003e` busy‑spin threshold for `strict` mode\n- `--no-throttle` disable pacing (measure max throughput)\n- `--no-output` disable stdout output (closer to tokenizer upper bound)\n\n### Stats\n\n- `--no-stats` disable stats output (stderr)\n- `--stats-interval \u003cn\u003e` stats interval seconds (default: 1)\n\n### Random Output Filters\n\n- `--no-skip-special` do not skip special tokens\n- `--allow-digits`\n- `--allow-punct`\n- `--allow-space`\n- `--allow-non-ascii`\n- `--no-require-letter`\n- `--no-require-cjk`\n\n### Seed\n\n- `--seed \u003cn\u003e` random seed\n\n### Examples\n\n```bash\n# Random rate range sampled every 2 seconds\ncargo run -p tokstream-cli -- --model gpt2 --mode english --rate-min 6 --rate-max 12 --rate-sample-interval 2\n\n# Text mode from file, repeat 5 times\ncargo run -p tokstream-cli -- --model gpt2 --mode text --text-file ./sample.txt --repeat 5\n\n# Infinite loop text\ncargo run -p tokstream-cli -- --model gpt2 --mode text --text \"Hello\" --loop-text\n\n# Throughput upper bound (no throttle, no output)\ncargo run -p tokstream-cli -- --model gpt2 --mode english --no-throttle --no-output\n```\n\n## npx CLI\n\n### Quick Start\n\n```bash\nnpx tokstream@latest --model gpt2 --mode english --rate 8\nnpx tokstream@latest --web --port 8787\n```\n\nFor local development in this repo:\n\n```bash\nnpx . --model gpt2 --mode english --rate 8\n```\n\n### Supported Flags (npx)\n\n- `--model \u003cid\u003e`\n- `--revision \u003crev\u003e`\n- `--hf-token \u003ctoken\u003e` (or env `HF_TOKEN` / `HUGGINGFACE_HUB_TOKEN`)\n- `--mode \u003cenglish|chinese|text\u003e`\n- `--text \u003ctext\u003e`\n- `--loop` (loop text forever)\n- `--repeat \u003cn\u003e`\n- `--rate \u003cn\u003e`\n- `--rate-min \u003cn\u003e` / `--rate-max \u003cn\u003e`\n- `--rate-sample-interval \u003cn\u003e`\n- `--seed \u003cn\u003e`\n- `--max-tokens \u003cn\u003e`\n- `--no-skip-special`\n- `--allow-digits` / `--allow-punct` / `--allow-space` / `--allow-non-ascii`\n- `--no-require-letter` / `--no-require-cjk`\n- `--no-stats` / `--stats-interval \u003cn\u003e`\n- `--no-throttle` / `--no-output`\n- `--web --port \u003cn\u003e`\n\nNotes:\n- `--loop-text`, `--text-file`, `--batch`, `--pace`, and `--spin-threshold-us` are Rust‑CLI only.\n\n## Web Demo\n\n```bash\nnpx tokstream@latest --web --port 8787\n# open http://localhost:8787\n```\n\nWhile running, you can drag the rate slider or enable random rate range. The page shows target and actual throughput. The output pane is fixed‑height and scrolls independently.\n\n## Accuracy Notes\n\n- Rust CLI `strict` uses sleep + short spin for high precision.\n- Web / npx are best‑effort due to event loop and I/O limits.\n- If actual throughput doesn’t change while raising target rates, you likely hit tokenizer limits.\n- For maximum throughput testing, use the Rust CLI with `--no-output --no-throttle`.\n\n## Build WASM (optional refresh)\n\n```bash\nnpm run build:wasm\n```\n\nWASM artifacts are committed and included in the npm package.\n\n## just Recipes\n\n```bash\njust\n```\n\n## Tests\n\n```bash\ncargo clippy --workspace\ncargo nextest run --workspace\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm4n5ter%2Ftokstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm4n5ter%2Ftokstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm4n5ter%2Ftokstream/lists"}