{"id":26225614,"url":"https://github.com/jwalsh/tokenviz","last_synced_at":"2026-04-22T11:36:15.012Z","repository":{"id":261239015,"uuid":"883698121","full_name":"jwalsh/tokenviz","owner":"jwalsh","description":"Lightweight X11/tmux-based token usage visualization for LLM systems. Simulates and monitors token consumption patterns using Unix tools (xload + named pipes) as a local development alternative to CloudWatch/Prometheus metrics. Useful for developing intuition about token usage patterns without deploying full observability stacks.","archived":false,"fork":false,"pushed_at":"2024-11-06T16:56:16.000Z","size":208,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-29T08:19:53.097Z","etag":null,"topics":["developer-tools","llm","metrics","monitoring","named-pipes","observability","tmux","token-usage","unix","visualization","x11","xload"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwalsh.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-11-05T12:24:06.000Z","updated_at":"2025-06-07T00:19:50.000Z","dependencies_parsed_at":"2024-11-05T14:03:59.069Z","dependency_job_id":null,"html_url":"https://github.com/jwalsh/tokenviz","commit_stats":null,"previous_names":["jwalsh/tokenviz"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jwalsh/tokenviz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwalsh%2Ftokenviz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwalsh%2Ftokenviz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwalsh%2Ftokenviz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwalsh%2Ftokenviz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwalsh","download_url":"https://codeload.github.com/jwalsh/tokenviz/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwalsh%2Ftokenviz/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32134782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T08:34:57.708Z","status":"ssl_error","status_checked_at":"2026-04-22T08:34:55.583Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["developer-tools","llm","metrics","monitoring","named-pipes","observability","tmux","token-usage","unix","visualization","x11","xload"],"created_at":"2025-03-12T19:17:22.558Z","updated_at":"2026-04-22T11:36:10.334Z","avatar_url":"https://github.com/jwalsh.png","language":"Shell","readme":"#+TITLE: TokenViz: Token Usage Visualization\n#+AUTHOR: Jason Walsh\n#+DATE: [2024-11-05]\n#+PROPERTY: header-args:bash :mkdirp t\n#+PROPERTY: header-args:makefile :mkdirp t\n#+PROPERTY: header-args:mermaid :mkdirp t :exports both\n#+PROPERTY: header-args :tangle yes\n#+STARTUP: showall\n\n* Overview\n\nTokenViz provides lightweight token usage visualization for LLM systems using Unix tools.\nThis simulates what would typically be monitored through enterprise observability platforms,\nmaking it useful for local development and pattern analysis.\n\n* Dashboard Preview\n\n#+CAPTION: TokenViz Dashboard showing real-time token usage visualization\n#+NAME: fig:tokenviz-dashboard\n[[file:tokenviz_20241106_104613.png]]\n\n* Quick Start\n\n#+begin_src bash\n# Launch dashboard (local)\nmake dashboard\n\n# For container environments\nDISPLAY=:1 make dashboard\n\n# View status\nmake status\n\n# Clean up\nmake stop\n#+end_src\n\n* Architecture\n\n** System Overview\n#+begin_src mermaid :file docs/images/architecture.png :tangle docs/architecture.mmd\nflowchart TD\n    subgraph Generators[\"Token Generators\"]\n        G1[\"Generator 1\\n(Random 0-3000)\"]\n        G2[\"Generator 2\\n(Random 0-3000)\"]\n        G3[\"Generator 3\\n(Random 0-3000)\"]\n    end\n\n    subgraph Storage[\"Storage Layer\"]\n        L1[\"/tmp/tokenload/gen1.log\"]\n        L2[\"/tmp/tokenload/gen2.log\"]\n        L3[\"/tmp/tokenload/gen3.log\"]\n    end\n\n    subgraph Processing[\"Aggregation Layer\"]\n        A1[\"Aggregator\\n(Sum all inputs)\"]\n        P1[\"Named Pipe\\n/tmp/tokenload_pipe\"]\n        D1[\"Data File\\n/tmp/tokenload_data\"]\n    end\n\n    subgraph Display[\"Visualization Layer\"]\n        V1[\"tmux pane 1\\nGenerator Logs\"]\n        V2[\"tmux pane 2\\nTotal Usage\"]\n        V3[\"tmux pane 3\\nxload Graph\"]\n    end\n\n    G1 --\u003e L1\n    G2 --\u003e L2\n    G3 --\u003e L3\n    L1 \u0026 L2 \u0026 L3 --\u003e A1\n    A1 --\u003e P1\n    A1 --\u003e D1\n    L1 \u0026 L2 \u0026 L3 --\u003e V1\n    D1 --\u003e V2\n    P1 --\u003e V3\n#+end_src\n\n** Data Flow\n#+begin_src mermaid :file docs/images/dataflow.png :tangle docs/dataflow.mmd\nsequenceDiagram\n    participant G as Generators\n    participant L as Log Files\n    participant A as Aggregator\n    participant P as Named Pipe\n    participant D as Display\n\n    loop Every Second\n        G-\u003e\u003eL: Write random token counts\n        L-\u003e\u003eA: Read latest values\n        A-\u003e\u003eP: Write sum to pipe\n        A-\u003e\u003eD: Update display\n    end\n#+end_src\n\n* Implementation\n** Configuration\n#+begin_src makefile :tangle Makefile\n# TokenViz Configuration\nSHELL := /bin/bash\n.PHONY: all clean test dashboard stop setup generators aggregator test-tmux test-xload status logs kill-all restart\n\n# Check if we're in a container and set DISPLAY accordingly\nCONTAINER_CHECK := $(shell test -f /.dockerenv \u0026\u0026 echo 1 || echo 0)\nifeq ($(CONTAINER_CHECK),1)\n\tXDISPLAY := :1\nelse\n\tXDISPLAY := :0\nendif\n\n# Paths\nPIPE := /tmp/tokenload_pipe\nDATA := /tmp/tokenload_data\nLOGDIR := /tmp/tokenload\nSESSION := tokenviz\n\n# Default target\n.DEFAULT_GOAL := help\n\nhelp: ## Show this help message\n\t@echo 'Usage: make [target]'\n\t@echo\n\t@echo 'Targets:'\n\t@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = \":.*?## \"}; {printf \"\\033[36m%-30s\\033[0m %s\\n\", $$1, $$2}'\n#+end_src\n\n** Core Components\n*** Setup\n#+begin_src makefile :tangle Makefile\nsetup: ## Setup\n\t@echo \"Setting up directories and files...\"\n\t@rm -rf $(LOGDIR) || true\n\t@mkdir -p $(LOGDIR)\n\t@rm -f $(PIPE) || true\n\t@mkfifo $(PIPE)\n\t@touch $(DATA)\n\t@for i in 1 2 3; do echo \"Initializing gen$$i...\" \u003e $(LOGDIR)/gen$$i.log; done\n#+end_src\n\n*** Generators\n#+begin_src makefile :tangle Makefile\ngenerators: setup ## Generators\n\t@for i in 1 2 3; do \\\n\t\t( \\\n\t\t\twhile true; do \\\n\t\t\t\tif [ -d \"$(LOGDIR)\" ]; then \\\n\t\t\t\t\techo \"gen$$i: $$((RANDOM % 3000))\" \u003e\u003e \"$(LOGDIR)/gen$$i.log\"; \\\n\t\t\t\telse \\\n\t\t\t\t\texit 0; \\\n\t\t\t\tfi; \\\n\t\t\t\tsleep 1; \\\n\t\t\tdone \\\n\t\t) \u0026 \\\n\tdone\n\naggregator: setup ## Aggregator\n\t@( \\\n\t\twhile true; do \\\n\t\t\tif [ -d \"$(LOGDIR)\" ]; then \\\n\t\t\t\tTOTAL=0; \\\n\t\t\t\tfor f in $(LOGDIR)/gen*.log; do \\\n\t\t\t\t\tif [ -f \"$$f\" ]; then \\\n\t\t\t\t\t\tVAL=$$(tail -n1 \"$$f\" 2\u003e/dev/null | grep -o '[0-9]*$$' || echo 0); \\\n\t\t\t\t\t\tTOTAL=$$((TOTAL + VAL)); \\\n\t\t\t\t\tfi; \\\n\t\t\t\tdone; \\\n\t\t\t\techo \"$$TOTAL\" \u003e \"$(PIPE)\" 2\u003e/dev/null || exit 0; \\\n\t\t\t\techo \"[`date '+%H:%M:%S'`] Total: $$TOTAL\" \u003e \"$(DATA)\" 2\u003e/dev/null || exit 0; \\\n\t\t\telse \\\n\t\t\t\texit 0; \\\n\t\t\tfi; \\\n\t\t\tsleep 1; \\\n\t\tdone \\\n\t) \u0026\n#+end_src\n\n** Process Management\n*** Dashboard\n#+begin_src makefile :tangle Makefile\ndashboard: setup ## Dashboard\n\t@echo \"Starting dashboard with DISPLAY=$(XDISPLAY)...\"\n\t@tmux new-session -d -s $(SESSION) -n 'TokenViz' \\; \\\n\t\tsplit-window -h \\; \\\n\t\tsplit-window -h \\; \\\n\t\tselect-layout even-horizontal \\; \\\n\t\tsend-keys -t 0 \"while true; do clear; tail -n 10 $(LOGDIR)/gen*.log 2\u003e/dev/null || echo 'Waiting for data...'; sleep 1; done\" C-m \\; \\\n\t\tsend-keys -t 1 \"while true; do clear; tail -n 10 $(DATA) 2\u003e/dev/null || echo 'Waiting for data...'; sleep 1; done\" C-m \\; \\\n\t\tsend-keys -t 2 \"DISPLAY=$(XDISPLAY) xload -geometry 400x200+100+100 -bg black -fg green -scale 5 \u003c $(PIPE)\" C-m \\; \\\n\t\tselect-pane -t 0\n\t@echo \"Starting generators...\"\n\t@$(MAKE) generators\n\t@echo \"Starting aggregator...\"\n\t@$(MAKE) aggregator\n\t@echo \"Attaching to session...\"\n\t@tmux attach -t $(SESSION)\n#+end_src\n\n*** Process Control\n#+begin_src makefile :tangle Makefile\nstop: ## Stop all processes\n\t@echo \"Stopping all processes...\"\n\t@pkill -f \"/bin/bash.*while true.*gen\" 2\u003e/dev/null || true\n\t@pkill -f \"while true.*TOTAL\" 2\u003e/dev/null || true\n\t@tmux kill-session -t $(SESSION) 2\u003e/dev/null || true\n\t@rm -f $(PIPE) $(DATA) 2\u003e/dev/null || true\n\t@rm -rf $(LOGDIR) 2\u003e/dev/null || true\n\t@echo \"All processes stopped\"\n\nkill-all: ## Emergency cleanup\n\t@echo \"Emergency cleanup in progress...\"\n\t@ps ax | grep \"gen.*RANDOM\" | grep -v grep | awk '{print $$1}' | xargs kill -9 2\u003e/dev/null || true\n\t@pkill -f \"while true.*TOTAL\" 2\u003e/dev/null || true\n\t@echo \"Emergency cleanup complete\"\n\nrestart: stop dashboard ## Restart all services\n#+end_src\n\n** Utility Functions\n#+begin_src makefile :tangle Makefile\nstatus: ## Status\n\t@echo \"TokenViz Status:\"\n\t@echo \"---------------\"\n\t@echo \"Environment: $$([ $(CONTAINER_CHECK) -eq 1 ] \u0026\u0026 echo 'Container' || echo 'Local')\"\n\t@echo \"Display: $(XDISPLAY)\"\n\t@echo \"\\nGenerator processes:\"\n\t@ps ax | grep \"while true.*gen\" | grep -v grep || echo \"No generators running\"\n\t@echo \"\\nAggregator process:\"\n\t@ps ax | grep \"while true.*TOTAL\" | grep -v grep || echo \"No aggregator running\"\n\t@echo \"\\nTmux session:\"\n\t@tmux has-session -t $(SESSION) 2\u003e/dev/null \u0026\u0026 echo \"Session $(SESSION) is running\" || echo \"No session running\"\n\nlogs: ## Logs\n\t@echo \"Last 5 lines from each generator:\"\n\t@for i in 1 2 3; do \\\n\t\techo \"\\nGenerator $$i:\"; \\\n\t\ttail -n 5 \"$(LOGDIR)/gen$$i.log\" 2\u003e/dev/null || echo \"No log file\"; \\\n\tdone\n\t@echo \"\\nLast 5 lines from aggregator:\"\n\t@tail -n 5 \"$(DATA)\" 2\u003e/dev/null || echo \"No aggregator data\"\n\ntest-display: ## Test display\n\t@echo \"Container detection: $(CONTAINER_CHECK)\"\n\t@echo \"Using DISPLAY=$(XDISPLAY)\"\n\t@echo \"Testing X11 connection...\"\n\t@if DISPLAY=$(XDISPLAY) xdpyinfo \u003e/dev/null 2\u003e\u00261; then \\\n\t\techo \"X11 connection successful\"; \\\n\telse \\\n\t\techo \"X11 connection failed\"; \\\n\t\texit 1; \\\n\tfi\n#+end_src\n\n** Container Support\n*** Test Display\n#+begin_src makefile :tangle Makefile\ntest-display: ## Test display\n\t@echo \"Container detection: $(CONTAINER_CHECK)\"\n\t@echo \"Using DISPLAY=$(XDISPLAY)\"\n\t@echo \"Testing X11 connection...\"\n\t@if DISPLAY=$(XDISPLAY) xdpyinfo \u003e/dev/null 2\u003e\u00261; then \\\n\t\techo \"X11 connection successful\"; \\\n\telse \\\n\t\techo \"X11 connection failed\"; \\\n\t\texit 1; \\\n\tfi\n#+end_src\n\n*** Dockerfile\n#+begin_src dockerfile :tangle Dockerfile\nFROM ubuntu:22.04\n\n# Install required packages\nRUN apt-get update \u0026\u0026 apt-get install -y \\\n    tmux \\\n    x11-apps \\\n    xauth \\\n    make \\\n    \u0026\u0026 rm -rf /var/lib/apt/lists/*\n\n# Set up working directory\nWORKDIR /app\n\n# Copy application files\nCOPY . .\n\n# Set display for X11\nENV DISPLAY=:1\n\n# Default command\nCMD [\"make\", \"dashboard\"]\n#+end_src\n\n* Cloud Equivalents\n** AWS Implementation\n#+begin_src mermaid :file docs/images/aws-impl.png\nflowchart LR\n    subgraph LLMs[\"LLM Services\"]\n        L1[\"Service 1\"]\n        L2[\"Service 2\"]\n        L3[\"Service 3\"]\n    end\n\n    subgraph Queue[\"Message Queue\"]\n        Q1[\"SNS Topic\\nToken Usage\"]\n        Q2[\"SQS Queue\\nAggregation\"]\n    end\n\n    subgraph Monitor[\"Monitoring\"]\n        M1[\"CloudWatch\\nMetrics\"]\n        M2[\"CloudWatch\\nDashboard\"]\n    end\n\n    L1 \u0026 L2 \u0026 L3 --\u003e Q1\n    Q1 --\u003e Q2\n    Q2 --\u003e M1\n    M1 --\u003e M2\n#+end_src\n\n** Kafka Implementation\n#+begin_src mermaid :file docs/images/kafka-impl.png\nflowchart LR\n    subgraph LLMs[\"LLM Services\"]\n        L1[\"Service 1\"]\n        L2[\"Service 2\"]\n        L3[\"Service 3\"]\n    end\n\n    subgraph Kafka[\"Kafka Cluster\"]\n        K1[\"Topic: token-usage\"]\n        K2[\"Topic: aggregated-usage\"]\n    end\n\n    subgraph Process[\"Processing\"]\n        P1[\"Kafka Streams\\nAggregation\"]\n    end\n\n    subgraph Monitor[\"Monitoring\"]\n        M1[\"Metrics API\"]\n        M2[\"Dashboard\"]\n    end\n\n    L1 \u0026 L2 \u0026 L3 --\u003e K1\n    K1 --\u003e P1\n    P1 --\u003e K2\n    K2 --\u003e M1\n    M1 --\u003e M2\n#+end_src\n\n** Prometheus/Grafana Implementation\n#+begin_src mermaid :file docs/images/prom-impl.png\nflowchart LR\n    subgraph LLMs[\"LLM Services\"]\n        L1[\"Service 1\\n/metrics\"]\n        L2[\"Service 2\\n/metrics\"]\n        L3[\"Service 3\\n/metrics\"]\n    end\n\n    subgraph Collect[\"Collection\"]\n        C1[\"Prometheus\\nServer\"]\n    end\n\n    subgraph Visual[\"Visualization\"]\n        V1[\"Grafana\\nDashboard\"]\n    end\n\n    L1 \u0026 L2 \u0026 L3 --\u003e C1\n    C1 --\u003e V1\n#+end_src\n\n* Contributing\n\n#+begin_src markdown :tangle CONTRIBUTING.md\n# Contributing to TokenViz\n\n## Development Setup\n\n1. Fork and clone the repository\n2. Ensure XQuartz is installed (macOS)\n3. Run tests: `make test`\n4. Submit PR with clear description\n\n## Container Development\n\n```bash\n# Build container\ndocker build -t tokenviz .\n\n# Run with X11 socket mounted\ndocker run -v /tmp/.X11-unix:/tmp/.X11-unix tokenviz\n```\n\n## Testing\n- Run `make test-display` to verify X11 setup\n- Run `make test` for full test suite\n- Ensure clean shutdown with `make stop`\n#+end_src\n\n* File Properties\n# Local Variables:\n# org-confirm-babel-evaluate: nil\n# org-src-preserve-indentation: t\n# org-edit-src-content-indentation: 0\n# whitespace-style: (face tabs spaces trailing lines space-before-tab newline indentation empty space-after-tab space-mark tab-mark newline-mark)\n# whitespace-mode: t\n# End:\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwalsh%2Ftokenviz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwalsh%2Ftokenviz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwalsh%2Ftokenviz/lists"}