{"id":16766144,"url":"https://github.com/cryscan/web-rwkv-inspector","last_synced_at":"2025-03-23T15:31:16.359Z","repository":{"id":235856402,"uuid":"791397453","full_name":"cryscan/web-rwkv-inspector","owner":"cryscan","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-21T16:08:28.000Z","size":43536,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T21:52:07.314Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cryscan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-24T16:40:08.000Z","updated_at":"2024-12-21T16:08:08.000Z","dependencies_parsed_at":"2024-04-24T19:15:44.809Z","dependency_job_id":"560b1742-1ea1-43b6-a2d2-e4fea9a543ad","html_url":"https://github.com/cryscan/web-rwkv-inspector","commit_stats":{"total_commits":49,"total_committers":2,"mean_commits":24.5,"dds":"0.020408163265306145","last_synced_commit":"3b44a7c87d708f24fa4307c17c142054ece3d5c5"},"previous_names":["cryscan/web-rwkv-inspector"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryscan%2Fweb-rwkv-inspector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryscan%2Fweb-rwkv-inspector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryscan%2Fweb-rwkv-inspector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryscan%2Fweb-rwkv-inspector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cryscan","download_url":"https://codeload.github.com/cryscan/web-rwkv-inspector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245122739,"owners_count":20564364,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T06:05:31.679Z","updated_at":"2025-03-23T15:31:12.522Z","avatar_url":"https://github.com/cryscan.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web-RWKV Inspector\n\nThis is an inspector for the language model of [RWKV X060](https://github.com/BlinkDL/RWKV-LM). It lets you look into the inner state of the model during inference.\n\n## Download\n\nDownload pre-built binaries at [Release](https://github.com/cryscan/web-rwkv-inspector/releases).\n\n## Introduction\n\n![3b](assets/screenshots/3b.png)\n\nThis is a screenshot of inspecting into a [3B RWKV model](https://huggingface.co/BlinkDL/rwkv-6-world/blob/main/RWKV-x060-World-3B-v2.1-20240417-ctx4096.pth).\nThe inspector computes the attention score (`QK` dot product for transformers and `R-WK` dot product for `RWKV`) between two tokens `i` and `j` (`i` is less than `j`), for each head for each layer.\nThe formula of the `R-WK` score is:\n\n![rwk](assets/screenshots/rwk.png).\n\nThe blue token is the token that is selected, and for each token before and after it, we compute an attention score between the selected and that token.\nRed ones have a positive score; green ones have a negative score.\n\nIn this picture, we can see that the head `L5 H10` has some kind of *retrieval behavior*: note that the selected token (\"based\") is part of a summary of the above text, and this head attends to the selected token to its follower (\"ambient\") in the source text, which indicated that this head is looking for possible next-word in the given text above.\n\nAlso there is another visualization mode, namely `WK` norm, which is essentially a head's influence span for a certain token:\n\n![wk](assets/screenshots/wk.png).\n\nThe prompt used in this example is:\n\n```text\nInstruction:\nSummarize the given input in one sentence.\n\nInput:\nA two pass method for calculating point based ambient occlusion was implemented\non the GPU using GLSL and OpenGL, in the\ncourse TNCG14 - Advanced Computer Graphics at Linkoping University. The program can ¨\nload in wavefront-object files and render them\nwith point-based ambient occlusion. The result is a good approximation of a ray-traced\nambient occlusion render but it is rendered\nwith a greater speed than with a ray-traced\nmethod. A point-based rendering visualization method called ”Affinely projected pointsprites” was implemented and compared with\nthe normal way of visualizing objects with triangles.\n\nResponse:\nA GPU-based ambient occlusion method was implemented using GLSL and OpenGL, and compared to a ray-traced method.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcryscan%2Fweb-rwkv-inspector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcryscan%2Fweb-rwkv-inspector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcryscan%2Fweb-rwkv-inspector/lists"}