{"id":34072525,"url":"https://github.com/ristoale97/centered-kernel-alignment","last_synced_at":"2025-12-14T08:00:54.065Z","repository":{"id":223054271,"uuid":"755159407","full_name":"RistoAle97/centered-kernel-alignment","owner":"RistoAle97","description":"CKA (Centered Kernel Alignment) implemented in PyTorch","archived":false,"fork":false,"pushed_at":"2025-10-04T12:45:24.000Z","size":592,"stargazers_count":43,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-07T18:07:36.418Z","etag":null,"topics":["deep-learning","neural-networks","representation-similarity"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RistoAle97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-09T14:48:37.000Z","updated_at":"2025-10-04T12:45:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e101df2-1b7d-49e3-988f-e7e5ec14fe26","html_url":"https://github.com/RistoAle97/centered-kernel-alignment","commit_stats":null,"previous_names":["ristoale97/centered-kernel-alignment"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/RistoAle97/centered-kernel-alignment","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RistoAle97%2Fcentered-kernel-alignment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RistoAle97%2Fcentered-kernel-alignment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RistoAle97%2Fcentered-kernel-alignment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RistoAle97%2Fcentered-kernel-alignment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RistoAle97","download_url":"https://codeload.github.com/RistoAle97/centered-kernel-alignment/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RistoAle97%2Fcentered-kernel-alignment/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27722636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-14T02:00:11.348Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","neural-networks","representation-similarity"],"created_at":"2025-12-14T08:00:53.083Z","updated_at":"2025-12-14T08:00:54.056Z","avatar_url":"https://github.com/RistoAle97.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🤖 CKA PyTorch 🤖\n**CKA (Centered Kernel Alignment) in PyTorch.**\n\n[![Python](https://img.shields.io/badge/Python-3776AB?style=for-the-badge\u0026logo=python\u0026logoColor=white)](https://github.com/python/cpython)\n[![Pytorch](https://img.shields.io/badge/PyTorch-EE4C2C?style=for-the-badge\u0026logo=pytorch\u0026logoColor=white)](https://github.com/pytorch/pytorch)\n\n[![PyPI](https://img.shields.io/pypi/v/ckatorch.svg?style=for-the-badge\u0026logo=pypi\u0026logoColor=white)](https://pypi.org/project/ckatorch/)\n[![Python versions](https://img.shields.io/pypi/pyversions/ckatorch.svg?style=for-the-badge\u0026logo=python\u0026logoColor=white)](https://pypi.org/project/ckatorch/)\n\n\n\u003c/div\u003e\n\n\u003e [!WARNING]\n\u003e This repository has been built mainly for personal and academic use since \u003cimg height=\"15\" width=\"15\" src=\"https://cdn.simpleicons.org/pytorch\"/\u003e[Captum](https://github.com/pytorch/captum) still needs to implement its variant of CKA. As such, do not expect this project to work for every model.\n\n---\n\n## ✒️ About\n\u003e [!NOTE]\n\u003e Centered Kernel Alignment (CKA) [1] is a similarity index between representations of features in neural networks, based on the Hilbert-Schmidt Independence Criterion (HSIC) [2]. Given a set of examples, CKA compares the representations of examples passed through the layers that we want to compare.\n\nGiven two matrices $X \\in \\mathbb{R}^{n\\times s_1}$ and $Y \\in \\mathbb{R}^{n\\times s_2}$ representing the output of two layers, we can define two auxiliary $n \\times n$ Gram matrices $K=XX^T$ and $L=YY^T$ and compute the *dot-product similarity* between them\n\n$$\\langle vec(XX^T), vec(YY^T)\\rangle = tr(XX^T YY^T) = \\lVert Y^T X \\rVert_F^2.$$\n\nThen, the $HSIC$ on $K$ and $L$ is defined as\n\n$$HSIC_0(K, L) = \\frac{tr(KHLH)}{(n - 1)^2},$$\n\nwhere $H = I_n - \\frac{1}{n}J_n$ is the centering matrix and $J_n$ is an $n \\times n$ matrix filled with ones. Finally, to obtain the CKA value we only need to normalize $HSIC_0$\n\n$$CKA(K, L) = \\frac{HSIC(K, L)}{\\sqrt{HSIC(K, K) HSIC(L, L)}}.$$\n\n\u003e [!NOTE]\n\u003e However, naive computation of linear CKA (i.e.: the previous equation) requires maintaining the activations across the entire dataset in memory, which is challenging for wide and deep networks [3].\n\nTherefore, we need to define the unbiased estimator of HSIC so that the value of CKA is independent of the batch size\n\n$$HSIC_1(K, L)=\\frac{1}{n(n-3)}\\left( tr(\\tilde{K}, \\tilde{L}) + \\frac{1^T\\tilde{K}11^T\\tilde{L}1}{(n-1)(n-2)} - \\frac{2}{n-2}1^T\\tilde{K}\\tilde{L}1\\right),$$\n\nwhere $\\tilde{K}$ and $\\tilde{L}$ are obtained by setting the diagonal entries of $K$ and $L$ to zero. Finally, we can compute the minibatch version of CKA by averaging $HSIC_1$ scores over $k$ minibatches\n\n$$CKA_{minibatch}=\\frac{\\frac{1}{k} \\displaystyle\\sum_{i=1}^{k} HSIC_1(K_i, L_i)}{\\sqrt{\\frac{1}{k} \\displaystyle\\sum_{i=1}^{k} HSIC_1(K_i, K_i)}\\sqrt{\\frac{1}{k} \\displaystyle\\sum_{i=1}^{k} HSIC_1(L_i, L_i)}},$$\n\nwith $K_i=X_iX_i^T$ and $L_i=Y_iY_i^T$, where $X_i \\in \\mathbb{R}^{m \\times p_1}$ and $Y_i \\in \\mathbb{R}^{m \\times p_2}$ are now matrices containing activations of the $i^{th}$ minibatch of $m$ examples sampled without replacement [3].\n\n---\n\n## 📦 Installation\nThis project requires python \u003e= 3.10.\n\n### Create a new venv\n```bash\n# If you have uv installed\nuv venv\n\n# Otherwise\npython -m venv .venv\n\n# Activate the virtual environment\nsource .venv/bin/activate  # if you are on Linux\n.\\.venv\\Scripts\\activate.bat  # if you are using the cmd on Windows\n.\\.venv\\Scripts\\Activate.ps1  # if you are using the PowerShell on Windows\n```\n\n### Install the package\n\u003e [!NOTE]\n\u003e This will install \u003cimg height=\"15\" width=\"15\" src=\"https://cdn.simpleicons.org/pytorch\"/\u003ePyTorch compiled with CUDA.\n\nYou can install the package:\n- _from PyPI_\n  ```bash\n  # Using uv\n  uv pip install ckatorch\n\n  # Using pip\n  pip install ckatorch\n  ```\n\n- _from this repo_\n  ```bash\n  # Using uv\n  uv pip install git+https://github.com/RistoAle97/centered-kernel-alignment\n\n  # Using pip\n  pip install git+https://github.com/RistoAle97/centered-kernel-alignment\n  ```\n\n- _by cloning the repository and installing the dependencies_\n  ```bash\n  git clone https://github.com/RistoAle97/centered-kernel-alignment\n\n  # If you have uv installed\n  uv pip install -e centered-kernel-alignment\n  uv pip install ckatorch --group dev  # if you want to also install the dev dependencies\n\n  # Otherwise\n  pip install -e centered-kernel-alignment\n  pip install ckatorch --group dev # same as for uv, remember to open a pull request afterwards\n  ```\n\nTake a look at the `examples` directory to understand how to compute CKA in two basic scenarios.\n\n---\n\n## 🖼️\tPlots\n\u003e [!NOTE]\n\u003e The comparison makes more sense if the models share a common architecture.\n\nModel compared with itself             |  Different models compared\n:-------------------------:|:-------------------------:\n![Model compared with itself](https://raw.githubusercontent.com/RistoAle97/centered-kernel-alignment/refs/heads/main/plots/model_comparison_itself.png)  |  ![Model comparison](https://raw.githubusercontent.com/RistoAle97/centered-kernel-alignment/refs/heads/main/plots/model_comparison.png)\n\n---\n\n## 📚 Bibliography\n[1] Kornblith, Simon, et al. [\"Similarity of neural network representations revisited.\"](https://arxiv.org/abs/1905.00414) *International Conference on Machine Learning*. PMLR, 2019.\n\n[2] Wang, Tinghua, Xiaolu Dai, and Yuze Liu. [\"Learning with Hilbert–Schmidt independence criterion: A review and new perspectives.\"](https://www.sciencedirect.com/science/article/pii/S0950705121008297) *Knowledge-based systems* 234 (2021): 107567.\n\n[3] Nguyen, Thao, Maithra Raghu, and Simon Kornblith. [\"Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth.\"](https://arxiv.org/abs/2010.15327) *arXiv preprint* arXiv:2010.15327 (2020).\n\nThis project is also based on the following repositories:\n- [representation_similarity](https://github.com/google-research/google-research/tree/master/representation_similarity) (original implementation).\n- [PyTorch-Model-Compare](https://github.com/AntixK/PyTorch-Model-Compare) (nice PyTorch implementation that employs hooks).\n- [CKA.pytorch](https://github.com/numpee/CKA.pytorch) (minibatch version of CKA and useful batched implementation of $HSIC_1$).\n\n---\n\n## 📝 License\nThis project is [MIT licensed](https://github.com/RistoAle97/centered-kernel-alignment/blob/main/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fristoale97%2Fcentered-kernel-alignment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fristoale97%2Fcentered-kernel-alignment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fristoale97%2Fcentered-kernel-alignment/lists"}