{"id":20099129,"url":"https://github.com/spack/spack-stack-catalog","last_synced_at":"2025-05-06T06:31:07.712Z","repository":{"id":37727261,"uuid":"369376845","full_name":"spack/spack-stack-catalog","owner":"spack","description":"A community catalog of spack stacks  ⭐","archived":false,"fork":false,"pushed_at":"2024-05-01T10:55:56.000Z","size":1904481,"stargazers_count":4,"open_issues_count":2,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-05-01T21:40:08.040Z","etag":null,"topics":["catalog","spack","spack-stack"],"latest_commit_sha":null,"homepage":"https://spack.github.io/spack-stack-catalog/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-21T01:03:49.000Z","updated_at":"2024-05-03T12:06:18.184Z","dependencies_parsed_at":"2024-05-03T12:16:22.728Z","dependency_job_id":null,"html_url":"https://github.com/spack/spack-stack-catalog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spack%2Fspack-stack-catalog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spack%2Fspack-stack-catalog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spack%2Fspack-stack-catalog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spack%2Fspack-stack-catalog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spack","download_url":"https://codeload.github.com/spack/spack-stack-catalog/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224490710,"owners_count":17319983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["catalog","spack","spack-stack"],"created_at":"2024-11-13T17:08:23.303Z","updated_at":"2025-05-06T06:31:07.706Z","avatar_url":"https://github.com/spack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spack Stack Catalog\n\n![assets/img/spack-stacks-blue.png](assets/img/spack-stacks-blue.png)\n\nThis is the spack stack community catalog, or spack.yaml files discovered\nin the wild across the community! It is inspired by the [snakemake-workflow-catalog](https://github.com/snakemake/snakemake-workflow-catalog)\nthat is released under an [MIT LICENSE](.github/SNAKEMAKE-LICENSE).\n\n⭐ [View the Catalog](https://spack.github.io/spack-stack-catalog/) ⭐\n\n**this repository is still under development!**\n\n## Usage\n\n### Install Dependencies\n\nThe update of repositories that have spack.yaml is done during an automated CI,\nhowever you might want to run the update script locally. You can first install\nrequirments:\n\n```bash\npython -m venv env\npip install -r requirements.txt\n```\n\nand spack should also be on your path.\n\n```bash\nwhich spack\n```\n\n### Environment\n\nYou'll need to export a personal access token as `GITHUB_TOKEN`:\n\n```bash\nexport GITHUB_TOKEN=xxxxxxxxxxx\n```\n\n### Run update script\n\nThen, run the script!\n\n```bash\npython scripts/generate-catalog.py\n```\n\nThis should be run nightly to update the stacks here.\n\n### Generate graph\n\nThe script [generate-graph-data.py](scripts/generate-graph-data.py) will parse\nthe current set of spack.yaml files and count the number of times packages are\ninstalled with one another, not accounting for versions. We then create\na similarity matrix based on the counts, and visualize that. More specifically:\n\n1. Make a matrix that counts \"similarity\" between packages. The similarity is based e.g., on the number of times packages that appear together. We don't need to worry about the diagonal (e.g., leave it as NaN).\n2. We need to normalize by packages that appear across a lot of packages. So we want to divide by the square root of the row sums and the columns sums. We do this so it's still symmetric, and we use a square root because we want the units to go away.\n3. Transform the counts to a distance e.g., [using one of these functions](https://stackoverflow.com/questions/4064630/how-do-i-convert-between-a-measure-of-similarity-and-a-measure-of-difference-di). TLDR - we do distance = 1/ our normalized count matrix.\n4. Finally, use dimensionality reduction to find coordinates for each package. Packages that appear frequently together will be closer. E.g., [isomap](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.Isomap.html) or [TSNE](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html).\n\n[![assets/img/spack-package-togetherness.png](assets/img/spack-package-togetherness.png)](https://spack.github.io/spack-stack/catalog/similarity/)\n\nThe embedding [on the catalog](https://spack.github.io/spack-stack/catalog/similarity/) shows packages that are commonly found with one another. The closer the dots, the more likely the two packages are found in a spack.yaml together. The size of the dot shows the number of distinct packages that a package appears with. This means that larger dots appear with more packages. The smallest black dots likely appear with few other packages. This does not include dependencies - only the list of specs asked for in a spack.yaml.\n\n## License\n\nSpack is distributed under the terms of both the MIT license and the\nApache License (Version 2.0). Users may choose either license, at their\noption.\n\nAll new contributions must be made under both the MIT and Apache-2.0\nlicenses.\n\nSee [LICENSE-MIT](https://github.com/spack/spack/blob/develop/LICENSE-MIT),\n[LICENSE-APACHE](https://github.com/spack/spack/blob/develop/LICENSE-APACHE),\n[COPYRIGHT](https://github.com/spack/spack/blob/develop/COPYRIGHT), and\n[NOTICE](https://github.com/spack/spack/blob/develop/NOTICE) for details.\n\nSPDX-License-Identifier: (Apache-2.0 OR MIT)\n\nLLNL-CODE-811652\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspack%2Fspack-stack-catalog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspack%2Fspack-stack-catalog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspack%2Fspack-stack-catalog/lists"}