{"id":24213703,"url":"https://github.com/sarodyatawatta/lshm","last_synced_at":"2026-02-03T04:07:08.517Z","repository":{"id":49264791,"uuid":"310357162","full_name":"SarodYatawatta/LSHM","owner":"SarodYatawatta","description":"LOFAR System Health Management","archived":false,"fork":false,"pushed_at":"2024-05-29T13:54:50.000Z","size":123350,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-23T18:13:47.556Z","etag":null,"topics":["autoencoder","cascade-autoencoder","clustering","fft","k-harmonic-means","k-means-clustering","lofar","spectrograms"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SarodYatawatta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-11-05T16:29:06.000Z","updated_at":"2024-05-29T13:54:54.000Z","dependencies_parsed_at":"2024-05-19T20:39:17.921Z","dependency_job_id":"c55c24bf-b111-47f0-afac-a35214625847","html_url":"https://github.com/SarodYatawatta/LSHM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SarodYatawatta/LSHM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarodYatawatta%2FLSHM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarodYatawatta%2FLSHM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarodYatawatta%2FLSHM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarodYatawatta%2FLSHM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SarodYatawatta","download_url":"https://codeload.github.com/SarodYatawatta/LSHM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarodYatawatta%2FLSHM/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261287356,"owners_count":23135895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoencoder","cascade-autoencoder","clustering","fft","k-harmonic-means","k-means-clustering","lofar","spectrograms"],"created_at":"2025-01-14T03:17:44.877Z","updated_at":"2026-02-03T04:07:03.491Z","avatar_url":"https://github.com/SarodYatawatta.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LSHM : LOFAR System Health Management\nWe use a cascaded autoencoder duo with k-harmonic clustering to learn features in LOFAR spectrograms. The autoencoders work in the real space and the Fourier space of the spectrograms. We combine [Deep K-means](https://arxiv.org/abs/1806.10069) and [K Harmonic means](https://www.hpl.hp.com/techreports/2000/HPL-2000-137.html) to implement deep-K-Harmonic means clustering.\n\nFiles included are:\n\n``` lofar_models.py ``` : Methods to read LOFAR H5 data and Autoencoder models.\n\n``` kharmonic_lofar.py ``` : Train K-harmonic autoencoders (in real and Fourier space) as well as perform clustering in latent space.\n\n``` evaluate_clustering.py ``` : Load trained models and print clustering results for given dataset.\n\n``` lbfgsnew.py ``` : Improved LBFGS optimizer.\n\n``` train_graph.py ``` : Build a line-graph using baselines and train a classifier (Pytorch Geometric).\n\n\u003cimg src=\"./figures/arch.png\" alt=\"Architecture of the full system\" width=\"900\"/\u003e\n\n\nThe above image summarizes the three autoencoders that extract latent space representations and the training scheme with various constraints.\n\n\n\u003cimg src=\"./figures/errors.png\" alt=\"LBFGS vs Adam\" width=\"500\"/\u003e\n\n## How to train\nSometimes Adam will diverge (see figure above), and LBFGS will give a more stable result. Here is a training strategy that will generally work:\n\n - Set alpha=beta=gamma=0.001 (a low value), use Adam for training the autoencoders, a few epochs.\n - Increase alpha, beta, gamma (say to 0.01 and thereafter 0.1) and use LBFGS for the remainder of the training.\n - Always keep an eye for the k-harmonic loss exploding (as shown in the figure above). Control this by tuning alpha.\n - Important: divide the training into three: i) 2D CNN, ii) 1D CNN and iii) K Harmonic model, and iteratively update parameters of each of these models. This will give a stable result.\n\n\n\u003cimg src=\"./figures/examplepatch.png\" alt=\"Example input/output\" width=\"300\"/\u003e\n\nThe above figure shows of an example of the autoencoders in action: top left: input **x**, bottom left: output **xhat** of the first autoencoder,\ntop right: input **y**, bottom right: output **yhat** of the second autoencoder.\n\n\nBelow is one example, the first figure shows the t-SNE plot, and following that, the spectrograms for two closest cluster ids.\n\n\u003cimg src=\"./figures/scatter.png\" alt=\"Example input/output\" width=\"500\"/\u003e\n\n\u003cimg src=\"./figures/cluster0.png\" alt=\"Example input/output\" width=\"300\"/\u003e\n\n\u003cimg src=\"./figures/cluster1.png\" alt=\"Example input/output\" width=\"300\"/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsarodyatawatta%2Flshm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsarodyatawatta%2Flshm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsarodyatawatta%2Flshm/lists"}