{"id":22012724,"url":"https://github.com/onolab-tmu/blinky-iva","last_synced_at":"2025-09-09T15:36:05.908Z","repository":{"id":103217442,"uuid":"153260816","full_name":"onolab-tmu/blinky-iva","owner":"onolab-tmu","description":"Multimodal formulation of IVA using conventional microphones and power sensing blinkies.","archived":false,"fork":false,"pushed_at":"2019-05-23T01:54:33.000Z","size":214,"stargazers_count":13,"open_issues_count":0,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-06T21:49:07.462Z","etag":null,"topics":["blind-source-separation","blinky","independent-vector-analysis","microphone-array","multi-modal","unsupervised-learning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1904.02334","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onolab-tmu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-10-16T09:41:14.000Z","updated_at":"2024-08-26T15:44:26.000Z","dependencies_parsed_at":"2023-06-29T01:15:39.073Z","dependency_job_id":null,"html_url":"https://github.com/onolab-tmu/blinky-iva","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/onolab-tmu/blinky-iva","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onolab-tmu%2Fblinky-iva","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onolab-tmu%2Fblinky-iva/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onolab-tmu%2Fblinky-iva/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onolab-tmu%2Fblinky-iva/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onolab-tmu","download_url":"https://codeload.github.com/onolab-tmu/blinky-iva/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onolab-tmu%2Fblinky-iva/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274318577,"owners_count":25263105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blind-source-separation","blinky","independent-vector-analysis","microphone-array","multi-modal","unsupervised-learning"],"created_at":"2024-11-30T03:14:11.692Z","updated_at":"2025-09-09T15:36:05.894Z","avatar_url":"https://github.com/onolab-tmu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Multi-modal Blind Source Separation with Microphones and Blinkies\n=================================================================\n\nThis repository provides implementations and code to reproduce the results\nof the paper\n\n\u003e Robin Scheibler and Nobutaka Ono, [*\"Multi-modal Blind Source Separation with Microphones and Blinkies,\"*](https://arxiv.org/abs/1904.02334) Proc. IEEE ICASSP, Brighton UK, 2019.\n\nAbstract\n--------\n\nWe propose a blind source separation algorithm that jointly exploits\nmeasurements by a conventional microphone array and an ad hoc array of low-rate\nsound power sensors called blinkies. While providing less information than\nmicrophones, blinkies circumvent some difficulties of microphone arrays in\nterms of manufacturing, synchronization, and deployment. The algorithm is\nderived from a joint probabilistic model of the microphone and sound power\nmeasurements. We assume the separated sources to follow a time-varying\nspherical Gaussian distribution, and the non-negative power measurement\nspace-time matrix to have a low-rank structure. We show that alternating\nupdates similar to those of independent vector analysis and Itakura-Saito\nnon-negative matrix factorization decrease the negative log-likelihood of the\njoint distribution. The proposed algorithm is validated via numerical\nexperiments. Its median separation performance is found to be up to 8 dB more\nthan that of independent vector analysis, with significantly reduced\nvariability.\n\nAuthors\n-------\n\n[Robin Scheibler](http://robinscheibler.org) and [Nobutaka\nOno](http://www.comp.sd.tmu.ac.jp/onolab/index-e.html) are with the Faculty of\nSystem Design at [Tokyo Metropolitan University](https://www.tmu.ac.jp/english/index.html).\n\n### Contact\n\n    Robin Scheibler (robin[at]tmu[dot]ac[dot]jp)\n    6-6 Asahigaoka\n    Hino, Tokyo\n    191-0065 Japan\n\nPreliminaries\n-------------\n\nThe preferred way to run the code is using [anaconda](https://www.anaconda.com/distribution/).\nAn `environment.yml` file is provided to install the required dependencies.\n\n    # create the minimal environment\n    conda env create -f environment.yml\n\n    # switch to new environment\n    conda activate 2019_scheibler_icassp_blinkiva\n\nTest BlinkIVA\n-------------\n\nThe algorithm can be tested and compared to others using the sample\nscript `mbss_oneshot.py`. It can be run as follows.\n\n    $ python ./mbss_oneshot.py --help\n    usage: mbss_oneshot.py [-h] [-b BLOCK] [-m MICS] [-s {1,2,3,4,5,6,7,8,9,10}]\n                           [-a {blinkiva,ilrma,auxiva,auxiva-gauss}] [--gui]\n                           [--save]\n\n    Demonstration of blind source separation using microphones and blinkies.\n\n    optional arguments:\n      -h, --help            show this help message and exit\n      -b BLOCK, --block BLOCK\n                            STFT block size\n      -m MICS, --mics MICS  Number of microphones\n      -s {1,2,3,4,5,6,7,8,9,10}, --srcs {1,2,3,4,5,6,7,8,9,10}\n                            Number of sources\n      -a {blinkiva,ilrma,auxiva,auxiva-gauss}, --algo {blinkiva,ilrma,auxiva,auxiva-gauss}\n                            Chooses BSS method to run\n      --gui                 Creates a small GUI for easy playback of the sound\n                            samples\n      --save                Saves the output of the separation to wav files\n\nFor example, we can run BlinkIVA with 4 microphones and 2 sources.\n\n    python ./mbss_oneshot.py -a blinkiva -m 4 -s 2\n\nReproduce the Results\n---------------------\n\nThe code can be run serially, or using multiple parallel workers via\n[ipyparallel](https://ipyparallel.readthedocs.io/en/latest/).\nMoreover, it is possible to only run a few loops to test whether the\ncode is running or not.\n\n1. Run **test** loops **serially**\n\n        python ./mbss_sim.py ./mbss_sim_config.json -t -s\n\n2. Run **test** loops in **parallel**\n\n        # start workers in the background\n        # N is the number of parallel process, often \"# threads - 1\"\n        ipcluster start --daemonize -n N\n\n        # run the simulation\n        python ./mbss_sim.py ./mbss_sim_config.json -t\n\n        # stop the workers\n        ipcluster stop\n\n3. Run the whole simulation\n\n        # start workers in the background\n        # N is the number of parallel process, often \"# threads - 1\"\n        ipcluster start --daemonize -n N\n\n        # run the simulation\n        python ./mbss_sim.py ./mbss_sim_config.json\n\n        # stop the workers\n        ipcluster stop\n\nThe results are saved in a new folder `data/\u003cdata\u003e-\u003ctime\u003e_mbss_sim_\u003cflag_or_hash\u003e`\ncontaining the following files\n\n    parameters.json  # the list of global parameters of the simulation\n    arguments.json  # the list of all combinations of arguments simulated\n    data.json  # the results of the simulation\n\nFigure 3. from the paper is produced then by running\n\n    python ./mbss_sim_plot.py data/\u003cdata\u003e-\u003ctime\u003e_mbss_sim_\u003cflag_or_hash\u003e -s\n\nData\n----\n\nFor the experiment, we concatenated utterances from the CMU ARCTIC speech corpus to\nobtain samples of at least 15 seconds long. The dataset thus created was stored on zenodo\nwith DOI [10.5281/zenodo.3066488](https://zenodo.org/record/3066489). The data is automatically\nretrieved upon running the scripts, but can also be manually downloaded with the `get_data.py` script.\n\n    python ./get_data.py\n\nIt is stored in the `samples` directory.\n\nUse BlinkIVA\n------------\n\nOur implementation of BlinkIVA can be found in the file `blinkiva_gauss.py`.\nIt can be used as follows.\n\n    from blinkiva_gauss import blinkiva_gauss\n\n    # STFT tensor, a numpy.ndarray with shape (frames, frequencies, channels)\n    X = ...\n\n    # The blinky matrix is an numpy.ndarray with shape (frames, blinkies)\n    U = ...\n\n    # perform separation, output Y has the same shape as X\n    Y = blinkiva_gauss(X, U, n_src=2, n_iter=50)\n\nThe function comes with docstrings.\n\n    blinkiva_gauss(X, U, n_src=None, n_iter=20, n_nmf_sub_iter=20, proj_back=True,\n                   W0=None, R0=None, seed=None, epsilon=0.5, sparse_reg=0.,\n                   print_cost=False, return_filters=False, callback=None)\n\n    Implementation of BlinkIVA algorithm for blind source separation using jointly\n    microphones and sound power sensors \"blinkies\". The algorithm was presented in\n\n    R. Scheibler and N. Ono, *Multi-modal Blind Source Separation with Microphones and Blinkies,*\n    Proc. IEEE ICASSP, Brighton, UK, May, 2019.  DOI: 10.1109/ICASSP.2019.8682594\n    https://arxiv.org/abs/1904.02334\n\n    Parameters\n    ----------\n    X: ndarray (nframes, nfrequencies, nchannels)\n        STFT representation of the signal\n    U: ndarray (nframes, nblinkies)\n        The matrix containing the blinky signals\n    n_src: int, optional\n        The number of sources or independent components\n    n_iter: int, optional\n        The number of iterations (default 20)\n    n_nmf_sub_iter: int, optional\n        The number of NMF iteration to run between two updates of the demixing\n        matrices (default 20)\n    proj_back: bool, optional\n        Scaling on first mic by back projection (default True)\n    W0: ndarray (nfrequencies, nchannels, nchannels), optional\n        Initial value for demixing matrix\n    R0: ndarray (nframes, nsrc), optional\n        Initial value of the activations\n    seed: int, optional\n        A seed to make deterministic the random initialization of NMF parts,\n        when None (default), the random number generator is used in its current state\n    epsilon: float, optional\n        A regularization value to prevent too large values after the division\n    sparse_reg: float\n        A regularization term to make the activation matrix sparse\n    print_cost: bool, optional\n        Print the value of the cost function at each iteration\n    return_filters: bool, optional\n        If true, the function will return the demixing matrix, gains, and activations too\n    callback: func\n        A callback function called every 10 iterations, allows to monitor convergence\n\n    Returns\n    -------\n    Returns an (nframes, nfrequencies, nsources) array. Also returns\n    the demixing matrix (nfrequencies, nchannels, nsources), gains (nsrc, nblinkies),\n    and activations (nframes, nchannels) if ``return_filters`` keyword is True.\n\nSummary of the Files in this Repo\n---------------------------------\n\n    environment.yml  # anaconda environment file\n\n    auxiva_gauss.py  # implementation of auxiva with time-varying Gauss source model\n    blinkiva_gauss.py  # implementation of blind source separation with microphones and blinkies\n    get_data.py  # script that gets the data necessary for the experiment\n    routines.py  # contains a bunch of helper routines for the simulation\n\n    mbss_oneshot.py  # test file for source separation, with audible output\n    mbss_sim.py  # script to run exhaustive simulation, used for the paper\n    mbss_sim_config.json  # simulation configuration file\n    mbss_sim_plot.py  # plots the figures from the output of overiva_sim.py\n\n    data  # directory containing simulation results\n    rrtools  # tools for parallel simulation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonolab-tmu%2Fblinky-iva","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonolab-tmu%2Fblinky-iva","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonolab-tmu%2Fblinky-iva/lists"}