{"id":18620510,"url":"https://github.com/simonsobs/mnms","last_synced_at":"2025-04-11T02:31:08.751Z","repository":{"id":96342804,"uuid":"607721443","full_name":"simonsobs/mnms","owner":"simonsobs","description":"Generate map-based noise models and noise simulations.","archived":false,"fork":false,"pushed_at":"2024-11-11T17:21:10.000Z","size":2433,"stargazers_count":2,"open_issues_count":7,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-25T08:11:32.568Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonsobs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-28T14:48:21.000Z","updated_at":"2025-02-12T14:24:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"4a51ea8f-d5c5-4dbe-b80e-7656e1aca7b3","html_url":"https://github.com/simonsobs/mnms","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fmnms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fmnms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fmnms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fmnms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonsobs","download_url":"https://codeload.github.com/simonsobs/mnms/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248329551,"owners_count":21085557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T04:06:37.815Z","updated_at":"2025-04-11T02:31:08.740Z","avatar_url":"https://github.com/simonsobs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `M`ap-based `N`oise `M`odel`S`\nServing up sugar-coated map-based models of SO/ACT data. Each model supports drawing map-based simulations. The only ingredients are data splits with independent realizations of the noise or equivalent, like an independent set of time-domain sims. \n\n## Contact\nFor any questions please reach out to Zach Atkins (email: [zatkins@princeton.edu](mailto:zatkins@princeton.edu), github: [@zatkins2](https://github.com/zatkins2)). If you use any released `mnms` products or this code in your own work, please cite [Atkins et. al. 2023](https://arxiv.org/abs/2303.04180).\n\n## Products\nProducts for the ACT DR6.01 release are available at `NERSC` and at Princeton (`della`). You can create a public account on `NERSC` following [these instructions](https://crd.lbl.gov/divisions/scidata/c3/c3-research/cosmic-microwave-background/cmb-data-at-nersc/). Follow the setup instructions for `mnms` and `sofind`, and utilize the appropriate `sofind` data model. \n\nAs an aid in forecasting studies, the noise curve products for figure 3 of [Atkins et. al. 2023](https://arxiv.org/abs/2303.04180) are available at https://phy-act1.princeton.edu/public/data/dr6_noise_v1, with an accompanying notebook.\n\n## Dependencies\nUsers wishing to filter data or generate noise simulations should have the following dependencies in their environment:\n* from `simonsobs`: [`pixell`](https://github.com/simonsobs/pixell), [`sofind`](https://github.com/simonsobs/sofind)\n* from the community: [`numba`](https://numba.pydata.org/), [`enlib`](https://github.com/amaurea/enlib)\n    * [`optweight`](https://github.com/AdriJD/optweight) is required if you are using `mnms` for any functionality beyond loading existing products to disk with `sofind`.\n\nAll other dependencies (e.g. `numpy` etc.) are required by packages listed here, especially by `pixell`.\n\nA note on [`enlib`](https://github.com/amaurea/enlib): all users need access to the top-level python modules. This is achieved just by adding the repo to your `PYTHONPATH`. **If you are only drawing new simulations or loading existing products from disk, you do not need to do anything else.** Only if you are generating new noise models, you **may** also try compiling the library `array_ops`.  This is done via `make array_ops` executed from within the top-level directory. Please see the enlib docs for more info on how to do this on your system. We have had success using an up-to-date intel `c` compiler with intel `mkl` loaded in your environment, if available. **However, enlib.array_ops is not necessary to generate new noise models.** If the `array_ops` module isn't compiled, `mnms` will fallback to using `optweight` to help make models.\n\n## Installation\nClone this repo and install it via pip:\n```shell\n$ pip install path/to/mnms\n```\nor \n```shell\n$ pip install -e path/to/mnms\n```\nto see changes to source code automatically updated in your environment. To check the installation, run tests from within `path/to/mnms`:\n\n```\n$ pytest\n```\nTests are still under construction!\n\n## Quick Setup\nAll users must create a file `.mnms_config.yaml` in their system's `HOME` directory. One is also generated for you when the repo is installed. This file encodes the location on their system of products read and written by `mnms`. This file contains only a `private_path` entry:\n```yaml\nprivate_path: \"/path/to/personal/mnms/products\"\n```\nThe `private_path` is unique to each user: it is where `mnms` will write any products the user generates themselves. This directory must contain the following subdirectories: `models` and `sims`.\n\n## Basic Usage\nThe simplest way to interact with `mnms` products in code is by instantiating a `BaseNoiseModel` subclass object from a configuration file, e.g.:\n```python\nfrom mnms import noise_models as nm\n\n# a qid is an identifier tag for a dataset, like a detector array.\n# see sofind for a list of possible qids depending on which data\n# model you load. thus, in the below, could also do ['pa4a', 'pa4b'] \n# or ['pa6a', 'pa6b']\nqids = ['pa5a', 'pa5b'] \n\n# this will load a baseline-map noise model for act_dr6v4. could also \n# do (for example) 'act_dr6v4_pwv_split' for pwv split maps (likewise el_split, inout_split), or `act_dr6.01` for dr6.01 products. these\n# correspond to the name of noise_model config files in the noise_model\n# product of sofind\nconfig_name = 'act_dr6v4' \n\n# this will load the tiled noise model. could also do 'fdw_cmbmask'\n# for directional wavelet model (or 'tile', 'wav', or 'fdw' for\n# dr6.01; see noise_models product configs in sofind). these correspond\n# to the blocks within the config file\nnoise_model_name = 'tile_cmbmask'\n\n# if you are loading a config that requires subproduct_kwargs (e.g.,  \n# 'act_dr6v4_pwv_split' maps require a 'pwv_split' argument), you need\n# to specify which subproduct_kwargs the model will include at object\n# creation. this could be nothing (e.g., for 'act_dr6v4'),\n# {'pwv_split': ['pwv1']} (e.g, for 'act_dr6v4_pwv_split'), or may be\n# a longer list like {'inout_split': ['inout1', 'inout2']} (e.g., for\n# 'act_dr6v4_inout_split'). in the latter case, passing a pair of qids\n# will result in 4 \"datasets\" (the outer product of all the qids and\n# subproduct_kwargs in the list) being jointly modeled/covaried.\nsubproduct_kwargs = {}\n# subproduct_kwargs = {'inout_split': ['inout1', 'inout2']}\n\n# instantiate NoiseModel object\ntnm = nm.BaseNoiseModel.from_config(\n    config_name,\n    noise_model_name,\n    *qids,\n    **subproduct_kwargs\n    )\n\n# grab a sim from disk, generate on-the-fly if does not exist\nmy_sim = tnm.get_sim(split_num=2, sim_num=16, lmax=10800)\n\n# grab a sim from disk, fail if does not exist on-disk\nmy_sim = tnm.get_sim(split_num=2, sim_num=16, lmax=10800, generate=False)\n\n# generate a sim on-the-fly whether or not exists on disk\nmy_sim = tnm.get_sim(split_num=2, sim_num=16, lmax=10800, check_on_disk=False)\n```\nThese method calls can also write products to disk by supplying `write=True`. Products written by users are **always** saved in their `private_path`! Note, a noise covariance matrix (i.e., a `model`) must exist before a simulation can be drawn. Such a covariance matrix can be produced via the `BaseNoiseModel.get_model` method.\n\n## Scripts\nIn addition to on-the-fly simulations as above, we provide ready-made scripts for users who wish to write a large batch of products to disk in a dedicated SLURM job. We have three kinds of scripts: `gen` only generates a noise covariance matrix from raw data inputs; `sim` only generates simulations given a noise covariance matrix on-disk; `all` first generates the covariance matrix, and then generates simulations as well. A noise covariance matrix (i.e., a `model`) must exist before a simulation can be drawn.\n\nTo protect against wasted computation, scripts check for existing products on-disk and only generate new products if they do not exist. The command-line options for each script are documented and available as \n```bash \npython noise_{all/gen/sim}.py --help\n```\nProducts written by users are **always** saved in their `private_path`!\n\n## Configs and Metadata\nThe recommended way to instantiate a `BaseNoiseModel` subclass of any type is by loading a configuration file (in fact, the provided scripts require this). A configuration file lives in the `sofind` repository, under `sofind/products/noise_models` (all configs are always `yaml` files).\n\nThe job of a config is to store the metadata that helps instantiate noise models in a centralized location. For instance, you wouldn't want to have to manage all the arguments for a `FDWNoiseModel` instance yourself! In addition, the config maps this metadata to filenames. Each config stores a `model_file_template` and a `sim_file_template` which will be populated by the kwargs in the config when those products are written. Users can experiment with changing the templates to match their preferred filename --- note, some additional kwargs provided at runtime to `get_model` and `get_sim` likely go here, e.g. `lmax` or `split_num`.\n\nTake care with filenames:\n* `mnms` will not prevent you from overwriting preexisting products in your `private_path`!\n* `mnms` **will** prevent you from writing a filename into your `private_path` if it already exists in the public location indicated by `sofind`. This ensures reading products from disk is always unambiguous as to where the product comes from.\n\n## Other Notes\n* All noise models can account for correlated detector-arrays. The array correlations are introduced automatically between `qids` if multiple `qids` are passed in the constructor to a noise model. Adding a list of subproduct_kwargs for a given subproduct key expands the datasets as the outer product of all `qids` and `subproduct_kwargs` lists. See the documentation of `noise_models.BaseNoiseModel.from_config` for more detail.\n\n* All noise models are multithreaded. Please ensure to set the environment variable `OMP_NUM_THREADS` appropriately in your slurm scripts. The bottleneck tends to be either spherical-harmonic transforms or fourier transforms. We've had success on 10-20 threads; more than that tends to incur too much overhead. The number of threads also plays a role in random number generation, so sims with identical parameters but run with a different number of threads will be different.\n\n* All map products assume the following axis assignment convention: (qid, split, polarization, y, x). If the map is loaded as an alm, the last two axes will only be one axis: the lm index. Because simulations are per-split, the second axis always has dimension 1!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fmnms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonsobs%2Fmnms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fmnms/lists"}