{"id":18620506,"url":"https://github.com/simonsobs/sofind","last_synced_at":"2025-05-07T08:22:51.415Z","repository":{"id":96342810,"uuid":"607279855","full_name":"simonsobs/sofind","owner":"simonsobs","description":"Simple, extendable framework for loading generic products from disk. ","archived":false,"fork":false,"pushed_at":"2025-04-29T13:09:24.000Z","size":245,"stargazers_count":0,"open_issues_count":3,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-29T14:28:47.139Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonsobs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-27T17:18:57.000Z","updated_at":"2025-04-18T17:46:40.000Z","dependencies_parsed_at":"2024-04-14T11:25:00.171Z","dependency_job_id":"fb91a234-51dd-418b-bc5c-cb91826db838","html_url":"https://github.com/simonsobs/sofind","commit_stats":{"total_commits":153,"total_committers":5,"mean_commits":30.6,"dds":0.542483660130719,"last_synced_commit":"07408630b90ca6654fdccbd8d34db10cf189c6fa"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsofind","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsofind/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsofind/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsofind/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonsobs","download_url":"https://codeload.github.com/simonsobs/sofind/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252839857,"owners_count":21812185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T04:06:37.737Z","updated_at":"2025-05-07T08:22:51.394Z","avatar_url":"https://github.com/simonsobs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `S`imons `O`bservatory `FI`le`N`ame `D`epot\nSimple, extendable framework for loading generic products from disk.\n\n## Contact\nFor any questions please reach out to Zach Atkins (email: [zatkins@princeton.edu](mailto:zatkins@princeton.edu), github: [@zatkins2](https://github.com/zatkins2)).\n\n## Dependencies\nCurrently:\n* from `simonsobs`: [`pixell`](https://github.com/simonsobs/pixell)\n    * [`mnms`](https://github.com/simonsobs/mnms) is required if you are using `sofind` for any functionality related to `mnms` products.\n\nAll other dependencies (e.g. `numpy` etc.) are required by packages listed here, especially by `pixell`.\n\n## Installation\nClone this repo and install it via pip:\n```shell\n$ pip install path/to/sofind\n```\nor \n```shell\n$ pip install -e path/to/sofind\n```\nto see changes to source code automatically updated in your environment.\n\n## Quick Setup\nUsers only need to set one environment variable, `SOFIND_SYSTEM`, corresponding to the name of the cluster they are working on. Supported systems are listed in `sofind/systems.py` (e.g. `della`, `perlmutter`).\n\n## Usage\nAll you need in your code is the following (e.g. for the `act_dr6v4` data model):\n```python\nfrom sofind import DataModel\n\n# use the from_config method to specify the data model at runtime\ndm = DataModel.from_config('act_dr6v4')\n\n# a qid is an identifier tag for a dataset, like a detector array\nmy_qid = 'pa4a'\n\n# have fun\nmy_default_map_filename = dm.get_map_fn(my_qid, **more_kwargs)\nmy_el_split_map = dm.read_map(my_qid, subproduct='el_split', el_split='el1')\n```\nUsers should only ever interface with the high-level `DataModel` class. This class inherits from all implemented `sofind` products! Available `qids` for a particular data model or product are available either by following the `qids_config` entry in the data model config itself (referring to a particular file in `sofind/qids`) or should be documented for the product (see, e.g., the `sofind/products/maps` README file).\n\n## If you would like to add a system to `sofind`\nThere are two simple steps:\n1. Add the system name to `sofind/systems.py`\n2. For a given product `.yaml` file, add the location of the product on-disk on that system under the `system_paths` block. Of course, this does not add that product to that system in that location; that must happen outside `sofind`.\n\n## If you would like to contribute a product to `sofind`\nThere are four steps:\n1. Create a new product folder in the `sofind/products` directory. Add to this folder a python module named `__init__.py`.\n    * There is a minimum prescription your subclass implementation must follow. To make it easy, a template of this implementation (for a product called `HotDog`) can be copied from `sofind/products/hotdogs/__init__.py`. You should *not* delete the template class declaration or `__init__` method (but you may add to them), and you *must* modify the `sofind.products.products.Product` methods decorated with `@productmethod`. Note the template has more detail on how to implement your product class. You can also look at e.g. `sofind.products.maps.Map` for inspiration.\n    * Anything beyond this minimal perscription can be added if your product has more complicated features!\n2. Make sure your product is imported directly by the `sofind.products` package. For instance, if your module was named `hotdogs` and your particular class named `HotDog`, then add this line to `sofind.products.__init__.py`:\n\n    ```python\n    from .hotdogs import HotDog\n    ```\n3. Add a config (or multiple configs if you have multiple product versions, or subproducts, etc.) to `sofind/products/{module_name}`. Following the hotdog example, there is a config file `hotdog_example.yaml` in `sofind/products/hotdogs`.\n    * This must be a `.yaml` file.\n    * You must have a `system_paths` block. This will have entries for a subset of the available clusters in `sofind/systems.py`. Each entry will point to a path giving the directory on that system in which the product files are located.\n    * You must have an `allowed_qids_configs` block. If your subproduct is agnostic to qids, this can be `all`; otherwise, it must list the individual `qids_config` files (in `sofind/qids`) that this subproduct may work with.\n    * You must have an `allowed_qids` block. If your product methods use a `qid` to provide keywords to filename templates, then the permissible `qid`(s) for that product must be listed here. May also be `all` or left blank if no `qid`(s) work with your subproduct. For instance, a call to `HotDog.read_hotdog` will raise an error if `hotdog_example.yaml` is used to configure the calling `DataModel` instance and the supplied `qid` is *not* one of `pa4a`, `pa5a`, or `pa6a`.\n    * You must have an `allowed_qids_extra_kwargs` block. If your template filename requires additional keywords for a given `qid` than are present in any `allowed_qids_configs` files (see `sofind/qids`), those keywords would need to be added here. For instance, the `act_dr6_default_qids.yaml` file only contains `array`, `freq`, `patch`, and `daynight` keywords. The `num_splits` keyword required for the `hotdog_file_template` is thus added for each permissible qid directly in `hotdog_example.yaml`, e.g.:\n        ```yaml\n        allowed_qids_extra_kwargs:\n            pa6a:\n                num_splits: 8\n        ```\n        The `allowed_qids_extra_kwargs` block may be empty if there are no extra keywords to add for any of the `allowed_qids`.\n    * This should contain any information to help get filenames for your product or load them from disk, such as a template filename. For instance, given a set of keyword arguments `array='pa6', freq='f090', num_splits=8, condiment='mustard'`, the `hotdog_file_template` string in `hotdog_example.yaml` would format to `pa6_f090_8way_mustard.txt` (the actual formatting would occur in your `HotDog` class's `get_hotdog_fn` method).\n4. Clearly document product implementations and product configs. For example, see the docstrings in the `Map` class (`sofind/products/maps/__init__.py`) as well as the `maps` readme (`sofind/products/maps/README.md`).\n    \nPlease commit and push your contribution to a new branch and open a PR on github! If you are updating an old config, please include it under a new file altogether so that historical products may still be loaded at runtime.\n    \n## If you would like to contribute a data model to `sofind`\nThere are one (maybe two) steps:\n1. Create a new data model config in `sofind/datamodels`.\n    * This config must have a block for each product this data model will load. Note, it is not necessary to have a block for every product in `sofind`, only those that will be functional in this data model. The same goes for subproducts of a product -- include only those that will be functional in this data model. To add a subproduct, include an entry under the associated product block like:\n        ```yaml\n        prod:\n            subprod_config: config.yaml\n        ``` \n        where `prod` and `subprod` are the names of the product and subproduct that may be called by the `DataModel`.\n    * This config file must also have an entry for a `qids_config` (at the top-level), indicating one of the qid config files under `sofind/qids`.\n2. Only if one of the included qid config files are not sufficient for your needs, you'll need to add another one with your qids.\n\nPlease commit and push your contribution to a new branch and open a PR on github! If you are updating an old config, please include it under a new file altogether so that historical products may still be loaded at runtime.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fsofind","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonsobs%2Fsofind","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fsofind/lists"}