{"id":18620485,"url":"https://github.com/simonsobs/sotrplib","last_synced_at":"2026-03-16T08:02:14.913Z","repository":{"id":259225777,"uuid":"870250093","full_name":"simonsobs/sotrplib","owner":"simonsobs","description":"Simons Observatory Time Domain Pipeline Library","archived":false,"fork":false,"pushed_at":"2025-05-06T20:02:39.000Z","size":512,"stargazers_count":1,"open_issues_count":8,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-07T08:14:14.845Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonsobs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-09T17:45:50.000Z","updated_at":"2025-05-06T16:20:38.000Z","dependencies_parsed_at":"2024-12-08T22:26:12.835Z","dependency_job_id":"e0cad4e4-e19d-475c-aca7-82f04e9fe6c7","html_url":"https://github.com/simonsobs/sotrplib","commit_stats":{"total_commits":34,"total_committers":3,"mean_commits":"11.333333333333334","dds":0.1470588235294118,"last_synced_commit":"ae85ad15571b2187f1682a5418c888cd7632d0a7"},"previous_names":["simonsobs/sotdplib","simonsobs/sotrplib"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsotrplib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsotrplib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsotrplib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonsobs%2Fsotrplib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonsobs","download_url":"https://codeload.github.com/simonsobs/sotrplib/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252839296,"owners_count":21812089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T04:06:19.440Z","updated_at":"2026-03-16T08:02:14.879Z","avatar_url":"https://github.com/simonsobs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sotrplib\nSimons Observatory Time Resolved Pipeline Library\n\nA pipeline to ingest fits maps, perform pre- and post-processing, forced photometry and blind searching for point sources.\n\nCurrently, the output is a pandas database in pickle format.\n\nSee `scripts/end-to-end/` for an example of a full pipeline run including socat and lightcurvedb\n\n## Development requirements\n\nTo get ready for development, create a virtual enviroment and install the package:\n```\nuv venv --python=3.12\nsource .venv/bin/activate\nuv pip install -e \".[dev]\"\npre-commit install\n```\nIf you don't have uv installed, you can install it with `pip install uv`, or\njust go ahead and use `pip install -e \".[dev]\"`. \n\nWe use `ruff` for formatting. When you go to commit your code, it will automatically be \nformatted thanks to the pre-commit hook.\n\nTests are performed using `pytest`.\n\n## extra required packages\nAny required packages should be listed in `pyproject.toml`\n\nIf a package is missing, you can manually install it with `uv install [package]`. Please then report this on the GitHub\n[issue tracker](https://github.com/simonsobs/sotrplib/issues).\n\n\n## Setting up and Running the Pipeline\n\nAfter following the development instructions above, you will be able to run the pipeline by running the following:\n\n`sotrp -c [path to config file]`\n\nThe config file is a .json which contains a dictionary of all the pipeline segments and inputs. \nYou can see several examples in the top level directory: `sample_*.json` \n\nA config file is read in by a basic handler (see `sotrplib/handlers/basic.py`), which de-serializes to Python objects using the\ncode in `sotrplib/config/config.py` and the relevant config files.\n\n### Source Catalog (socat)\n\nWe have implemented the source catalog as a mock version of `socat` (https://github.com/simonsobs/socat/).\nTo make things work with an old ACT type catalog, socat includes a runnable script `socat-act-fits` which ingests the .fits file into a pickle file which can be converted into a mock socat object in the pipeline.\n\nThus to run the pipeline with an ACT source catalog, you first create the socat pickle file:\n\n`socat-act-fits -f [act_catalog.fits] -o socat.pickle`\n\nThen tell socat where to find it:\n\n`export socat_client_client_type=pickle`\n`export socat_client_pickle_path=socat.pickle`\n\nSetting the sotrplib config to use `socat` as one of it's source catalogs:\n\n```\n\"source_catalogs\": [\n        {\n            \"catalog_type\": \"socat\",\n            \"flux_lower_limit\": \"0.01 Jy\"\n        }\n    ],\n```\nthen generates a mock socat client given the info above.\n\nThis source catalog can then be used for forced photometry, pointing, blind-search crossmatching, etc.\n\n### Map Catalog (mapcat)\n\nOne way to ingest maps into the pipeline is to manually add them into the config, like the example `sample_read_unfiltered_map.json` . \nThis is convenient for testing a specific map, or a one-off, etc. \n\nHowever, running on a full set of maps and keeping track of map metadata, etc. requires a map database.\nWe call this `mapcat` (https://github.com/simonsobs/mapcat) and again have an ingestion script for ACT-like map sets.\n\nTo ingest ACT depth1 maps into a mapcat sqlite db, you would run the script `actingest` after setting the relevant mapcat environment variables; \n\n```\nexport MAPCAT_DEPTH_ONE_PARENT=/path/to/depth1/maps\nexport MAPCAT_DATABASE_NAME=/path/to/mapcat.sqlite\n```\nThis tells the map catalog where to look for the maps and where the database lives. \n\nWith the existance of a mapcat database, the pipeline can be configured to read from there via :\n\n```json\n\"maps\": {\n  \"map_generator_type\": \"mapcat_database\",\n  \"number_to_read\": 1,\n  \"instrument\": \"SOLAT\",\n  \"frequency\": \"f090\",\n  \"array\": \"i6\",\n  \"rerun\": \"True\"\n},\n\n```\nfor example, which tells the runner to read in 1 map at f090, from array i6 and to rerun it if it has already been analyzed.\n\n### How to configure the pipeline\n\nThe .json config file is from where the pipeline runner reads.\nAllowed methods and their properties can be accessed in the `sotrplib/config/` directory.\nEach file contains the relevant configurations and required methods for each type of object; i.e. maps, preprocessors, forced_photometries, etc.\nThese configurations are read-in, converted from JSON to pydantic models (`sotrplib/config/config.py`), and used by the pipeline handler to construct the pipeline.\nThe basic handler can be found in `sotrplib/handlers/base.py`\n\nLet's follow one example through from .json config to understand what is happening.\nWe'll use maps.\nThe pipeline expects maps to be a list of `ProcessableMap` objects.\nYou'll notice in the samples there are two different settings for `maps`; a dictionary or a list of dictionaries.\nIf the converting function sees a list, it knows that they are lists of map objects, so it processes each one individually.\nIf the conversion sees a dictionary it knows to expect a map_generator, which, in the case of mapcat_database, it will query the `mapcat.sqlite` db and construct a list of map objects.\n\nLet's take the case of `sample_read_unfiltered_map.json`. Here we have \n\n```json\n\"maps\": [\n  {\n    \"map_type\": \"inverse_variance\",\n    \"intensity_map_path\": \"./depth1_1538613353_pa5_f090_map.fits\",\n    \"weights_map_path\": \"./15386/depth1_1538613353_pa5_f090_ivar.fits\",\n    \"time_map_path\": \"./depth1_1538613353_pa5_f090_time.fits\",\n    \"frequency\": \"f090\",\n    \"band\": \"pa5\",\n    \"intensity_units\": \"K\",\n    \"box\": [\n      {\n        \"ra\": {\n          \"value\": 138.52,\n          \"unit\": \"deg\"\n        },\n        \"dec\": {\n          \"value\": -13.095,\n          \"unit\": \"deg\"\n        }\n      },\n      {\n        \"ra\": {\n          \"value\": 140.52,\n          \"unit\": \"deg\"\n        },\n        \"dec\": {\n          \"value\": -11.095,\n          \"unit\": \"deg\"\n        }\n      }\n    ]\n  }\n],\n```\nso we can see `map_type` is `inverse_variance`. Going to `config/maps.py`, you can find where map_type is inverse_variance; i.e. the `InverseVarianceMapConfig` class.\nYou can see what the required / default arguments are and what the pipeline does when it converts that input `to_map` -- it creates a ProcessableMap class of subclass IntensityAndInverseVarianceMap.\n\nIf you look at the other example, `sample_read_mapcat.json`, you will see \n\n```json\n\"maps\": {\n  \"map_generator_type\": \"mapcat_database\",\n  \"number_to_read\": 1,\n  \"instrument\": \"SOLAT\",\n  \"frequency\": \"f090\",\n  \"array\": \"i6\",\n  \"rerun\": \"True\"\n},\n```\nwhich clearly shows `map_generator_type` as the descriptor, not `map_type`. This implies that it will generate maps from the source (which is listed as mapcat_databse here).\nChecking `config/maps.py` we see the subclass with that map_generator_type is `MapCatDatabaseConfig` which returns a `MapCatDatabaseReader` instance; returning a list of map objects corresponding to what is configured.\n\nOnce the map objects are loaded, the pipeline handler then builds the maps and injects them into the rest of the pipeline.\n\nThe various other components of the pipeline are built in a similar manner, for example map preprocessing is built by configuring a list of `PreProcessor` objects, etc.\n\nThe pipeline then runs as per your config, and the steps in the handler script.\n\n### Pipeline Outputs\n\nThe current default is to output to pickle files because these are simply converted from the pydantic models transferred between the pipeline components in production mode.\n\nThe output format can be found in `outputs/core.py`, and in the default case is the `PickleSerializer`.\n\nEssentially this is just dictionaries of lists of MeasuredSource objects (and InjectedSource objects in the case you're simulating sources).\n\nThese MeasuredSource objects contain information about their measurement and even cutouts.\n\n\n### Running with prefect\n\n[prefect](https://docs.prefect.io/v3/get-started) is a workflow orchestrator that provides a conveneient web interface for monitoring and running the pipeline.\nInstalling and invoking the pipeline using prefect follows the same basic pattern as above.\n\n```console\nuv sync --extra prefect\nsource .venv/bin/activate\nexport sotrp_runner=prefect\nsotrp -c [path to config file]\n```\n\nThis will start a temporary prefect server, if you want a persistent server you can start one as described in the [prefect docs](https://docs.prefect.io/v3/get-started/quickstart#open-source).\n\nThe runner can also be specified via the configuration file or as a command-line argument.\n\n```console\nprefect server start --host [HOSTNAME, e.g., localhost] --port [PORT, e.g., 8899] --background\n```\n\nThis will start a prefect server and provide a URL to the dashboard, in this case http://localhost:8484.\n`sotrp-prefect` can then be invoked either by manually specifying the `PREFECT_API_URL` as an environment variable, e.g.,\n\n```console\nPREFECT_API_URL=http://localhost:8484/api sotrp-prefect -c [path to config file]\n```\n\nor by using the prefect tool\n\n```console\nprefect config set PREFECT_API_URL=http://localhost:8484/api\n```\n\nThe server can be stopped with\n\n```console\nprefect server stop\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fsotrplib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonsobs%2Fsotrplib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonsobs%2Fsotrplib/lists"}