{"id":21735126,"url":"https://github.com/caltech-ipac/spherex_butler_poc","last_synced_at":"2025-03-21T00:20:37.495Z","repository":{"id":54195412,"uuid":"282083095","full_name":"Caltech-IPAC/spherex_butler_poc","owner":"Caltech-IPAC","description":"proof-of-concept code for using dax_butler","archived":false,"fork":false,"pushed_at":"2021-03-03T21:27:00.000Z","size":71,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-01-25T20:43:48.199Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Caltech-IPAC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-24T00:12:35.000Z","updated_at":"2021-03-03T21:21:51.000Z","dependencies_parsed_at":"2022-08-13T08:51:05.111Z","dependency_job_id":null,"html_url":"https://github.com/Caltech-IPAC/spherex_butler_poc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Caltech-IPAC%2Fspherex_butler_poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Caltech-IPAC%2Fspherex_butler_poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Caltech-IPAC%2Fspherex_butler_poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Caltech-IPAC%2Fspherex_butler_poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Caltech-IPAC","download_url":"https://codeload.github.com/Caltech-IPAC/spherex_butler_poc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244712708,"owners_count":20497543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T05:12:00.693Z","updated_at":"2025-03-21T00:20:37.476Z","avatar_url":"https://github.com/Caltech-IPAC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spherex_butler_poc\n\nProof-of-concept code for using [dax_butler](https://github.com/lsst/daf_butler) – Rubin/LSST Gen3 Butler and \nPipelineTask framework in SPHEREx pipelines.\n\n_Butler_ organizes _datasets_ (units of stored data) in data repositories, identifying them by a combination of \n_dataset type_, _data id_, and _collection_. It encapsulates all I/O done by pipeline code. \n\n_PipelineTask_ is a framework for writing and packaging algorithmic code that enables generating a pipeline execution \nplan, in the form of a directed acyclic graph (DAG). PipelineTask is built on top of Butler.\n\n\n#### Why using Butler?\n\nLSST Butler is a Python 3 only package, which provides data access framework for LSST Data Management team.\nThe main function of such a framework is to organize the discrete entities of stored data to facilitate its search and retrieval.\n\nLSST Butler abstracts:\n- **where data live on the storage**   \nThe data can live in a POSIX datastore, on Amazon S3, or elsewhere.\nThe framework users deal with Python rather than stored representations of data entities.\n- **data format and how to deal with it**  \nFormatters are used to to move between stored and Python representations of data entities. \nAs a result users deal with Python objects, such as astropy Table or CCDData instead of \nthe stored formats, such as VOTable, FITS or HDF5.\n- **calibrations**   \nYou ask for a calibration for an image, and it returns you the right file. (Use case: bias reference image - \nit is averaged over a few days. The algorithmic code does not need to be aware of how the bias reference is obtained.)\n\n#### Butler concepts\n\n- **Dataset** is a discrete entity of stored data, uniquely identified by a _Collection_ and _DatasetRef_.  \n- **Collection** is an entity that contains _Datasets_.  \n- **DatasetRef** is an identifier for a _Dataset_.  \n- **Registry** is a database that holds metadata and provenance for _Datasets_.  \n- **Dimension** is a concept used to organize and label _Datasets_. _Dimension_ is analogous to a coordinate axis \nin coordinate space. _Dataset_ can be viewed as a point in this space with the position is defined by \n_DataCoordinate_ or _data id_. For example, SPHEREx raw image might be identified by the observation \n(pointing of the telescope at a particular time) and the detector array that took this image. For this reason, \n_observation_ and and _detector_ might be good _Dimensions_ to describe raw image _Datasets_. \n- **DatasetType** is a named category of _Datasets_ (ex. raw image)  \nTogether, _DatasetType_ and _DataCoordinate_ \nmake a unique _Dataset_ identifier, see [DatasetRef](https://pipelines.lsst.io/v/weekly/py-api/lsst.daf.butler.DatasetRef.html#lsst.daf.butler.DatasetRef).   \n\n#### Resources\n\n- [10/2020 Gen3 Middleware Tutorial](https://github.com/lsst-dm/obs-bootcamp-2020-bosch-tutorial/)\n- [07/2020 Gen3 Middleware Tutorial](https://github.com/lsst-dm/dm-demo-notebooks/tree/u/jbosch/desc-2020-07/workshops/desc-2020-07/gen3-butler)\n    - https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/organizing.html\n    - https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/queries.html\n    - https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/dimensions.html\n\n#### Proof-of-concept directions\n\n- Custom python representations of stored _Datasets_  \nVerified that we can create a custom _Formatter_, that maps stored representation of a particular dataset type \ninto its Python representation. (Using `butler.put()` to store Python object into file system datastore and \n`butler.get()` to retrieve Python object from the stored file.)\n- Custom set of _Dimensions_  \nVerified that we can organize our _Datasets_ around custom set of _Dimensions_, see Caveats below.\n- File ingest (using `butler.ingest()`)   \nButler allows to configure file templates (`datastore.templates`), which allows to use _Collection_ name,\n_DatasetType_, and any of the field in _Dimensions_ tables to create directory structure and file name \nof _Dataset_ stored representation.  \nVerified that the data can be ingested into datastore according to the defined template and \nthe transfer type (ex. copy, symlink).\n- Custom butler command   \nIt is possible to use butler framework to create butler [subcommands](https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/writing-subcommands.html#adding-butler-subcommands). \nVerified this capability by adding `ingest-simulated` subcommand.\n- Simple task and example pipeline  \nCreated SubtractTask pipeline task, which accepts two images and subtracts the second from the first.\nCreated an example pipeline that runs this task, see `pipelines/ExamplePipeline.yaml`.\n\nProof-of-concept is designed around Python unit tests that run in a container\non GitHub-hosted machines as a part of GitHub's built-in continuous integration service,\nsee `.github/workflows/unit_test.yaml`  \nUnfortunately, pipeline tasks can not be validated with GitHub actions, because they rely on\n[pipe_base](https://github.com/lsst/pipe_base) and [ctrl_mpexec](https://github.com/lsst/ctrl_mpexec) packages with \ndeeper rooted dependencies. Running example pipeline requires installing Rubin/LSST environment, where packages \nare managed with [EUPS](https://developer.lsst.io/stack/eups-tutorial.html).\n\nDependency management is one of the main concerns when using Rubin/LSST pipeline framework.\n\n#### Caveats\n\nButler allows to override parts of its configuration. The overwritten configuration is merged \nwith the default configuration. As of November 2020, it's possible to completely overrode dimensions, \nbut not possible to completely replace formatters and storage classes.\n\nThere is an implied requirement in `ctrl_mpexec` package that `instrument` dimension table must have \na reference to instrument class. \n\nOther known issues:\n- Butler relies on `lsst.sphgeom` lower level C++ library, which does not support HEALPix pixelization at the moment\n\n\n### Testing Gen3 Butler and Pipeline Task framework\n\n#### Testing locally\n\n##### Install latest weekly\n\nTo install the latest pipeline distribution `lsst_distrib` built by Rubin/LSST project, follow [newinstall](https://pipelines.lsst.io/install/newinstall.html#run-newinstall-sh) recipe:\n\n```\n# from an empty directory - \ncurl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh\n# continue a previous failed install, if any, in batch mode, and prefer tarballs\nbash newinstall.sh -cbt \nsource loadLSST.bash\n# install weekly 46 for 2020\neups distrib install -t w_2020_46 lsst_distrib  \n# fix shebangs - tarballs have shebangs encoded at build time that need to be fixed at install time\ncurl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python\n# use with tag option if other versions installed: setup -t w_2020_46 lsst_distrib\nsetup lsst_distrib\n```\n[What newinstall does](https://pipelines.lsst.io/install/newinstall.html#newinstall-background)\n\nYou only need to do newinstall when conda base environment changes.\nCheck the last modified date of [conda-system](https://eups.lsst.codes/stack/osx/10.9/conda-system/).\n\nIf newest weekly is installed without running newinstall.sh, the previous versions can be removed\nwith [this script](https://github.com/lsst/lsstsw/blob/u/jbosch/bin/pruneTags). The script will remove \nall packages except locally set up and those with the given tag. Use `--dry-run`\noption to avoid surprises:\n```\npruneTags w_2020_44 --delete-untagged --dry-run\n```\n\n##### Running example pipeline\n\nTo run the example pipeline defined in these repository follow these steps:\n\n- Install the latest `lsst_distrib` (see above)\n\n- Set up spherex_butler_poc repository with EUPS package manager:\n```\ngit checkout https://github.com/Caltech-IPAC/spherex_butler_poc.git\ncd spherex_butler_poc\n# set up the package in the eups stack\nsetup -r . -t $USER\n# review set up packages (optional)\neups list -s\n```\n- Create a directory where the buttler repository will live:\n```\nmkdir ../test_spherex\ncd ../test_spherex\n```\n- Run SPHEREx simulator to produce simulated files. \nThe simulated files have exposure and detector id embedded in the file names.\n\n- Create empty butler repository (DATA):\n```\nbutler create --override --seed-config ../spherex_butler_poc/python/spherex/configs/butler.yaml --dimension-config ../spherex_butler_poc/python/spherex/configs/dimensions.yaml DATA\n```\n- Ingest simulated images:\n```\nbutler ingest-simulated DATA /\u003cabspath\u003e/simulator_files\n```\n- Ingest simulated dark current images (the group is set to the ingest date, hence ingesting raw and dark images should be done on the same date):\n```\nbutler ingest-simulated --regex dark_current.fits --ingest-type dark DATA /\u003cabspath\u003e/simulator_files\n```\n- Examine butler database\n```\nsqlite3 DATA/spherex.sqlite3\n\u003e .header on\n\u003e .tables\n\u003e select * from file_datastore_records;\n\u003e .exit\n```\n- Create pipeline execution plan as a qgraph.dot file\n```\npipetask qgraph -p ../spherex_butler_poc/pipelines/ExamplePipeline.yaml --qgraph-dot qgraph.dot -b DATA -i rawexpr,darkr -o subtractr\n```\n- Convert `qgraph.dot` into `pdf` (`graphvis` required):\n```\ndot -Tpdf qgraph.dot -o qgraph.pdf\n```\n- Run example pipeline:\n```\npipetask run -p ../spherex_butler_poc/pipelines/ExamplePipeline.yaml -b DATA --register-dataset-types -i rawexpr,darkr -o subtractr\n```\n- Optionally: rerun replacing (`--replace-run`) and removing (`--prune-replaced=purge`) the previous run:\n```\npipetask run -p ../spherex_butler_poc/pipelines/ExamplePipeline.yaml -b DATA -o subtractr --replace-run --prune-replaced=purge\n```\n- Examine butler repository in `DATA` directory\n\n- Explore the contents of butler repository using command line tools:\n```\n\u003e butler query-collections DATA\n\u003e butler query-collections DATA --collection-type CHAINED\n\u003e butler query-collections DATA --flatten-chains subtractr\n```\n\n### Testing in a container (using weekly image):\n\n- make sure you have test data (Git LFS repo) and test scripts:\n```\n\u003e git clone https://github.com/lsst/testdata_ci_hsc\n\u003e git clone https://github.com/lsst/ci_hsc_gen3\n```\n\n- start up container:\n```\n\u003e docker run -it -v `pwd`:/home/lsst/mnt docker.io/lsstsqre/centos:7-stack-lsst_distrib-w_latest\n```\n\n- in container: \n```\n$ source /opt/lsst/software/stack/loadLSST.bash\n$ setup lsst_distrib\n$ cd /home/lsst/mnt\n$ setup -j -r testdata_ci_hsc\n$ setup -j -r ci_hsc_gen3\n$ echo $TESTDATA_CI_HSC_DIR; echo $CI_HSC_GEN3_DIR\n$ cd ci_hsc_gen3/$ scons\n$ sqlite3_analyzer /home/lsst/mnt/ci_hsc_gen3/DATA/gen3.sqlite3 \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaltech-ipac%2Fspherex_butler_poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaltech-ipac%2Fspherex_butler_poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaltech-ipac%2Fspherex_butler_poc/lists"}