{"id":21375353,"url":"https://github.com/lsst/ci_hsc_gen3","last_synced_at":"2025-07-13T09:32:52.981Z","repository":{"id":38429275,"uuid":"185243508","full_name":"lsst/ci_hsc_gen3","owner":"lsst","description":"An integration testing package that uses third generation middleware","archived":false,"fork":false,"pushed_at":"2024-11-06T16:17:50.000Z","size":301,"stargazers_count":3,"open_issues_count":4,"forks_count":0,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-11-06T17:27:22.835Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lsst.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-06T17:40:44.000Z","updated_at":"2024-10-25T04:11:38.000Z","dependencies_parsed_at":"2023-02-10T22:15:52.985Z","dependency_job_id":"cdd45e32-fd7f-4fde-a7a5-92b9d270c991","html_url":"https://github.com/lsst/ci_hsc_gen3","commit_stats":null,"previous_names":[],"tags_count":369,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Fci_hsc_gen3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Fci_hsc_gen3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Fci_hsc_gen3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsst%2Fci_hsc_gen3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lsst","download_url":"https://codeload.github.com/lsst/ci_hsc_gen3/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225871094,"owners_count":17537173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T09:09:53.263Z","updated_at":"2024-11-22T09:09:53.960Z","avatar_url":"https://github.com/lsst.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"######\nci_hsc\n######\n\n``ci_hsc`` provides scripts which use the LSST stack to perform single frame and coadd processing based on engineering test data from Hyper Suprime-Cam.\n\nObtaining test data\n===================\n\n``ci_hsc`` uses the the files in `testdata_ci_hsc`_ to run its tests.\nThat package must be setup first, before running ``scons`` (see below).\n\n .. _testdata_ci_hsc: https://github.com/lsst/testdata_ci_hsc/\n\nRunning the tests\n=================\n\nSet up the package\n------------------\n\nBoth `testdata_ci_hsc`_ both and ``ci_hsc`` must be setup in eups in order to run the tests in this package.\nOne way to accomplish this is as follows::\n\n  $ cd PATH_TO_TESTDATA_CI_HSC\n  $ setup -r .\n  $ cd PATH_TO_CI_HSC\n  $ setup -kr .\n\n\nRunning the tests\n-----------------\n\nExecute ``scons -jN``, where ``N`` is the number of CPU cores to use.\nNote that running these tests can take a few hours, depending on the speed of your machine and the number of cores available.\n\nThis will create a butler repository at ``DATA/``, ingest the raw data into ``HSC/raw/all``, create a chained ``HSC/defaults`` collection for all of the input data, and write the output of the pipeline run to ``HSC/runs/ci_hsc``.\nIt will also run various checks of the data integrity of the processed output.\nThe resulting repository in ``DATA/`` will take up about 18GB.\n\nDebugging ``HSC/runs/ci_hsc``\n-----------------------------\n\nIf the run fails on a given task for the primary ``HSC/runs/ci_hsc`` run, you can rerun a specific task in your failed run with the following command (replacing ``taskLabelToRerun`` with the appropriate label from the DRP pipeline):\n\n.. code-block:: bash\n\n    pipetask run -b DATA -j 1 -i HSC/runs/ci_hsc -o u/$USER/testing -p \"${DRP_PIPE_DIR}/pipelines/HSC/DRP-ci_hsc.yaml#taskLabelToRerun\" -d \"skymap='discrete/ci_hsc' AND tract=0 AND patch=69\"\n\nThis will pick up the outputs from everything that had been run prior to the task that failed and rerun the task with the output going to a new collection called ``u/$USER/testing``.\n\nAfter fixing any problems, the processing can be resumed with the following command (note the ``..`` at the end of the task label):\n\n.. code-block:: bash\n\n    pipetask run -b DATA -j NPROCESS -i HSC/runs/ci_hsc -o HSC/runs/ci_hsc -p \"${DRP_PIPE_DIR}/pipelines/HSC/DRP-ci_hsc.yaml#taskLabelToRerun..\" -d \"skymap='discrete/ci_hsc' AND tract=0 AND patch=69\"\n\nCleaning up\n-----------\nAfter each run of this test (and, in particular, before re-running it), the repository should be cleaned as follows::\n\n  $ scons --clean\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flsst%2Fci_hsc_gen3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flsst%2Fci_hsc_gen3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flsst%2Fci_hsc_gen3/lists"}