{"id":38903968,"url":"https://github.com/cgosmeyer/wfc3_cte_monitor","last_synced_at":"2026-01-17T15:05:45.975Z","repository":{"id":275072738,"uuid":"95674292","full_name":"cgosmeyer/wfc3_cte_monitor","owner":"cgosmeyer","description":"Pipeline and SQL database interface for the Hubble WFC3/UVIS charge transfer efficiency (CTE) monitor. ","archived":false,"fork":false,"pushed_at":"2017-06-28T16:11:33.000Z","size":157,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-10T00:31:51.834Z","etag":null,"topics":["astronomy","charge-transfer-efficiency","cte","database","hubble","photometry","pipeline","sqlite","wfc3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cgosmeyer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-28T13:53:11.000Z","updated_at":"2021-10-30T21:37:44.000Z","dependencies_parsed_at":"2025-01-31T02:45:32.886Z","dependency_job_id":null,"html_url":"https://github.com/cgosmeyer/wfc3_cte_monitor","commit_stats":null,"previous_names":["cgosmeyer/wfc3_cte_monitor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cgosmeyer/wfc3_cte_monitor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgosmeyer%2Fwfc3_cte_monitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgosmeyer%2Fwfc3_cte_monitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgosmeyer%2Fwfc3_cte_monitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgosmeyer%2Fwfc3_cte_monitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cgosmeyer","download_url":"https://codeload.github.com/cgosmeyer/wfc3_cte_monitor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgosmeyer%2Fwfc3_cte_monitor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28510928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T13:38:16.342Z","status":"ssl_error","status_checked_at":"2026-01-17T13:37:44.060Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["astronomy","charge-transfer-efficiency","cte","database","hubble","photometry","pipeline","sqlite","wfc3"],"created_at":"2026-01-17T15:05:41.904Z","updated_at":"2026-01-17T15:05:45.957Z","avatar_url":"https://github.com/cgosmeyer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WFC3/UVIS External CTE Monitor\n\nAuthor: C.M. Gosmeyer\n\nCreated: 10 Apr. 2015\n\nLast Update: 12 Feb. 2017\n\n**NOTE:**\n**Originally located in private repository `spacetelescope/detectors/scripts` as `uvis_external_cte`.**\n**Written for the Hubble WFC3 team at Space Telescope Science Institute.**\n**Some modificiations have been made to this version to keep file structures hidden.**\n**All Hubble data used in this monitor are public.**\n\n\n-------------------------------------------------------------------------------\n\n\n## Contents\n\n0. Introduction\n\n1. Instrument Science Reports and Relevant Documents\n\n2. Proposals\n\n3. Running the Pipeline\n\n   3.1 Initial Setup\n   \n   3.2 Procedure for Ingesting ALL the Data\n   \n   3.3 Procedure for General Run of the Pipeline\n   \n   3.4 Procedure for Processing the 180-Degree ('Same-Chip') Dataset\n   \n   3.5 Procedure for Switching Between Python and IRAF Photometry\n   \n   3.6 Example Use Cases of the Pipeline\n\n4. Care and Feeding of the Database\n\n   4.1 Procedure for Adding/Renaming a Column\n  \n   4.2 Procedure for Removing a Column\n   \n   4.3 Procedure for Adding a Table\n   \n   4.4 Procedure for Re-Setting and Re-Populating Database\n   \n   4.5 Help! The Database Locked Me Out!\n\n5. Summary of Scripts and Modules\n\n   5.1 Pipeline\n   \n   5.2 Database\n   \n   5.3 For Tests and ISRs\n\n6. Issues and Future Improvements\n\n\n-------------------------------------------------------------------------------\n\n\n## 0. Introduction\n\nThis document is meant as a guide for setting up and running the pipeline \nof the external Charge Transfer Efficiency (CTE) monitor. All scripts are \ndescended from Kai Noeske’s “PHOTCODE” suite. The originals can be found \nby going way back to the first commits in Quicklook's\n`spacetelescope/detectors/scripts/uvis_external_cte`. \n\nThis program monitors the CTE using observations of star clusters (hence \nthe 'external' bit), and via these data, produces a table of coefficients \nthat can be used to correct for CTE loss given an observation’s epoch, \ndistance from amplifiers, source flux, and exposure length. \n\nThis program has many, many parameters to keep in mind.\n* Two star clusters of different densities are observed: sparse NGC 6791 \nand dense NGC 104 (47 Tuc).\n* Two filters were used until Cycle 20: F502N and F606W. \n* Following Cycle 20 (inclusive) all observations are taken only in F502N \nso there will be time to take post flash images of each field. In \naddition, it was deemed unnecessary to continue F606W because post flash \ncould provide, as the two filters did, a variety of backgrounds. Non-post \nflash images are still taken for continuity. \n* Two flash levels are sampled: 6 and 12 e-/pxl, of both clusters.\n* For dense NGC104, flash exposures 1.24, 1.59, 13, and 21.5 are sampled \nto get a range of background levels. (Filter 502N has effectively no \nbackground.) This gives us effectively flash levels of 33, 55, 91, and \n116 e-/pxl.\nStarting in 14378, we sample finer gradients of flash levels: 18 and 24 \ne-/pxl.\n* Flash 12 e-/pxl is most used in the analysis, as it is the one recommended \nto observers for general use.\n* All images have the pixel-based CTE correction run over them. Every FLT, \nthen, should have a corresponding FLC. \n* Photometry is run on apertures 3, 5, and 10 pixels. Aperture 3 is the \none used in most of the analysis use because it is most reliable - any \nlarger aperture allows in too many contaminating sources.\n\nSee ISRs and Proposals below for further background information.  Again,\nthis document's focus is on how to run the pipeline, not on the monitor's \nobservation setup or on analysis of results. \n\n\n-------------------------------------------------------------------------------\n\n\n## 1. Instrument Science Reports and Relevant Documents\n\n\"The Efficacy of Post- Flashing for Mitigating CTE-Losses in WFC3/UVIS Images.\"\nJ. Anderson, J. MacKenty, S. Baggett, and K. Noeske, K., 2012\nhttp://www.stsci.edu/hst/wfc3/ins_performance/CTE/ANDERSON_UVIS_POSTFLASH_EFFICACY.pdf\n\nWFC3 ISR 2012-09: \"WFC3 UVIS Charge Transfer Efficiency October 2009 to October 2011\"\nNoeske et al., 08 Jun. 2012\nhttp://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2012-09.pdf\n\nACS ISR 2012-05: \"A new accurate CTE photometric correction formula for ACS/WFC\"\nChiaberge, M., Oct. 2012\nhttp://www.stsci.edu/hst/acs/documents/isrs/isr1205.pdf\n\nWFC3 ISR 2015-03: \"WFC3/UVIS Charge Transfer Efficiency 2009 - 2015\"\nS. Baggett, C. Gosmeyer, K. Noeske, 31 March 2015\nhttp://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2015-03.pdf\n\nWFC3 ISR 2016-17:  \"WFC3/UVIS External CTE Monitor: Single-Chip CTE Measurements\"\nC. Gosmeyer and S. Baggett, 14 Dec. 2016\nhttp://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2016-17.pdf\n\nWFC3 ISR 2017-09: \"WFC3/UVIS External CTE Monitor: 2016 Updates on Coefficients and Analysis Pipeline\"\nC. Gosmeyer and S. Baggett, 2017\nhttp://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2017-09.pdf\n\nWFC3 Instrument Handbook for Cycle 23, \"6.9 Charge Transfer Efficiency\"\nhttp://www.stsci.edu/hst/wfc3/documents/handbooks/currentIHB/c06_uvis10.html\n\n\n-------------------------------------------------------------------------------\n\n\n## 2. Proposals\n\n(Based on Table 1 of ISR 2015-03)\n\nProgram\t| Cycle\t| Obs Dates\t|\t\tFilters\t|\tTargets | URL\n+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n\n11924\t| 17 Oct 2009,Mar 2010,Sep 2010\t| F502N, F606W |\tNGC6791 | \nhttp://www.stsci.edu/hst/phase2-public/11924.pdf\n\n12348 | 18\tSep 2010\t|\t\tF502N\t|\tNGC104 | \nhttp://www.stsci.edu/hst/phase2-public/12348.pdf\n\n12379\t| 18\tNov 2010,Mar 2011,Apr 2011\t| F502N, F606W\t| NGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/12379.pdf\n\n12692\t| 19\tOct 2011,Mar 2012,Jun/Jul 2012 | F502N, F606W\t| NGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/12692.pdf\n\n13083\t| 20\tNov 2012,Mar 2013,Jul/Aug 2013 |\tF502N\t|\tNGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/13083.pdf\n\n13566 |\t21\tJan 2014,Jul 2014 |\tF502N\t| NGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/13566.pdf\n\n14012 | \t22\tFeb 2015,Jun 2015\t|\tF502N\t|\tNGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/14012.pdf\n\n14378 | 23  Jan 2016, Jul-Aug 2016 | F502N |  NGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/14378.pdf\n\n14541 | 24 Jan 2017, ... | F502N | NGC6791, NGC104 | \nhttp://www.stsci.edu/hst/phase2-public/14541.pdf\n\nNotes:\n\n* 12348 was not a monitor proposal; its purpose was to test the charge injection\n  (CI) mode as a means of mitigating CTE. But we use some of the images\n  because one of our standard clusters, NGC104, was observed in non-CI mode. \n  (The code is capable of sorting through CI and non-CI modes. For now it just\n  tosses the CI mode images out, but this could always be modified in \n  run_results.pro if you wish.)\n\n* 12379, Visit 13 not only was imaged with CI but also had a guide star failure. \n  (See http://www.stsci.edu/cgi-bin/get-visit-status\nid=12379\u0026markupFormat=html\u0026observatory=HST)\n\n* 12692, Visits 10 and 11, consists of the 180-degree dataset, observed with\n  a non-nominal field in ngc6583.\n\n* 14012, Visit 05 is not part of the normal monitor (the data are binned for a \n  separate experiment) and should be removed from the proposal directory.\n\n\n-------------------------------------------------------------------------------\n\n\n## 3. Running the Pipeline\n\nThe scripts are written in Python.  They were originally in IDL, Fortran, \nand IRAF, written by V. Platais (geometric distortions) and K. Noeske \n(everything else).  They were refurbished by C.M. Gosmeyer in Spring-Summer \n2016.  \n\nYou might find the originals by going back in the master branch's history.\nThe new pipeline was merged on 30 Nov. 2016.\n\nThe pipeline is capable of performing on many combinations of data modes \n(postflash, pixel-based cte-corrected, aperture radius sizes, etc.). In the\nsubsections below we describe the most general cases for running the pipeline.\nWe will not go into the inner workings of each module and function here. \nPlease refer to their doc strings and comment lines. \n\n\n### 3.1 Initial Setup\n\nA new user of the scripts should first complete all steps in this section. \n\nI have checked that the pipeline (with Python-based photometry) works in \nAstroconda with Python 3.  You should try setting up in a Python 3 environment\ninitially.\nTo run the pipeline with IRAF photometry, you'll need a Python 2.7 environment\nand you may need to wrestle with it to get IRAF running... \n\n1. Clone the `wfc3_cte_monitor` repo, to wherever convenient. The scripts\n   are independent of the input data and the output file’s locations.\n   Run the `setup.py`:\n   \n   ```\n   \u003e\u003e\u003e python setup.py develop\n   ```\n\n2. In your `wfc3_cte_monitor\\wfc3_cte_monitor` directory run the script\n\n   ```\n   \u003e\u003e\u003e python initial_setup.py\n   ```\n\n   and follow the prompts. You should now have a “data” and an “outputs” \n   directory where you specified.\n\n   In addition, your `config.py` will be modified to contain the paths you \n   specified to the script. Have a look at `config.py` to check all paths are \n   correct. If you ever change the locations of your repo, \"data\", or \n   \"outputs\" directories, you only need modify this script.\n\n   The “data” directory will eventually contain *flt.fits and *flc.fits \n   files sorted into directories by proposal number. Each Cycle a new \n   directory will be added.  These proposal directories will be generated \n   when new data is processed with `run_tweakreg.py`.\n\n   \"data\" should contain the following directories (initially empty).\n       master_cat - contains the Master catalogs\n       newdata - the holding directory for new data\n       PA_MAPS - contains the pixel area maps\n   \n   The “outputs” directory consists of the following four sub-directories.\n   They initially contain only READMEs describing their contents.\n       finalresults - contains final plots and coefficients.\n       photom - contains coo, mag, and diagonistic plots from photometry.\n       tempfiles - holding folder for temporary files.\n           (this may be degenerate, as nothing in the pipeline currently \n           needs to store temp files)\n       tweakreg - contains intermediate files and diagonistic plots from\n           TWEAKREG. \n\n3. Clone cgosmeyer's `photutils_plus` repo. Go to its top level and install\n   as\n\n   ```\n   \u003e\u003e\u003e python setup.py develop\n   ```\n\n   You should then be able to import the 'photutils_plus' package.  Check this \n   in ipython.\n\n   ```\n   python\u003e from photutils_plus import photutils_plus.\n   ```\n\n4. Likewise, clone cgosmeyer's `analysis_tools` repo and run its `setup.py`.\n\n5. Obtain the Pixel Area Maps.\n    \n   Into \"data/PA_MAPS\" download the pixel area maps for UVIS chips 1 and 2 \n   from http://www.stsci.edu/hst/wfc3/pam/pixel_area_maps\n\n   You should now have \n\n       UVIS1wfc3_map.fits\n       UVIS2wfc3_map.fits     \n\n6. Copy the Master catalogs for the three clusters.  Currently the *cat files\n   and the drizzled images used to create them are located on C.M. Gosmeyer's\n   central store.  \n\n   In \"data/master_cat\" copy directory from [C.M. Gosmeyer will provide you with path]\n\n   The pipeline will expect to find the *cat files in \"data/master_cat/final\".\n\n7. Set up the SQLITE database.  \n\n   If you are in a hurry and want an already filled-in sqlite database file\n   \"uvis_external_cte.db\", copy it from C.M. Gosmeyer's central store to \n   your \"database\" directory. \n   Note that if you do this, you may not be able to repopulate the FileInfo\n   and Phot tables with the `database_reset.py` functions because they rely\n   on the paths to the photometry files stored in the database. You could \n   probably get around this with some hacks.  In any case, if you are \n   taking over this monitor, you probably should populate the database from\n   scratch in your own environment at some point.\n\n   To create the \"uvis_external_cte.db\" file and initialize the tables in\n   the location you specified in `config.py` run \n\n   ```\n   \u003e\u003e\u003e python database_interface.py   \n   ```\n\n   To fill in the Master tables run\n\n   ```\n   \u003e\u003e\u003e populate_master_tables.py\n   ```\n\n   Check that the database is correctly filled in. Go to the location of\n   the SQLITE database and enter it with the following command.\n\n   ```\n   \u003e\u003e\u003e sqlite3 uvis_external_cte.db\n   ```\n\n   Then do\n\n   ```\n   sqlite\u003e .tables\n   ```\n\n   You should see all the following tables.\n\n       ngc104_fileinfo   ngc104_results    ngc6583_phot      ngc6791_master  \n       ngc104_master     ngc6583_fileinfo  ngc6583_results   ngc6791_phot    \n       ngc104_phot       ngc6583_master    ngc6791_fileinfo  ngc6791_results \n\n   Then to check that each table has all the correct columns, do, for \n   example,\n    \n   ``` \n   sqlite\u003e pragma table_info(ngc104_master);\n   ```\n\n   You should see all the columns that were defined in the files located \n   in \"detectors/scripts/uvis_external_cte/table_definitions\"\n\n   Finally check that the Master tables are filled in. For example,\n\n   ```\n   sqlite\u003e select id,xpix,ypix,ra,dec from ngc6583_master;\n   ```\n\n   You should see columns printed containing the catalog information\n   from the .cat files located in \"master_cat/final/\".\n\n   To quit SQLITE, do\n\n   ```\n   sqlite\u003e .exit\n   ```\n\n   Note that there will be *no* entries in the columns until you ingest\n   the data, as outlined in Section 3.2. \n\n8. You should now to ready to run the scripts! Follow the procedure below\n   for ingeseting ALL the data if you haven't yet copied the data into\n   \"outputs\".\n\n** A note about the `config.py`**  \nYou should *not* commit this once you add your personal paths.  \nIn this file you will also find static lists of flux bin ranges, proposal\nIDs, target names, filters, and flash levels.  If you ever need these lists\njust import them from here so you know they will always be consistently the \nsame across all functions (and hopefully all correct!).\nAn example import of the targets would be \n\n    `from config import target_list`\n\nNote that all the looping lists and switches in the pipeline are either set \nin the config.py or are arguments (some of them with default values) in the\nprimary wrapper, `run_uvis_external_cte.py`. \nThe idea is that, to change what subsets of data it is legal for the \npipeline to chew on, modify the config.py. To constrain the pipeline from\nlooping over and doing EVERYTHING, use the command-line switches parsed by\n`run_uvis_external_cte.py`. \nThis idea has been imperfectly implemented as yet; see Section 6. \n\n\n### 3.2 Procedure for Ingesting ALL the Data\n\n1. If you do not already have them in your \"data\" directory, retrieve \n   all the FLCs and FLTs from MAST for all proposal numbers listed above\n   in Section 2 and store them in \"data/newdata\".\n   Alternatively, copy all the RAWs from Quicklook to \"data/newdata\" \n   and run calwf3 over them. You want all the images to be calibrated \n   with the same version of the calibration pipeline.\n\n2. Procede as in Step 3 of \"3.3 Procedure for General Run of the Pipeline\".\n   All images should be copied into subdirectories named for the proposal\n   numbers in the \"data\" directory.\n\n3. If you are running all data for the first time (i.e., you have not yet \n   filled in the FileInfo or Phot tables), do\n\n   ```\n   \u003e\u003e\u003e python run_uvis_external_cte.py --pr 'all' \n   ```\n\n   This step also works if you want to re-fill in the database, i.e.,\n   if you found an error in photometry. All entries will get overwritten.\n\n   The script `run_image_extraction.py` is smart enough to skip over bad\n   or irrelevant datasets within the proposals, such as those described\n   above in Section 2. See the script's functions' docstrings for more \n   details.\n\n4. Wait. Filling in the database could take dayyys unless you can figure \n   out how to multi-process things... The slow-down seems to be writing\n   the photometry results to the database.\n   See Section 6 for more details.\n\n5. Make a copy of your filled database in a seperate, safe directory (even \n   better, on a different partition or machine) so you can fall back\n   to it should anything go horribly wrong with the active database.  \n   Back it up as needed.\n\n6. Read Section 3.3 for more information on individual steps.\n\n\n### 3.3 Procedure for General Run of the Pipeline\n\nFollow this procedure if you have a new epoch to ingest.\nIf starting a new Cycle, be sure that the proposal list in `config.py` \nis updated. (And, of course, update this file's \"Programs\" section!)\n\n1. Copy the FLTs and FLCs from MAST or Quicklook directories to \n   \"data/newdata\".\n   If you are copying ALL the data to be run through scripts for the first\n   time, you should either go through MAST or run calwf3 on the RAWs yourself \n   so you can ensure all files will be calibrated with the same and most \n   recent version of calwf3.\n\n2. Do a quick quality check of all the data. (Open in DS9 or look at\n   the Quicklook JPEGs.) Make sure all expected data for a given visit \n   is there. You can check this by comparing to the proposal pdf.\n   You might also be clever and write a script that checks all expected\n   data is in fact there.\n\n3. In \"data/newdata\" run \n\n   ```\n   \u003e\u003e\u003e python run_tweakreg.py\n   ```\n\n   This will correct the WCS of each image so that the photometry scripts \n   can match the stars exactly to the Master catalog using RAs and Decs.\n   Don't be discouraged if this is a bit slow!\n\n   Once the script is complete, go to the directory \"outputs/tweakreg\"\n   and look through the diagonistic plots.\n   \n   * hist2d_*_flc.png - 2d histogram bins should be centered.\n   * residuals_*_flc.png - scatter points should be evenly distributed\n     around line without systematics.\n   * vector_*_flc.png - vectors should be random and evenly distributed.\n\n   The script will move the FITS files to the appropriate proposal directory\n   in \"outputs/data\". Now you should be ready to run the pipeline.\n\n4. Run the pipeline wrapper `run_uvis_external_cte.py`. It does the \n   following over CTE-corrected on non-CTE-corrected images of NGC104 and \n   NGC6791 for all available flash levels and exposure lengths (long and \n   short).\n   * Matches sources to the Master catalog.\n   * Performs photometry on the matched sources.\n   * Matches the image pairs.\n   * Measures CTE decline per image pair by slopes of Chip 1/Chip 2 flux-\n     ratios vs Chip 2 y-positions.\n   * Creates plots of CTE decline vs log10 flux, fitting polynomials to each \n     epoch, obtaining coefficients for the aperture photometry correction \n     formula. Displays these plots on the Quicklook website.\n   * Creates plots of CTE decline vs MJD. Displays these plots on the \n     Quicklook website.\n   * Fills in the FileInfo, Phot, and Results tables in the database. \n\n   The wrapper's parameters are by default set to assume the most recent \n   ('last') proposal is being ingested. So to run the pipeline in this case, \n   you only need specify one parameter,\n\n   ```\n   \u003e\u003e\u003e python run_uvis_external_cte.py --pr 'last' \n   ```\n\n   Read the doc strings of `run_uvis_external_cte.py` for more options.\n   You can also see all the parameter options on the commandline by\n\n   ```\n   \u003e\u003e\u003e python run_uvis_external_cte.py -h\n   ```\n\n   The pipeline can be run over a specific proposal or over all the \n   proposals, as well as many other variations of plots created and options \n   of whether to insert new data into the database, whether to copy new \n   files to \"automated_outputs\" for display on Quicklook website, etc.\n\n   Note that the 'last' option will re-run *all* the visits through the\n   pipeline for that most recent proposal, which you may not want if you\n   are time-constrained and you already ran the first visits through.\n   It's not bad if a visit gets re-run.  It will just take extra time. \n\n   This functionality could be changed. Really is up to you; it only exists \n   as it is so you don't necessarily need to remember the proposal name to \n   run the pipeline on the newest data nor remember whether you already ran \n   it with the previous visit in that proposal.  It was made to be a dummy-\n   proof. There may have been instances in the old pipeline days where the\n   runner pushed the most recent data through and could not figure out why \n   that proposal's first visit from 6 months ago was missing from the plots \n   (she forgot she hadn't run that one through yet...) or why the pipeline \n   wasn't generating photometry files for the most recent proposal (she had \n   been specifying the name of an older proposal for the pipeline to run...)\n   \n   Anyway! You can instead specify the exact proposal number and exact \n   visit number(s) like in this example:\n\n   ```\n   \u003e\u003e\u003e python run_uvis_external_cte.py --pr '14012' --v 3 4\n   ```\n\n\n### 3.4 Procedure for Processing the 180-Degree ('Same-Chip') Dataset\n\nThe 180-degree dataset is from proposal 12692, visits 10 and 11,\nimaging the non-nominal star cluster NGC 6583. Its purpose was\nto provide a dataset from which the CTE loss of Chip 1 and Chip 2\nindividually could be measured. See the proposal and ISR 2016-17\nfor more information.\n\nBecause it uses its own star cluster, this dataset has its own \ntables in the database: ngc6583_master, etc. \n\nRun the FLTs and FLCs through the tweakreg scripts as you normally would \nfor the other datasets. Then to run the pipeline on these data, simply do\n\n```\n\u003e\u003e\u003e python run_uvis_external_cte.py --pr '180'\n```\n\nOutputs will be ingested into the database and the files will be\nsorted just like the other datasets. For example, outputs/finalresults/pf0/most_recent/ngc6583_F502N_l\n\nNote that there will only be flashlvl=0, filter=F502N (long and short \nexposures), and filter=F606W (long exposure only).\n\nIncluded within the `uvis_external_cte_plots.py` module is a special\n180-degree dataset plot from the function `plot_180_slope_vs_expt_setup`,\nwhich is run automatically from the pipeline in `run_outputs.py` when \nyou set from the commandline, --pr '180'. This function creates plots in \noutputs/finalresults/pf0/most_recent/180cte_expt/\n\nIn addition to the nominal plots (which by themselves really aren't\ntoo enlightening) there is a special plotting script for this dataset:\n\n* cte180test_plots.py -- Overplots the 180-degree dataset onto the \n    nominal NGC 104 data from the same 12692 proposal onto a CTE loss\n    vs log10 flux plot. This was the primary plot of ISR 2016-17.\n\nA few words about how the 180-degree dataset is run through the pipeline.\nYou can see that throughout there are '180' if-else switches. But\nreally the photometry, source-to-master-catalog matching, etc are all\nthe same.  The one spot where there's a substantial difference is in how \nthe image pairs are matched.  \n\nRight now the image-pair matching is a bit hokey. Go to `run_image_extraction.py`\nin the function `create_param_dict`.  There is the line\n\n    if '6583' in targname:\n\nThis procedes to assign images to chip numbers based on hard-coded strings, \nwhich are unique letters in each of the image's filenames.  As a one-off test, \nit's not the worst thing ever...  But if the monitor starts taking these \nimages regularly, you should come up with something more automated!\n\nThe matched pairs are later retrieved using the database querying function\n`query_for_180pair` from module `database_queries.py`. You can see it \nutilized in the 180-degree plotting functions described above.\nIt relies on the chip numbers assigned in `run_image_extraction.py`.\nMatching pairs uniquely is tricky in the 180-degree test. The nominal\ndataset's pairs can be matched based solely on the different chip \nnumbers (where all other parameters match). But in the 180-degree\ntest pairs *have the same chip number*.  Thus the `query_for_180pair`\nfunction relies on the observation setup to provide pairs whose\nfilenames come in alphabetical order, should all other parameters\nbe the same - for example, there are two pairs of long exposures \nthat I wanted to ensure would be unique. Except for consistency, there \nwas no other reason to match one image to the other. (Relying on \nalphabetical order was the lazy way to do this. Could do a more \nsophisticated retrieval of the 'obstime' from the headers or whatever.)\n\n\n\n### 3.5 Procedure for Switching Between Python and IRAF Photometry\n\nI actually have two SQLITE databases which are identical except that\none contains photometry from Python's `photutils.aperture_photometry`\nfunction and the other from IRAF (pyraf-wrapped) `phot` procedure.\n\nTo switch between them, you need do two things.  \n\nFirst, you need change the name of the database in the `config.py` file.\n\nFor my Python-based database:\n\n     db_name = 'uvis_external_cte.db'\n\nFor my IRAF-based databse:\n\n     db_name = 'uvis_external_cte_iraf.db'\n\nSecond, when you run the full pipeline (with the intent to do photometry), \nyou need switch IRAF on with the following keyword in order for IRAF-readable \ncoordinate files to be generated, photometry be performed with the `iraf.phot` \nfunction, and for outputs to be placed in the proper *_iraf subdirectories \nof \"outputs\".\n\n```\n\u003e\u003e\u003e run_uvis_external_cte.py --irafphot 'y'\n```\n\nBy default this option is 'n'.\n\nIf you are just making plots, you don't need to specify IRAF on the \ncommandline because it will extract data from the active database,\nwhich if you modified `config.py` correctly, should be the IRAF one.\n(One could make this more dummy-proof, but works for now...)\n\nSee Section 6 for an explanation why I stuck to IRAF photometry for the\ntime being.\n\n\n### 3.6 Example Use Cases of the Pipeline\n\nBelow we give examples for running the scrips in several scenarios.\n\n* If re-running photometry and databse ingestion over just a specific \n  program (ie, 12379) for aperture 3 and only want to recalculate the\n  slope plots,\n\n  ```\n  \u003e\u003e\u003e python run_uvis_external_cte.py --pr '12379' --ap 3 --pl 'ratio_ypos'\n  ```\n\n\n* If want to do all photometry and plots for 180-degree dataset,\n\n  ```\n  \u003e\u003e\u003e python run_uvis_external_cte.py --pr '180' --pl 'ratio_ypos' '180cte_expt' --ao 'n'\n  ```\n\nAnd so on.   \n\nAlso, this pipeline carries with it an extensive database of stars uniquely\nidentified over many observing modes through over six years of continuous \nobservation. Many side projects could result from this!  \n\n\n\n-------------------------------------------------------------------------------\n\n## 4. Care and Feeding of the Database\n\nThe database is SQLITE, basically a file with the extension DB.  It can be\ncopied, moved, or whatever. Each database 'file' is made up of tables (e.g., \nmaster_ngc104, etc.). Each table is made up of columns (e.g., exptime, filter, \netc.). Entries can be overwritten. \n\nThe columns for each table are defined in the text files contained in the \nsubdirectory \"table_definitions\". \n\nSee Section 5 for a list of all modules and scripts relevent for interacting\nwith the database. \n\n\n### 4.1 Procedure for Adding/Renaming a Column\n\n1. In the \"table_definitions\" subdirectory of this repo, open the table\n   for which you want to add a column. Then type the name of the column\n   followed by the datatype (String, Float, or Integer).\n\n2. In `database_update.py`, go to the appropriate `results*dict` function\n   and add the lists, etc., which all should be obvious in context.\n   The hardest one to modify is `return_phot_dict_list`, since that has \n   so many columsn already for the different photometry apertures and\n   because you need enter them differently for IRAF vs Python-based \n   photometry.\n\n3. Copy the database for safety.\n   \n   ```\n   \u003e\u003e\u003e cp uvis_external_cte.db uvis_external_cte_copy.db\n   ```\n\n4. Enter SQL to drop the table for which you want to add/rename a column.\n   This is permanent!  Be sure you saved a copy of the database!!\n\n   ```\n   \u003e\u003e\u003e sqlite3 uvis_external_cte.db\n\n   sqlite\u003e drop table table_name;\n\n   sqlite\u003e .exit\n   ```\n\n5. Re-generate the table. It should now contain the new/renamed column.\n\n   ```\n   \u003e\u003e\u003e python database_interface.py  \n   ```\n\n   Check that the table and its new/renamed column is there by entering \n   SQL again.\n\n   ```\n   \u003e\u003e\u003e sqlite3 uvis_external_cte.db\n\n   sqlite\u003e .tables\n\n   sqlite\u003e pragma table_info(table_name);\n\n   sqlite\u003e .exit\n   ```\n\n6. Re-populate the table.  Feel free to modify this step as needed.  It\n   will depend on your column. You might be able to just use the \n   `database_reset.py` script to repopulate columns and write another \n   small script to populate the new column.  Or you may have to re-run\n   the pipeline entirely, especially if your column changes have to do\n   with photometry.\n\n7. If you are satisfied that nothing went horribly wrong, delete the\n   copy of the database (presumably you'll have another backup\n   somewhere in a different safe directory anyway!).\n\n\n### 4.2 Procedure for Removing a Column\n\n1. In the \"table_definitions\" subdirectory of this repo, open the table\n   that contains the column and delete its entry.\n\n2. In `database_update.py`, remove all references to it in the appropriate \n   `results*dict` function.\n\n3. Procede as in Section 4.1 to drop the table and repopulate it without\n   the column.\n\n\n### 4.3 Procedure for Adding a Table\n\nLikely any new table you create will need have counterparts for each\ntarget. For examle: master_ngc104, master_ngc6791, master_ngc6583.\nEach instance of a table class will have the same column names and use \nthe same updating and retrieving functions.\n\n1. Create a new file in the \"table_definitions\" subdirectory of this repo.\n   The name of the file must be the base name of the new table. Look into\n   the existing files for how to name the columns within the file.\n\n2. In `database_interface.py`, create an \"orm\" function for the table class.\n   Then intilize it for each target, following the naming convention you'll\n   see, under the ``# Initialize classes`` line.\n   Finally, add the new table instance names to the ``__all__`` list at the\n   top of the module.\n\n3. In `database_update.py` create an \"update\" and \"return\" function for\n   the table class.\n   Add imports from `database_interface` for your new table's instances \n   to the top of the file. For example,\n\n       from database_interface import NGC104_\u003cnewtable\u003e\n       from database_interface import NGC6791_\u003cnewtable\u003e\n\n   Note that in any of the modules that need access to the tables in\n   order to query or update them, you'll need add these imports.\n\n4. Run `database_interface.py` to generate the new tables.\n\n   ```\n   \u003e\u003e\u003e python database_interface.py  \n   ```\n\n   Go into SQL to check that the new tables are there with the correct \n   columns.\n\n   ```\n   \u003e\u003e\u003e sqlite3 uvis_external_cte.db\n\n   sqlite\u003e .tables\n\n   sqlite\u003e pragma table_info(table_name);\n\n   sqlite\u003e .exit\n   ```\n\n5. You may consider adding a \"reset\" function to `database_reset.py`.  \n   May or may not be viable for your new table class.\n\n\n4.4 Procedure for Re-Setting and Re-Populating Database\n-------------------------------------------------------\n\nYou *could* drop all the tables as described in step 4 of Section 4.1. \nBut it's probably easiest just to delete (in reality, stash away in case\nthe worst happens) the *db file and re-generate the database as described\nin Step 5 of Section 3.1.  Then you can, if you wish, re-populate the\nMaster, FileInfo, and Phot tables using the data from the old database.\nGo into `database_reset.py` and modify the Main in order to reset the\ntables you desire.\n\n```\n\u003e\u003e\u003e python database_reset.py\n```\n\nNote that there is not currently a function for reseting the Results tables.\nYou'll still need to run the pipeline like this:\n\n```\n\u003e\u003e\u003e python run_uvis_external_cte.py --pr 'all' --dofind 'n' --dophot 'n' --pl 'ratio_ypos'\n```\n\nThe Master tables always need be generated first.  If starting from scratch\nand you don't want to re-populate with old data, just repopulate the Master\ntables with:\n\n```\n\u003e\u003e\u003e python populate_master_tables.py\n```\n\n\n### 4.5 Help! The Database Locked Me Out!\n\nSometimes the database 'locks' when it the pipeline crashes mid-injestion.\nFortunately unlocking it is a two-command procedure. Go to where the database\nis located and do\n\n    \u003e\u003e\u003e fuser uvis_external_cte_iraf.db\n\n\nYou will get a message about what process locked the database. For example,\n\n    uvis_external_cte_iraf.db: 1234\n\nTo unlock the database, you just need to kill the process (here, '1234'):\n\n    \u003e\u003e\u003e kill -9 1234\n\n\n\n-------------------------------------------------------------------------------\n\n\n## 5. Summary of Scripts and Modules\n\n### 5.1 Pipeline \n\n* config.py -- filled by initial_setup.py. Used for setting global lists\n   and paths, for import.\n\n* initial_setup.py -- run this to initialize directories for pipeline.\n\n* run_adriz.py -- runs AstroDrizzle to create CR mask and to obtain a \n    background measurement in MDRIZSKY keyword. It will be up to results\n    of your testing whether to officially make this the second prep step\n    before running actual pipeline.\n\n* run_image_extraction.py -- performs the setup of the images. This includes \n   pulling out all an storing all relevant header information from the images \n   and performing source-finding, catalog-matching, and photometry on them. \n   From here the FileInfo and Phot tables get filled. \n\n* run_outputs.py -- measures CTE and outputs various plots used for analysis.\n   From here the Results table (CTE measurements) gets filled.\n\n* run_tweakreg.py -- runs TweakReg to correct WCS in new image's headers\n   to WCS in master image of that field. The first prep step before \n   running pipeline itself.\n\n* run_uvis_external_cte.py -- the pipeline-running script. \n\n* set_paths_to_outputs.py -- handles all logic to name paths to outputs \n   in which we want flashlvl-ctecorrected-timestamp ordered directories. \n\n* uvis_external_cte_plots.py -- contains plot-creation functions. \n\n\n### 5.2 Database\n\n* database_interface -- Module to interface and connect to the database.  \n  When run as script on command line, creates initial empty database.\n\n* database_queries -- Module containing the database queries.\n\n* database_reset -- Module for re-populating the database's FileInfo, \n  Master, and Phot tables, without having to redo the photometry calculations.\n  Can be run as a script by modifying the Mean.\n\n* database_update -- Module for interfacing with database to update the \n  various tables either by inserting new records or updating existing ones.\n\n* populate_master_tables -- Populates all the Master tables.\n\n\n### 5.3 For Tests and ISRs\n\n* compare_idl_python.py -- overplots IDL pipeline CTE measurements on\n   Python pipeline measurements. For ISR 2017-xx.\n\n* cte180test_plots.py -- overplots 180-degree log10 flux vs CTE degredation\n   on nominal data from same proposal. For ISR 2016-17.\n\n* photom_tests_modeldata.py -- for comparing iraf.phot to photutils.\n    aperture_photometry on a simulated 2-d gaussian star.\n\n* plot_model_on_reality.py -- plots the model CTE-measuring slopes on the \n    observational slopes vs time. For ISR 2017-xx.\n\n* print_coeff_latextable.py -- prints a base LaTeX table from coefficient\n    text files, for ease of incorporating into ISR 2017-xx.\n\n\n-------------------------------------------------------------------------------\n\n\n## 6. Issues and Future Improvements\n\nThese can be used as exercises for the new monitor lead to get used to the scripts\nand database.  They are in no particular order of importance. \n\n1. Master catalogs could be better. \n   There's an issue with the wide-band F606W data. The stars present in \n   the narrow F502N images way saturate and are useless for our CTE \n   measurements. And further, because the master drizzled image was created \n   from F502N images, the master catalog does not contain many of F606W's \n   faint sources.  One might want to create a second master image/catalog \n   for F606W images only. This is not high priority since F606W\n   observations were discontinued in 2012. But still might be nice to have\n   the F606W results available for the new pipeline.\n   This actually might be a little complicated when filling out the \n   photometry database. Will need to make sure that source IDs for sources\n   that also appear in F502N match.\n\n2. Speed up the database ingestion?\n   I believe the slowest part to the entire pipeline is the ingestion\n   of the photometry table entries.\n   Since it is a SQLITE database, you cannot parallelize the ingestion\n   of entries. You *could* switch to a MySQL database. I chose not to begin\n   with a MySQL database because it lives on a server and I would not have\n   been able to access it offline to do testing (I was working offline\n   in random location A LOT while I was developing this...). It also \n   gave me the freedom to make a bunch of copies to try out one thing on\n   one version of the database. In fact, currently two working versions\n   exist one based on Python photometry and the other on IRAF photometry. \n   A final concern is dealing with IT to setup and maintain the MySQL \n   database (and stuff goes down a lot in this place, in spite of IT's \n   best attempts...). I like the automaty of the SQLITE database, but it's\n   up to you. \n   Once IT has given you the access to MySQL, switching to MySQL is \n   literally one line of code (setting the connection to the server instead \n   of sqlite in `load_connection` function in `database_interface.py`).\n\n3. Switch from IRAF's 'phot' routine to Python's 'aperture_photometry'\n   function from the photutils package.  I already started this. The\n   wrappers to the Python photometry exist in the cgosmeyer repo \n   `photutils_plus`.  I did not use the Python\n   photometry in my ISRs, however because of the disreprency I found between\n   the IRAF photometry (see Bajaj \u0026 Khandrika 2017), and I at the time \n   wanted consistency with the photometry from the IDL pipeline, which also \n   used the IRAF 'phot' routine. \n   In addition, I'm not 100% trusting of photutils's functions. They don't\n   have all the functionality of the IRAF routines, and my wrappers to them\n   in detector_tools could do only so much. For instance, centroiding from\n   photutils still does not work on images with multiple sources, and I believe\n   that to be a contributing factor to why my CTE measurements from Python\n   photometry look so much more gross than the ones from IRAF photometry.\n\n   I have added an option to mask cosmic rays and bad pixels on top of the \n   Python photometry (see Issues 6 and 7). CTE measurements with Python\n   still look gross. \n\n   See Section 3.5 for how to switch between the Python and IRAF\n   photometry.\n\n4. Improve the 2-d polynomial fitting function in `uvis_external_cte_plots.py`. \n   Relevant functions are `fit_empirical_model` and `polyfit2d`.\n   Right now it deals poorly with the ends, so that the low flux bin is not \n   fitting at all, and I end up tossing out that bin entirely. Probably not the \n   best way to do this. The IDL pipeline’s sfit procedure is clearly better \n   at this. (but IDL also has more sophisticated fitting procedures…).  I just \n   pulled the 2-d poly fitter off stack overflow and worked better than anything \n   else I tried.  It would be worth your time to play with this. \n\n5. In the 180-degree dataset, test fixing where the fit for the flux-ratio vs chip2 \n   y-position crosses the 1.0 ratio to always be at 1024 pixels.\n   There might be a statistical error in our measurement of CTE, which depends on \n   the fit crossing at 1024 pixels. For the long exposures this does not seem to \n   vary much image-pair to image-pair.  For the short exposures, especially the low\n   flux bins, you do see the fit crossing at different y-positions, although often\n   these bins are so messy anyway we don’t count them for much… I’m not 100% convinced\n   this extra correction makes sense because there’s additional sources of noise\n   such as cosmic rays, possibly stars being read out by different amplifiers, flat\n   field fluxuations, geometric distortion, etc.  Really, might make more sense to\n   quantify it as an uncertainty and report it as that, rather than try to correct\n   it?  \n   We do NOT do this for the nominal dataset because correcting the fit always\n   cross at y-position 1024 at ratio 1.0 assumes (1) the CTE on the two chips behaves\n   approximately the same and (2) the source at the center (y=1024) on one chip\n   falls at the center on the second chip.  \n   For the nominal monitor assumption (1) fails the CTE degredation in the two chips\n   cannot be assumed to be the same because the two chips were cut from different \n   wafers and likely they had different populations of charge traps. ACS can get away\n   with this assumption because the two chips on WFC were indeed cut from the same\n   wafer. You can convince yourself of the inaccuracy of assumption (2) just\n   by blinking the image pairs. The target fields are very often offset by many pixels.\n   Note that 'fixing the fit' was not done in the original CTE monitor written by\n   Kai Noeske and I believe he did this deliberately because he adopted almost all the other\n   methods employed by M. Chiaberge in his external CTE monitor for ACS/WFC. \n   (see the ACS ISR 2012-05)\n   But anyway you should try to see what happens when you correct the fit to always cross\n   at y-position 1024 at ratio 1.0 in the 180-degree dataset. In ISR 2016-17 I gave only\n   a back of the evelope calculation. \n\n6. Test masking bad pixels and cosmic rays. I wrote scripts to do create the masks and\n   have resulting photometry and CTE measurements, but never stringently looked \n   through the cosmic ray masks to see if they weren't over-masking and never did \n   a strict comparision between the masked and non-masked photometry and CTE measurements. \n   Should also quantify how the masking affects where the linear fit crosses 1.0\n   ratio (as described in Issue 5).\n   The script is called `run_adriz.py` and if it is used, it needs be run as the \n   second prep step after `run_tweakreg.py`.\n\n7. Related to Issue 6...\n   If decide to make `run_adriz.py` the second prep step, might consider some of\n   following changes.\n   * Create seperate columns in the photometry table for bad pixel and cosmic ray-masked \n   results?\n   (I already have a seperate database filled with bad pixel and cosmic-ray masked \n   Python-based photometry. This may be cleaner than creating seperate columns in the\n   main database, and anyway, before investing the time, it would depend on whether \n   we think doing cosmic-ray masking is actually a good idea.)\n   * Change the name of the 'MNCLIP_BKGRD' column in the FILEINFO table to 'MDRIZSKY'.\n   See above in section 4.1, Procedure for Adding/Renaming a Column.\n\n8. Make the FLT/FLC loop and flashlvl loop user-specified in `run_uvis_external_cte.py`. \n   So the user should be able to specify that they just want to run over flashlvl=6 or\n   they just want to run the pipeline over FLCs. \n   It's silly I hard-coded them when everything else is set as a user-given argument\n   or set in config.py.\n   You'll need add them to the arg_parse function and call, to the top docstring, and \n   of course to the run_uvis_exernal_cte function.\n\n9. Pull the hard-coded 'bad visits' from `run_uvis_external_cte.py` to some sort of\n   lists in config.py, in the same vein as Issue 8.\n   There are some data subsets that the pipeline is constrained from running over. \n   See the myriad of comments in `run_uvis_external_cte.py`.\n   The primary subsets are the charge injection visits. From the standpoint of best\n   coding practices, I should not be throwing all these data subsets out in the script\n   but rather by reading some list of \"forbidden visits\" in config.py. It would \n   be effectively the same thing, but keeping with the idea that config.py should be the primary \n   judge of what is legal for the pipeline to run on. \n   But why not just remove the offending images entirely from the data directory?\n   I thought to do this when I began setting up the pipeline, but then rejected it for \n   two primary reasons.\n   (1) When copying data from MAST or Quicklook, or re-grabbing everything after re-calibrating,\n   it's annoying to weed out the bad ones and humans being humans you always miss some, and\n   I've done it before.  Hardcoding the forbidden lists instead of hoping I didn't copy\n   charge-injected image by accident showed itself to be more dummy-proof. \n   (2) I had a vision that the pipeline should be able to run over some of the stranger\n   datasets, such as the Charge Injection ones. So, being opimistic, I retained all those\n   images in the data directory. The pipeline doesn't run properly on them now, but that might be\n   a matter of adding some special functions to handle them (and what else is a pipeline for??). \n   The pipeline should be broad enough that it should be possible for a user to say \n   they want CTE measurements of whatever subset were observed in the 6+ years of the CTE monitor.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcgosmeyer%2Fwfc3_cte_monitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcgosmeyer%2Fwfc3_cte_monitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcgosmeyer%2Fwfc3_cte_monitor/lists"}