{"id":22215700,"url":"https://github.com/jobovy/gaia_tools","last_synced_at":"2025-09-06T20:31:41.816Z","repository":{"id":11019594,"uuid":"68023376","full_name":"jobovy/gaia_tools","owner":"jobovy","description":"Tools for working with the @ESAGaia data and related data sets","archived":false,"fork":false,"pushed_at":"2022-08-04T19:42:56.000Z","size":741,"stargazers_count":63,"open_issues_count":2,"forks_count":19,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-12-30T04:52:52.810Z","etag":null,"topics":["astronomy","astrophysics","gaia","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jobovy.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-12T15:47:05.000Z","updated_at":"2024-12-12T17:54:37.000Z","dependencies_parsed_at":"2022-08-07T06:01:02.330Z","dependency_job_id":null,"html_url":"https://github.com/jobovy/gaia_tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobovy%2Fgaia_tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobovy%2Fgaia_tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobovy%2Fgaia_tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jobovy%2Fgaia_tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jobovy","download_url":"https://codeload.github.com/jobovy/gaia_tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232142347,"owners_count":18478482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["astronomy","astrophysics","gaia","python"],"created_at":"2024-12-02T21:42:29.424Z","updated_at":"2025-09-06T20:31:41.803Z","avatar_url":"https://github.com/jobovy.png","language":"Python","funding_links":[],"categories":["Repositories","Uncategorized"],"sub_categories":["Other","Uncategorized"],"readme":"gaia_tools\n-----------\n\nTools for working with the `ESA/Gaia \u003chttp://sci.esa.int/gaia/\u003e`__\ndata and related data sets (`APOGEE\n\u003chttp://www.sdss.org/surveys/apogee/\u003e`__, `GALAH\n\u003chttps://galah-survey.org/\u003e`__, `LAMOST DR2\n\u003chttp://dr2.lamost.org/\u003e`__, and `RAVE\n\u003chttps://www.rave-survey.org/project/\u003e`__).\n\n**PLEASE NOTE THAT THIS PROJECT IS CURRENTLY UNMAINTAINED, AS OF 01/01/2025**\n\n.. contents::\n\nAUTHORS\n========\n\n * Jo Bovy - bovy at astro dot utoronto dot ca\n\nwith contributions from\n\n * Henry Leung\n * Miguel de Val-Borro\n * Nathaniel Starkman\n * Simon Walker\n\nACKNOWLEDGING USE OF THIS CODE\n==============================\n\nPlease refer back to this repository when using this code in your\nwork. If you use the TGAS selection-function code in\n``gaia_tools.select.tgasSelect``, please cite `Bovy (2017)\n\u003chttp://adsabs.harvard.edu/abs/2017MNRAS.470.1360B\u003e`__.\n\nINSTALLATION\n============\n\nStandard python setup.py build/install\n\nEither\n\n``sudo python setup.py install``\n\nor install to a custom directory with\n\n``python setup.py install --prefix=/some/directory/``\n\nor in your home directory with\n\n``python setup.py install --user``\n\nDEPENDENCIES, PYTHON VERSIONS, AND OPERATING SYSTEMS\n====================================================\n\nThis package requires `NumPy \u003chttp://www.numpy.org/\u003e`__, `astropy\n\u003chttp://www.astropy.org/\u003e`__, `astroquery\n\u003chttps://astroquery.readthedocs.io/en/latest/\u003e`__, `tqdm\n\u003chttps://github.com/noamraph/tqdm\u003e`__, and `dateutil\n\u003chttps://dateutil.readthedocs.io\u003e`__. The query functions require\n`psycopg2 \u003chttps://pypi.org/project/psycopg2/\u003e`__. Additionally, some\nfunctions require `Scipy \u003chttp://www.scipy.org/\u003e`__ and `galpy\n\u003chttps://github.com/jobovy/galpy\u003e`__. The selection-function code\nrequires `healpy \u003chttps://github.com/healpy/healpy\u003e`__ (most\nconveniently installed using ``conda``); the\neffective-selection-function code requires `mwdust\n\u003chttps://github.com/jobovy/mwdust\u003e`__ for dealing with extinction. If\nthe `apogee \u003chttps://github.com/jobovy/apogee\u003e`__ package is\ninstalled, this package will use that to access the APOGEE data;\notherwise they are downloaded separately into the **GAIA_TOOLS_DATA**\ndirectory (see below). With standard installation, astropy is used to\nread FITS files. Installing `fitsio\n\u003chttp://github.com/esheldon/fitsio\u003e`__ will cause those routines to\ntake precedence over `astropy\n\u003chttp://docs.astropy.org/en/stable/io/fits/index.html\u003e`__.\n\nThis package should work in both python 2 and 3. Please open an `issue\n\u003chttps://github.com/jobovy/gaia_tools/issues\u003e`__ if you find a part of the\ncode that does not support python 3.\n\nNote that this code is primarily developed on UNIX-like systems. While most of the code runs on Windows without special handling, it is possible that some parts of the code are incompatible with Windows. Please open an `issue \u003chttps://github.com/jobovy/gaia_tools/issues\u003e`__ if you find any part of the code that does not work on Windows, or open a `pull request \u003chttps://github.com/jobovy/gaia_tools/pulls\u003e`__ to fix it.\n\nDATA FILES AND ENVIRONMENT VARIABLES\n=====================================\n\nThis code will download and store various data files. The top-level\nlocation of where these are stored is set by the **GAIA_TOOLS_DATA**\nenvironment variable, which is the path of the top-level directory\nunder which the data will be stored. To use the `apogee\n\u003chttps://github.com/jobovy/apogee\u003e`__ functionality, you also need to\nset the environment variables appropriate for that package.\n\nBASIC USE\n==========\n\nCatalog reading\n^^^^^^^^^^^^^^^\n\nThe basic use of the code is to read various data files and match them\nto each other. For example, to load the `TGAS \u003chttp://www.cosmos.esa.int/web/gaia/iow_20150115\u003e`__ data, do::\n\n    import gaia_tools.load as gload\n    tgas_cat= gload.tgas()\n\nThe first time you use this function, it will download the TGAS data\nand return the catalog (the data is stored locally in a manner that\nmirrors the Gaia Archive, so downloading only happens once).\n\nSimilarly, you can load the RV subsample of `Gaia DR2 \u003chttps://www.cosmos.esa.int/web/gaia/dr2\u003e`__ using::\n\n    gaiarv_cat= gload.gaiarv()\n\nwhich again downloads the data upon the first invocation (and also converts it to fits format for faster access in the future; note that the original CSV files are retained under ``$GAIA_TOOLS_DATA/Gaia/gdr2/gaia_source_with_rv/csv`` and you might want to delete these to save space).\n\n``gaia_tools`` can also load data from various additional surveys, for example, for the `GALAH \u003chttps://galah-survey.org/\u003e`__ survey's DR3 data, do (older DRs are also available)::\n\n    galah_cat= gload.galah()\n\nGALAH DR3 also has Value-Added Catalogs with ages and orbital info (computed using `galpy \u003chttps://www.galpy.org\u003e`__!), which can be loaded using the ``ages=True`` and ``dynamics=True`` keywords.\n    \nThrough an interface to the more detailed `apogee\n\u003chttps://github.com/jobovy/apogee\u003e`__ package, you can also load\nvarious APOGEE data files, for example::\n\n\tapogee_cat= gload.apogee()\n\trc_cat= gload.apogeerc()\n\nIf you don't have the `apogee \u003chttps://github.com/jobovy/apogee\u003e`__\npackage installed, the code will still download the data, but less\noptions for slicing the catalog are available. If you are using APOGEE\nDR14 and want to use the (less noisy) parameters and abundances\nderived using the `astroNN\n\u003chttps://github.com/henrysky/astroNN_spectra_paper_figures\u003e`__ method\nof `Leung \u0026 Bovy (2019a) \u003chttps://arxiv.org/abs/1808.04428\u003e`__, the\ndistances to APOGEE stars determined using a neural-network approach\ntrained on Gaia by Leung \u0026 Bovy (2019b; in prep.), and the ages from\n`Mackereth, Bovy, Leung, et al. (2019)\n\u003chttp://arxiv.org/abs/1901.04502\u003e`__ do::\n\n   apogee_cat= gload.apogee(use_astroNN=True)\n   rc_cat= gload.apogeerc(use_astroNN=True)\n\nThe recommended distances are ``weighted_dist`` (pc) and the ages are\n``astroNN_age``. You can load only the astroNN abundances, only the\ndistances, or only the ages using ``use_astroNN_abundances``,\n``use_astroNN_distances``, and ``use_astroNN_ages``, respectively.\n\nThe ``GALAH``, ``apogee``, and ``apogeerc`` catalog can also be\ncross-matched to Gaia EDR3 upon loading, e.g., as::\n\n  rc_cat, gaia2_matches= gload.apogeerc(xmatch='gaiaedr3')\n\nthrough an interface to the ``gaia_tools.xmatch.cds`` function\ndescribed below (keywords of that function can be specified here as\nwell). Use ``xmatch='gaiadr2'`` to match to the earlier Gaia DR2\ndata. This will return only the stars in the overlap of the two\ncatalogs. The results from the cross-match are cached such that this\nfunction will run much faster the second time if run with the same\nparameters. Note that the caching ignores the option\n``gaia_all_columns`` described below; if you first do the cross-match\nwith that option, that result will be saved, otherwise not; the cached\nresult will be returned regardless of the value of\n``gaia_all_columns`` in the call (remove the cached file to re-do the\ncross-match; the cached file is in the same directory as the data\nfile; see ``gaia_tools.load.path``). This cross-matching capability is\nnot implemented for the next catalogs at the time of writing. Note\nthat cross-matching can take more than an hour.\n\nSimilarly, you can load the `RAVE\n\u003chttps://www.rave-survey.org/project/\u003e`__ and `RAVE-on\n\u003chttps://zenodo.org/record/154381#.V-D27pN97ox\u003e`__ data as::\n\n\trave_cat= gload.rave()\n\traveon_cat= gload.raveon()\n\nLast but not least, you can also load the `LAMOST DR2\n\u003chttp://dr2.lamost.org/\u003e`__ data as::\n\n\tlamost_cat= gload.lamost()\n\nor::\n\n\tlamost_star_cat= gload.lamost(cat='star')\n\nfor just the stars.\n\nCross-matching\n^^^^^^^^^^^^^^\n\nTo match catalogs to each other, use the tools in\n``gaia_tools.xmatch``. For example, to match the GALAH and APOGEE-RC\ncatalogs loaded above and compare the effective temperatures for the\nstars in common, you can do::\n\n\t from gaia_tools import xmatch\n\t m1,m2,sep= xmatch.xmatch(rc_cat,galah_cat,colDec2='dec')\n\t print(rc_cat[m1]['TEFF']-galah_cat[m2]['Teff'])\n\t      Teff\n\t      K\n\t --------------\n\t -12.3999023438\n\t  0.39990234375\n\nwhich matches objects using their celestial coordinates using the\ndefault maximum separation of 2 arcsec. To match catalogs with\ncoordinates at epoch 2000.0 to the TGAS data, which is at epoch 2015.,\ngive the ``epoch1`` and ``epoch2`` keyword. For example, to\ncross-match the APOGEE-RC data and TGAS do::\n\n\t    tgas= gload.tgas()\n\t    aprc= gload.apogeerc()\n\t    m1,m2,sep= xmatch.xmatch(aprc,tgas,colRA2='ra',colDec2='dec',epoch2=2015.)\n\t    aprc= aprc[m1]\n\t    tgas= tgas[m2]\n\nIf your catalogs contain duplicates that differ in another property\nand you want to match on both position and the other property, you can\nmatch both by specifying ``col_field``. An example use case is for\nAPOGEE DR16 which contains stars that are observed by two different\ntelescopes (the APO 2.5m in the northern hemisphere and the LCO 2.5m\nin the southern hemisphere). An example to\ndemonstrate the usage::\n\n        from gaia_tools import xmatch\n\t# Create simple example data catalogs\n        from astropy.table import QTable\n        mc1 = {'RA': [10, 10, 30, 10, 10], 'DEC': [10, 10, 30, 10, 10], 'LENS': ['A', 'B', 'A', 'C', 'A']}\n        mc2 = {'RA': [10, 20, 10, 20, 30], 'DEC': [10, 20, 10, 20, 30], 'LENS': ['A', 'A', 'B', 'B', 'A']}\n        cat1 = QTable()\n        for key in mc1.keys():\n            cat1[key] = mc1[key]\n        cat2 = QTable()\n        for key in mc2.keys():\n            cat2[key] = mc2[key]\n\t# Match mc1 and mc2 on (RA,Dec) and LENS\n        idx1, idx2, sep = xmatch.xmatch(cat1,cat2,col_field='LENS')\n        # array([0, 1, 2, 4])\n        print(idx2)\n        # array([0, 2, 4, 0])\n\nFurther, it is possible to cross-match any catalog to the catalogs in\nthe CDS database using the `CDS cross-matching service\n\u003chttp://cdsxmatch.u-strasbg.fr/xmatch\u003e`__. For example, to match the\nGALAH DR2 catalog to the Gaia DR2 catalog, do the following::\n\n   gaia2_matches, matches_indx= xmatch.cds(galah_cat,colRA='raj2000',colDec='dej2000',xcat='vizier:I/345/gaia2')\n   print(galah_cat['raj2000'][matches_indx[0]],gaia2_matches['ra_epoch2000'][0],gaia2_matches['pmra'][matches_indx[0]],gaia2_matches['pmdec'][matches_indx[0]])\n   (0.00047,0.00049021022,22.319,-10.229)\n\nUse ``xcat='vizier:I/350/gaiaedr3'`` to match to Gaia EDR3 instead. If\nyou want *all* columns in Gaia DR2 or Gaia EDR3, specify\n``gaia_all_columns=True``. This will first run the CDS cross-match,\nthen upload the result to the Gaia Archive, and join to the\n``gaia_source`` table to return all columns. If the Gaia Archive\ncannot be reached for some reason, the limited subset of columns\nreturned by CDS is returned instead.\n\nIf you want to download a catalog from CDS, you can use\n``gaia_tools.load.download.vizier``.\n\nReading RVS or XP spectra\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nTo read the Gaia RVS or sampled XP spectra released in DR3, here is an example for a single star::\n\n    from gaia_tools.load.spec import load_rvs_spec, load_xp_sampled_spec\n\n    # load Gaia DR3 2771993642553377280 xp spectrum\n    wavelength, flux, flux_err = load_xp_sampled_spec(2771993642553377280)\n\n    # load Gaia DR3 2771993642553377280 rvsspectrum\n    wavelength, flux, flux_err = load_rvs_spec(2771993642553377280)\n\n\nYou can also supply a list of source ids::\n\n    from gaia_tools.load.spec import load_rvs_spec, load_xp_sampled_spec\n\n    # load Gaia DR3 2771993642553377280 and 383167952467292288\n    wavelength, flux, flux_err = load_xp_sampled_spec([2771993642553377280, 383167952467292288])\n\n    # also support loading a list of source id with duplicated entries (e.g. from APOGEE allstar some are duplicated)\n    wavelength, flux, flux_err = load_xp_sampled_spec([2771993642553377280, 383167952467292288, 2771993642553377280])\n\n    # also support loading a list of source id with some stars not having corresponding spectra, returning zero array for that star with warnings\n    wavelength, flux, flux_err = load_xp_sampled_spec([2771993642553377280, 1234567891234567891])\n\nThese functions assume that you have downloaded the RVS/XP spectra to ``$GAIA_TOOLS_DATA/Gaia/gdr3/Spectroscopy`` in their respective folders (mirroring the Gaia Archive). Automagic downloading of these spectra is currently not supported.\n    \nTools for querying the Gaia Archive\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe large amount of data in Gaia's DR2 and beyond means that to access\nthe full catalog, the easiest way is to perform ADQL or SQL queries\nagainst the `Gaia Archive database\n\u003chttps://gea.esac.esa.int/archive/\u003e`__. Some tools to help with this\nare located in ``gaia_tools.query``.\n\nThe base function in this module is ``query.query``, which can be used\nto send a query either to the central Gaia Archive or to a local\nPostgres copy of the database. When using a local copy of the\ndatabase, the main Gaia table is best named ``gaiadr2_gaia_source``\n(for ``gaiadr2.gaia_source`` on the Gaia Archive) and similarly\n``gaiadr2_gaia_source_with_rv`` for the RV subset (and similar for\n``gaiaedr3_gaia_source`` for the EDR3 data, but note that local Gaia\nEDR3 queries are currently not supported). In this case, the *same*\nquery can be run locally or remotely (``query.query`` will\nautomatically adjust the tablename), making it easy to mix use of the\nlocal database and the Gaia Archive. The name and user of the local\ndatabase can be set using the ``dbname=`` and ``user=``\noptions. Queries can be timed using ``timeit=True``.\n\nAdvanced tools to create and execute complex ADQL queries are included in this\nmodule via `query.make_query` and `query.make_simple_query`. Both functions are\ndescribed in the following section as well as this `example document`_\n\nTo setup your own local database with Gaia DR2 or EDR3, you can follow\nthe steps described about halfway down `this section\n\u003chttp://astro.utoronto.ca/~bovy/group/data.html#2mass\u003e`__. Note that\nyou will need \u003e1TB of space and be familiar with Postgres database\nmanagement.\n\nFor example, to generate the average proper motion maps displayed\n`here \u003chttps://twitter.com/jobovy/status/992455544291049472\u003e`__, do::\n\n      pm_query= \"\"\"SELECT hpx5, AVG((c1*pmra+c2*pmdec)/cos(b_rad)) AS mpmll,\n      AVG((-c2*pmra+c1*pmdec)/cos(b_rad)) AS mpmbb\n      FROM (SELECT source_id/562949953421312 as hpx5,pmra,pmdec,radians(b) as b_rad,parallax,\n      0.4559838136873017*cos(radians(dec))-0.889988068265628*sin(radians(dec))*cos(radians(ra-192.85947789477598)) as c1,\n      0.889988068265628*sin(radians(ra-192.85947789477598)) as c2 FROM gaiadr2.gaia_source\n      WHERE phot_g_mean_mag \u003c 17.) tab\n      GROUP BY hpx5;\"\"\"\n      # Add and random_index between 0 and 1000000 to the WHERE line for a quicker subset\n\nand then run the query locally as::\n\n    out= query.query(pm_query,local=True)\n\nSetting ``local=False`` will run the query on the Gaia Archive (but\nnote that without the additional ``and random_index between 0 and\n1000000`` the query will likely time out on the Gaia Archive; this is\none reason to have a local copy!)\n\nSimilarly, ``query.query`` can automatically translate queries that\njoin against the 2MASS catalog. For example, the query::\n\n  twomass_query= \"\"\"SELECT gaia.source_id,gaia.bp_rp, gaia.phot_bp_mean_mag as bp, gaia.phot_rp_mean_mag as rp,\n  gaia.phot_g_mean_mag as g, tmass.j_m as j, tmass.h_m as h, tmass.ks_m as k\n  FROM gaiadr2.gaia_source AS gaia\n  INNER JOIN gaiadr2.tmass_best_neighbour AS tmass_match ON tmass_match.source_id = gaia.source_id\n  INNER JOIN gaiadr1.tmass_original_valid AS tmass ON tmass.tmass_oid = tmass_match.tmass_oid\n  WHERE gaia.random_index \u003c 1000000\n  and gaia.phot_g_mean_mag \u003c 13.;\"\"\"\n\ncan be run locally. For this to work, the two ``INNER JOIN`` lines in\nthis query need to be exactly as written here (thus, you need to call\nthe 2MASS table ``tmass``).\n\nSimilarly, ``query.query`` can automatically translate queries that\njoin against the PanSTARRS1 catalog. For example, the query::\n\n  panstarrs_query= \"\"\"SELECT gaia.source_id,gaia.bp_rp, gaia.phot_bp_mean_mag as bp, gaia.phot_rp_mean_mag as rp,\n  gaia.phot_g_mean_mag as g, panstarrs1.g_mean_psf_mag as pg, panstarrs1.r_mean_psf_mag as pr\n  FROM gaiadr2.gaia_source AS gaia\n  INNER JOIN gaiadr2.panstarrs1_best_neighbour AS panstarrs1_match ON panstarrs1_match.source_id = gaia.source_id\n  INNER JOIN gaiadr2.panstarrs1_original_valid AS panstarrs1 ON panstarrs1.obj_id = panstarrs1_match.original_ext_source_id\n  WHERE gaia.random_index \u003c 100000\n  and gaia.phot_g_mean_mag \u003c 13.;\"\"\"\n\ncan be run locally. Again, for this to work, the two ``INNER JOIN``\nlines in this query need to be exactly as written here (thus, you need\nto call the PanSTARRS1 table ``panstarrs1``).\n\n``query.query`` by default also maintains a cache of queries run\npreviously. That is, if you run the exact same query a second time,\nthe cached result is returned rather than re-running the query (which\nmight take a while); this is useful, for example, when re-running a\npiece of code for which running the query is only a single part. The\nlocation of the cache directory is ``$HOME/.gaia_tools/query_cache``\nwhere ``$HOME`` is your home directory. The results from queries are\ncached as pickles, with filenames consisting of the date/time of when\nthe query was run and a hash of the query. You may rename cached\nqueries, as long as you retain the hash in the filename; this is\nuseful to keep track of queries that you do not want to lose and\nknowing what queries they represent. The function\n``gaia_tools.query.cache.nickname(sql_query,nick)`` can be used to\nrename a cached query ``sql_query`` by giving it a nickname ``nick``\n(e.g., ``nick`` can be ``gaia_cmd``, it should not be a filename,\nbecause an appropriate filename is generated by the code). To clean\nthe cache, do::\n\n\tfrom gaia_tools.query import cache\n\tcache.clean()\n\nwhich removes all cached files with the default ``date/time_hash.pkl``\nfilename format (that is, if you have renamed a cached file, it is not\nremoved by ``cache.clean()``). To remove absolutely all files\n(including renamed ones), use ``cache.cleanall()``. Upon loading the\n``gaia_tools.query`` module, cached files with the default\n``date/time_hash.pkl`` filename format *older than one week* are\nremoved.\n\nTo turn off caching, run queries using ``use_cache=False``.\n\n\nExtended tools for querying the Gaia Archive\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nTwo functions are provided for creating **and** executing queries:\n``make_query`` and ``make_simple_query``. Both functions have robust default\nADQL queries, allow for complex user input, can automatically perform *2MASS*\nand *panSTARRS1* crossmatches, and then perform the query and cache the results.\n\nThere is an example document demonstrating varying uses and options for both\n``make_query`` and ``make_simple_query`` in this `example document`_.\n\n.. _`example document`: ./examples/make_gaia_query_examples.ipynb\n\nThe call signature of ``make_query`` is::\n\n    make_query(\n        # ADQL Options\n        WHERE=None, ORDERBY=None, FROM=None, random_index=None,\n        user_cols=None, all_columns=False,\n        gaia_mags=False, panstarrs1=False, twomass=False,\n        use_AS=False, user_ASdict=None, defaults='default',\n        inmostquery=False,\n        units=False,\n        # Query Options\n        do_query=False, local=False, cache=True, timeit=False,\n        verbose=False, dbname='catalogs', user='postgres',\n        # Extra Options\n        _tab='    ', pprint=False):\n\n``make_simple_query`` is a wrapper for ``make_query``, but optimized for\nsingle-layer queries. The options ``use_AS`` and ``inmostquery``\nare forced to ``True`` and ``_tab`` is not included.\n\nThe ADQL options of `make_query` are:\n\n\n``WHERE``\n\toptional user-input `WHERE' argument.\n\t\n\t* None: skips\n\t* str: used in query\n\n\texample::\n\n\t\t`1=CONTAINS(POINT('ICRS',gaia.ra,gaia.dec),\n\t\t\t    CIRCLE('ICRS',200.,65.,5.))`\n\n\n``ORDERBY``\n\toptional user-input `ORDER BY' argument.\n\n\t* None: skips\n\t* str: used in query\n\n\texample::\n\n\t\t`gaia.source_id`\n\n\n``FROM``\n\toptional user-input `FROM' argument.\n\n\t* None: skips\n\t* str: used in query\n\n\n\tThe ``FROM`` argument is the most powerful part of the ADQL functions.\n\tBy calling ``make_query`` in ``FROM`` it is very easy to create nested ADQL functions.\n\n\texample::\n\n\t\t# Innermost Query\n\t\tFROM=make_query(\n\t\t\t...\n\t\t\tinmostquery=True, # telling system this is the innermost level\n\t\t)\n\n\tNote that it is necessary to specify ``inmostquery`` if the query is the innermost\n\tquery. It is for this reason ``make_simple_query`` is provided: to preclude specifying\n\ta query is single-levelled.\n\n\n``random_index``\n\tthe gaia.random_index for fast querying\n\n\t* None: skips\n\t* int: appends ``AND random_index \u003c ...`` to ``WHERE``\n\n\n``user_cols``\n\tData columns in addition to default columns\n\n\t* None: skips\n\t* str: uses columns\n\n\t``user_cols`` specified in an outer level of a query must have corresponding ``user_cols``\n\tin all inner levels, so that the columns can properly propagate through the query.\n\tFor convenience, ``user_cols`` will automatically remove trailing commas, which would\n\totherwise break the ADQL query and be difficult to debug.\n\n\texample::\n\n\t\tmake_query(\n\t\t\tuser_cols=\"gaia.L, gaia.B,\"  # \u003c- trailing , automatically trimmed\n\t\t\tFROM=make_query(\n\t\t\t\tuser_cols=\"gaia.L, gaia.B\"\n\t\t\t)\n\t\t)\n\n\n``all_columns``\n\twhether to include all columns.\n\n``gaia_mags``\n\twhether to include Gaia magnitudes as specified in ``defaults``\n\n``panstarrs1``\n\twhether to INNER JOIN Panstarrs1, using columns specified in ``defaults``\n\n``twomass``\n\twhether to INNER JOIN 2MASS, using columns specified in ``defaults``\n\n\t* ``use_AS``: add 'AS __' to the data columns, as specified in ``defaults``\n\t\n\tThis is good for the outer part of the query so as to have convenient names in the output data table.\n\t``use_AS`` should never by used for an inner-level query.\n\n``user_ASdict``\n\tdictionary with `AS' arguments for ``user_cols``\n\n``defaults``\n\tfile for default columns, units, AS specifications, etc\n\n\t* 'default': the default file\n\t* 'empty': only sourc_id is built-in\n\t* 'full': a more verbose set of columns\n\t* other \u003cstr\u003e: a custom defaults file\n\t* \u003cdict\u003e: a custom defaults file\n\n\tFor the included columns for `default`, `empty`, and `full`, check out the \n\t`example document`_.\n\n\tFor an example of a custom ``defaults`` file, see this `example json file`_.\n\n.. _`example json file`: ./examples/custom_defaults.json\n\n\n``inmostquery``\n\tneeded if in-most query\n\n``units``\n\tadds units to a query, as specified in ``defaults``\n\n``do_query``\n\tperforms the query\n\n``local``\n\tto perform locally or on Gaia servers\n\n``cache``\n\tto cache the result, with nickname specification\n\n\t* True (False): does (not) cache\n\t* str: caches with nickname = str\n\n``timeit``\n\tif True, print how long the query ran\n\n``verbose``\n\tif True, up verbosity level\n\n``dbname``\n\tif local, the name of the postgres database\n\n``user``\n\tif local, the name of the postgres user\n\n``_tab``\n\tthe tab. In general, this need not be changed\n\n``pprint``\n\tto print the query\n\n\n\nThe TGAS selection function\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n`Bovy (2017) \u003chttp://adsabs.harvard.edu/abs/2017MNRAS.470.1360B\u003e`__\ndetermines the raw TGAS selection function over the 48% of the sky\nwhere the TGAS selection is well behaved. This selection function\ngives the fraction of true point-like objects observed as a function\nof *(J,J-Ks)* 2MASS photometry and as a function of position on the\nsky. Bovy (2017) also discusses how to turn this raw selection\nfunction into an effective selection function that returns the\nfraction of true stars contained in the TGAS catalog as a function of\ndistance and position on the sky, for a given stellar population and\nhow to compute the fractional volume of a given spatial region that is\neffectively contained in TGAS (this is the denominator in N/V when\ncomputing bias-corrected densities based on TGAS star counts in a\ncertain spatial region). Tools to work with the raw and effective\nselection functions are contained in the\n``gaia_tools.select.tgasSelect`` sub-module.\n\nThe raw selection function is contained in an object and can be\ninstantiated as follows::\n\n\t     \u003e\u003e\u003e import gaia_tools.select\n\t     \u003e\u003e\u003e tsf= gaia_tools.select.tgasSelect()\n\nWhen you run this code for the first time, a ~200 MB file that\ncontains 2MASS counts necessary for the selection function will be\ndownloaded. When instantiating the ``tgasSelect`` object, it is\npossible to make different choices for some of the parameters\ndescribed by Bovy (2017), but it is best to leave all keywords at\ntheir default values. To then evaluate the fraction observed at\n*J=10*, *J-Ks* = 0.5, RA= 10 deg, Dec= 70.deg, do::\n\n\t \u003e\u003e\u003e tsf(10.,0.5,10.,70.)\n\t array([ 0.7646336])\n\nAnother example::\n\n\t\u003e\u003e\u003e tsf(10.,0.5,10.,20.)\n\tarray([ 0.])\n\nThe latter is exactly zero because the (RA,Dec) combination falls\noutside of the part of the sky over which the selection function is\nwell behaved. The method ``tsf.determine_statistical`` can return the\npart of your TGAS sub-sample that is part of the sky over which the\nselection function is well behaved. For example, to plot the data in\nTGAS for which the selection function is determined, do::\n\n     \u003e\u003e\u003e import gaia_tools.load as gload\n     \u003e\u003e\u003e tgas_cat= gload.tgas()\n     \u003e\u003e\u003e twomass= gload.twomass()\n     \u003e\u003e\u003e indx= tsf.determine_statistical(tgas_cat,twomass['j_mag'],twomass['k_mag'])\n     \u003e\u003e\u003e import healpy\n     \u003e\u003e\u003e healpy.mollview(title=\"\")\n     \u003e\u003e\u003e healpy.projplot(tgas_cat['l'][indx],tgas_cat['b'][indx],'k,',lonlat=True,alpha=0.03)\n\nwhich gives\n\n.. image:: _readme_files/tgas_stat.png\n\nWe can turn the raw TGAS selection function into an effective\nselection function that is a function of distance rather than\nmagnitude for a given stellar population by specifying a sampling of\ntrue intrinsic absolute *M_J* and true *J-Ks* for this stellar\npopulation. We also require a three-dimensional extinction map,\nalthough by default the extinction is set to zero (for this, you need\nto install `mwdust \u003chttps://github.com/jobovy/mwdust\u003e`__). A simple\nexample of this is the following instance::\n\n\t\u003e\u003e\u003e import mwdust\n\t\u003e\u003e\u003e tesf= gaia_tools.select.tgasEffectiveSelect(tsf,dmap3d=mwdust.Zero(),MJ=-1.,JK=0.65)\n\nwhich is close to a red-clump effective selection function. We can\nthen evaluate ``tesf`` as a function of (distance,RA,Dec) to give the\nfraction of stars with absolute *M_J = -1* and *J-Ks* = 0.65 contained\nin TGAS, for example at 1 kpc distance and (RA,Dec) = (10,70)::\n\n   \u003e\u003e\u003e tesf(1.,10.,70.)\n   array([ 0.89400531])\n\nWe could do the same taking extinction into account::\n\n   \u003e\u003e\u003e tesf_ext= gaia_tools.select.tgasEffectiveSelect(tsf,dmap3d=mwdust.Combined15(filter='2MASS J'),MJ=-1.,JK=0.65)\n   \u003e\u003e\u003e tesf_ext(1.,10.,70.)\n   array([ 0.27263462])\n\nThis is much lower, because the extinction toward (RA,Dec) = (70,10)\n=~ (l,b) = (122,7.1) is very high (A_J =~ 0.7). Note that the ``MJ``\nand ``JK`` inputs can be arrays, in which case the result will be\naveraged over these, and they can also be changed on-the-fly when\nevaluating the effective selection function.\n\nWe can also compute the effective volume as defined by Bovy\n(2017). For this, we need to define a function that defines the volume\nover which we want to compute the effective volume. For example, a\ncylindrical volume centered on the Sun is::\n\n   def cyl_vol_func(X,Y,Z,xymin=0.,xymax=0.15,zmin=0.05,zmax=0.15):\n       \"\"\"A function that bins in cylindrical annuli around the Sun\"\"\"\n       xy= numpy.sqrt(X**2.+Y**2.)\n       out= numpy.zeros_like(X)\n       out[(xy \u003e= xymin)*(xy \u003c xymax)*(Z \u003e= zmin)*(Z \u003c zmax)]= 1.\n       return out\n\nWe can then compute the effective volume for a cylinder of radius 0.15\nkpc from z=0.1 kpc to 0.2 kpc as::\n\n    \u003e\u003e\u003e dxy= 0.15\n    \u003e\u003e\u003e zmin= 0.1\n    \u003e\u003e\u003e zmax= 0.2\n    \u003e\u003e\u003e tesf.volume(lambda x,y,z: cyl_vol_func(x,y,z,xymax=dxy,zmin=zmin,zmax=zmax),ndists=101,xyz=True,relative=False)\n    0.0023609512382473932\n\nSetting ``relative=True`` would return the fractional effective\nvolume, that is, the effective volume divided by the true spatial\nvolume; computing the relative volume and multiplying it with the true\nvolume is a more robust method for computing the effective volume\n(because pixelization effects in the computation of the effective\nvolume cancel out). Compare::\n\n       \u003e\u003e\u003e tesf.volume(lambda x,y,z: cyl_vol_func(x,y,z,xymax=dxy,zmin=zmin,zmax=zmax),ndists=101,xyz=True,relative=False)/(numpy.pi*dxy**2.*(zmax-zmin))\n       0.33400627552533657\n\nwith::\n\n\t\u003e\u003e\u003e tesf.volume(lambda x,y,z: cyl_vol_func(x,y,z,xymax=dxy,zmin=zmin,zmax=zmax),ndists=101,xyz=True,relative=True)\n\t0.3332136527277989\n\nAs you are running these examples, you will notice that evaluating the\neffective volume is much faster the second time you do it (even for a\ndifferent volume). This is because the evaluation of the selection\nfunction gets cached and re-used. Taking extinction into account (that\nis, running these examples using ``tesf_ext`` rather than ``tesf``)\ntakes *much* longer. Tools to use multiprocessing are available in\nthis case.\n\nFor more examples of how to use this code, please see the\n`tgas-completeness \u003chttps://github.com/jobovy/tgas-completeness\u003e`__\nrepository, which contains all of the code to reproduce the results of\nBovy (2017).\n\nRECIPES\n========\n\nMatch APOGEE or APOGEE-RC to Gaia DR2\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWe can do this with the `CDS xMatch Service \u003chttp://cdsxmatch.u-strasbg.fr/\u003e`__ using the ``gaia_tools.xmatch.cds`` routine::\n\n    apogee_cat= gaia_tools.load.apogee()\n    gaia2_matches, matches_indx= gaia_tools.xmatch.cds(apogee_cat,xcat='vizier:I/345/gaia2')\n    apogee_cat= apogee_cat[matches_indx]\n    print(len(apogee_cat))\n    264423\n\n(takes about fifteen minutes). Make the first line ``apogee_cat= gaia_tools.load.apogeerc()`` for the APOGEE-rc catalog.\n\nMatch RAVE to TGAS taking into account the epoch difference\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nRAVE celestial positions (and more generally all of the positions in\nthe spectoscopic catalogs) are given at epoch J2000, while TGAS\nreports positions at J2015. To match stars between RAVE and TGAS, we\ntherefore have to take into account the proper motion to account for\nthe 15 year difference. This can be done as follows::\n\n    tgas= gaia_tools.load.tgas()\n    rave_cat= gaia_tools.load.rave()\n    m1,m2,sep= gaia_tools.xmatch.xmatch(rave_cat,tgas,\n\t\t\t\t\tcolRA1='RAdeg',colDec1='DEdeg',\n\t\t\t\t\tcolRA2='ra',colDec2='dec',\n\t\t\t\t\tepoch1=2000.,epoch2=2015.,swap=True)\n    rave_cat= rave_cat[m1]\n    tgas= tgas[m2]\n    print(len(rave_cat))\n    216201\n\nThe ``xmatch`` function is setup such that the second catalog is the\none that contains the proper motion if the epochs are different. This\nis why TGAS is the second catalog. Normally, ``xmatch`` finds matches\nfor all entries in the first catalog. However, RAVE contains\nduplicates, so this would return duplicate matches and the resulting\nmatched catalog would still contain duplicates. Because TGAS does not\ncontain duplicates, we can do the match the other way around using\n``swap=True`` and get a catalog without duplicates. There is currently\nno way to rank the duplicates by, e.g., their signal-to-noise ratio in\nRAVE.\n\nMatch LAMOST to TGAS\n^^^^^^^^^^^^^^^^^^^^^\n\nSimilar to RAVE above, we do::\n\n    tgas= gaia_tools.load.tgas()\n    lamost_cat= gaia_tools.load.lamost()\n    m1,m2,sep= gaia_tools.xmatch.xmatch(lamost_cat,tgas,\n\t\t\t\t\tcolRA1='ra',colDec1='dec',\n\t\t\t\t\tcolRA2='ra',colDec2='dec',\n\t\t\t\t\tepoch1=2000.,epoch2=2015.,swap=True)\n    lamost_cat= lamost_cat[m1]\n    tgas= tgas[m2]\n    print(len(lamost_cat))\n    108910\n\nMatch APOGEE or APOGEE-RC to TGAS\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nSimilar to RAVE above, we do::\n\n    tgas= gaia_tools.load.tgas()\n    apogee_cat= gaia_tools.load.apogee()\n    m1,m2,sep= gaia_tools.xmatch.xmatch(apogee_cat,tgas,\n\t\t\t\t\tcolRA2='ra',colDec2='dec',\n\t\t\t\t\tepoch1=2000.,epoch2=2015.,swap=True)\n    apogee_cat= apogee_cat[m1]\n    tgas= tgas[m2]\n    print(len(apogee_cat))\n    20113\n\nMake that second line ``apogee_cat= gaia_tools.load.apogeerc()`` for\nthe APOGEE-RC catalog.\n\nMatch GALAH DR1 to TGAS\n^^^^^^^^^^^^^^^^^^^^^^^^\n\nSimilar to RAVE above, we do::\n\n    tgas= gaia_tools.load.tgas()\n    galah_cat= gaia_tools.load.galah(dr=1)\n    m1,m2,sep= gaia_tools.xmatch.xmatch(galah_cat,tgas,\n\t\t\t\t\tcolRA1='RA',colDec1='dec',\n\t\t\t\t\tcolRA2='ra',colDec2='dec',\n\t\t\t\t\tepoch1=2000.,epoch2=2015.,swap=True)\n    galah_cat= galah_cat[m1]\n    tgas= tgas[m2]\n    print(len(galah_cat))\n    7919\n\nAPI\n====\n\n(May or may not be fully up-to-date)\n\n * ``gaia_tools.load``\n     * ``gaia_tools.load.apogee``\n     * ``gaia_tools.load.apogeerc``\n     * ``gaia_tools.load.astroNN``\n     * ``gaia_tools.load.astroNNDistances``\n     * ``gaia_tools.load.gaiarv``\n     * ``gaia_tools.load.galah``\n     * ``gaia_tools.load.lamost``\n     * ``gaia_tools.load.rave``\n     * ``gaia_tools.load.raveon``\n     * ``gaia_tools.load.tgas``\n         * ``gaia_tools.load.download.vizier``\n * ``gaia_tools.query``\n     * ``gaia_tools.query.query``\n     * ``gaia_tools.query.cache``\n        * ``gaia_tools.query.cache.autoclean``\n        * ``gaia_tools.query.cache.clean``\n        * ``gaia_tools.query.cache.cleanall``\n        * ``gaia_tools.query.cache.current_files``\n        * ``gaia_tools.query.cache.file_path``\n        * ``gaia_tools.query.cache.load``\n        * ``gaia_tools.query.cache.nickname``\n        * ``gaia_tools.query.cache.save``\n    * ``gaia_tools.query.make_query``\n    * ``gaia_tools.query.make_simple_query``\n * ``gaia_tools.select``\n     * ``gaia_tools.select.tgasSelect``\n         * ``__call__``\n\t * ``determine_statistical``\n\t * ``plot_mean_quantity_tgas``\n\t * ``plot_2mass``\n\t * ``plot_tgas``\n\t * ``plot_cmd``\n\t * ``plot_magdist``\n     * ``gaia_tools.select.tgasEffectiveSelect``\n         * ``__call__``\n\t * ``volume``\n * ``gaia_tools.xmatch``\n     * ``gaia_tools.xmatch.xmatch``\n     * ``gaia_tools.xmatch.cds``\n     * ``gaia_tools.xmatch.cds_matchback``\n * ``gaia_tools.util``\n     * ``gaia_tools.json``\n        * ``gaia_tools.json.strjoinall``\n        * ``gaia_tools.json.strjoinkeys``\n        * ``gaia_tools.json.prettyprint``\n     * ``gaia_tools.table_utils``\n        * ``gaia_tools.table_utils.neg_to_nan``\n        * ``gaia_tools.table_utils.add_units_to_Table``\n        * ``gaia_tools.table_utils.add_color_col``\n        * ``gaia_tools.table_utils.add_calculated_col``\n        * ``gaia_tools.table_utils.add_abs_pm_col``\n        * ``gaia_tools.table_utils.rename_columns``\n        * ``gaia_tools.table_utils.drop_colnames``\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjobovy%2Fgaia_tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjobovy%2Fgaia_tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjobovy%2Fgaia_tools/lists"}