https://github.com/mithaystack/scikit-dataaccess

Scikit Data Access: Data Interfaces for Python
https://github.com/mithaystack/scikit-dataaccess

Last synced: 12 days ago
JSON representation

Scikit Data Access: Data Interfaces for Python

Host: GitHub
URL: https://github.com/mithaystack/scikit-dataaccess
Owner: MITHaystack
License: other
Created: 2016-05-16T16:13:27.000Z (almost 9 years ago)
Default Branch: master
Last Pushed: 2019-06-18T18:30:57.000Z (almost 6 years ago)
Last Synced: 2024-11-06T04:48:16.624Z (6 months ago)
Language: Python
Homepage:
Size: 38 MB
Stars: 45
Watchers: 12
Forks: 12
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

Scikit Data Access

- Import scientific data from various sources through one easy Python API.
- Use iterator patterns for each data source (configurable data generators + functions to get next data chunk).
- Skip parser programming and file format handling.
- Enjoy a common namespace for all data and unleash the power of data fusion.
- Handle data distribution in different modes: (1) local download, (2) caching of accessed data, or (3) online stream access
- Easily pull data on cloud servers through Python scripts and facilitate large-scale parallel processing.
- Build on an extensible plattform: Adding access to a new data source only requires addition of its "DataFetcher.py".
- Open source (MIT License)

Scikit Data Access Overview

Supported data sets:

Namespace

Preview
^(link)

Description & Data Source

^Astronomy

^astro.kepler

^{Light curves for stars imaged by the NASA Kepler Space Telescope
Source: https://keplerscience.arc.nasa.gov}

^{astro.spectra}

^{Spectra from the Sloan Digital Sky Survey
Source: https://www.sdss.org/dr14/spectro/}

^{astro.tess.data}

^{Light curves from TESS Data Alerts
Source: https://archive.stsci.edu/prepds/tess-data-alerts/}

^{astro.tess.simulated}

^{Simulated light curves from TESS End-to-End 6
Source: https://archive.stsci.edu/prepds/tess-data-alerts/}

^{astro.voyager}

^{Data from the Voyager mission.
Source: https://spdf.gsfc.nasa.gov/}

^Engineering

^{engineering.la.traffic_counts}

^{Traffic Count data in Los Angeles.
Source: https://data.lacity.org/A-Livable-and-Sustainable-City/LADOT-Traffic-Counts-Summary/94wu-3ps3}

^{engineering.webcam.mit_sailing}

^{Images from webcams located at the MIT Sailing Pavilion
Source: http://sailing.mit.edu/webcam.php}

^Finance

^{finance.timeseries}

^{Financial time series data retrieved using Alpha Vantage API.
Source: https://www.alphavantage.co/}

^Geoscience

^{geo.era_interim}

^{Era-Interim data at different pressure values from
the European Centre for Medium-Range Weather Forecasts accessed through the University Corporation for Atmospheric Research.
Source: https://rda.ucar.edu/datasets/ds627.0/}

^geo.gldas

^{Land hydrology model produced by NASA. This version of the data is generated to match the GRACE temporal and spatial characteristics and is available as a complementary data product.
Source: https://grace.jpl.nasa.gov/data/get-data/land-water-content}

^geo.grace

^{NASA GRACE Tellus Monthly Mass Grids. 30-day measurements of changes in Earth’s gravity field to quantify equivalent water thickness.
Source: https://grace.jpl.nasa.gov/data/get-data/monthly-mass-grids-land}

^{geo.grace.mascon}

^{NASA GRACE Tellus Monthly Mass Grids - Global Mascons. 30-day measurements of changes in Earth’s gravity field to quantify equivalent water thickness. Source: https://grace.jpl.nasa.gov/data/get-data/jpl_global_mascons}

^{geo.groundwater}

^{United States groundwater monitoring wells measuring the depth to water level. Source: https://waterservices.usgs.gov}

^{geo.magnetometer}

^{Data collected at magnetic observatories operated by the U.S. Geological Survey. Source: https://geomag.usgs.gov}

^{geo.mahali.rinex}

^{Rinex files from the MIT led NSF project studying the Earth’s ionosphere with GPS.
Web: http://mahali.mit.edu}

^{geo.mahali.tec}

^{Total Electron Content from the MIT led NSF project studying the Earth’s ionosphere with GPS.
Web:http://mahali.mit.edu}

^{geo.mahali.temperature}

^{Temperature data from the MIT led NSF project studying the Earth’s ionosphere with GPS.
Web: http://mahali.mit.edu}

^geo.modis

^{Spectroradiometer aboard the NASA Terra and Aqua image satellites. Generates approximately daily images of the Earth’s surface.
Source:https://modis.gsfc.nasa.gov}

^geo.pbo

^{EarthScope - Plate Boundary Observatory (PBO): Daily GPS displacement time series measurements throughout the United States.
Source: http://www.unavco.org/projects/major-projects/pbo/pbo.html}

^{geo.sentinel_1}

^{Sentinel-1 TOPSAR data from the European Space Agency retrieved from the Alaska Satellite Facility.
Source:https://www.asf.alaska.edu/}

^geo.srtm

^{Elevation data at a one arc second resolution from the Shuttle Radar Topography Mission (SRTMGL1).
Source: https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003}

^geo.uavsar

^{UAVSAR SLC data from JPL.
Source: https://uavsar.jpl.nasa.gov/}

^{geo.wyoming_sounding}

^{Sounding data from the University of Wyoming.
Source: http://weather.uwyo.edu/upperair/sounding.html}

^{Planetary Science}

^{planetary.ode}

^{Mars planetary data from PDS Geosciences Node's Orbital Data Explorer.
Source: http://pds-geosciences.wustl.edu/default.htm}

^{Solar Science}

^solar.sdo

^{Images from the Solar Dynamics Observatory.
Source: https://sdo.gsfc.nasa.gov/}

### Install
```python
pip install scikit-dataaccess
```

### Documentation

- User Manual: [/docs/skdaccess_manual.pdf](https://github.com/MITHaystack/scikit-dataaccess/blob/master/skdaccess/docs/skdaccess_manual.pdf)

- Code documentation (Doxygen): [/docs/skdaccess_doxygen.pdf](https://github.com/MITHaystack/scikit-dataaccess/blob/master/skdaccess/docs/skdaccess_doxygen.pdf)
- Code visualization (treemap): [/docs/skdaccess_treemap.png](https://github.com/MITHaystack/scikit-dataaccess/blob/master/skdaccess/docs/skdaccess_treemap.png)
- Code class diagrams: [/docs/class_diagrams](https://github.com/MITHaystack/scikit-dataaccess/tree/master/skdaccess/docs/class_diagrams)

### Contributors

Project lead: [Victor Pankratius (MIT)](http://www.victorpankratius.com)

Contributors: Cody M. Rude, Justin D. Li, David M. Blair, Michael G. Gowanlock, Guillaume Rongier, Victor Pankratius

New contributors welcome! Contact to contribute and add interface code for your own datasets :smile:

### Acknowledgements

We acknowledge support from NASA AIST14-NNX15AG84G, NASA AIST16-80NSSC17K0125, NSF ACI-1442997, and NSF AGS-1343967.

## Examples

Code examples (Jupyter notebooks) for all datasets listed above are available at: [/skdaccess/examples](https://github.com/MITHaystack/scikit-dataaccess/tree/master/skdaccess/examples)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/mithaystack/scikit-dataaccess

Awesome Lists containing this project

README