{"id":13789895,"url":"https://github.com/geospace-code/georinex","last_synced_at":"2025-05-15T04:04:22.182Z","repository":{"id":30739803,"uuid":"34296204","full_name":"geospace-code/georinex","owner":"geospace-code","description":"Python RINEX 2 / 3 NAV / OBS / sp3 reader \u0026 batch convert to HDF5 with C-like speed","archived":false,"fork":false,"pushed_at":"2025-03-09T21:14:42.000Z","size":8119,"stargazers_count":245,"open_issues_count":36,"forks_count":94,"subscribers_count":27,"default_branch":"main","last_synced_at":"2025-05-15T04:03:14.894Z","etag":null,"topics":["gnss","gps","rinex"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geospace-code.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["scivision"],"ko_fi":"scivision"}},"created_at":"2015-04-21T01:19:29.000Z","updated_at":"2025-05-07T03:47:03.000Z","dependencies_parsed_at":"2023-02-14T06:15:18.843Z","dependency_job_id":"2447b6bd-ab60-4301-a2b8-d478a3996652","html_url":"https://github.com/geospace-code/georinex","commit_stats":{"total_commits":406,"total_committers":13,"mean_commits":31.23076923076923,"dds":0.08374384236453203,"last_synced_commit":"c689a5a6bc2ffb68bc055f150f1da1b6bab12812"},"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geospace-code%2Fgeorinex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geospace-code%2Fgeorinex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geospace-code%2Fgeorinex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geospace-code%2Fgeorinex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geospace-code","download_url":"https://codeload.github.com/geospace-code/georinex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270640,"owners_count":22042858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gnss","gps","rinex"],"created_at":"2024-08-03T22:00:34.285Z","updated_at":"2025-05-15T04:04:22.142Z","avatar_url":"https://github.com/geospace-code.png","language":"Python","funding_links":["https://github.com/sponsors/scivision","https://ko-fi.com/scivision"],"categories":["Open Source Software, Tools and APIs"],"sub_categories":["Python"],"readme":"# GeoRinex\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2580306.svg)](https://doi.org/10.5281/zenodo.2580306)\n![ci](https://github.com/geospace-code/georinex/workflows/ci/badge.svg)\n[![PyPi Download stats](http://pepy.tech/badge/georinex)](http://pepy.tech/project/georinex)\n\nRINEX 4, RINEX 3 and RINEX 2 reader and batch conversion to NetCDF4 / HDF5 in Python or Matlab.\nBatch converts\n[NAV](./Readme_NAV.md)\nand\n[OBS](./Readme_OBS.md)\nGPS RINEX (including Hatanaka compressed OBS) data into\n[xarray.Dataset](http://xarray.pydata.org/en/stable/api.html#dataset)\nfor easy use in analysis and plotting.\nThis gives remarkable speed vs. legacy iterative methods, and allows for HPC / out-of-core operations on massive amounts of GNSS data.\nGeoRinex has over 125 unit tests driven by Pytest.\n\nPure compiled language RINEX processors such as within Fortran NAPEOS give perhaps 2x faster performance than this Python program--that's pretty good for a scripted language like Python!\nHowever, the initial goal of this Python program was to be for one-time offline conversion of ASCII (and compressed ASCII) RINEX to HDF5/NetCDF4,\nwhere ease of cross-platform install and correctness are primary goals.\n\n![RINEX plot](./src/georinex/tests/example_plot.png)\n\n## Input data types\n\n* RINEX 4.x, RINEX 3.x, or RINEX 2.x\n  * NAV\n  * OBS\n* Plain ASCII or seamlessly read compressed ASCII in:\n  * `.gz` GZIP\n  * `.Z` LZW\n  * `.bz2` bzip2\n  * `.zip`\n* Hatanaka compressed RINEX (plain `.crx` or `.crx.gz` etc.)\n* Python `io.StringIO` text stream RINEX\n\nAlso SP3 ephemeris:\n\n* [SP3-a](https://files.igs.org/pub/data/format/sp3.txt)\n* [SP3-c](https://files.igs.org/pub/data/format/sp3c.txt)\n* [SP3-d](https://files.igs.org/pub/data/format/sp3d.pdf)\n\n## Output\n\n* File: NetCDF4 (subset of HDF5), with `zlib` compression.\nThis yields orders of magnitude speedup in reading/converting RINEX data and allows filtering/processing of gigantic files too large to fit into RAM.\n* In-memory: Xarray.Dataset. This allows all the database-like indexing power of Pandas to be unleashed.\n\n## Install\n\nLatest stable release:\n\n```sh\npip install georinex\n```\n\nCurrent development version:\n\n```sh\ngit clone https://github.com/geospace-code/georinex\n\npython -m pip install -e ./georinex\n```\n\n### Selftest\n\nIt can be useful to check the setup of your system with:\n\n```sh\npython -m pytest\n```\n\n```\n158 passed, 1 skipped\n```\n\n## Usage\n\nThe simplest command-line use is through the top-level `python -m georinex.read` script.\nNormally you'd use the `-p` option with single files to plot, if not converting.\n\nRead single RINEX 4, RINEX 3, or RINEX 2 Obs or Nav file:\n\n```sh\npython -m georinex.read myrinex.XXx\n```\n\nRead times from a file (helpful for debugging a file that doesn't read properly):\n\n```sh\npython -m georinex.gtime myrinex.XXx\n```\n\nRead NetCDF converted RINEX data:\n\n```sh\npython -m georinex.read myrinex.nc\n```\n\nBatch convert RINEX to NetCDF4 / HDF5:\n\n```sh\npython -m georinex.rinex2hdf5 ~/data \"*o\" -o ~/data\n```\n\nin this example, the suffix `.nc` is appended to the original RINEX filename: `my.15o` =\u003e `my.15o.nc`\n\nIt's suggested to save the GNSS data to NetCDF4 (a subset of HDF5) with the `-o`option,\nas NetCDF4 is also human-readable, yet say 1000x faster to load than RINEX.\n\nYou can also of course use the package as a python imported module as in\nthe following examples. Each example assumes you have first done:\n\n```python\nimport georinex as gr\n```\n\n### Time limits\n\nTime bounds can be set for reading -- load only data between those time bounds:\n\n```sh\n--tlim start stop\n```\n\noption, where `start` and `stop` are formatted like `2017-02-23T12:00`\n\n```python\ndat = gr.load('my.rnx', tlim=['2017-02-23T12:59', '2017-02-23T13:13'])\n```\n\n## read RINEX\n\nThis convenience function reads several formats (including compressed, Hatanaka) RINEX 2, RINEX 3, RINEX 4 OBS/NAV or `.nc` file:\n\n```python\nobs = gr.load('tests/demo.10o')\n```\n\n## Analysis\n\nA significant reason for using `xarray` as the base class of GeoRinex is that big data operations are fast, easy and efficient.\nIt's suggested to load the original RINEX files with the `-use` or `use=` option to greatly speed loading and conserve memory.\n\nA copy of the processed data can be saved to NetCDF4 for fast reloading and out-of-core processing by:\n\n```python\nobs.to_netcdf('process.nc', group='OBS')\n```\n\n`georinex.__init.py__` shows examples of using compression and other options if desired.\n\n### Join data from multiple files\n\nPlease see documentation for `xarray.concat` and `xarray.merge` for more details.\nAssuming you loaded OBS data from one file into `obs1` and data from another file into `obs2`, and the data needs to be concatenated in time:\n\n```python\nobs = xarray.concat((obs1, obs2), dim='time')\n```\n\nThe `xarray.concat`operation may fail if there are different SV observation types in the files.\nyou can try the more general:\n\n```python\nobs = xarray.merge((obs1, obs2))\n```\n\n### Receiver location\n\nWhile `APPROX LOCATION XYZ` gives ECEF location in RINEX OBS files, this is OPTIONAL for moving platforms.\nIf available, the `location` is written to the NetCDF4 / HDF5 output file on conversion.\nTo convert ECEF to Latitude, Longitude, Altitude or other coordinate systems, use\n[PyMap3d](https://github.com/scivision/pymap3d).\n\nRead location from NetCDF4 / HDF5 file can be accomplished in a few ways:\n\n* `python -m georinex.loc` to load and plot all RINEX and .nc files in a directory\n* using `xarray`\n\n  ```python\n  obs = xarray.open_dataset('my.nc)\n\n  ecef = obs.position\n  latlon = obs.position_geodetic  # only if pymap3d was used\n  ```\n* Using `h5py`:\n\n  ```python\n  with h5py.File('my.nc') as f:\n      ecef = h['OBS'].attrs['position']\n      latlon = h['OBS'].attrs['position_geodetic']\n  ```\n\n## Convert to Pandas DataFrames\n\nAlthough Pandas DataFrames are 2-D, using say `df = nav.to_dataframe()` will result in a reshaped 2-D DataFrame.\nSatellites can be selected like `df.loc['G12'].dropna(0, 'all')` using the usual\n[Pandas Multiindexing methods](http://pandas.pydata.org/pandas-docs/stable/advanced.html).\n\n## Benchmark\n\nAn Intel Haswell i7-3770 CPU with plain uncompressed RINEX 2 OBS processes in about:\n\n* [6 MB file](ftp://data-out.unavco.org/pub/rinex/obs/2018/021/ab140210.18o.Z): 5 seconds\n* [13 MB file](ftp://data-out.unavco.org/pub/rinex/obs/2018/021/ab180210.18o.Z): 10 seconds\n\nThis processing speed is about within a factor of 2 of compiled RINEX parsers, with the convenience of Python, Xarray, Pandas and HDF5 / NetCDF4.\n\nOBS2 and NAV2 currently have the fast pure Python read that has C-like speed.\n\n### Obs3\n\nOBS3 / NAV3 are not yet updated to new fast pure Python method.\n\nOn Haswell laptop:\n\n```sh\ntime python -m georinex.read tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz -u E\n```\n\n\u003e real 48.6 s\n\n```sh\ntime python -m georinex.read tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz -u E -m C1C\n```\n\n\u003e real 17.6 s\n\n### Profiling\n\nusing\n\n```sh\nconda install line_profiler\n```\n\nand `ipython`:\n\n```ipython\n%load_ext line_profiler\n\n%lprun -f gr.obs3._epoch gr.load('tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz', use='E', meas='C1C')\n```\n\nshows that `np.genfromtxt()` is consuming about 30% of processing time, and `xarray.concat` and xarray.Dataset` nested inside `concat` takes over 60% of time.\n\n## Notes\n\n* GPS satellite position is given for each time in the NAV file as Keplerian parameters, which can be [converted to ECEF](https://ascelibrary.org/doi/pdf/10.1061/9780784411506.ap03).\n* https://downloads.rene-schwarz.com/download/M001-Keplerian_Orbit_Elements_to_Cartesian_State_Vectors.pdf\n\n### Number of SVs visible\n\nWith the GNSS constellations in 2018, per the\n[Trimble Planner](https://www.gnssplanning.com/)\nthe min/max visible SV would be about:\n\n* Maximum: ~60 SV maximum near the equator in Asia / Oceania with 5 degree elev. cutoff\n* Minimum: ~6 SV minimum at poles with 20 degree elev. cutoff and GPS only\n\n### RINEX OBS reader algorithm\n\n1. read overall OBS header (so we know what to expect in the rest of the OBS file)\n2. fill the xarray.Dataset with the data by reading in blocks -- another key difference from other programs out there, instead of reading character by character, I ingest a whole time step of text at once, helping keep the processing closer to CPU cache making it much faster.\n\n### Data\n\nFor\n[capable Android devices](https://developer.android.com/guide/topics/sensors/gnss.html),\nyou can\n[log RINEX 3](https://play.google.com/store/apps/details?id=de.geopp.rinexlogger)\nusing the built-in GPS receiver.\n\nUNAVCO [site map](https://www.unavco.org/instrumentation/networks/map/map.html#/): identify the 4-letter callsign of a station, and look in the FTP sites below for data from a site.\n\nUNAVCO RINEX 3 data:\n\n* OBS: ftp://data-out.unavco.org/pub/rinex3/obs/\n* NAV: ftp://data-out.unavco.org/pub/rinex3/nav/\n\nUNAVCO RINEX 2 data:\n\n* OBS: ftp://data-out.unavco.org/pub/rinex/obs/\n* NAV: ftp://data-out.unavco.org/pub/rinex/nav/\n\n### Hatanaka compressed RINEX .crx\n\nCompressed Hatanaka CRINEX files are supported seamlessly via\n[hatanaka](https://github.com/valgur/hatanaka)\nPython package.\nThese are distinct from the supported `.rnx`, `.gz`, or `.zip` RINEX files.\n\nHatanaka, Y. (2008), A Compression Format and Tools for GNSS Observation Data, Bulletin of the Geospatioal Information Authority of Japan, 55, 21-30.\n(available at http://www.gsi.go.jp/ENGLISH/Bulletin55.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeospace-code%2Fgeorinex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeospace-code%2Fgeorinex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeospace-code%2Fgeorinex/lists"}