{"id":42569265,"url":"https://github.com/intelligent-data-design-inc/nep","last_synced_at":"2026-05-28T23:01:47.221Z","repository":{"id":324221929,"uuid":"1013626631","full_name":"Intelligent-Data-Design-Inc/NEP","owner":"Intelligent-Data-Design-Inc","description":"NetCDF Expansion Pack","archived":false,"fork":false,"pushed_at":"2026-05-28T14:13:19.000Z","size":23001,"stargazers_count":14,"open_issues_count":5,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T16:10:49.283Z","etag":null,"topics":["compression","netcdf"],"latest_commit_sha":null,"homepage":"https://intelligent-data-design-inc.github.io/NEP/","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Intelligent-Data-Design-Inc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-04T07:51:51.000Z","updated_at":"2026-03-26T22:29:19.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Intelligent-Data-Design-Inc/NEP","commit_stats":null,"previous_names":["intelligent-data-design-inc/nep"],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/Intelligent-Data-Design-Inc/NEP","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelligent-Data-Design-Inc%2FNEP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelligent-Data-Design-Inc%2FNEP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelligent-Data-Design-Inc%2FNEP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelligent-Data-Design-Inc%2FNEP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Intelligent-Data-Design-Inc","download_url":"https://codeload.github.com/Intelligent-Data-Design-Inc/NEP/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelligent-Data-Design-Inc%2FNEP/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33629560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","netcdf"],"created_at":"2026-01-28T21:04:22.467Z","updated_at":"2026-05-28T23:01:47.214Z","avatar_url":"https://github.com/Intelligent-Data-Design-Inc.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"![NEP Logo](docs/images/logo_small.png)\n\n# NEP (NetCDF Expansion Pack)\n\n**[📚 Full Documentation](https://intelligent-data-design-inc.github.io/NEP/)**\n\n## High-Performance Compression + Multi-Format Data Access\n\nNEP extends NetCDF-4 with powerful new capabilities for scientific data workflows:\n\n- **Ultra-Fast LZ4 Compression**: 2-3x faster than DEFLATE with excellent compression ratios - ideal for real-time data processing and HPC workflows\n- **High-Ratio BZIP2 Compression**: Superior compression for archival storage - reduce storage costs while maintaining data integrity\n- **NASA CDF File Reader**: Access Common Data Format files directly through the familiar NetCDF API - no conversion needed\n- **GeoTIFF File Reader**: Read GeoTIFF geospatial raster files through the NetCDF API with CF-1.8 compliant CRS metadata\n- **GRIB2 File Reader**: Read GRIB2 meteorological and oceanographic data files (NWP model output, wave forecasts) through the NetCDF API\n- **Drop-In Compatibility**: Works with existing NetCDF-4 applications without code changes\n\n## Why NEP?\n\nScientific data producers need better tools to handle growing data volumes:\n\n- **Storage Costs**: Petabyte-scale datasets require efficient compression without sacrificing performance\n- **Processing Speed**: DEFLATE compression creates bottlenecks in data pipelines and analysis workflows\n- **Data Format Silos**: CDF and NetCDF communities use different tools despite similar data structures\n- **Limited Options**: NetCDF-4 needs more compression algorithms optimized for different use cases\n\n## What NEP Delivers\n\n### LZ4 Compression: Speed Without Compromise\n\n**Performance**: 2-3x faster compression and decompression than DEFLATE while achieving 2.2x compression ratios on typical scientific datasets.\n\n**Use Cases**:\n- Real-time satellite data processing\n- High-throughput simulation output\n- Interactive data analysis workflows\n- Cloud-based data pipelines\n\n**How It Works**: LZ4 compression is provided as an HDF5 filter plugin. Simply set `HDF5_PLUGIN_PATH` and use standard NetCDF-4 compression APIs - no code changes required.\n\n### BZIP2 Compression: Maximum Storage Efficiency\n\n**Performance**: 6.7x compression ratios on scientific datasets - significantly better than DEFLATE for long-term archival storage.\n\n**Use Cases**:\n- Long-term data archives\n- Reducing cloud storage costs\n- Datasets with repetitive patterns\n- Bandwidth-constrained data transfers\n\n**How It Works**: Like LZ4, BZIP2 integrates as an HDF5 filter plugin with zero code changes to existing applications.\n\n### CDF File Reader: Unified Data Access\n\n**Capability**: Read NASA Common Data Format files using the NetCDF API you already know.\n\n**Benefits**:\n- No file conversion required - access CDF data directly\n- Automatic type mapping (CDF types → NetCDF types)\n- Support for TT2000 time variables and multi-dimensional arrays\n- Unified analysis tools for both NetCDF and CDF datasets\n\n**Use Cases**:\n- Space physics and heliophysics research\n- NASA mission data analysis (IMAP, MMS, Van Allen Probes)\n- Cross-format data integration\n- Legacy CDF data access in modern workflows\n\n**How It Works**: NEP provides a User-Defined Format (UDF) handler that transparently reads CDF files through standard NetCDF functions like `nc_open()`, `nc_get_var()`, and `nc_get_att()`.\n\n## Key Benefits\n\n- **Choose Your Trade-Off**: Select LZ4 for speed or BZIP2 for compression ratio - optimize for your specific workflow\n- **No Code Changes**: Drop-in replacement for existing NetCDF-4 applications via HDF5 filter plugins\n- **Multi-Format Support**: Work with both NetCDF and CDF files using a single API\n- **Production Ready**: Full CMake and Autotools build support, comprehensive test suites, CI validation\n- **HPC Optimized**: Designed for large-scale scientific computing with Spack package manager support\n- **Cost Savings**: Reduce storage and bandwidth costs without sacrificing data access performance\n\n## Compression Performance\n\nThe following benchmarks compare compression methods on a 150 MB NetCDF-4 dataset. For comparison, I include ZSTD, which was recently added to NetCDF and provides better performance than ZLIB, though not as fast as LZ4:\n\n### All Compression Methods\n\n![Compression Performance](docs/compression_performance.svg)\n\n### Fast Compression Methods (Excluding BZIP2)\n\nFor better visualization of the faster compression methods:\n\n![Fast Compression Performance](docs/compression_performance_fast.svg)\n\n| Method | Write Time (s) | File Size (MB) | Read Time (s) | Compression Ratio | Write Speed | Read Speed |\n|--------|----------------|----------------|---------------|-------------------|-------------|------------|\n| none   | 0.27          | 150.01         | 0.14          | 1.0×              | 1.0×        | 1.0×       |\n| lz4    | 0.34          | 68.95          | 0.16          | 2.2×              | 0.79×       | 0.88×      |\n| zstd   | 0.66          | 34.94          | 0.26          | 4.3×              | 0.41×       | 0.54×      |\n| zlib   | 1.83          | 41.78          | 0.59          | 3.6×              | 0.15×       | 0.24×      |\n| bzip2  | 22.14         | 22.39          | 5.90          | 6.7×              | 0.01×       | 0.02×      |\n\n**Key Insights:**\n- **LZ4** offers the best balance: 2.2× compression with minimal performance impact (79% write speed, 88% read speed)\n- **ZSTD** (recently added to NetCDF) provides excellent compression (4.3×) with moderate performance impact (41% write speed, 54% read speed)\n- **ZLIB** (standard DEFLATE) shows 3.6× compression but is slower than both LZ4 and ZSTD\n- **BZIP2** achieves the highest compression ratio (6.7×) but is significantly slower (1% write speed, 2% read speed)\n- **Read performance** generally mirrors write performance, with LZ4 being fastest and BZIP2 slowest\n\n---\n\n## CDF Reader\n\nNEP includes support for reading NASA Common Data Format (CDF) files through a NetCDF-like API. CDF is a self-describing data format designed for the storage and manipulation of multi-dimensional data sets, widely used in space physics and solar research communities.\n\n### What is CDF?\n\nThe Common Data Format (CDF) is a conceptually similar format to NetCDF, developed and maintained by NASA's Space Physics Data Facility (SPDF). CDF files are commonly used for storing time-series and multi-dimensional scientific data from space missions and ground-based observations.\n\n**Key characteristics:**\n- Self-describing format with metadata\n- Support for multiple data types and dimensions\n- Platform-independent binary format\n- Optimized for space physics data\n\n### Resources\n\n- **[NASA CDF Homepage](https://cdf.gsfc.nasa.gov/)** - Official CDF library and documentation\n- **[CDF C Reference Manual](https://spdf.gsfc.nasa.gov/pub/software/cdf/doc/cdf_C_RefManual.pdf)** - Complete C API reference\n\n### CDF Support in NEP\n\nNEP provides a User-Defined Format (UDF) handler that allows reading CDF files using NetCDF-style API calls. This enables applications to work with both NetCDF and CDF files through a unified interface.\n\nTo enable CDF support during build, use the `--enable-cdf` (Autotools) or `-DENABLE_CDF=ON` (CMake) configuration option. You must have the NASA CDF library installed on your system.\n\n---\n\n## GeoTIFF Reader\n\nNEP v1.6.0 provides full-featured reading of GeoTIFF files through the NetCDF API, including CF-1.8 compliant CRS metadata, coordinate variables, and coordinate bounds.\n\n### What is GeoTIFF?\n\nGeoTIFF is a public domain metadata standard that allows georeferencing information to be embedded within TIFF image files. It's the de facto standard for geospatial raster data exchange and is widely used in remote sensing, GIS applications, and Earth observation missions.\n\n**Key characteristics:**\n- Standard TIFF format with geospatial extensions\n- Embedded coordinate reference system (CRS) information\n- Support for various map projections and datums\n- Multi-band raster data support\n- Platform-independent format\n\n### Resources\n\n- **[GeoTIFF Homepage](https://www.geotiff.org/)** - Official GeoTIFF specification\n- **[libgeotiff](https://github.com/OSGeo/libgeotiff)** - Open source GeoTIFF library\n\n### GeoTIFF Support in NEP\n\nNEP provides a User-Defined Format (UDF) handler that allows reading GeoTIFF files using NetCDF-style API calls. This enables applications to work with GeoTIFF, NetCDF, and CDF files through a unified interface.\n\n**Current Features (v1.6.0):**\n- ✅ Automatic format detection — GeoTIFF files recognized by magic number\n- ✅ **BigTIFF support** — handles files \u003e4 GB via dual UDF registration\n- ✅ Open/close via standard `nc_open()` and `nc_close()`\n- ✅ Dimension and data type extraction\n- ✅ **CF-1.8 CRS metadata** — `crs` grid-mapping variable with `grid_mapping_name`, `semi_major_axis`, `inverse_flattening`, and projection-specific attributes\n- ✅ **Coordinate variables** — `lon`/`lat` (degrees) for geographic CRS, `x`/`y` (metres) for projected CRS\n- ✅ **Coordinate bounds** — `lon_bnds`/`lat_bnds` or `x_bnds`/`y_bnds` for pixel-as-area rasters\n- ✅ **Pixel raster type** — pixel-as-point and pixel-as-area handling via `GTRasterTypeGeoKey`\n- ✅ **Multi-band raster reading** — 3D variable access (band, y, x)\n- ✅ **Single-band raster reading** — optimised 2D access\n- ✅ **Hyperslab operations** — efficient subsetting and partial reads\n- ✅ Both tiled and striped TIFF organisations; PLANARCONFIG_CONTIG and PLANARCONFIG_SEPARATE\n- ✅ Graceful degradation — files without CRS tags still open and data is readable\n- ⏳ Coordinate transformations via PROJ (future release)\n\nSee [docs/cf-compliance.md](docs/cf-compliance.md) for the full CF grid mapping attribute specification.\n\n**Usage Example:**\n\n```c\n#include \u003cnetcdf.h\u003e\n\nint ncid, varid, crs_varid, lon_varid;\nint retval;\n\n/* Open GeoTIFF file - automatically detected (standard TIFF and BigTIFF) */\nif ((retval = nc_open(\"satellite_image.tif\", NC_NOWRITE, \u0026ncid)))\n    ERR(retval);\n\n/* Get raster data variable (always named \"data\" for single-band files) */\nif ((retval = nc_inq_varid(ncid, \"data\", \u0026varid)))\n    ERR(retval);\n\n/* Read a single pixel [y=100, x=200] */\nsize_t start[2] = {100, 200};\nsize_t count[2] = {1, 1};\nunsigned char pixel;\nif ((retval = nc_get_vara_uchar(ncid, varid, start, count, \u0026pixel)))\n    ERR(retval);\n\n/* Read a 100x100 hyperslab from the top-left corner */\nsize_t start2[2] = {0, 0};\nsize_t count2[2] = {100, 100};\nunsigned char *buffer = malloc(100 * 100);\nif ((retval = nc_get_vara_uchar(ncid, varid, start2, count2, buffer)))\n    ERR(retval);\n\n/* Access CF-1.8 CRS metadata on the 'crs' grid-mapping variable */\nchar grid_mapping_name[NC_MAX_NAME + 1];\ndouble semi_major;\nif ((retval = nc_inq_varid(ncid, \"crs\", \u0026crs_varid)) == NC_NOERR) {\n    nc_get_att_text(ncid, crs_varid, \"grid_mapping_name\", grid_mapping_name);\n    nc_get_att_double(ncid, crs_varid, \"semi_major_axis\", \u0026semi_major);\n}\n\n/* Multi-band files: use 3D access (band, y, x) */\nsize_t start3[3] = {0, 100, 200};  /* band 0, y=100, x=200 */\nsize_t count3[3] = {1, 50, 50};    /* 1 band, 50×50 pixels */\nunsigned char *multiband = malloc(50 * 50);\nif ((retval = nc_get_vara_uchar(ncid, varid, start3, count3, multiband)))\n    ERR(retval);\n\n/* Close file */\nif ((retval = nc_close(ncid)))\n    ERR(retval);\n```\n\n**Build Configuration:**\n\nGeoTIFF support is enabled by default. To disable:\n\n```bash\n# CMake\ncmake -B build -DENABLE_GEOTIFF=OFF\n\n# Autotools\n./configure --disable-geotiff\n```\n\nYou must have libgeotiff and libtiff installed on your system.\n\n**Useful Tools for Working with GeoTIFF Files:**\n\n```bash\n# Essential tools for GeoTIFF inspection and manipulation\nsudo apt install gdal-bin        # GDAL command-line utilities\nsudo apt install libtiff-tools   # TIFF utilities (tiffinfo, tiffdump)\nsudo apt install qgis            # GUI for viewing/analyzing GeoTIFF files (optional)\n```\n\n**Key GDAL commands:**\n- `gdalinfo \u003cfile.tif\u003e` - Display detailed file information (bands, CRS, metadata)\n- `gdal_translate` - Convert between formats and extract subsets\n- `gdalwarp` - Reproject and transform raster data\n- `gdal_merge.py` - Merge multiple GeoTIFF files\n\n**Key libtiff commands:**\n- `tiffinfo \u003cfile.tif\u003e` - Display TIFF structure and tags\n- `tiffdump \u003cfile.tif\u003e` - Dump TIFF directory contents\n- `tiffcp` - Copy and convert TIFF files\n\n**Example usage:**\n```bash\n# Check if a file is multi-band\ngdalinfo satellite_image.tif | grep \"Band\"\n\n# View planar configuration\ntiffinfo satellite_image.tif | grep \"Planar Configuration\"\n\n# Get basic file info\ntiffinfo satellite_image.tif | grep -E \"(Image Width|Image Length|Samples)\"\n```\n\n**Use Cases:**\n- NASA Earth observation data (MODIS, Landsat, Sentinel)\n- Satellite imagery analysis\n- Digital elevation models (DEMs)\n- Land cover classification maps\n- Climate and weather model outputs\n- Integration of GIS data with NetCDF workflows\n\n### GeoTIFF Performance Characteristics\n\nNEP's GeoTIFF implementation is optimized for selective data access patterns typical of NetCDF applications. Performance characteristics vary significantly between full raster reads and hyperslab (partial) reads:\n\n#### Hyperslab Read Performance\n\nFor hyperslab operations (reading subsets of data), NEP demonstrates excellent performance:\n\n| Operation | File Size | NEP Performance vs Native |\n|-----------|-----------|---------------------------|\n| Small hyperslab (10×10) | 4800×4800 | **99% faster** |\n| Medium hyperslab (100×100) | 4800×4800 | **84% faster** |\n| Small hyperslab (10×10) | 41013×55877 | **90% faster** |\n| Medium hyperslab (100×100) | 41013×55877 | **98% faster** |\n\nNEP outperforms naive native libgeotiff approaches for hyperslab reads because:\n- Tiled implementation reads only necessary data\n- Optimized for selective access patterns\n- Efficient tile-based I/O minimizes data transfer\n\n#### Full Raster Read Performance\n\nFull raster reads (reading entire images) show different characteristics:\n\n| Operation | File Size | NEP vs Native |\n|-----------|-----------|---------------|\n| Full raster read | 4800×4800 | ~60× slower |\n\nFull raster reads are slower because:\n- NEP reads tile-by-tile through the NetCDF API layer\n- Native comparison uses highly optimized bulk read functions (`TIFFReadRGBAImageOriented`)\n- This operation is not the primary use case for NetCDF API\n\n#### Performance Context\n\nThe NetCDF API is designed for selective data access (hyperslabs, strided reads, single variables) rather than bulk file reads. NEP's GeoTIFF implementation reflects this design philosophy:\n\n- **Optimized for**: Subsetting, regional analysis, time-series extraction, multi-file workflows\n- **Not optimized for**: Reading entire rasters in a single operation\n\nFor applications requiring full raster reads, consider using native GeoTIFF tools (GDAL, libgeotiff) or pre-processing data into NetCDF format. For typical scientific workflows involving selective data access, NEP provides excellent performance.\n\n**Benchmark Details**: Performance measurements conducted on tiled GeoTIFF files using 1 iteration per test. Results represent typical performance on modern hardware. See `docs/performance.md` for complete methodology and detailed results.\n\n---\n\n## GRIB2 Reader\n\nNEP v1.7.0 provides reading of GRIB2 meteorological and oceanographic data files through the standard NetCDF API.\n\n### What is GRIB2?\n\nGRIB2 (General Regularly-distributed Information in Binary form, Edition 2) is the standard format used by NOAA, ECMWF, and other agencies to distribute numerical weather prediction (NWP) model output and wave forecast data.\n\n**Key characteristics:**\n- Gridded binary format optimized for meteorological data\n- Products organized by discipline, category, and parameter number\n- Bitmap-based land/sea masking\n- Used by NOAA GFS, NAM, HRRR, GDAS, and global wave forecast models\n\n### GRIB2 Support in NEP\n\n**Current Features (v1.7.0):**\n- ✅ Automatic format detection — GRIB2 files recognized by `GRIB` magic number\n- ✅ Open/close via standard `nc_open()` and `nc_close()`; `ncdump` works directly\n- ✅ Product inventory — all GRIB2 products exposed as named `NC_FLOAT` variables\n- ✅ Shared `[y, x]` dimensions — all variables on the same grid share one dim pair\n- ✅ Full grid expansion — `g2_getfld(expand=1)` fills complete `[ny][nx]` grid\n- ✅ Bitmap handling — land/masked points filled with `_FillValue = 9.999e20f`\n- ✅ Variable names from `g2c_param_abbrev()`; duplicates uniquified with `_2`, `_3` suffixes\n- ✅ Per-variable attributes: `long_name`, `_FillValue`, `GRIB2_discipline`, `GRIB2_category`, `GRIB2_param_number`\n- ✅ Global attributes: `Conventions = \"GRIB2\"`, `GRIB2_edition = 2`\n- ✅ `.ncrc` autoload support\n\n**Usage Example:**\n\n```c\n#include \u003cnetcdf.h\u003e\n\nint ncid, varid;\nfloat data[151 * 241];  /* ny=151, nx=241 for GDAS West Coast wave grid */\n\n/* Open GRIB2 file - automatically detected */\nif ((retval = nc_open(\"gdaswave.t00z.wcoast.0p16.f000.grib2\", NC_NOWRITE, \u0026ncid)))\n    ERR(retval);\n\n/* Look up a variable by GRIB2 parameter abbreviation */\nif ((retval = nc_inq_varid(ncid, \"WIND\", \u0026varid)))\n    ERR(retval);\n\n/* Read full [ny][nx] grid; land points = 9.999e20 (_FillValue) */\nif ((retval = nc_get_var_float(ncid, varid, data)))\n    ERR(retval);\n\nif ((retval = nc_close(ncid)))\n    ERR(retval);\n```\n\nOr use `ncdump` directly:\n\n```bash\nncdump -h gdaswave.t00z.wcoast.0p16.f000.grib2\nncdump -v WIND gdaswave.t00z.wcoast.0p16.f000.grib2\n```\n\n**Build Configuration:**\n\nGRIB2 support is enabled by default. To disable:\n\n```bash\n# CMake\ncmake -B build -DENABLE_GRIB2=OFF\n\n# Autotools\n./configure --disable-grib2\n```\n\nRequires NOAA NCEPLIBS-g2c (\u003e= 2.1.0) and libjasper (\u003e= 3.0.0). Supply the g2c install path via `G2C_PATH` or `--with-g2c` at configure time.\n\n**Note**: GRIB2 and CDF are mutually exclusive — both use UDF slot 2. Enable one or the other, not both.\n\n---\n\n## UDF Autoloading via .ncrc\n\nNEP installs a `.ncrc` configuration file that enables NetCDF-C's UDF self-loading\nmechanism. Once configured, any application can open GeoTIFF, CDF, and GRIB2 files through the\nstandard `nc_open()` API without calling `NC_GEOTIFF_initialize()`, `NC_CDF_initialize()`,\nor `NC_GRIB2_initialize()` explicitly.\n\n### Quickstart\n\nAfter installing NEP, merge the configuration into your `~/.ncrc`:\n\n```bash\ncat /usr/local/share/nep/.ncrc \u003e\u003e ~/.ncrc\n```\n\nThen open GeoTIFF, CDF, or GRIB2 files from any application without extra initialization:\n\n```c\nint ncid;\nnc_open(\"satellite_image.tif\",                        NC_NOWRITE, \u0026ncid);  /* GeoTIFF */\nnc_open(\"data.cdf\",                                   NC_NOWRITE, \u0026ncid);  /* CDF */\nnc_open(\"gdaswave.t00z.wcoast.0p16.f000.grib2\",       NC_NOWRITE, \u0026ncid);  /* GRIB2 */\n```\n\n### Alternate: per-session via NETCDF_RC\n\n```bash\nexport NETCDF_RC=/usr/local/share/nep\n```\n\n### Install Path Override\n\n| Build system | Default | Override |\n|---|---|---|\n| CMake | `${prefix}/share/nep/.ncrc` | `-DNEP_NCRC_INSTALL_DIR=\u003cpath\u003e` |\n| Autotools | `${datarootdir}/nep/.ncrc` | `--with-ncrc-dir=\u003cpath\u003e` |\n\nFor full details see [docs/build-options.md](docs/build-options.md#udf-autoloading-via-ncrc-v155)\nand the [NetCDF UDF documentation](https://docs.unidata.ucar.edu/netcdf/NUG/user_defined_formats.html).\n\n---\n\n## Example Programs\n\nNEP v3.5.1 includes comprehensive example programs in C and Fortran to help you learn NetCDF API usage. These examples demonstrate both read and write operations, covering basic to advanced features.\n\n### What's Included\n\n**Classic NetCDF Examples:**\n- Quickstart introduction to NetCDF\n- Basic 2D arrays and coordinate variables\n- Format variants (classic, 64-bit offset, CDF-5)\n- Size limits and unlimited dimensions\n- 4-dimensional variables\n\n**NetCDF-4 Examples:**\n- NetCDF-4 file creation\n- Compression filters (deflate, shuffle)\n- Chunking strategies and performance\n- Multiple unlimited dimensions\n- User-defined compound and enum types\n- Hierarchical groups and dimension visibility\n\n**Dual Language Support:**\n- All examples provided in both C and Fortran\n- Equivalent functionality across languages\n- Demonstrates language-specific API usage\n\n### Running Examples\n\nExamples are built automatically and run as tests:\n\n```bash\n# CMake\ncmake -B build\ncmake --build build\nctest --test-dir build\n\n# Autotools\n./configure\nmake\nmake check\n```\n\nEach example creates a NetCDF file and reads it back to verify correctness, demonstrating both write and read operations.\n\n### Disabling Examples\n\nIf you don't need the examples:\n\n```bash\n# CMake\ncmake -B build -DBUILD_EXAMPLES=OFF\n\n# Autotools\n./configure --disable-examples\n```\n\nFor more details, see `examples/README.md`.\n\n---\n\n## Installation\n\n### Prerequisites\n\nNEP v1.5.0 requires the following dependencies:\n\n- **NetCDF-C library** (v4.9+)\n- **HDF5 library** (v1.12+)\n- **CMake** (v3.9+) or **Autotools** for building\n- **LZ4 library** for LZ4 compression support\n- **BZIP2 library** for BZIP2 compression support\n- **NetCDF-Fortran** (optional, for Fortran wrappers)\n- **NASA CDF library** (v3.9+, optional, for CDF file support)\n- **libgeotiff** (latest stable, optional, for GeoTIFF file support)\n- **libtiff** (latest stable, optional, required by libgeotiff)\n- **Doxygen** (optional, for building documentation)\n\n### Spack Installation (Recommended for HPC)\n\nNEP and CDF can be installed using Spack for simplified dependency management:\n\n```bash\n# Install NEP with all features\nspack install nep\n\n# Install NEP with minimal features\nspack install nep~docs~fortran\n\n# Install CDF library separately\nspack install cdf\n\n# Load packages\nspack load nep\nspack load cdf\n```\n\n**Status**: NEP and CDF packages submitted to spack/spack-packages repository (PR pending approval).\n\nFor more details on Spack installation options and variants, see **[Spack Installation Guide](docs/spack.md)**.\n\n### Test Data\n\nNEP includes comprehensive test suites with real-world sample data files located in `test/data/`:\n\n#### CDF Test Files\n\n**`imap_mag_l1b-calibration_20240229_v001.cdf`** (3.2 KB)\n- NASA IMAP (Interstellar Mapping and Acceleration Probe) magnetometer calibration data\n- L1B calibration dataset from February 29, 2024\n- Contains multi-dimensional arrays and TT2000 time variables\n- Used for testing CDF UDF handler functionality\n- Source: NASA Space Physics Data Facility (SPDF)\n\n**`imap_mag_cdfdump.txt`** (8.9 KB)\n- Reference output from NASA's `cdfdump` utility for validation\n- Used to verify correct metadata extraction and data reading\n\n#### GeoTIFF Test Files\n\n**`MCDWD_L3_F1C_NRT.A2025353.h00v02.061.tif`** (41 KB)\n- MODIS/Aqua+Terra Global Flood Product (tile h00v02)\n- Single-band raster: 4800×4800 pixels, 8-bit unsigned integer\n- Resolution: 250m (~0.002° pixel size)\n- Coverage: 70°N to 60°N latitude, 180°W to 170°W longitude\n- Planar configuration: Single image plane (PLANARCONFIG_CONTIG)\n- Used for testing GeoTIFF single-band reading and organization detection\n\n**`MCDWD_L3_F1C_NRT.A2025353.h00v03.061.tif`** (383 KB)\n- MODIS/Aqua+Terra Global Flood Product (tile h00v03)\n- Single-band raster: 4800×4800 pixels, 8-bit unsigned integer\n- Resolution: 250m (~0.002° pixel size)\n- Coverage: 60°N to 50°N latitude, 180°W to 170°W longitude\n- Planar configuration: Single image plane (PLANARCONFIG_CONTIG)\n- Used for testing GeoTIFF reading with different data patterns\n\n**Data Source:** NASA LANCE (Land, Atmosphere Near real-time Capability for EOS)\n- Product: MCDWD_L3_F1C_NRT v6.1\n- Description: 1-day composite flood detection with cloud shadow masks\n- DOI: 10.5067/MODIS/MCDWD_L3_F1C_NRT.061\n\n**`ABBA_2022_C61_HNL.tif`** (5.4 MB)\n- Arctic Boreal Annual Burned Area for 2022 (tile HNL)\n- Single-band raster: 55,877×41,013 pixels, 8-bit unsigned integer with palette\n- Resolution: 463m pixel size in Sinusoidal projection\n- Coverage: Circumpolar boreal forest and tundra regions above 50°N\n- Planar configuration: Single image plane (PLANARCONFIG_CONTIG)\n- Cloud-optimized GeoTIFF with multiple overview levels\n- Used for testing large GeoTIFF files and tiled organization\n- Citation: Loboda, T. V., Hall, J. V., Chen, D., Hoffman-Hall, A., Shevade, V. S., Argueta, F., \u0026 Liang, X. (2024). Arctic Boreal Annual Burned Area, Circumpolar Boreal Forest and Tundra, V2, 2002-2022 (Version 2). ORNL Distributed Active Archive Center. https://doi.org/10.3334/ORNLDAAC/2328 Date Accessed: 2025-12-30\n\n**Note:** Current test files are single-band GeoTIFFs. For testing multi-band raster reading (Phase 3.3), additional test files with multiple bands (e.g., Landsat, Sentinel-2) are recommended. See the GeoTIFF section above for data sources.\n\n### CMake Build and Installation\n\n```bash\n# Configure\ncmake -B build -DCMAKE_INSTALL_PREFIX=/usr/local\n\n# Build\ncmake --build build\n\n# Build documentation (optional)\ncmake --build build --target docs\n\n# Install\ncmake --install build\n\n# Uninstall (if needed)\ncmake --build build --target uninstall\n```\n\n**Note:** If HDF5 is installed in a non-standard location, you may need to specify `HDF5_ROOT`:\n\n```bash\ncmake -B build -DHDF5_ROOT=/path/to/hdf5 -DCMAKE_INSTALL_PREFIX=/usr/local\n```\n\nFor example, if HDF5 is installed in `/usr/local/hdf5-1.14.6`:\n\n```bash\ncmake -B build -DHDF5_ROOT=/usr/local/hdf5-1.14.6 -DCMAKE_INSTALL_PREFIX=/usr/local\n```\n\n### Autotools Build and Installation\n\n```bash\n# Bootstrap and configure\n./autogen.sh\n./configure --prefix=/usr/local --enable-lz4 --enable-bzip2\n\n# Build\nmake\n\n# Build documentation (optional)\nmake docs\n\n# Install\nmake install\n\n# Uninstall (if needed)\nmake uninstall\n```\n\n### Configuration Options\n\n| CMake Option | Autotools Option | Default | Description |\n|--------------|------------------|---------|-------------|\n| `-DBUILD_DOCUMENTATION=ON/OFF` | `--enable-docs/--disable-docs` | ON/enabled | Build API documentation |\n| `-DBUILD_EXAMPLES=ON/OFF` | `--enable-examples/--disable-examples` | ON/enabled | Build example programs (v3.5.1+) |\n| `-DENABLE_FORTRAN=ON/OFF` | `--enable-fortran/--disable-fortran` | ON/enabled | Fortran wrappers and tests |\n| `-DENABLE_CDF=ON/OFF` | `--enable-cdf/--disable-cdf` | OFF/disabled | CDF UDF handler build (v1.3.0+) |\n| `-DENABLE_GEOTIFF=ON/OFF` | `--enable-geotiff/--disable-geotiff` | OFF/disabled | GeoTIFF UDF handler build (v1.5.0+) |\n| N/A | `--enable-lz4/--disable-lz4` | enabled | LZ4 compression support |\n| N/A | `--enable-bzip2/--disable-bzip2` | enabled | BZIP2 compression support |\n\n**Note on CDF Support (v1.3.0):** The `--enable-cdf` option enables building the CDF UDF handler library (`libnccdf`) with full read support for CDF files. To use this option, you must have the NASA CDF library installed. Download from: https://spdf.gsfc.nasa.gov/pub/software/cdf/dist/latest/\n\nThe CDF UDF handler library is installed to `${prefix}/lib/libnccdf.so` (CMake) or `${prefix}/lib/libnccdf.la` (Autotools) when CDF support is enabled.\n\n**Spack Users:** Install CDF separately with `spack install cdf` (v1.4.0+). The CDF variant will be added back to the NEP Spack package once the CDF package is accepted into the main Spack repository.\n\n### Using NEP in Your Project\n\nLZ4 and BZIP2 compression are provided as HDF5 filter plugins. Simply set the `HDF5_PLUGIN_PATH` environment variable to the NEP installation directory, and use standard NetCDF-4 compression APIs.\n\n```bash\nexport HDF5_PLUGIN_PATH=/usr/local/lib/plugin\n```\n\n---\n\n## Documentation\n\nFor more detailed information about the project:\n\n- **[PR/FAQ](docs/prfaq.md)** - Press release and frequently asked questions\n- **[Roadmap](docs/roadmap.md)** - Development roadmap and release schedule\n- **[Product Requirements](docs/prd_1.md)** - Detailed product requirements and specifications (v1.0.0)\n- **[Design Document](docs/design_1.md)** - Technical architecture and design details (v1.0.0)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintelligent-data-design-inc%2Fnep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintelligent-data-design-inc%2Fnep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintelligent-data-design-inc%2Fnep/lists"}