Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/miikama/ncdl-gen

A Parser for NetCDF cdl language
https://github.com/miikama/ncdl-gen

cpp netcdf parser vscode

Last synced: 5 months ago
JSON representation

A Parser for NetCDF cdl language

Awesome Lists containing this project

README 

          
# ncdlgen



`ncdlgen` in a nutshell: A library that enables creating interfaces for reading and writing structured typed data in multiple formats.



ncdlgen is a C++ library that provides a [NetCDF](https://github.com/Unidata/netcdf-c) cdl-_parser_ and a _code generator_ for C++ code that reads/writes NetCDF files to reads/writes data with [ZeroMQ](https://zeromq.org/).



NetCDF is a platform independent data file format for structured data. It is commonly used in Earth observation missions as data storage format. NetCDF cdl-files describe structured data and are used together with the NetCDF library tools. The Netcdf CDL grammar is available [here](https://manpages.ubuntu.com/manpages/focal/man1/ncgen.1.html).



ZeroMQ provides sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast.



In the future `ncdlgen` could be used as a code generation tool for other structured data formats as well.



## Example



See concrete example for using `ncdlgen` as a library to enable code generation and writing to different pipes under `examples/`. Main current features include generating interface code based on `.cdl` file and reading the same data in multiple formats.



Interface is described via .cdl file:



```

netcdf Data {

  variables:

    int foo ;

    float bar ;

}

```



And passing this data



```c++

// Data in generated during compilation with the interface code generator

generated::Data data{{1}, {2.0}};



// Create ZeroMQPipe at local socket and send data

ncdlgen::ZeroMQPipe pipe();

generated::write(pipe, data);



// Receive the data (other process, node etc.)

generated::read(pipe, data);



// Write received data to file

ncdlgen::NetCDFPipe ncpipe("data.nc");

generated::read(ncpipe, data);

```



## Installation



Clone the repository



```sh

cd ncdl-gen

git clone git@github.com:miikama/ncdl-gen.git

```



### Dependencies



Get all dependencies with conan. Alternatively, download dependencies and configure `cmake` by hand



```sh

conan install .

```



### Install with conan



Install the library for downstream consumers (with conan)



```sh

conan build .

conan export .

```



### Install with cmake



After installing libraries with conan `ncdlgen` can be install with



```sh

cd build

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=~/ncdlgen -DCMAKE_PREFIX_PATH=$(pwd)/Release/generators ..

make -j6 && make install

```



After running `conan install .` all the required `packageConfig.cmake` files for `ncdlgen`are now under `build/Release/generators` for finding the dependencies. It is also possible to manually download all the primary and transitive dependencies without Conan. Currently primary dependencies are



- fmt

- netCDF

- GTest

- cppzmq



> NOTE: add permanent setting with `conan profile update settings.compiler.libcxx=libstdc++11 default`



To build and run the tests, enable them separately by setting `cmake -DBUILD_TESTING=ON .. && make && make test`.



## Parser



Take an example cdl-file



```

netcdf simple {



    group: foo {



        dimensions:

            dim = 5 ;



        variables:

            int bar ;

            float baz ;

            ushort bee(dim) ;

            int foobar(dim, dim) ;

    }

}

```



Result of parsing the file contents:



```sh

${installation_directory}/parser data/simple.cdl

Group simple

  Group foo

      Dimensions

          dim = 5

      Variables

          int bar

          float baz

          ushort bee (dim)

          int foobar (dim, dim)

```



## Code generator



Take the same example `data/simple.cdl` file but use it as an input for the code-generator:



```sh

${installation_directory}/generator data/simple.cdl --header --target_pipes NetCDFPipe --interface_class_name generated_simple

```



results in the following generated code



```c++

#pragma once



#include "stdint.h"



#include "netcdf_pipe.h"



#include 



#include "vector_interface.h"



namespace ncdlgen {



struct simple

{

  struct foo

  {

      int bar;

      float baz;

      std::vector bee;

      std::vector> foobar;

  };



  foo foo_g{};

};



void read(NetCDFPipe& pipe, simple&);



void read(NetCDFPipe& pipe, simple::foo&);



void write(NetCDFPipe& pipe, const simple&);



void write(NetCDFPipe& pipe, const simple::foo&);



};

```



The corresponding source file can be generated with



```sh

${installation_directory}/generator data/simple.cdl --source --target_pipes NetCDFPipe --interface_class_name generated_simple

```



### Generation as part of CMake build



This can be integrated as part of a CMake build (as done for the test/CMakelFiles.txt)



```cmake

# Run generator to create test wrappers

add_custom_command(

                   OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h

                   OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp

                   COMMAND generator ${CMAKE_SOURCE_DIR}/data/simple.cdl --header > ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h

                   COMMAND generator ${CMAKE_SOURCE_DIR}/data/simple.cdl --source > ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp

                   DEPENDS generator

                   DEPENDS ${CMAKE_SOURCE_DIR}/data/simple.cdl

                   VERBATIM

                   )

# Add dependency to generated code

add_custom_target(generated-test-code

                  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h

                  DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp  )

set_source_files_properties(${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.h

                            ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp

                            PROPERTIES GENERATED TRUE)

set(GENERATED_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/generated_simple.cpp)

```



### Generator configurability and extra pipes



If you want, the available pipe read/write entries can either be disabled with



```shell

./generator data/simple.cdl --header --target_pipes  {} --interface_class_name generated_simple --interface_namespace_name generated

```



Or you can generate for all the supported pipes



```shell

./generator data/simple.cdl --header --target_pipes NetCDFPipe ZeroMQPipe --interface_class_name generated_simple --interface_namespace_name generated

```



Which generates the following additional interfaces on



```c++

#pragma once



#include "stdint.h"



#include "pipes/netcdf_pipe.h"

#include "pipes/zeromq_pipe.h"



#include 



#include "vector_interface.h"



namespace generated {



struct simple

{

  struct foo

  {

      int bar;

      float baz;

      std::vector bee;

      std::vector> foobar;

  };



  foo foo_g{};

};



void read(NetCDFPipe& pipe, simple&);



void read(ZeroMQPipe& pipe, simple&);



void read(NetCDFPipe& pipe, simple::foo&);



void read(ZeroMQPipe& pipe, simple::foo&);



void write(NetCDFPipe& pipe, const simple&);



void write(ZeroMQPipe& pipe, const simple&);



void write(NetCDFPipe& pipe, const simple::foo&);



void write(ZeroMQPipe& pipe, const simple::foo&);



};

```



### Using generated code



The generated code for reading and writing to pipes can be used to read/write the contents of the entire file or its subgroups



```c++

#include "generated_simple.h"



ncdlgen::simple root;

ncdlgen::NetCDFPipe pipe{"generated.nc"};

pipe.open();



// Write contents of 'simple' struct to a netcdf file

ncdlgen::write(pipe, root);



// Read the contents of a netcdf file into 'simple' struct

read(pipe, root);



pipe.close();



// Configure ZeroMQPipe

ncdlgen::ZeroMQPipe zeromq_pipe {};



// Push the contents through ZeroMQPipe

ncdlgen::write(zeromq_pipe, root);



// Read the contents through ZeroMQPipe

ncdlgen::read(zeromq_pipe, root);

```



## ncdlgen as dependency



See example for downstream usage under the [example](examples) directory.



Clone the `ncdlgen` repository. In repository root, run



```sh

conan install .

conan build .

conan export .

```



If using conan yourself, you can make a small conanfile with the wanted dependencies



```

[requires]

 ncdlgen/0.3.0

 netcdf/4.8.1

 cppzmq/4.10.0

```



And run `conan install . --build=missing`. Which compiles and installs the `ncdlgen` library.



Then, configure your project with `cmake` with



```sh

cd build

cmake -DCMAKE_PREFIX_PATH=$(pwd)/Release/generators -DCMAKE_BUILD_TYPE=RELEASE ..

```



Then compile.



> The example uses the `generator` binary compiled with the ncdlgen build. This has to be in `PATH`



```sh

export PATH="$PATH:"

make

```



After installing ncdlgen you can use the library in your projects `CMakeLists.txt`



```cmake

cmake_minimum_required(VERSION 3.16)

project("ncdlgen-examples")



find_package(ncdlgen REQUIRED)



add_executable(custom_parser custom_parser.cpp)

target_link_libraries(custom_parser PRIVATE ncdlgen::ncdlgen)

`

```



The example `CMakeLists.txt` runs the interface generator to build `example_data.h` interface during compilation. This file is included in the example `custom_parser.cpp` file.



## Build using Docker



There is a `Dockerfile` that setups a build environment for the current user. Build the docker file with



```sh

docker build \

    --build-arg USER=$(whoami) \

    --build-arg USER_ID=$(id -u $(whoami)) \

    --build-arg GROUP_ID=$(id -g $(whoami)) \

    --tag=ncdgen \

    .

```



After building, start the container with



```sh

docker run --rm -it -v $(pwd):/home/$(whoami) ncdgen bash

```



This mounts the repository at the home directory of the container user.



## Changelist



Main features for each release



0.1.0



- Initial relase with NetCDF cdl parser

- Code generator for generating code for interface reading/writing



0.2.0



- Support Conan 2

- Support multidimensional containers in interfaces

- Support multidimensional containers in code generation

- Update gtest version

- Improve code generation configurability

- Support global attributes outside of variables: section

- Resolve untyped attribute types by finding corresponding variable

- Make NetCDF and optional dependency



0.3.0



- Rename NetCDFInterface as NetCDFPipe

- Introduce ZeroMQPipe

- Add cppzmq/4.10.0 optional dependency

- Add cli11/2.4.2 dependency

- Improve code generation configurability

- Make project exportable with conan



## Building VSCode extension



Official guides



[Getting started](https://code.visualstudio.com/api/get-started/your-first-extension) with extensions



[VSCode language extensions](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide)



The language extensions add a `language` [Contribution Point](https://code.visualstudio.com/api/references/contribution-points)



Text mate grammar guides



[Writing grammar](https://macromates.com/manual/en/language_grammars)



[Notes about textmate language grammar](https://www.apeth.com/nonblog/stories/textmatebundle.html)



Some inspiration is derived from the first example I found, which is the Jakt language in SerenityOS



[Jakt language syntax](https://github.com/SerenityOS/jakt/blob/main/editors/vscode/syntaxes/jakt.tmLanguage.json)