{"id":19772189,"url":"https://github.com/recordevolution/tdmtermite","last_synced_at":"2025-04-30T17:33:22.108Z","repository":{"id":52194689,"uuid":"183450365","full_name":"RecordEvolution/TDMtermite","owner":"RecordEvolution","description":"Extract and read data from National Instruments LabVIEW tdx/tdm files and export them as csv files","archived":false,"fork":false,"pushed_at":"2024-09-17T21:31:07.000Z","size":745,"stargazers_count":23,"open_issues_count":3,"forks_count":2,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-09-18T02:11:42.561Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RecordEvolution.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-25T14:24:53.000Z","updated_at":"2024-09-17T21:31:11.000Z","dependencies_parsed_at":"2024-09-18T01:29:22.230Z","dependency_job_id":null,"html_url":"https://github.com/RecordEvolution/TDMtermite","commit_stats":{"total_commits":179,"total_committers":6,"mean_commits":"29.833333333333332","dds":0.4134078212290503,"last_synced_commit":"759c88dfb411e19527d664ce5e2b679849b8215e"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FTDMtermite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FTDMtermite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FTDMtermite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FTDMtermite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RecordEvolution","download_url":"https://codeload.github.com/RecordEvolution/TDMtermite/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224219688,"owners_count":17275477,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T05:05:42.451Z","updated_at":"2025-04-30T17:33:22.092Z","avatar_url":"https://github.com/RecordEvolution.png","language":"C++","readme":"\n[![LICENSE](https://img.shields.io/github/license/RecordEvolution/TDMtermite)](https://img.shields.io/github/license/RecordEvolution/TDMtermite)\n[![STARS](https://img.shields.io/github/stars/RecordEvolution/TDMtermite)](https://img.shields.io/github/stars/RecordEvolution/TDMtermite)\n![CI Build Wheel](https://github.com/RecordEvolution/TDMtermite/actions/workflows/pypi-deploy.yml/badge.svg?branch=\u0026event=push)\n[![PYPI](https://img.shields.io/pypi/v/TDMtermite.svg)](https://pypi.org/project/tdmtermite/)\n\n# TDMtermite\n\n_TDMtermite_ is a C++ based library that decodes the proprietary\nfile format _TDM/TDX_ for measurement data. First introduced by\n[National Instruments](https://www.ni.com), the TDM format relies on the\n_technical data management_ data model and is employed by\n[LabVIEW](https://www.ni.com/de-de/shop/labview.html), LabWindows™/CVI™,\nMeasurement Studio, SignalExpress, and [DIAdem](https://www.ni.com/de-de/shop/data-acquisition-and-control/application-software-for-data-acquisition-and-control-category/what-is-diadem.html).\n\nThe [Record Evolution Platform](https://www.record-evolution.de/en/home-en/) uses TDMtermite to integrate measurement data into ETL processes. The TDMtermite library is available both as a command line tool and as a Python module. The Python module of TDMtermite enables data scientists to conveniently include TDM formats in their existing data pipelines by providing access to both raw data and metadata in terms of native Python objects.    \n\n## Overview\n\n* [TDM file format](#Dataformat)\n* [Build and Installation](#Installation)\n* [Usage and Examples](#Usage)\n* [References](#References)\n\n## Dataformat\n\nDatasets encoded in the TDM/TDX format come in pairs comprised of a\n.tdm (header) file and a .tdx (data) file. While the .tdm file is a human-readable\nfile providing meta information about the dataset, the .tdx file is a binary file\ncontaining the actual data. The .tdm based on the _technical data management_\nmodel is an XML file. It describes what data the .tdx file contains and how\nto read it. The\n[TDM data model](https://www.ni.com/de-de/support/documentation/supplemental/10/ni-tdm-data-model.html)\nstructures the data hierarchically with respect to _file_, _(channel)_ _groups_ and\n_channels_. The file-level XML may contain any number of (channel) groups, each\nof which is made up of an arbitrary number of channels. Thus, the XML tree in\nthe [TDM header file](https://zone.ni.com/reference/de-XX/help/370858P-0113/tdmdatamodel/tdmdatamodel/tdm_headerfile/)\nlooks like this:\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\" ?\u003e\n\u003cusi:tdm xmlns:usi=\"http://www.ni.com/Schemas/USI/1_0\" version=\"1.0\"\u003e\n\n  \u003cusi:documentation\u003e\n    \u003cusi:exporter\u003eNational Instruments USI\u003c/usi:exporter\u003e\n    \u003cusi:exporterVersion\u003e1.5\u003c/usi:exporterVersion\u003e\n  \u003c/usi:documentation\u003e\n\n  \u003cusi:model modelName=\"National Instruments USI generated meta file\" modelVersion=\"1.0\"\u003e\n    \u003cusi:include nsUri=\"http://www.ni.com/DataModels/USI/TDM/1_0\"/\u003e\n  \u003c/usi:model\u003e\n\n  \u003cusi:include\u003e\n    \u003cfile byteOrder=\"littleEndian\" url=\"example.tdx\"\u003e\n    ...\n    \u003cblock byteOffset=\"0\" id=\"inc0\" length=\"1000\" valueType=\"eFloat64Usi\"/\u003e\n    ...\n    \u003cblock_bm id=\"inc4\" blockOffset=\"100\" blockSize=\"7\" byteOffset=\"0\" length=\"4\" valueType=\"eInt8Usi\"/\u003e\n    ...\n  \u003c/usi:include\u003e\n\n  \u003cusi:data\u003e\n    ...\n  \u003c/usi:data\u003e\n\n\u003c/usi:tdm\u003e\n```\n\nThe XML tree is comprised of _four_ main XML elements: `usi:documentation`, `usi:model`,\n`usi:include` and `usi:data`. The element `usi:include` references the data file\n`example.tdx` and reveals one of _two_ possible orderings of the mass data (.tdx):\n\n1. either _channel-wise_ (`\u003cblock\u003e`) - all values of a specific channel follow subsequently\n1. or _block-wise_ (`\u003cblock_bm\u003e`) - all values of a specific measurement time follow subsequently.\n\nThe supported _numerical data types_ are:\n\n| datatype    | channel datatype | numeric | value sequence  | size  | description             |\n|-------------|------------------|---------|-----------------|-------|-------------------------|\n| eInt16Usi   | DT_SHORT         | 2       | short_sequence  | 2byte | signed 16 bit integer   |\n| eInt32Usi   | DT_LONG          | 6       | long_sequence   | 4byte | signed 32 bit integer   |\n| eUInt8Usi   | DT_BYTE          | 5       | byte_sequence   | 1byte | unsigned 8 bit integer  |\n| eUInt16Usi  | DT_SHORT         | 2       | short_sequence  | 2byte | unsigned 16 bit integer |\n| eUInt32Usi  | DT_LONG          | 6       | long_sequence   | 4byte | unsigned 32 bit integer |\n| eFloat32Usi | DT_FLOAT         | 3       | float_sequence  | 4byte | 32 bit float            |\n| eFloat64Usi | DT_DOUBLE        | 7       | double_sequence | 8byte | 64 Bit double           |\n| eStringUsi  | DT_STRING        | 1       | string_sequence |       | text                    |\n\nThe XML element `\u003cusi:data\u003e` is comprised of _five_ different types of\nelements that are `\u003ctdm_root\u003e`, `\u003ctdm_channelgroup\u003e`, `\u003ctdm_channel\u003e`, `\u003clocalcolumn\u003e`\nand `\u003csubmatrix\u003e`. The root element `\u003ctdm_root\u003e` describes the general properties\nof the dataset and lists the _ids_ of all channel groups that belong to\nthe dataset. The element `\u003ctdm_channelgroup\u003e` divides the _channels_ into groups\nand has a unique _id_ that is referenced by its root element. The `\u003cchannels\u003e`\nelement in `\u003ctdm_channelgroup\u003e` lists the unique ids of all channels that belong\nto that group. Finally, the element `\u003ctdm_channel\u003e` describes a single column of\nactual data including its datatype. The remaining element types are\n`\u003clocalcolumn\u003e`\n\n```xml\n\u003clocalcolumn id=\"usiXY\"\u003e\n  \u003cname\u003eUntitled\u003c/name\u003e\n  \u003cmeasurement_quantity\u003e#xpointer(id(\"usiAB\"))\u003c/measurement_quantity\u003e\n  \u003csubmatrix\u003e#xpointer(id(\"usiMN\"))\u003c/submatrix\u003e\n  \u003cglobal_flag\u003e15\u003c/global_flag\u003e\n  \u003cindependent\u003e0\u003c/independent\u003e\n  \u003csequence_representation\u003e ... \u003c/sequence_representation\u003e\n  \u003cvalues\u003e#xpointer(id(\"usiZ\"))\u003c/values\u003e\n\u003c/localcolumn\u003e\n```\n\nwith a unique id, the `\u003cmeasurement_quantity\u003e` referring to one specific channel,\nthe `\u003csubmatrix\u003e` and its id respectively, the type of representation in\n`\u003csequence_representation\u003e` - being one of _explicit_, _implicit linear_ or\n_rawlinear_ - and the `\u003cvalues\u003e` element, which refers to one _value sequence_,\nand the element `\u003csubmatrix\u003e`\n\n```xml\n\u003csubmatrix id=\"usiXX\"\u003e\n  \u003cname\u003eUntitled\u003c/name\u003e\n  \u003cmeasurement\u003e#xpointer(id(\"usiUV\"))\u003c/measurement\u003e\n  \u003cnumber_of_rows\u003eN\u003c/number_of_rows\u003e\n  \u003clocal_columns\u003e#xpointer(id(\"usiMN\"))\u003c/local_columns\u003e\n\u003c/submatrix\u003e\n```\n\nthat references the channel group in `\u003cmeasurement\u003e` to which it belongs and provides\nthe _number of rows_ in the channels listed in `\u003clocal_columns\u003e`.\n\n## Installation\n\nThe library can be used both as a _CLI_-based tool and as a _Python_ module.\n\n### CLI tool\n\nTo install the CLI tool _TDMtermite_, do\n\n```Shell\nmake install\n```\n\nwhich uses `/usr/local/bin` as an installation directory. On _macOSX_, please first\nbuild the binary locally with `make` and install it in your preferred location.\n\n### Python\n\nIn order to build a _Python module_ from the _C++_ code base, the\n[Cython](https://cython.readthedocs.io/en/latest/index.html) package must be\navailable. It may be installed via `python3 -m pip install cython` .\nThe [Numpy](https://numpy.org) package is recommended\nto pass arrays of data from the C++ kernel to Python. The _makefile_ provides\nthe target `make cython-requirements` to install all required Python modules.\nFinally, to build the Python extension _tdm_termite_ locally or install\nit, the targets `make cython-build` and `make cython-install` are provided.\nTo install the Python module on the system, simply do\n\n```Shell\nmake cython-requirements\nmake cython-install\n```\n\nwhich makes the module available for import by `import tdm_termite` .\n\n#### Installation with pip\n\nThe package is also available via the [Python Package Index](https://pypi.org) at\n[TDMtermite](https://pypi.org/project/tdmtermite/). To install the latest version simply do\n\n```Shell\npython3 -m pip install tdmtermite\n```\n\n##### Unix\n\nNote, that _python3_setuptools_ and _gcc version \u003e= 10.2.0_ are required to\nsuccessfully install and use it.\n\n## Usage\n\n### CLI tool\n\nThe usage of the CLI tool is sufficiently clarified by its help message displayed\nby `tdmtermite --help`. To extract the data decoded in the pair of\nfiles `samples/SineData.tdm` and `samples/SineData.tdx` into the directory\n`/home/jack/data/`:\n\n```Shell\ntdmtermite samples/SineData.tdm samples/SineData.tdx --output /home/jack/data\n```\n\nThe tool can also be used to list the available objects in the TDM dataset, which\nare i.a. _channels_, _channelgroups_ and TDX _blocks_. To list\nall channels and channelgroups (without writing any file output):\n\n```Shell\ntdmtermite samples/SineData.tdm samples/SineData.tdx --listgroups --listchannels\n```\n\nThe user may also submit a _filenaming rule_ to control the names of the files the\nchannel(group)s are written to. To this end, the _magic flags_ `%G` `%g`, `%C`\nand `%c` representing the group id, group name, channel index and channel name\nare defined. The default filenaming option is:\n\n```Shell\ntdmtermite samples/SineData.tdm samples/SineData.tdx --output /home/jack/data --filenames channelgroup_%G.csv\n```\n\nThis makes the tool write _all channels_ grouped into files according to their\ngroup association, while all channelgroup filenames obey the pattern `channelgroup_%G.csv`,\nwith `%G` being replaced by the group id. The filenaming rule also enables the user\nto extract only a single channel(group) by providing a particular channel(group)\nid in the filenaming flag. For example,\n\n```Shell\ntdmtermite samples/SineData.tdm samples/SineData.tdx --output /home/jack/data -f channel_usi16_%c.csv --includemeta\n```\n\nThis will write the single channel with the id `usi16` to the file\n`/home/jack/data/channel_usi16_A4.csv`, including its meta-data as a file header.\n\n### Python\n\nTo be able to use the Python module _tdm_termite_, it first has to be built locally\nor installed on the system. In the Python interpreter, simply do:\n\n```Python\nimport tdmtermite\n```\n\nThis will import the module. The TDM files are provided by creating an instance of\nthe _tdmtermite_ class:\n\n```Python\n# create 'tdmtermite' instance object\ntry :\n    jack = tdmtermite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')\nexcept RuntimeError as e:\n    print(\"failed to load/decode TDM files: \" + str(e))\n```\n\nAfter initializing the _tdmtermite_ object, it can be used to extract any of the\navailable data. For instance, to list the included channelgroups and channels:\n\n```Python\n# list ids of channelgroups\ngrpids = jack.get_channelgroup_ids()\n\n\n# list ids of channels\nchnids = jack.get_channel_ids()\n```\n\nAs a use case, we have a look at listing the ids of all channelgroups and printing\ntheir data to separate files:\n\n```Python\nimport tdmtermite\nimport re\n\n# create 'tdmtermite' instance object\ntry :\n    jack = tdmtermite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')\nexcept RuntimeError as e :\n    print(\"failed to load/decode TDM files: \" + str(e))\n\n# list ids of channelgroups\ngrpids = jack.get_channelgroup_ids()\ngrpids = [x.decode() for x in grpids]\nprint(\"list of channelgroups: \",grpids)\n\nfor grp in grpids :\n\n    # obtain meta data of channelgroups\n    grpinfo = jack.get_channelgroup_info(grp.encode())\n    print( json.dumps(grpinfo,sort_keys=False,indent=4) )\n\n    # write this channelgroup to file\n    try :\n        grpname = re.sub('[^A-Za-z0-9]','',grpinfo['name'])\n        grpfile = \"channelgroup_\" + str(grp) + \"_\" + str(grpname) + \".csv\"\n        jack.print_channelgroup(grp.encode(),      # id of group to be printed\n                                grpfile.encode(),  # filename\n                                True,              # include metadata as fileheader\n                                ord(' ')           # delimiter char\n                                )\n    except RuntimeError as e :\n        print(\"failed to print channelgroup: \" + str(grp) + \" : \" + str(e))\n```\n\nFor details, see this [extensive example](python/usage.py)\nand the absolute minimal example [minimal usage](python/minimal.py). In order\nto simply extract all data of the TDM datatset and dump it to files in a given\n(existing!) directory, do\n\n```Python\nimport tdmtermite\njack = tdmtermite.tdmtermite(b'samples/SineData.tdm',b'samples/SineData.tdx')\njack.write_all(b\"./my_tdm_data_directory/\")\n```\n\nThe interface allows you to construct customized file/column headers from any\nmeta-data and provide these headers for usage in file output (see this\n[example](python/custom.py)).\n\n## References\n\n### TDM\n\n- https://www.ni.com/de-de/support/documentation/supplemental/10/ni-tdm-data-model.html\n- https://zone.ni.com/reference/en-XX/help/371361R-01/lvconcepts/fileio_tdms_model/\n- https://zone.ni.com/reference/en-XX/help/371361R-01/lvhowto/ni_test_data_exchange/\n- https://www.ni.com/de-de/support/documentation/supplemental/06/the-ni-tdms-file-format.html\n- https://zone.ni.com/reference/de-XX/help/370858P-0113/tdmdatamodel/tdmdatamodel/tdm_headerfile/\n- https://www.ni.com/content/dam/web/product-documentation/c_dll_tdm.zip\n\n### IEEE Standard and datatypes\n\n- https://en.wikipedia.org/wiki/IEEE_754\n- https://www.ias.ac.in/public/Volumes/reso/021/01/0011-0030.pdf\n- https://en.cppreference.com/w/cpp/language/types\n\n### Implementation\n\n- https://en.cppreference.com/w/\n- https://pugixml.org/\n- https://github.com/zeux/pugixml\n- https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html\n\n### Packaging\n\n#### Documentation\n\n- https://packaging.python.org/tutorials/packaging-projects/\n- https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html\n- https://test.pypi.org/account/register/\n- https://github.com/pypa/auditwheel\n- https://github.com/pypa/python-manylinux-demo\n- https://github.com/pypa/manylinux\n\n#### C/C++ Extensions\n\n- https://docs.python.org/3/extending/building.html\n\n#### Articles\n\n- https://martinsosic.com/development/2016/02/08/wrapping-c-library-as-python-module.html\n- https://malramsay.com/post/perils-of-packaging/\n- https://github.com/neuronsimulator/nrn/issues/329\n- https://levelup.gitconnected.com/how-to-deploy-a-cython-package-to-pypi-8217a6581f09\n- https://medium.com/swlh/distributing-python-packages-protected-with-cython-40fc29d84caf\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecordevolution%2Ftdmtermite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frecordevolution%2Ftdmtermite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecordevolution%2Ftdmtermite/lists"}