{"id":19772190,"url":"https://github.com/recordevolution/imctermite","last_synced_at":"2025-04-06T20:13:06.105Z","repository":{"id":54433629,"uuid":"242105335","full_name":"RecordEvolution/IMCtermite","owner":"RecordEvolution","description":"Enables extraction of measurement data from binary files with extension 'raw' used by proprietary software imcFAMOS/imcSTUDIO and facilitates its storage in open source file formats","archived":false,"fork":false,"pushed_at":"2025-03-09T21:02:00.000Z","size":7397,"stargazers_count":31,"open_issues_count":7,"forks_count":10,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-30T19:11:12.635Z","etag":null,"topics":["binary","csv","fileformat","imccronos","imcfamos","imcstudio","measurement","parquet","raw","time-series"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RecordEvolution.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-21T09:45:10.000Z","updated_at":"2025-03-18T15:44:55.000Z","dependencies_parsed_at":"2024-12-24T18:10:41.699Z","dependency_job_id":"260dc3f0-839d-4aa9-9874-85b87ca9735f","html_url":"https://github.com/RecordEvolution/IMCtermite","commit_stats":{"total_commits":202,"total_committers":6,"mean_commits":"33.666666666666664","dds":"0.34158415841584155","last_synced_commit":"ff69c329ccc45c2f7ad5944f24f76573460d98cf"},"previous_names":[],"tags_count":54,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FIMCtermite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FIMCtermite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FIMCtermite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RecordEvolution%2FIMCtermite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RecordEvolution","download_url":"https://codeload.github.com/RecordEvolution/IMCtermite/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247543593,"owners_count":20955865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","csv","fileformat","imccronos","imcfamos","imcstudio","measurement","parquet","raw","time-series"],"created_at":"2024-11-12T05:05:42.461Z","updated_at":"2025-04-06T20:13:06.067Z","avatar_url":"https://github.com/RecordEvolution.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[![LICENSE](https://img.shields.io/github/license/RecordEvolution/IMCtermite)](https://img.shields.io/github/license/RecordEvolution/IMCtermite)\n[![STARS](https://img.shields.io/github/stars/RecordEvolution/IMCtermite)](https://img.shields.io/github/stars/RecordEvolution/IMCtermite)\n![CI Build Wheel](https://github.com/RecordEvolution/IMCtermite/actions/workflows/pypi-deploy.yml/badge.svg?branch=\u0026event=push)\n[![PYPI](https://img.shields.io/pypi/v/IMCtermite.svg)](https://pypi.org/project/imctermite/)\n\n# IMCtermite\n\n_IMCtermite_ provides access to the proprietary data format\n_IMC2 Data Format_ with the file extension _.raw_ (or .dat) introduced and developed by\n[imc Test \u0026 Measurement GmbH](https://www.imc-tm.de/). This data format is\nemployed i.a. by the measurement hardware\n[imc CRONOSflex](https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/)\nto dump and store data and the software packages\n[imc Studio](https://www.imc-tm.de/produkte/messtechnik-software/imc-studio/ueberblick/)\n\u0026 [imc FAMOS](https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/)\nfor measurement data control and analysis. Thanks to the integrated Python module,\nthe extracted measurement data can be stored in any open-source file format\naccessible by Python like i.a. _csv_, _json_ or _parquet_.\n\nOn the [Record Evolution Platform](https://www.record-evolution.de/en/home-en/),\nthe library can be used both as a command line tool for interactive usage and as a\nPython module to integrate the _.raw_  format into any ETL workflow.\n\n## Overview\n\n* [File format](#Fileformat)\n* [Build and Installation](#Installation)\n* [Usage and Examples](#Usage)\n* [References](#References)\n\n## File format\n\nA file of the _IMC2 Data Format_ type with extension _.raw_ (or .dat) is a _mixed text/binary\nfile_ featuring a set of markers (keys) that indicate the start of various blocks\nof data that provide meta information and the actual measurement data. Every single\nmarker is introduced by the character `\"|\" = 0x 7c` followed by two uppercase letters that\ncharacterize the type of marker. Each block is further divided into several\nparameters separated by commata `\",\" = 0x 2c` and terminated by a semicolon\n`\";\" = 0x 3b`. For instance, the header - first 600 bytes - of a raw file may\nlook like this (in UTF-8 encoding):\n\n```\n|CF,2,1,1;|CK,1,3,1,1;\n|NO,1,86,0,78,imc STUDIO 5.0 R10 (04.08.2017)@imc DEVICES 2.9R7 (25.7.2017)@imcDev__15190567,0,;\n|CG,1,5,1,1,1; |CD,2,  63,  5.0000000000000001E-03,1,1,s,0,0,0,  0.0000000000000000E+00,1;\n|NT,1,16,1,1,1980,0,0,0.0;       |CC,1,3,1,1;|CP,1,16,1,4,7,32,0,0,1,0;\n|CR,1,60,0,  1.0000000000000000E+00,  0.0000000000000000E+00,1,4,mbar;|CN,1,27,0,0,0,15,pressure_Vacuum,0,;\n|Cb,1, 117,1,0,    1,         1,         0,      9608,         0,      9608,1,  2.0440300000000000E+03,  1.2416717060000000E+09,;\n|CS,1,      9619,         1,�oD\t�nD6�nD)�nD�\n```\n\nLine breaks are introduced for readability. Most of the markers introduce\nblocks of text, while only the last block identified by `|CS` contains binary data.\nThe format supports the storage of _multiple data sets (channels)_ in a single\nfile. The channels may be ordered in _multiplex_ mode (ordering w.r.t. time) or\n_block_ mode (ordering w.r.t. to channels).\n\nThe markers (keys) are introduced by `\"|\" = 0x 7c` followed by two uppercase\nletters. There are _two types_ of markers distinguished by the first letter:\n\n1. _critical_ markers: introduced by `|C` featuring uppercase `C`\n1. _noncritical_ markers: introduced by `|N` featuring uppercase `N`  \n\nThe second letter represents further details of the specific key. Note that\nwhile the _noncritical_ keys are optional, any _.raw_ file _cannot be_ correctly\ndecoded if any of the _critical_ markers are misinterpreted, invalid or damaged.\nThe second uppercase letter is followed by the first comma and the _version_\nof the key starting from 1. After the next comma, an _(long) integer_ (in text\nrepresentation) specifies the length of the entire block, i.e. the number of\nbytes between the following comma and the block-terminating semicolon. The further\nstructure of a block is not defined and may feature different numbers of additional\nparameters. The format allows for any number of carriage returns (`CR = 0x0d`)\nand line feeds (`LF = 0x 0a`) between keys, i.e. the block-terminating semicolon\nand the vertical bar (pipe) of the next key. The following _critical markers_\nare defined:\n\n\n| marker | description                                                                                         |\n|--------|-----------------------------------------------------------------------------------------------------|\n| CF     | format version and processor                                                                        |\n| CK     | start of group of keys, no. parameters = 3, indicates (in)correct closure of the measurement series |\n| CB     | defines a group of channels                                                                         |\n| CT     | text definition including group association index                                                   |\n| CG     | introduces group of components corresponding to CC keys                                             |\n| CD1,2  | old/new version of abscissa description                                                             |\n| CZ     | scaling of z-axis for segments                                                                      |\n| CC     | start of a component                                                                                |\n| CP     | information about buffer, datatype and samples of component                                         |\n| Cb     | buffer description                                                                                  |\n| CR     | permissible range of values in component                                                            |\n| CN     | name and comment of channel                                                                         |\n| CS     | raw binary data                                                                                     |\n| CI     | single numerical value (including unit)                                                             |\n| Ca     | add reference key                                                                                   |\n\nAmong the _noncritical_ markers, there are\n\n| marker | description                                |\n|--------|--------------------------------------------|\n| NO     | origin of data                             |\n| NT     | timestamp of trigger                       |\n| ND     | (color) display properties                 |\n| NU     | user defined key                           |\n| Np     | property of a channel                      |\n| NE     | extraction rule for channels from BUS data |\n\nThe format loosely defines some rules for the ordering of the markers in the\nfile stream. The rules for critical keys include: _CK_ has to follow up on _CF_,\n_CK_ may be followed by any number of _CG_ blocks, each _CG_ has to be followed\nby (any number of) component sequences comprised of the series _CC_ , _CP_,\n(_CR_), (_ND_) and terminated by either _CS_ or the start of a new group,\ncomponent, text field or buffer.\n\n## Installation\n\nThe _IMCtermite_ library may be employed both as a _CLI_ tool and a _python_\nmodule.\n\n### CLI tool\n\nTo build the CLI tool locally, use the default target `make` resulting\nin the binary `imctermite`. To ensure system-wide availability, the installation\nof the tool (in the default location `/usr/local/bin`) is done via\n\n```\nmake install\n````\n\nwhich may require root permissions.\n\n### Python\n\nTo integrate the library into a customized ETL toolchain, several python targets\nare available. For a local build that enables you to run the examples, use:\n\n```\nmake python-build\n```\n\n#### Installation with pip\n\nThe package is also available in the [Python Package Index](https://pypi.org)\nat [imctermite](https://pypi.org/project/imctermite/).\nTo install the latest version simply do\n\n```Shell\npython3 -m pip install imctermite\n```\n\nwhich provides binary wheels for multiple architectures on _Windows_ and _Linux_\nand most _Python 3.x_ distributions. However, if your platform/architecture is\nnot supported you can still compile the source distribution yourself, which\nrequires _python3_setuptools_ and an up-to-date compiler supporting C++11\nstandard (e.g. _gcc version \u003e= 10.2.0_).\n\n## Usage\n\n### CLI\n\nThe usage of the `imctermite` binary looks like this:\n\n```\nimctermite \u003craw-file\u003e [options]\n```\n\nYou have to provide a single _raw_ file and any option to specify what\nto do with the data. All available options can be listed with `imctermite --help`:\n\n```\nOptions:\n\n -c, --listchannels      list channels\n -b, --listblocks        list IMC key-blocks\n -d, --output            output directory to print channels\n -s, --delimiter         csv delimiter/separator char for output\n -h, --help              show this help message\n -v, --version           display version\n```\n\nFor instance, to show a list of all channels included in `sample-data.raw`, you\ndo `imctermite sample-data.raw --listchannels`. No output files are\nwritten by default. Output files are written only when an existing (!) directory\nis provided as argument to the `--output` option. By default, every output file\nis written using a `,` delimiter. You may provide any custom separator with the\noption `--delimiter`. For example, in order to use `|`, the binary is called with\noptions `imctermite sample-data.raw -b -c -s '|'`.\n\n### Python\n\nGiven the `IMCtermite` module is available, we can import it and declare an instance\nof it by passing a _raw_ file to the constructor:\n\n```Python\nimport imctermite\n\nimcraw = imctermite.imctermite(b\"sample/sampleA.raw\")\n```\n\nAn example of how to create an instance and obtain the list of channels is:\n\n```Python\nimport IMCtermite\n\n# declare and initialize instance of \"imctermite\" by passing a raw-file\ntry :\n    imcraw = IMCtermite.imctermite(b\"samples/sampleA.raw\")\nexcept RuntimeError as e :\n    print(\"failed to load/parse raw-file: \" + str(e))\n\n# obtain list of channels as list of dictionaries (without data)\nchannels = imcraw.get_channels(False)\nprint(channels)\n```\n\nA more complete [example](python/examples/usage.py), including the methods for\nobtaining the channels, i.a. their data and/or directly printing them to files,\ncan be found in the `python/examples` folder.\n\n## References\n\n### IMC\n\n- https://www.imc-tm.de/produkte/messtechnik-software/imc-famos/funktionen/im-und-export/\n- https://www.imc-tm.de/produkte/messtechnik-hardware/imc-cronosflex/ueberblick/\n- https://www.imc-tm.de/download-center/produkt-downloads/imc-famos/handbuecher\n- https://www.imc-tm.de/fileadmin/Public/Downloads/Manuals/imc_FAMOS/imcGemeinsameKomponenten.pdf\n- https://github.com/Apollo3zehn/ImcFamosFile\n- https://apollo3zehn.github.io/ImcFamosFile/api/ImcFamosFile.FamosFileKeyType.html\n\n### Cython\n\n- https://cython.readthedocs.io/en/latest/src/userguide/wrapping_CPlusPlus.html\n\n### PyPI\n\n- https://pypi.org/help/#apitoken\n- https://sgoel.dev/posts/uploading-binary-wheels-to-pypi-from-github-actions/\n- https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsrun\n- https://github.com/pypa/cibuildwheel/blob/main/examples/github-deploy.yml\n- https://cibuildwheel.readthedocs.io/en/stable/deliver-to-pypi/\n- https://github.com/actions/download-artifact#download-all-artifacts\n- https://github.com/actions/download-artifact?tab=readme-ov-file#download-multiple-filtered-artifacts-to-the-same-directory\n\n### iconv\n\n- https://www.gnu.org/software/libiconv/\n- https://vcpkg.io/en/packages.html\n- https://vcpkg.io/en/getting-started\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecordevolution%2Fimctermite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frecordevolution%2Fimctermite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecordevolution%2Fimctermite/lists"}