{"id":24555821,"url":"https://github.com/crambl/fastpasta","last_synced_at":"2025-04-16T21:44:01.981Z","repository":{"id":151624398,"uuid":"620975954","full_name":"CramBL/fastPASTA","owner":"CramBL","description":"Mirror of the repository on CERN's Gitlab. CLI for viewing and verifying data integrity on the raw binary data read out from the ALICE detector and its subdetectors.","archived":false,"fork":false,"pushed_at":"2024-05-23T19:38:49.000Z","size":12027,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T05:41:36.721Z","etag":null,"topics":["cli","daq","data-verification","rust","verification"],"latest_commit_sha":null,"homepage":"https://gitlab.cern.ch/mkonig/fastpasta","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CramBL.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-29T18:38:35.000Z","updated_at":"2024-05-23T19:38:54.000Z","dependencies_parsed_at":"2023-09-23T06:14:27.937Z","dependency_job_id":"ff05b802-6084-411b-bfde-698df02c2947","html_url":"https://github.com/CramBL/fastPASTA","commit_stats":{"total_commits":1409,"total_committers":3,"mean_commits":469.6666666666667,"dds":0.3101490418736693,"last_synced_commit":"e70afbb60477c17a028c5672133d93200913c590"},"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CramBL%2FfastPASTA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CramBL%2FfastPASTA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CramBL%2FfastPASTA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CramBL%2FfastPASTA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CramBL","download_url":"https://codeload.github.com/CramBL/fastPASTA/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249280535,"owners_count":21243138,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","daq","data-verification","rust","verification"],"created_at":"2025-01-23T04:32:43.662Z","updated_at":"2025-04-16T21:44:01.965Z","avatar_url":"https://github.com/CramBL.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# fastPASTA\n[![pipeline status](https://gitlab.cern.ch/mkonig/fastpasta/badges/master/pipeline.svg)](https://gitlab.cern.ch/mkonig/fastpasta/-/commits/master)\n [![coverage report](https://gitlab.cern.ch/mkonig/fastpasta/badges/master/coverage.svg)](https://gitlab.cern.ch/mkonig/fastpasta/-/commits/master)\n [![Latest Release](https://gitlab.cern.ch/mkonig/fastpasta/-/badges/release.svg)](https://gitlab.cern.ch/mkonig/fastpasta/-/releases)\n\n![docs.rs](https://img.shields.io/docsrs/fastpasta)\n![GitHub commit activity (branch)](https://img.shields.io/github/commit-activity/m/crambl/fastpasta)\n![Crates.io](https://img.shields.io/crates/d/fastpasta)\n\n\n\n\n## fast Protocol Analysis Scanner Tool for ALICE\n\nFor an exhaustive list of the data verification done via the `check` subcommand, see [doc/checks_list.md](https://gitlab.cern.ch/mkonig/fastpasta/-/blob/master/doc/checks_list.md).\n\nReleases and associated changelogs can be found at [releases](https://gitlab.cern.ch/mkonig/fastpasta/-/releases) or [CHANGELOG.md](CHANGELOG.md).\n\nSee more example including advanced senarios in [examples document](examples.md).\n\nLooking for more details? see the [Documentation for developers](#documentation-for-developers) section.\n\n## Purpose\n\nTo verify or view curated content of the scanned raw binary data from the ALICE detector at CERN.\n\n## Demo\n![demo-gif](doc/fastpasta_demo.gif)\n\n# Table of Contents\n- [fastPASTA](#fastpasta)\n  - [fast Protocol Analysis Scanner Tool for ALICE](#fast-protocol-analysis-scanner-tool-for-alice)\n  - [Purpose](#purpose)\n  - [Demo](#demo)\n- [Table of Contents](#table-of-contents)\n- [Quickstart](#quickstart)\n  - [Prerequisite](#prerequisite)\n  - [Install via cargo (comes with Rust)](#install-via-cargo-comes-with-rust)\n  - [Add shell completions](#add-shell-completions)\n  - [See help, including examples of use](#see-help-including-examples-of-use)\n  - [Examples of use](#examples-of-use)\n    - [Read from stdin -\\\u003e filter link -\\\u003e view RDHs](#read-from-stdin---filter-link---view-rdhs)\n    - [Read from file -\\\u003e filter by link -\\\u003e validate](#read-from-file---filter-by-link---validate)\n    - [Read from file -\\\u003e view ITS readout frames with `less`](#read-from-file---view-its-readout-frames-with-less)\n  - [Command flow](#command-flow)\n- [Customize checks](#customize-checks)\n  - [Config with custom checks](#config-with-custom-checks)\n  - [Output comprehensive statistics (and input them for validation)](#output-comprehensive-statistics-and-input-them-for-validation)\n    - [Output statistics](#output-statistics)\n    - [Example](#example)\n    - [Use statistics for data validation](#use-statistics-for-data-validation)\n    - [Example](#example-1)\n- [Error messages](#error-messages)\n    - [Messages are formatted as follows:](#messages-are-formatted-as-follows)\n    - [Example of failed RDH sanity check](#example-of-failed-rdh-sanity-check)\n    - [Error codes](#error-codes)\n  - [Verbosity levels](#verbosity-levels)\n  - [Running tests](#running-tests)\n- [License](#license)\n- [Project status](#project-status)\n- [Benchmarks and comparisons](#benchmarks-and-comparisons)\n    - [Verifying all RDHs of 1.5 GB file with data from 1 GBT link](#verifying-all-rdhs-of-15-gb-file-with-data-from-1-gbt-link)\n    - [Verifying all RDHs in 3 GB file with data from 2 different GBT links](#verifying-all-rdhs-in-3-gb-file-with-data-from-2-different-gbt-links)\n    - [Verifying all RDHs and payloads in 260 MB file with data from 1 GBT link](#verifying-all-rdhs-and-payloads-in-260-mb-file-with-data-from-1-gbt-link)\n- [Need more performance?](#need-more-performance)\n    - [Background](#background)\n    - [To install the nightly toolchain (and check your installation)](#to-install-the-nightly-toolchain-and-check-your-installation)\n    - [Compile the optimized `release-nightly` experimental build profile](#compile-the-optimized-release-nightly-experimental-build-profile)\n    - [Path to binary: `/target/release-nightly/fastpasta`](#path-to-binary-targetrelease-nightlyfastpasta)\n  - [Documentation for developers](#documentation-for-developers)\n\n# Quickstart\n## Prerequisite\nThe [rust toolchain](https://www.rust-lang.org/tools/install) is required to compile the binary. Use the link to download a Windows installer. On macOS, Linux or other Unix-like OS simply run\n```shell\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n```\nand follow the on-screen instructions.\n\n## Install via cargo (comes with Rust)\n```shell\ncargo install fastpasta\n```\nUpdating fastpasta simply requires rerunning `cargo install fastpasta`\n\n## Add shell completions\n\nGenerate completion script for bash/zsh/fish/powershell/elvish with:\n```shell\nfastpasta --generate-completions \u003cSHELL\u003e \u003e path/to/your/completion/scripts/_fastpasta\n```\n## See help, including examples of use\n\n```shell\nfastpasta -h\n```\n\n## Examples of use\n\n### Read from stdin -\u003e filter link -\u003e view RDHs\n```shell\n\nlz4 -d input.raw -c | fastpasta --filter-link 3 | fastpasta view rdh\n#        ^^^^                      ^^^^                       ^^^^\n#       INPUT       ---\u003e          FILTER          ---\u003e        VIEW\n# Decompressing with `lz4`\n```\n\nPiping is often optional and avoiding it will improve performance. e.g. the following is equivalent to the previous example, but saves significant IO overhead, by using one less pipe.\n```shell\nlz4 -d input.raw -c | fastpasta --filter-link 3 view rdh\n```\n### Read from file -\u003e filter by link -\u003e validate\nEnable all generic checks: `sanity` (stateless) AND `running` (stateful)\n```shell\nfastpasta input.raw --filter-link 0 check all\n```\nEnable all `sanity` checks and include checks applicable to `ITS` only\n```shell\nfastpasta input.raw check sanity its --filter-link 0\n```\n\n### Read from file -\u003e view ITS readout frames with `less`\nGenerate ITS readout frame view\n```shell\nfastpasta input.raw view its-readout-frames | less\n```\nView only readout frames from link #3\n```shell\nfastpasta input.raw view its-readout-frames -f 3 | less\n```\n\n\n## Command flow\n```mermaid\nflowchart TD;\n  start[\"fastpasta\"] --\u003e top_sub_cmd{\"Check or view?\"};\n\n  top_sub_cmd -- \"view\" --\u003e view_type{\"Type of view\"};\n  view_type -- \"rdh\" --\u003e view_rdh{{$ fastpasta view rdh}};\n  view_type -- \"its-readout-frames\" --\u003e view_ro_frames{{$ fastpasta view its-readout-frames}};\n  view_type -- \"its-readout-frames-data\" --\u003e view_ro_frames_data{{$ fastpasta view its-readout-frames-data}};\n\n  top_sub_cmd -- \"check\" --\u003e check_type{\"Type of check\"};\n\n  check_type -- \"sanity\" --\u003e check_sanity{\"$ fastpasta check sanity\n  or\n  target system\"};\n  check_sanity -- \"its\" --\u003e check_sanity_its{{$ fastpasta check sanity its}};\n\n  check_type -- \"all\" --\u003e check_all{\"$ fastpasta check all\n  or\n  target system\"};\n  check_all -- \"its\" --\u003e check_all_its{{$ fastpasta check all its}};\n  check_all -- \"its-stave\" --\u003e check_all_its_stave{{$ fastpasta check all its-stave}};\n```\n\n# Customize checks\n\n## Config with custom checks\nTo perform very specific checks on the raw data, it is possible to supply a `TOML` file with the `--checks-toml \u003cPATH\u003e` option.\n\nTo get started use the `--generate-checks-toml` flag to generate a template that shows which custom checks are available, along with descriptions, and examples.\n\nThe generated TOML file will contain content like this:\n```toml\n# Number of Physics (PhT) Triggers expected in the data\n# Example: 0, 10\n#triggers_pht = None [ u32 ] # (Uncomment and set to enable this check)\n```\nTo enable the check for 1 Physics Trigger in the raw data, edit the file like this:\n```toml\n# Number of Physics (PhT) Triggers expected in the data\n# Example: 0, 10\ntriggers_pht = 1 # This data should contain 1 PhT trigger.\n```\nFinally run fastPASTA as usual e.g.\n```shell\nfastpasta check all its input-data.raw --checks-toml custom_checks.toml\n```\n\n## Output comprehensive statistics (and input them for validation)\n### Output statistics\nA large variety of statistics are collected during data analysis. These statistics can be written to file/stdout in JSON/TOML and could for example serve as input to a script that verifies these statistics further.\n### Example\nCheck everything applicable to ITS on stave level for the data in `bin.raw`, save stats as `stats.json`\n```shell\nfastpasta check all its-stave --output-stats stats.json --stats-format json bin.raw\n```\n### Use statistics for data validation\nThe output statistics can also serve as the input to fastPASTA along with checks on some raw data, using the option `--input-stats-file \u003cfile\u003e`. This will run a full comparison between the input stats and the stats collected during analysis, and output an error message for each mismatching value.\n### Example\nVerify that analysis of `bin.raw` finds the same exact stats as listed in `stats.json`.\n```shell\nfastpasta check all its-stave --input-stats-file stats.json bin.raw\n```\nEven if you are not 100% sure that all the stats are correct, running one analysis and then using the output stats file as a reference in CI, will let you know if the data output ever changed in terms of these statistics, which could serve as a hint that something has gone wrong (or confirm a correct change in behaviour).\n\n# Error messages\n### Messages are formatted as follows:\n\n```shell\nMEMORY_OFFSET: [ERROR_CODE] ERROR_MESSAGE\n```\n### Example of failed RDH sanity check\n```shell\n0xE450FFD: [E10] RDH sanity check failed: data_format = 255\n```\n\n### Error codes\nError codes are unique and can between 2 and 4 digits. The first digit signifies a category for the error.\nThe following is a list of error codes and their meaning, `x` is a placeholder for any number 0-9.\n* [Ex0] - Sanity check\n* [E1x] - RDH\n* [E3x] - IHW\n* [E4x] - TDH\n* [E5x] - TDT\n* [E6x] - DDW0\n* [E7x] - Data word (Even number: IB, Odd number: OB) E70 is sanity check for both IB/OB.\n* [E8x] - CDW\n* [E9xxx] - Errors from [custom checks](#customize-checks)\n\n## Verbosity levels\n- 0: Errors\n- 1: Errors and warnings **[default]**\n- 2: Errors, warnings and info\n- 3: Errors, warnings, info and debug\n- 4: Errors, warnings, info, debug and trace\n\n\n## Running tests\nRun the full test suite with:\n```shell\ncargo test\n```\n# License\nApache 2.0 or MIT at your option.\n\n# Project status\nPassively Maintained. There are no plans for new features, but the maintainer intends to respond to issues that get filed.\n\n# Benchmarks and comparisons\nIn the tables below `fastPASTA` is compared with `rawdata-parser` and `decode.py` in typical verification tasks. Hyperfine is used for benchmarking, with `cache warmup`.\n### Verifying all RDHs of 1.5 GB file with data from 1 GBT link\n| Tool           | Command                                                   |   Mean ± σ [s] | Min [s] | Max [s] |\n| :------------- | :-------------------------------------------------------- | -------------: | ------: | ------: |\n| fastPASTA      | `fastpasta input.raw check all`                           |  0.195 ± 0.002 |   0.191 |   0.198 |\n| rawdata-parser | `./rawdata-parser --skip-packet-counter-checks input.raw` |  1.638 ± 0.066 |   1.575 |   1.810 |\n| decode.py      | `python3 decode.py -i 20522 -f input.raw --skip_data`     | 94.218 ± 0.386 |  93.914 |  94.811 |\n\n### Verifying all RDHs in 3 GB file with data from 2 different GBT links\n| Tool           | Command                                                  |  Mean ± σ [s] | Min [s] | Max [s] |\n| :------------- | :------------------------------------------------------- | ------------: | ------: | ------: |\n| fastPASTA      | `fastpasta input.raw check all`                          | 0.409 ± 0.004 |   0.402 |   0.417 |\n| rawdata-parser | `rawdata-parser input.raw`                               | 3.068 ± 0.028 |   3.012 |   3.105 |\n| decode.py      | Verifying multiple links simultaneously is not supported |           N/A |     N/A |     N/A |\n\n### Verifying all RDHs and payloads in 260 MB file with data from 1 GBT link\n| Tool           | Command                                   |   Mean ± σ [s] | Min [s] | Max [s] |\n| :------------- | :---------------------------------------- | -------------: | ------: | ------: |\n| fastPASTA      | `fastpasta input.raw check all its`       |  0.106 ± 0.002 |   0.103 |   0.111 |\n| rawdata-parser | Verifying payloads is not supported       |            N/A |     N/A |     N/A |\n| decode.py      | `python3 decode.py -i 20522 -f input.raw` | 55.903 ± 0.571 |  54.561 |  56.837 |\n\n\n# Need more performance?\nThe primary release profile of fastPASTA is already very fast, but if you absolutely need 10-20% more speed, a faster build profile exists that utilizes the experimental `rust nightly` toolchain.\n\n### Background\nThe rust compiler `rustc` does not yet provide access to all the features that its backend `LLVM` has. But the experimental `nightly` rust toolchain allows passing flags directly to `LLVM`. fastPASTA includes configuration for a build profile `release-nightly` which utilizes `LLVM` to achieve more speed at the cost of compilation time and binary size. As of this writing, the stable channel of Rust does not have a way to pass compiler flags to the LLVM backend. The increased speed is mainly achieved through configuring a higher threshold for inlining, which will increase speed but also compilation time and binary size, and most crucially, cache pressure. The performance impact will be highly dependent on the machine fastPASTA runs on. Better/more CPU cache will lead to a higher performance gain. With \u003e1 GB individual link data, the performance on one particular CERN machine running CentOS Stream 8, as measured by [`hyperfine`](https://github.com/sharkdp/hyperfine) increased by ~17%.\n\n### To install the nightly toolchain (and check your installation)\n```shell\nrustup toolchain install nightly\nrustup run nightly rustc --version\n```\n### Compile the optimized `release-nightly` experimental build profile\n```shell\ncargo +nightly build --profile release-nightly\n```\n### Path to binary: `/target/release-nightly/fastpasta`\n\n## Documentation for developers\n\n[Just](https://just.systems/) is used to run common commands in the project, install it with `cargo install just` and checkout the [main justfile](./justfile) for the available recipes.\n\nRunning `just` in the project root will display the available (public) recipes.\n\nTo see how data is passed around at runtime, see [data flow documentation](doc/data_flow.md).\n\nFor extensive documentation of source code see [documentation](https://docs.rs/fastpasta/latest/fastpasta/index.html) or invoke ```cargo doc --open```.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrambl%2Ffastpasta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrambl%2Ffastpasta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrambl%2Ffastpasta/lists"}