{"id":43338770,"url":"https://github.com/paulscherrerinstitute/sf_daq_buffer","last_synced_at":"2026-02-02T00:40:08.502Z","repository":{"id":45714234,"uuid":"259962323","full_name":"paulscherrerinstitute/sf_daq_buffer","owner":"paulscherrerinstitute","description":"POC for detector buffering at SwissFEL","archived":false,"fork":false,"pushed_at":"2024-05-21T17:57:09.000Z","size":2746,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-05-21T19:03:57.453Z","etag":null,"topics":["daq","jungfrau","swissfel"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulscherrerinstitute.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-29T15:18:22.000Z","updated_at":"2024-05-21T17:57:12.000Z","dependencies_parsed_at":"2024-05-21T19:10:39.082Z","dependency_job_id":null,"html_url":"https://github.com/paulscherrerinstitute/sf_daq_buffer","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/paulscherrerinstitute/sf_daq_buffer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulscherrerinstitute%2Fsf_daq_buffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulscherrerinstitute%2Fsf_daq_buffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulscherrerinstitute%2Fsf_daq_buffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulscherrerinstitute%2Fsf_daq_buffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulscherrerinstitute","download_url":"https://codeload.github.com/paulscherrerinstitute/sf_daq_buffer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulscherrerinstitute%2Fsf_daq_buffer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28997227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T23:10:54.274Z","status":"ssl_error","status_checked_at":"2026-02-01T23:10:47.298Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["daq","jungfrau","swissfel"],"created_at":"2026-02-02T00:40:07.818Z","updated_at":"2026-02-02T00:40:08.487Z","avatar_url":"https://github.com/paulscherrerinstitute.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sf_daq_buffer\n\nThe goal of the project is to provide a detector buffering and writing solution \nthat meets the needs of SwissFEL data taking. By dividing the problem into \nsmall components (Microservice architecture) we hope to achieve better \nmaintainability and long term stability of the codebase. We try to make the \ncode as simple as possible and write unit tests extensively to facilitate \nfuture changes.\n\nOverview of current architecture and component interaction:\n\n![Overview image](docs/sf_daq_buffer-overview.jpg)\n\nDocumentation of individual components:\n\n- [sf-buffer](sf-buffer) (Receive UDP and write buffer files)\n- [sf-stream](sf-stream) (Live streaming of detector data)\n- [sf-writer](sf-writer) (Read from buffer and write H5)\n- [sf-utils](sf-utils) (Small utilities for debugging and testing)\n- [jf-live-writer](jf-live-writer) (Live writer to high performance detectors)\n\n## Design goals\n\n- Simplest thing that works.\n    - Save time and iterate more quickly.\n    - Less moving parts, easier to debug.\n    - Avoid external libraries as much as possible.\n- Start optimizing only when things break.\n    - Many optimization possibilities, but not actually needed.\n    - Makes possible to refactor and change code faster.\n- Small debuggable and profileable processes.\n    - Needs to be able to run on your local machine in a debugger.\n    - Asses code performance without guessing.\n- As little dependency between processes as possible.\n    - Run only the process you want to test.\n    - Write unit tests.\n\n## Scope\n\nThe scope should be as limited as possible and we should abstain from \nimplementing additional features. We isolated the core functionality this buffer \nneeds as the following:\n\n- Receiving and assembling detector data.\n- Writing detector data to HDF5 files.\n- Live streaming detector data to external components.\n- Provide logs and analytics for each component.\n\nWe focus on the service and storage layer (see overview picture). Everything \nelse is not part of this project and should be addresses elsewhere.\n\n## Terminology\n\nIn order to unify the way we write code and talk about concepts the following \nterminology definitions should be followed:\n\n- frame (data from a single module)\n- image (assembled frames)\n- start_pulse_id and stop_pulse_id is used to determine the \ninclusive range (both start and stop pulse_id are included) of pulses.\n- pulse_id_step (how many pulses to skip between each image).\n- GPFS buffer (detector buffering mechanism based on binary files on GPFS)\n- detector_folder (root folder of the buffer for a specific detector on disk)\n- module_folder (folder of one module inside the detector_folder)\n- data_folder (folder where we group more buffer files based on pulse_id range)\n- data_file (the files where the actual data is stored, inside data_folder)\n\n## Build\n\nTo compile this repo you will need to install the following packages on RH7:\n- devtoolset-9\n- cmake3\n- zeromq-devel\n- hdf5-devel\n\n```bash\nyum install devtoolset-9\nyum install cmake3\nyum install zeromq-devel\nyum install hdf5-devel\n```\n\nStep by step procedure to build the repo:\n\n```bash\nscl enable devtoolset-9 bash\ngit clone https://github.com/paulscherrerinstitute/sf_daq_buffer.git\ncd sf_daq_buffer\nmkdir build\ncd build/\ncmake3 ..\nmake\n```\n\nIt is recommended to create symbolic links to the executables you will be using \ninside your PATH.\n\nExample:\n```bash\nln -s \"$(pwd)\"\"/\"\"sf_buffer\" /usr/bin/sf_buffer\nln -s \"$(pwd)\"\"/\"\"sf_stream\" /usr/bin/sf_stream\nln -s \"$(pwd)\"\"/\"\"sf_writer\" /usr/bin/sf_writer\n```\n\n### Warnings\n\n#### Zeromq\n\nZeromq version 4.1.4 (default on RH7) has a LINGER bug. Sometimes, the last \nmessage is not sent (the connection gets dropped before the message is in the buffer).\nWe are currently not using LINGER to extend our processes lifetime, so this is \nnot critical. But in the future this might change, so updating to the latest \nversion of ZMQ should help us prevent future bug hunting sessions.\n\nPlease install a later version:\n```bash\ncd /etc/yum.repos.d/\nwget https://download.opensuse.org/repositories/network:messaging:zeromq:release-stable/RHEL_7/network:messaging:zeromq:release-stable.repo\nyum remove zeromq\nyum remove openpgm\nyum install libsodium-devel\nyum install zeromq-devel\n```\n\n## Useful links\n\nThis is a collections of best links we came across so far during the development of \nthis project. They cover various topics somehow related to what we are trying to \nachieve.\n\n### Architecture\n- POSIX compliant write order test on GPFS\nhttps://svn.hdfgroup.org/hdf5/branches/hdf5_1_10_0/test/POSIX_Order_Write_Test_Report.pdf\n- Best Practice Guide - Parallel I/O\nhttps://prace-ri.eu/wp-content/uploads/Best-Practice-Guide_Parallel-IO.pdf\n- MPI-IO/GPFS, an Optimized Implementation of MPI-IO on top of GPFS\nhttps://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=1592834\n- 10GE network tests with UDP - European XFEL\nhttps://indico.cern.ch/event/212228/contributions/1507212/attachments/333941/466017/10GE_network_tests_with_UDP.pdf\n- How to choose between Kafka and RabbitMQ \nhttps://tarunbatra.com/blog/comparison/How-to-choose-between-Kafka-and-RabbitMQ/\n\n\n### Software\n- Intro to lock free programming\nhttps://preshing.com/20120612/an-introduction-to-lock-free-programming/\n- JSON library benchmarks\nhttps://github.com/miloyip/nativejson-benchmark\n- Kernel bypass\nhttps://blog.cloudflare.com/kernel-bypass/\n- PACKET_MMAP\nhttps://www.kernel.org/doc/Documentation/networking/packet_mmap.txt\n- Hyperslab selection \nhttps://support.hdfgroup.org/HDF5/Tutor/phypecont.html\nhttps://support.hdfgroup.org/HDF5/Tutor/selectsimple.html\n- Caching and Buffering in HDF5\nhttps://de.slideshare.net/HDFEOS/caching-and-buffering-in-hdf5\n- Chunking in HDF5\nhttps://portal.hdfgroup.org/display/HDF5/Chunking+in+HDF5\n- Setting Raw Data Chunk Cache Parameters in HDF5\nhttps://support.hdfgroup.org/pubs/rfcs/RFC_chunk_cache_functions.pdf\n- Memory model synchronization modes\nhttps://gcc.gnu.org/wiki/Atomic/GCCMM/AtomicSync\n- Is Parallel Programming Hard, And, If So, What Can You Do About It?\nhttps://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.2018.12.08a.pdf\n- Linux kernel profiling with perf\nhttps://perf.wiki.kernel.org/index.php/Tutorial\n- Introduction to parallel HDF5\nhttps://portal.hdfgroup.org/display/HDF5/Introduction+to+Parallel+HDF5\n- Introduction to Scientific I/O\nhttps://www.nersc.gov/users/training/online-tutorials/introduction-to-scientific-i-o/?show_all=1\n- Lecture 36: MPI, Hybrid Programming, and Shared Memory\nhttp://wgropp.cs.illinois.edu/courses/cs598-s15/lectures/lecture36.pdf\n- Ansi escape codes\nhttps://www.lihaoyi.com/post/BuildyourownCommandLinewithANSIescapecodes.html\n\n\n### Linux configuration\n- CFS: Completely fair process scheduling in Linux\nhttps://opensource.com/article/19/2/fair-scheduling-linux\n- perf sched for Linux CPU scheduler analysis\nhttp://www.brendangregg.com/blog/2017-03-16/perf-sched.html\n- Tuning CPU scheduler for reducing latency\nhttps://www.scylladb.com/2016/06/10/read-latency-and-scylla-jmx-process/\n- RHEL7: How to get started with CGroups.\nhttps://www.certdepot.net/rhel7-get-started-cgroups/\n- Cpusets\nhttps://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt\n- Understanding mlx5 ethtool Counters\nhttps://community.mellanox.com/s/article/understanding-mlx5-ethtool-counters\n- Red Hat Enterprise Linux Network Performance Tuning Guide\nhttps://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf\n- Low latency 10Gbps Ethernet\nhttps://blog.cloudflare.com/how-to-achieve-low-latency/\n- Monitoring and Tuning the Linux Networking Stack: Receiving Data\nhttps://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/#procnetsoftnet_stat\n- Making linux do hard real-time\nhttps://www.slideshare.net/jserv/realtime-linux\n- Linux timing and scheduling granularity\nhttps://fritshoogland.wordpress.com/2018/03/13/linux-timing-and-scheduling-granularity/\n- Raw Ethernet Programming: Basic Introduction - Code Example\nhttps://community.mellanox.com/s/article/raw-ethernet-programming--basic-introduction---code-example\n- Performance Tuning for Mellanox Adapters\nhttps://community.mellanox.com/s/article/performance-tuning-for-mellanox-adapters\n- UEFI Workload-based Performance and TuningGuide for HPE ProLiant Gen10 \nhttps://support.hpe.com/hpesc/public/docDisplay?docId=a00016408en_us\n- Low Latency Performance Tuning for Red Hat Enterprise Linux 7\nhttps://access.redhat.com/sites/default/files/attachments/201501-perf-brief-low-latency-tuning-rhel7-v2.1.pdf\n- Understanding PCIe performance for end hostnetworking\nhttps://www.cl.cam.ac.uk/research/srg/netos/projects/pcie-bench/neugebauer2018understanding.pdf\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulscherrerinstitute%2Fsf_daq_buffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulscherrerinstitute%2Fsf_daq_buffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulscherrerinstitute%2Fsf_daq_buffer/lists"}