{"id":13514920,"url":"https://github.com/p-ranav/binary_log","last_synced_at":"2025-09-09T21:46:38.701Z","repository":{"id":63522138,"uuid":"431650874","full_name":"p-ranav/binary_log","owner":"p-ranav","description":"Fast binary logger for C++","archived":false,"fork":false,"pushed_at":"2025-01-31T23:22:01.000Z","size":1717,"stargazers_count":223,"open_issues_count":2,"forks_count":16,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-05-24T11:06:38.383Z","etag":null,"topics":["binary-log","blazingly-fast","concepts","cpp20","cpp20-library","fastest","file-writing","fmt","fmtlib","header-only","logger","logger-go-brrr","logging","logging-library","mit-license","single-threaded","string-formatting","super-fast"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/p-ranav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-11-24T23:01:02.000Z","updated_at":"2025-05-08T19:33:55.000Z","dependencies_parsed_at":"2025-05-17T10:05:57.853Z","dependency_job_id":"980a7aa3-f97e-494c-9afa-53f4b4693e32","html_url":"https://github.com/p-ranav/binary_log","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/p-ranav/binary_log","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-ranav%2Fbinary_log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-ranav%2Fbinary_log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-ranav%2Fbinary_log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-ranav%2Fbinary_log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/p-ranav","download_url":"https://codeload.github.com/p-ranav/binary_log/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p-ranav%2Fbinary_log/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274367628,"owners_count":25272299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-log","blazingly-fast","concepts","cpp20","cpp20-library","fastest","file-writing","fmt","fmtlib","header-only","logger","logger-go-brrr","logging","logging-library","mit-license","single-threaded","string-formatting","super-fast"],"created_at":"2024-08-01T05:01:03.805Z","updated_at":"2025-09-09T21:46:38.678Z","avatar_url":"https://github.com/p-ranav.png","language":"C++","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg height=\"80\" src=\"images/logo.png\"/\u003e  \n\u003c/p\u003e\n\n# Highlights\n\n* Logs messages in a compact binary format\n* Fast\n  * ***Hundreds of millions*** of logs per second\n  * Average latency of ***1-3 ns*** for basic data types\n  * See [benchmarks](https://github.com/p-ranav/binary_log#benchmarks)\n* Provides an [unpacker](https://github.com/p-ranav/binary_log/tree/master/tools/unpacker) to deflate the log messages\n* Uses [fmtlib](https://github.com/fmtlib/fmt) to format the logs\n* Synchronous logging - not thread safe\n* Header-only library\n  - Single header file version available [here](https://github.com/p-ranav/binary_log/blob/master/single_include/binary_log/binary_log.hpp)\n* Requires C++20\n* MIT License\n\n# Usage and Performance\n\nThe following code logs 1 billion integers to file.\n\n```cpp\n#include \u003cbinary_log/binary_log.hpp\u003e\n\nint main()\n{\n  binary_log::binary_log log(\"log.out\");\n\n  for (int i = 0; i \u003c 1E9; ++i)\n    BINARY_LOG(log, \"Hello logger, msg number: {}\", i);\n}\n```\n\nOn a [modern workstation desktop](#system-details), the above code executes in `~2s`.\n\n| Type            | Value               |\n| --------------- | --------------------|\n| Time Taken      | 1.935 s             | \n| Throughput      | 2.06 Gb/s           |\n| Performance     | 516 million logs/s  |\n| Average Latency | 1.73 ns             |\n| File Size       | ~4 GB               |\n\n```console\nfoo@bar:~/dev/binary_log$ time ./build/examples/billion_integers/billion_integers\n\nreal    0m1.935s\nuser    0m0.906s\nsys     0m1.000s\n\nfoo@bar:~/dev/binary_log$ ls -lart log.out*\n-rw-r--r-- 1 pranav pranav         10 Sep 20 11:46 log.out.runlength\n-rw-r--r-- 1 pranav pranav         33 Sep 20 11:46 log.out.index\n-rw-r--r-- 1 pranav pranav 4000000002 Sep 20 11:46 log.out\n```\n\n## Deflate the logs\n\nThese binary log files can be deflated using the provided [unpacker](https://github.com/p-ranav/binary_log/tree/master/tools/unpacker) app:\n\n```console\nfoo@bar:~/dev/binary_log$ time ./build/tools/unpacker/unpacker log.out \u003e log.deflated\n\nreal    2m19.853s\nuser    1m16.078s\nsys     0m50.969s\n\nfoo@bar:~/dev/binary_log$ ls -lart log.deflated\n-rw-r--r-- 1 pranav pranav 35888888890 Dec  6 08:09 log.deflated\n\nfoo@bar:~/dev/binary_log$ wc -l log.deflated\n1000000000 log.deflated\n\nfoo@bar:~/dev/binary_log$ $ head log.deflated\nHello logger, msg number: 0\nHello logger, msg number: 1\nHello logger, msg number: 2\nHello logger, msg number: 3\nHello logger, msg number: 4\nHello logger, msg number: 5\nHello logger, msg number: 6\nHello logger, msg number: 7\nHello logger, msg number: 8\nHello logger, msg number: 9\n\nfoo@bar:~/dev/binary_log$ tail log.deflated\nHello logger, msg number: 999999990\nHello logger, msg number: 999999991\nHello logger, msg number: 999999992\nHello logger, msg number: 999999993\nHello logger, msg number: 999999994\nHello logger, msg number: 999999995\nHello logger, msg number: 999999996\nHello logger, msg number: 999999997\nHello logger, msg number: 999999998\nHello logger, msg number: 999999999\n```\n\n| Type                | Value      |\n| ------------------- | ---------- |\n| Time Taken          | 2m 19s     | \n| Throughput          | 258 MB/s   |\n| Original File Size  | ~5 GB      |\n| Deflated File Size  | ~35 GB     |\n| Log Compression     | ***7x***   |\n\nSee [benchmarks](https://github.com/p-ranav/binary_log/blob/master/README.md#benchmarks) section for more performance metrics.\n\n# Design Goals \u0026 Decisions\n\n* Implement a single-threaded, synchronous logger - Do not provide thread safety\n  - If the user wants multi-threaded behavior, the user can choose and implement their own queueing solution\n  - There are numerous well-known lock-free queues available for this purpose ([moody::concurrentqueue](https://github.com/cameron314/concurrentqueue), [atomic_queue](https://github.com/max0x7ba/atomic_queue) etc.) - let the user choose the technology they want to use.\n  - The latency of enqueuing into a lock-free queue is large enough to matter\n    - Users who do not care about multi-threaded scenarios should not suffer the cost\n    - Looking at the [atomic_queue benchmarks](https://max0x7ba.github.io/atomic_queue/html/benchmarks.html), the average round-trip latency across many state-of-the-art multi-producer, multi-consumer queues, to send and receive a 4-byte integer (between 2 threads, using 2 queues) is around 150-250 ns.\n* Avoid writing static information more than once\n  - Examples of static information: the format string, the number of format args, and type of each format arg\n  - Store the static information in an \"index\" file \n  - Store the dynamic information in the log file (refer to the index file where possible)\n* Do as little work as possible in the runtime hot path\n  - No formatting of any kind\n  - All formatting will happen offline using an unpacker that deflates the binary logs\n\n# How it Works\n\n`binary_log` splits the logging into three files:\n\n\u003cp\u003e\n  \u003cimg height=\"600\" src=\"images/how_it_works.png\"/\u003e  \n\u003c/p\u003e\n\n1. ***Index file*** contains all the static information from the logs, e.g., format string, number of args, type of each arg etc.\n   - If a format argument is marked as constant using `binary_log::constant`, the value of the arg is also stored in the index file\n2. ***Log file*** contains two pieces of information per log call:\n   1. An index into the index table (in the index file) to know which format string was used\n      - If runlength encoding is working, this index might not be written, instead the final runlength will be written to the runlengths file\n   3. The value of each argument\n3. ***Runlength file*** contains runlengths - If a log call is made 5 times, this information is stored here (instead of storing the index 5 times in the log file)\n   - NOTE: Runlengths are only stored if the runlength \u003e 1 (to avoid the inflation case with RLE)\n\n## Constants\n\nOne can specify a log format argument as a constant by wrapping the value with `binary_log::constant(...)`. When this is detected, the value is stored in the index file instead of the log file as it is now considered \"static information\" and does not change between calls. \n\n```cpp\nfor (auto i = 0; i \u003c 1E9; ++i) {\n  BINARY_LOG(log, \"Joystick {}: x_min={}, x_max={}, y_min={}, y_max={}\",\n             binary_log::constant(\"Nintendo Joycon\"),\n             binary_log::constant(-0.6),\n             binary_log::constant(+0.65),\n             binary_log::constant(-0.54),\n             binary_log::constant(+0.71));\n}\n```\n\nThe above loop runs in under `500 ms`. The final output is compact at just `118 bytes` and contains all the information needed to deflate the log (if needed). \n\n| File               | Size      |\n| ------------------ | --------- |\n| log.out            | 1 byte    | \n| log.out.runlength  | 6 bytes   |\n| log.out.index      | 111 bytes |\n\n```console\nfoo@bar:~/dev/binary_log$ ls -lart log.out*\n-rw-r--r-- 1 pranav pranav   6 Dec  5 08:41 log.out.runlength\n-rw-r--r-- 1 pranav pranav 111 Dec  5 08:41 log.out.index\n-rw-r--r-- 1 pranav pranav   1 Dec  5 08:41 log.out\n\nfoo@bar:~/dev/binary_log$ hexdump -C log.out.index\n00000000  33 4a 6f 79 73 74 69 63  6b 20 7b 7d 3a 20 78 5f  |3Joystick {}: x_|\n00000010  6d 69 6e 3d 7b 7d 2c 20  78 5f 6d 61 78 3d 7b 7d  |min={}, x_max={}|\n00000020  2c 20 79 5f 6d 69 6e 3d  7b 7d 2c 20 79 5f 6d 61  |, y_min={}, y_ma|\n00000030  78 3d 7b 7d 05 0c 0b 0b  0b 0b 01 0f 4e 69 6e 74  |x={}........Nint|\n00000040  65 6e 64 6f 20 4a 6f 79  63 6f 6e 01 33 33 33 33  |endo Joycon.3333|\n00000050  33 33 e3 bf 01 cd cc cc  cc cc cc e4 3f 01 48 e1  |33..........?.H.|\n00000060  7a 14 ae 47 e1 bf 01 b8  1e 85 eb 51 b8 e6 3f     |z..G.......Q..?|\n0000006f\n```\n\n# Benchmarks\n\n### System Details\n\n| Type            | Value                                                                                                     |\n| --------------- | --------------------------------------------------------------------------------------------------------- |\n| Processor       | 11th Gen Intel(R) Core(TM) i9-11900KF @ 3.50GHz   3.50 GHz                                                |\n| Installed RAM   | 32.0 GB (31.9 GB usable)                                                                                  |\n| SSD             | [ADATA SX8200PNP](https://www.adata.com/upload/downloadfile/Datasheet_XPG%20SX8200%20Pro_EN_20181017.pdf) |\n| OS              | Ubuntu 20.04 LTS running on WSL in Windows 11                                                             |\n| C++ Compiler    | g++ (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0                                                                 |\n\n```console\nfoo@bar:~/dev/binary_log$ ./build/benchmark/binary_log_benchmark\n2022-09-20T12:59:39-05:00\nRunning ./build/benchmark/binary_log_benchmark\nRun on (16 X 3504 MHz CPU s)\nLoad Average: 0.52, 0.58, 0.59\n------------------------------------------------------------------------------------------------------------------\nBenchmark                                                        Time             CPU   Iterations UserCounters...\n------------------------------------------------------------------------------------------------------------------\nBM_binary_log_static_integer\u003cuint8_t\u003e/42                      1.22 ns         1.20 ns    560000000 Latency=1.19978ns Logs/s=833.488M/s\nBM_binary_log_static_integer\u003cuint16_t\u003e/395                    1.43 ns         1.43 ns    448000000 Latency=1.42997ns Logs/s=699.317M/s\nBM_binary_log_static_integer\u003cuint32_t\u003e/3123456789             1.89 ns         1.84 ns    373333333 Latency=1.84152ns Logs/s=543.03M/s\nBM_binary_log_static_integer\u003cuint64_t\u003e/9876543123456789       5.45 ns         2.76 ns    248888889 Latency=2.76228ns Logs/s=362.02M/s\nBM_binary_log_static_integer\u003cint8_t\u003e/-42                      1.25 ns         1.26 ns    560000000 Latency=1.25558ns Logs/s=796.444M/s\nBM_binary_log_static_integer\u003cint16_t\u003e/-395                    1.54 ns         1.57 ns    448000000 Latency=1.56948ns Logs/s=637.156M/s\nBM_binary_log_static_integer\u003cint32_t\u003e/-123456789              1.94 ns         1.97 ns    373333333 Latency=1.96708ns Logs/s=508.369M/s\nBM_binary_log_static_integer\u003cint64_t\u003e/-9876543123456789       4.11 ns         2.92 ns    235789474 Latency=2.91574ns Logs/s=342.967M/s\nBM_binary_log_static_float                                    1.82 ns         1.84 ns    407272727 Latency=1.84152ns Logs/s=543.03M/s\nBM_binary_log_static_double                                   3.29 ns         2.73 ns    263529412 Latency=2.7274ns Logs/s=366.65M/s\nBM_binary_log_static_string                                   4.93 ns         2.92 ns    235789474 Latency=2.91574ns Logs/s=342.967M/s\nBM_binary_log_random_integer\u003cuint8_t\u003e                         5.75 ns         5.72 ns    112000000 Latency=5.71987ns Logs/s=174.829M/s\nBM_binary_log_random_integer\u003cuint16_t\u003e                        6.08 ns         6.14 ns    112000000 Latency=6.13839ns Logs/s=162.909M/s\nBM_binary_log_random_integer\u003cuint32_t\u003e                        7.51 ns         7.67 ns     89600000 Latency=7.67299ns Logs/s=130.327M/s\nBM_binary_log_random_integer\u003cuint64_t\u003e                        15.0 ns         15.0 ns     44800000 Latency=14.9972ns Logs/s=66.6791M/s\nBM_binary_log_random_integer\u003cint8_t\u003e                          5.70 ns         5.72 ns    112000000 Latency=5.71987ns Logs/s=174.829M/s\nBM_binary_log_random_integer\u003cint16_t\u003e                         5.84 ns         5.86 ns    112000000 Latency=5.85938ns Logs/s=170.667M/s\nBM_binary_log_random_integer\u003cint32_t\u003e                         7.89 ns         7.67 ns     89600000 Latency=7.67299ns Logs/s=130.327M/s\nBM_binary_log_random_integer\u003cint64_t\u003e                         14.9 ns         15.0 ns     44800000 Latency=14.9972ns Logs/s=66.6791M/s\nBM_binary_log_random_real\u003cfloat\u003e                              6.29 ns         6.25 ns    100000000 Latency=6.25ns Logs/s=160M/s\nBM_binary_log_random_real\u003cdouble\u003e                             11.6 ns         11.7 ns     64000000 Latency=11.7188ns Logs/s=85.3333M/s\nBM_binary_log_billion_integers                          2320246800 ns   1765625000 ns            1 Latency=1.76562ns Logs/s=566.372M/s\n```\n\n## Supported Format Argument Types\n\n`binary_log` supports a limited number of types of format arguments. They are:\n\n```\nbool,\nchar, \nuint8_t, uint16_t, uint32_t, uint64_t\nint8_t, int16_t, int32_t, int64_t,\nfloat, double,\nconst char*,\nstd::string,\nstd::string_view\n```\n\n# Building and installing\n\nSee the [BUILDING](BUILDING.md) document.\n\n## Generating Single Header\n\n```bash\npython3 utils/amalgamate/amalgamate.py -c single_include.json -s .\n```\n\n# Contributing\n\nSee the [CONTRIBUTING](CONTRIBUTING.md) document.\n\n# License\n\nThe project is available under the [MIT](https://opensource.org/licenses/MIT) license.\n","funding_links":[],"categories":["C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp-ranav%2Fbinary_log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fp-ranav%2Fbinary_log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp-ranav%2Fbinary_log/lists"}