{"id":21689094,"url":"https://github.com/tinram/rnd64","last_synced_at":"2025-04-12T09:23:47.332Z","repository":{"id":83640108,"uuid":"47898678","full_name":"Tinram/RND64","owner":"Tinram","description":"Fast multi-threaded data generator.","archived":false,"fork":false,"pushed_at":"2024-05-29T19:48:14.000Z","size":139,"stargazers_count":12,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-06T21:18:22.558Z","etag":null,"topics":["benchmarking","dummy-data","garbage","generation-rates","huge-files","junk","large-files","multi-threading","random","random-bytes","random-data-generation","randomdata","randomjunk","stress-test","stress-testing","test-files","testfile","testing-tool","trash"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tinram.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-12-12T23:26:06.000Z","updated_at":"2024-05-29T19:41:15.000Z","dependencies_parsed_at":"2023-09-25T01:51:28.052Z","dependency_job_id":"fccfe0a9-54d9-4585-bef2-dbd752ff8ba4","html_url":"https://github.com/Tinram/RND64","commit_stats":{"total_commits":88,"total_committers":1,"mean_commits":88.0,"dds":0.0,"last_synced_commit":"a04d8d6bac169277b3a18488bc4d2b09c6a68d31"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FRND64","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FRND64/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FRND64/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tinram%2FRND64/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tinram","download_url":"https://codeload.github.com/Tinram/RND64/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248544648,"owners_count":21121990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","dummy-data","garbage","generation-rates","huge-files","junk","large-files","multi-threading","random","random-bytes","random-data-generation","randomdata","randomjunk","stress-test","stress-testing","test-files","testfile","testing-tool","trash"],"created_at":"2024-11-25T17:21:06.941Z","updated_at":"2025-04-12T09:23:47.291Z","avatar_url":"https://github.com/Tinram.png","language":"C","readme":"\r\n# RND64\r\n\r\n### Fast multi-threaded file / stream junk data generator.\r\n\r\n#### Linux and Windows\r\n\r\n##### v.0.42\r\n\r\n\r\n[1]: https://tinram.github.io/images/rnd64.png\r\n![rnd64][1]\r\n\r\n\r\n\u0026nbsp;  \r\n\r\n[**Purpose**](#purpose)  \r\n[**Usage**](#usage)  \r\n[**Downloads**](#downloads)  \r\n[**Speed**](#speed)  \r\n[**Build**](#build)\r\n\r\n\u0026nbsp;  \r\n\r\n\r\n## Purpose \u003ca id=\"purpose\"\u003e\u003c/a\u003e\r\n\r\nGenerate large files (over 4 GB, non-sparse) or large streams of binary / character data (200 GB+) at fast generation rates  \r\n\u003csmall\u003e(~8.5 GB/sec i3 desktop, ~4.6 GB/sec AWS microinstance using null byte stream output on Linux)\u003c/small\u003e.\r\n\r\n*And the purpose of junk data?*\r\n\r\nUses:\r\n\r\n+ test files for development, network transfer etc,\r\n+ system stress testing,\r\n+ network speed tests.\r\n\r\nA few Windows programs exist to create large files, and there are plenty of shell scripts using `dd`  \r\nYet, I just needed something cross-platform with simple command-line options.\r\n\r\n\r\n## OS Support\r\n\r\n+ Linux x64\r\n+ Windows x64\r\n+ Mac arm64\r\n\r\n\r\n## Usage \u003ca id=\"usage\"\u003e\u003c/a\u003e\r\n\r\n```bash\r\n    rnd64 [option] \u003csize\u003e \u003cfile\u003e\r\n    rnd64 [option] \u003csize\u003e | \u003cprogram\u003e\r\n```\r\n\r\n#### Options\r\n\r\n    -a     (all)             binary bytes               includes control codes\r\n    -f     (fastest)         null character (0)         fastest generator\r\n    -r     (restrict)        characters 33 to 126       7-bit printable ASCII, safe for terminal output\r\n    -c     (crypto)          crypto-sourced bytes       Linux: /dev/urandom, Windows: CryptGenRandom (slow)\r\n\r\n    size   1K, 100M, 8G\r\n\r\n\r\n### Usage Examples\r\n\r\n    rnd64.exe or rnd64   (Windows)             display command-line options, as above\r\n    ./rnd64              (Linux)\r\n\r\n    rnd64 -a 1k f.txt                          output 1 kB of random binary bytes to the file 'f.txt'\r\n    rnd64 -f 1k f.txt                          output 1 kB of null bytes to 'f.txt'\r\n    rnd64 -r 1k f.txt                          output the restricted range of 7-bit ASCII characters (33 to 126) to 'f.txt'\r\n    rnd64 -f 4g | pv \u003e /dev/null               send 4 GB of null bytes to /dev/null with 'pv' displaying the throughput rate (Linux)\r\n    rnd64 -c 1k | ent                          pipe 1 kB of crypto bytes to the program 'ent'\r\n    rnd64 -a 1k | nc 192.168.1.20 80           pipe 1 kB of random bytes to 'netcat' to send to 192.168.1.20 on port 80\r\n    rnd64 -f 100g | pv \u003e /dev/null             stress a system\r\n\r\n    nc -lk -p 3000 \u003e /dev/null                 local network speed test (machine receiving, 192.168.1.20)\r\n    rnd64 -f 1g | pv | nc 192.168.1.20 3000    (machine sending)\r\n\r\n\r\n### Warning!\r\n\r\nWhen using RND64 to generate large files (over 1 GB):\r\n\r\n+ HDDs: consider the mechanical drive's age and performance,\r\n+ SSDs: consider the potential write wear.\r\n\r\nAs well as warming the CPU, RND64 is quite capable of exhausting all drive space, finishing off failing HDDs, and reducing SSD lifetimes.\r\n\r\n\r\n## Downloads \u003ca id=\"downloads\"\u003e\u003c/a\u003e\r\n\r\nDownload from [Releases](https://github.com/Tinram/RND64/releases/latest)  \r\nor directly:\r\n\r\n+ Linux: [rnd64](https://github.com/Tinram/RND64/raw/master/bin/rnd64)\r\n+ Windows: [rnd64.exe](https://github.com/Tinram/RND64/raw/master/bin/rnd64.exe)\r\n\r\n\r\n## Speed \u003ca id=\"speed\"\u003e\u003c/a\u003e\r\n\r\n### Linux\r\n\r\n**RND64 is fast:**\r\n\r\n*i3-4170 CPU 3.70GHz, 4.4 kernel*:\r\n\r\n        martin@xyz ~ $ rnd64 -f 4g | pv \u003e /dev/null\r\n        4GiB 0:00:00 [8.51GiB/s] [  \u003c=\u003e  ]\r\n\r\n`dd` on same machine:\r\n\r\n        martin@xyz ~ $ dd if=/dev/zero of=/dev/null bs=4G count=1 iflag=fullblock\r\n        4294967296 bytes (4.3 GB, 4.0 GiB) copied, 0.959431 s, 4.5 GB/s\r\n\r\n*AWS Xeon E5-2670 2.50GHz, single core*:\r\n\r\n        [ec2-user@ip-172-31-7-109 ~]$ rnd64 -f 4g | pv \u003e /dev/null\r\n        4GiB 0:00:00 [4.61GiB/s] [     \u003c=\u003e     ]\r\n\r\n+ Null byte stream generation rates `-f` are decent on Linux (~8 GB/sec on vanilla i3-4170), and the [PCG](http://www.pcg-random.org/) random number generator (`-a` switch) is pretty fast (~4 GB/sec on same CPU) compared to most other RNGs.\r\n\r\n**... but not that fast:**\r\n\r\n+ File generation rates are slower and subject to a multitude of factors including: OS, OS activity, kernel version, kernel patches, HDD versus SSD drive, SSD interface and underlying SSD technology etc.\r\n\r\nIn the code, there are faster ways to create files than using C's `fwrite()`, which RND64 uses.\r\n\r\nOn Linux, `write()` can be up to 4 times faster than `fwrite()` on some machines (using a single-threaded version of RND64, with file descriptor unclosed).  However, `write()` will only transfer a maximum of 2.1 GB, even on 64-bit systems [[write(2)](http://man7.org/linux/man-pages/man2/write.2.html)]. `fwrite()` does not have this limitation, and 4 GB+ output is what I sought.\r\n\r\nMulti-threading has its own speed impacts, such as thread-waiting and data streams being combined.\r\n\r\n**... and output is 'slowing down':**\r\n\r\n... apparently on my i3-4170, courtesy of the Spectre/Meltdown kernel revisions.\r\n\r\n\r\n### Windows\r\n\r\nWith Windows lacking `pv` or equivalent, stream output speed is somewhat more difficult to assess.\r\n\r\nOne way is to provide the stats output on *stderr*.  \r\nThis can be enabled by setting a flag in the source.\r\n\r\nIn *rnd64.h*, set the following macro value to 1:\r\n\r\n```c\r\n    #define STREAM_STATS 0 /* Win stream stats */\r\n```\r\n\r\nThen compile the source as in the *Build \u003e Windows* section.\r\n\r\n        C:\\rnd64.exe -f 4g \u003e nul\r\n        time: 9 s 938 ms\r\n        MB/s: 412.15\r\n\r\n\r\n## Build \u003ca id=\"build\"\u003e\u003c/a\u003e\r\n\r\n```bash\r\n    git clone https://github.com/Tinram/RND64.git\r\n    cd RND64/src\r\n```\r\n\r\n### Linux\r\n\r\n```bash\r\n    make\r\n```\r\n\r\nor full process:\r\n\r\n```bash\r\n    make \u0026\u0026 make install \u0026\u0026 make clean\r\n```\r\n\r\n(Default compiler is GCC. For Clang, just rename the *makefiles*.)\r\n\r\n\r\n### Mac\r\n\r\nCompiles on Mac with a few code changes:\r\n\r\n+ Delete the line `#include \u003csys/sysinfo.h\u003e` in *rnd64.h*\r\n+ Auto-replace all instances of `__linux` to `__APPLE__` in *rnd64.h* and *rnd64.c*\r\n+ In *rnd64.c*, change `iNumThreads = (unsigned int) get_nprocs();` to `iNumThreads = (unsigned int) sysconf(_SC_NPROCESSORS_ONLN);`\r\n+ Rename *makefile.clang* to *makefile* and `make`\r\n\r\n----\r\n\r\n#### Manual compilation:\r\n\r\n**GCC:**\r\n\r\n```bash\r\n    gcc rnd64.c -o rnd64 -lpthread -O3 -Wall -Wextra -Wuninitialized -Wunused -Werror -std=gnu99 -s\r\n```\r\n\r\n**Clang:**\r\n\r\n```bash\r\n    clang rnd64.c -o rnd64 -lpthread -O3 -Wall -Wextra -Wuninitialized -Wunused -Werror -s\r\n```\r\n\r\n##### Further Optimisation\r\n\r\n    -mtune=native -march=native                    current CPU\r\n    -flto                                          linker optimize\r\n\r\n    -march=core-avx2 -mtune=core-avx2              Intel Haswell\r\n    -march=skylake-avx512 -mtune=skylake-avx512    Intel Skylake\r\n\r\n----\r\n\r\n### Windows\r\n\r\n```bash\r\n    gcc rnd64.c -o rnd64.exe -O3 -Wall -Wextra -Wuninitialized -Wunused -Werror -std=c99 -s\r\n```\r\n\r\n\r\n## Other\r\n\r\nThe switches `-a` and `-c` are dangerous options when both a filename and pipe symbol are omitted. The wide range of output bytes, including control characters, are printed in the terminal, which can cause the terminal to lock or crash (especially on Windows).\r\n\r\nOn both Linux and Windows, it's more convenient for RND64 to be available from any directory location via the *$PATH* system variable.\r\n\r\n#### Linux\r\n\r\n```bash\r\n    make install\r\n```\r\n\r\nOr move the RND64 executable to a location such as */usr/local/bin* (location must be present in *$PATH*).\r\n\r\n#### Windows\r\n\r\n[Windows/Super key + Break] \u003e Advanced tab \u003e Environmental Variables button \u003e click Path line \u003e Edit button \u003e Variable value \u0026ndash; append at the end of existing line information: *C:\\directory\\path\\to\\rnd64.exe\\;*\r\n\r\n\r\n## Testing\r\n\r\n+ Bfbtester\r\n+ CppCheck\r\n+ Fuzz\r\n+ Valgrind\r\n+ dev/null_detect.c\r\n\r\n\r\n## Credits\r\n\r\n+ [Professor Melissa E. O'Neill](https://www.cs.hmc.edu/~oneill/index.html): creator of the fast [PCG](http://www.pcg-random.org/) RNG.\r\n+ [Damir Cohadarevic](https://github.com/cohadar): inspiration, highlighting PCG.\r\n+ [Aleksandr Sergeev](https://github.com/sergeevabc): thorough testing, recommendations.\r\n+ [Vort](https://github.com/Vort): testing.\r\n+ MSDN: Windows crypto.\r\n+ Ben Alpert: microsecond timer.\r\n\r\n\r\n## License\r\n\r\nRND64 is released under the [GPL v.3](https://www.gnu.org/licenses/gpl-3.0.html).\r\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinram%2Frnd64","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinram%2Frnd64","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinram%2Frnd64/lists"}