{"id":13586411,"url":"https://github.com/intel/fiovisualizer","last_synced_at":"2025-04-07T15:31:52.794Z","repository":{"id":24071111,"uuid":"27457529","full_name":"intel/fiovisualizer","owner":"intel","description":null,"archived":true,"fork":false,"pushed_at":"2024-04-08T16:08:41.000Z","size":96,"stargazers_count":120,"open_issues_count":10,"forks_count":33,"subscribers_count":17,"default_branch":"master","last_synced_at":"2024-04-14T09:39:05.384Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-12-02T22:50:33.000Z","updated_at":"2024-08-01T16:32:27.685Z","dependencies_parsed_at":"2024-08-01T16:42:39.683Z","dependency_job_id":null,"html_url":"https://github.com/intel/fiovisualizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Ffiovisualizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Ffiovisualizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Ffiovisualizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Ffiovisualizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intel","download_url":"https://codeload.github.com/intel/fiovisualizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247679559,"owners_count":20978073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:05:33.341Z","updated_at":"2025-04-07T15:31:52.441Z","avatar_url":"https://github.com/intel.png","language":"Python","readme":"# DISCONTINUATION OF PROJECT #  \nThis project will no longer be maintained by Intel.  \nIntel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.  \nIntel no longer accepts patches to this project.  \n If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.  \n  \n# fiovisualizer\n\nThis tool is a realtime data visualisation tool for [FIO](http://freecode.com/projects/fio) written using [PyQtGraph](http://www.pyqtgraph.org).\n\n## Overview\n\nThis application is written in Python, using PyQtGraph to graph FIO data in realtime. Running as root is required because this application accesses block devices.\n\n## Package Content\n- README.md\n  - This readme file\n- LICENSE.txt\n  - License file   \t\n- fio_visualizer.py\n  - The frontend for this application\n- realtime_back.py\n  - The backend for this application\n- mainwindow.ui \n  - GUI definition part of the script\n- Workloads:\n  - Device\n    - Block device workloads covering typical specifications, separated in SATA and NVMe sections\n  - Precondition\n    - instruction and script files to prepare SSD for the testing.\n  - Simulation\n    - Workloads to simulate real applications.\n  - MultiDevice\n    - Workloads to test two or more drives at a time. Currently has limitted support due to improper handling of total performance in the script. \n\n# Setup\n\n## Installation\nRequires:\n\n   - FIO 2.15 or newer (2.1.x is no longer supported) found [here](https://github.com/axboe/fio/releases)\n   - PyQtGraph found [here](http://www.pyqtgraph.org/)\n   - Python 2.7 or 3+ [here](https://www.python.org/downloads/)\n   - PyQt 4.8+ or PySide\n   - numpy\n   - scipy\n\n#### CentOS 7 steps\n\n0. You should have python and PyQt installed with this distro\n\n1. Install pyqtgraph-develop (0.9.9 required) from [here](http://www.pyqtgraph.org)\n   - `$ python setup.py install`\n\n2. Install Cyphon from [here](http://cython.org) Version 0.21 or higher is required.\n   - `$ python setup.py install`\n\n3. Install Numpy from [here](http://numpy.org)\n   - `$ python setup.py build`\n   - `$ python setup.py install`\n\n4. Install FIO from [here](http://freecode.com/projects/fio)\n   - `# ./configure`\n   - `# make`\n   - `# make install`\n\n5. Run Visualizer as root.\n   - `# ./fio_visualizer.py`\n\n\n#### Ubuntu steps\n\n1. Add the following repository `deb http://luke.campagnola.me/debian dev/` into your `/etc/apt/sources.list` and then run `$ sudo apt-get install fio python-pyqtgraph`\n\n2. Install FIO from [here](http://freecode.com/projects/fio)\n   - `# ./configure`\n   - `# make`\n   - `# make install`\n\n3. Run Visualizer as root.\n   - `# ./fio_visualizer.py`\n\n# Development notes\n\n## Limitations\n1. Increased CPU load with numjobs \u003e 4 and all threads plotting. Avoid plotting all threads with numjobs \u003e 4 or disable unneccessary ones to save CPU resources.\n2. Multi-jobs configurations files (i.e. [job1]... [job2]) are not supported at the moment. The total amount of jobs if every workload has \"numjobs\" specified too are not correctly parsed. This can still be implemented manually specifing it in the code.\n\n## Todo\nContinue working on workload profiles for new Intel SSDs. \n\n## Maintainers\n   - Andrey Kudryavtsev andrey.o.kudryavtsev@intel.com\n   - Alexey Ponomarev alexey.ponomarev@intel.com\n\n## Realtime backend processor\n\nfiovisualizer implements its realtime backend processor on the file\nrealtime_back.py. This section documents the backend processor.\n\n### Starting fio\n\nThe routine start_fio() kicks off the fio process, and it\nkicks of a process which more or less looks something like the following:\n\n``` bash\nfio --minimal --eta=never --status-interval=1 path-to-fio.ini\n```\n### Realtime resolution\n\nSince `--status-interval=1` is currently used statically, it means a resolution\nof 1 second is used between updates.\n\n### Meaning of data being processed\n\nSince `--status-interval` is used the values being processed currently do not\nprovide per period measurements. The values being processed reflect the\noverall cumulative (from job start) values at the specified time intervals.\n\n#### fiovisualizer support for output format\n\nfio supports different output formats. The smallest and most compact output\nformat currently is the terse output, and so fiovisualizer uses this to minimize\nthe amount of processing done on the output and also to minimize the amount of\ndata being transfered from a remote fio server.\n\n##### fiovisualizer terse output format support\n\nfio supports different terse format versions, the format version can be\nspecified via the argument `--terse-version`. If no version is specified the\nversion used will depend on the fio release.\n\n##### fio terse parsing version notes\n\nWhen `--minimal` is used it is equivalent to using `--output-format=terse`.\n\nThe `--output-format` parameter was added via fio commit f3afa57e3 (\"Add\n--output-format command line option\") merged since fio-2.0.10.\n\nfiovisualizer started supporting fio since fio-2.1.14. The release of\nfio-2.1.14 uses by default the terse output format version 3. fiovisualizer's\nparser only currently supports terse output format version 3.\n\nThe `--terse-version` parameter was added to fio via commit f57a9c59e36\n(\"Add terse version output format command line parameter\"), this commit was\nmerged on the fio-1.58 release. This fio-1.58 release however used the terse\nversion format 2. Development for the terse version format 3 started\nsince fio commit 312b4af22018a (\"Add IOPS to terse output\") supported since\nfio-1.99.6.\n\nThe current default terse output format on the latest fio release is terse\nversion format 3.\n\n##### fio terse output format version 3\n\nBelow is a short description of each or the column outputs for fio terse\noutput version format 3.\n\n| 0 | terse_version_3 |\n| 1 | fio_version |\n| 2 | jobname |\n| 3 | groupid |\n| 4 | error |\n| 5 | read_kb |\n| 6 | read_bandwidth |\n| 7 | read_iops |\n| 8 | read_runtime_ms |\n| 9 | read_slat_min |\n| 10 | read_slat_max |\n| 11 | read_slat_mean |\n| 12 | read_slat_dev |\n| 13 | read_clat_min |\n| 14 | read_clat_max |\n| 15 | read_clat_mean |\n| 16 | read_clat_dev |\n| 17 | read_clat_pct01 |\n| 18 | read_clat_pct02 |\n| 19 | read_clat_pct03 |\n| 20 | read_clat_pct04 |\n| 21 | read_clat_pct05 |\n| 22 | read_clat_pct06 |\n| 23 | read_clat_pct07 |\n| 24 | read_clat_pct08 |\n| 25 | read_clat_pct09 |\n| 26 | read_clat_pct10 |\n| 27 | read_clat_pct11 |\n| 28 | read_clat_pct12 |\n| 29 | read_clat_pct13 |\n| 30 | read_clat_pct14 |\n| 31 | read_clat_pct15 |\n| 32 | read_clat_pct16 |\n| 33 | read_clat_pct17 |\n| 34 | read_clat_pct18 |\n| 35 | read_clat_pct19 |\n| 36 | read_clat_pct20 |\n| 37 | read_tlat_min |\n| 38 | read_lat_max |\n| 39 | read_lat_mean |\n| 40 | read_lat_dev |\n| 41 | read_bw_min |\n| 42 | read_bw_max |\n| 43 | read_bw_agg_pct |\n| 44 | read_bw_mean |\n| 45 | read_bw_dev |\n| 46 | write_kb |\n| 47 | write_bandwidth |\n| 48 | write_iops |\n| 49 | write_runtime_ms |\n| 50 | write_slat_min |\n| 51 | write_slat_max |\n| 52 | write_slat_mean |\n| 53 | write_slat_dev |\n| 54 | write_clat_min |\n| 55 | write_clat_max |\n| 56 | write_clat_mean |\n| 57 | write_clat_dev |\n| 58 | write_clat_pct01 |\n| 59 | write_clat_pct02 |\n| 60 | write_clat_pct03 |\n| 61 | write_clat_pct04 |\n| 62 | write_clat_pct05 |\n| 63 | write_clat_pct06 |\n| 64 | write_clat_pct07 |\n| 65 | write_clat_pct08 |\n| 66 | write_clat_pct09 |\n| 67 | write_clat_pct10 |\n| 68 | write_clat_pct11 |\n| 69 | write_clat_pct12 |\n| 70 | write_clat_pct13 |\n| 71 | write_clat_pct14 |\n| 72 | write_clat_pct15 |\n| 73 | write_clat_pct16 |\n| 74 | write_clat_pct17 |\n| 75 | write_clat_pct18 |\n| 76 | write_clat_pct19 |\n| 77 | write_clat_pct20 |\n| 78 | write_tlat_min |\n| 79 | write_lat_max |\n| 80 | write_lat_mean |\n| 81 | write_lat_dev |\n| 82 | write_bw_min |\n| 83 | write_bw_max |\n| 84 | write_bw_agg_pct |\n| 85 | write_bw_mean |\n| 86 | write_bw_dev |\n| 87 | cpu_user |\n| 88 | cpu_sys |\n| 89 | cpu_csw |\n| 90 | cpu_mjf |\n| 91 | cpu_minf |\n| 92 | iodepth_1 |\n| 93 | iodepth_2 |\n| 94 | iodepth_4 |\n| 95 | iodepth_8 |\n| 96 | iodepth_16 |\n| 97 | iodepth_32 |\n| 98 | iodepth_64 |\n| 99 | lat_2us |\n| 100 | lat_4us |\n| 101 | lat_10us |\n| 102 | lat_20us |\n| 103 | lat_50us |\n| 104 | lat_100us |\n| 105 | lat_250us |\n| 106 | lat_500us |\n| 107 | lat_750us |\n| 108 | lat_1000us |\n| 109 | lat_2ms |\n| 110 | lat_4ms |\n| 111 | lat_10ms |\n| 112 | lat_20ms |\n| 113 | lat_50ms |\n| 114 | lat_100ms |\n| 115 | lat_250ms |\n| 116 | lat_500ms |\n| 117 | lat_750ms |\n| 118 | lat_1000ms |\n| 119 | lat_2000ms |\n| 120 | lat_over_2000ms |\n| 121 | disk_name |\n| 122 | disk_read_iops |\n| 123 | disk_write_iops |\n| 124 | disk_read_merges |\n| 125 | disk_write_merges |\n| 126 | disk_read_ticks |\n| 127 | write_ticks |\n| 128 | disk_queue_time |\n| 129 | disk_util |\n\n### Processing\n\nBelow is an example output of one line out using:\n\n``` bash\nfio  --client=some-hostname --eta=never --status-interval=1 --terse-version=3 some.ini\n```\nThe output:\n\n```\n3;fio-3.8-41-g7302;job1;0;0;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;1886832;188645;23580;10002;1;87;2.561636;0.999742;6;6575;165.280554;282.700285;1.000000%=11;5.000000%=13;10.000000%=15;20.000000%=19;30.000000%=25;40.000000%=35;50.000000%=51;60.000000%=89;70.000000%=148;80.000000%=240;90.000000%=452;95.000000%=675;99.000000%=1286;99.500000%=1679;99.900000%=2736;99.950000%=3293;99.990000%=5013;0%=0;0%=0;0%=0;12;6577;167.875769;282.729144;138363;314224;25.026987%;188630.900000;35637.687291;5.179482%;9.399060%;151970;0;51;0.1%;0.1%;100.0%;0.0%;0.0%;0.0%;0.0%;0.00%;0.00%;0.01%;21.49%;27.82%;12.61%;18.75%;10.68%;4.67%;2.00%;1.66%;0.28%;0.03%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;nvme0n1;44;934217;0;0;9;152472;152997;99.24%\n```\n\nFrom the above fiovisualizer's realtime_back.py records the following 6 columns\nand graphs them, below listed from the respective value from the output above:\n\n| 6 | read_bandwidth | 0 |\n| 7 | read_iops | 0 |\n| 15 | read_clat_mean | 0.000000 |\n| 47 | write_bandwidth | | 188645 |\n| 48 | write_iops | 23580 |\n| 56 | write_clat_mean | 165.280554 |\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintel%2Ffiovisualizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintel%2Ffiovisualizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintel%2Ffiovisualizer/lists"}