{"id":18782140,"url":"https://github.com/gpuopen-librariesandsdks/directstoragesample","last_synced_at":"2025-07-27T20:07:31.340Z","repository":{"id":210969875,"uuid":"696830161","full_name":"GPUOpen-LibrariesAndSDKs/DirectStorageSample","owner":"GPUOpen-LibrariesAndSDKs","description":"Demonstrating the advantages of using DirectStorage over standard file I/O asset loading. Includes the API and changes required to make such a pipeline work.","archived":false,"fork":false,"pushed_at":"2023-11-14T16:50:10.000Z","size":4242,"stargazers_count":30,"open_issues_count":1,"forks_count":3,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-20T12:42:41.632Z","etag":null,"topics":["directstorage","dx12"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GPUOpen-LibrariesAndSDKs.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-26T14:07:27.000Z","updated_at":"2025-03-17T20:16:41.000Z","dependencies_parsed_at":"2023-12-05T20:36:48.915Z","dependency_job_id":null,"html_url":"https://github.com/GPUOpen-LibrariesAndSDKs/DirectStorageSample","commit_stats":null,"previous_names":["gpuopen-librariesandsdks/directstoragesample"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/GPUOpen-LibrariesAndSDKs/DirectStorageSample","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GPUOpen-LibrariesAndSDKs%2FDirectStorageSample","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GPUOpen-LibrariesAndSDKs%2FDirectStorageSample/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GPUOpen-LibrariesAndSDKs%2FDirectStorageSample/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GPUOpen-LibrariesAndSDKs%2FDirectStorageSample/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GPUOpen-LibrariesAndSDKs","download_url":"https://codeload.github.com/GPUOpen-LibrariesAndSDKs/DirectStorageSample/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GPUOpen-LibrariesAndSDKs%2FDirectStorageSample/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267417664,"owners_count":24083839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["directstorage","dx12"],"created_at":"2024-11-07T20:35:01.281Z","updated_at":"2025-07-27T20:07:31.319Z","avatar_url":"https://github.com/GPUOpen-LibrariesAndSDKs.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DirectStorageSample \n\nThis sample demonstrates the advantages of using DirectStorage instead of standard file I/O for loading assets. It shows the API and changes required to make such a pipeline work. \n\nThe following are high-level examples of what can be explored with this sample:\n- DirectStorage with compressed or uncompressed assets vs Win32 I/O with traditional run-time decompression\n- Compression Options: Compression levels, and decompression location (CPU vs GPU)\n- Bandwidth amplification affect\n- Performance impact on rendering when streaming in resources\n- Impact of cancelling requests when using DirectStorage to conserve bandwidth and processing cycles at the decompression location.\n\nThis sample provides profiling output [on-screen](#profiler-window-f2), through [CSV files](#built-in-profiling-output), and through PIX markers. \n\nAll of the above can be controlled and measurements can be taken via [command-line options](#command-line-options). Some of the controls are also available at run-time.\n\n\n# Screenshot\n\n![Animatedscreenshot](images/animatedscreenshot.webp)\n\n\n# How\nThe program uses a [list of assets and bounding volumes](src/Common/DirectStorageSample.json). When the camera enters a bounding volume, the asset associated with that bounding volume begins loading. The asset appears on screen as soon as loading is complete. When the camera exits the bounding volume, the asset disappears and it is unloaded.\n\nIf the camera exits a bounding volume for a given asset prior to loading, it will not become visible. The loading of an asset could continue until it is loaded even if it never becomes visible, or loading and associated requests can be cancelled so that loading never completes to conserves bandwidth and enable more relevant assets to load. Cancellation of a loading requests is supported via [command-line option](#allow-cancellation-allowcancellation).\n\n# Prerequisites\n\nTo build DirectStorageSample, you must first install the following tools:\n\n- [CMake 3.2x](https://cmake.org/download/)\n- [Visual Studio 2019/2022](https://visualstudio.microsoft.com/downloads/)\n- [Windows 10 SDK 10.0.18362.0](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk)\n\n# Simplified Build/Run Instructions\n## DirectStorage with Compressed Assets\n\n1. Run Build.bat\n2. Run BuildMediaCompressed.bat\n3. Run RunDirectStorage.bat\n\n## DirectStorage with Non-Compressed Assets\n\n1. Run Build.bat\n2. Run BuildMediaUncompressed.bat\n3. Run RunDirectStorage.bat\n\n## No DirectStorage\n\n1. Run Build.bat\n2. Run RunNoDirectStorage.bat\n\n\n# More Detailed Build Instructions\n\n## Code\n\nRun BuildCode.bat\n\nThis will create the sample solution and build the RelWithDebInfo configuration of the sample.\n\n## Assets\n\nRunning with DirectStorage requires pre-processed assets. The assets may or may not be compressed.\n\n- For compressed assets, run BuildMediaCompressed.bat.\n- For non-compressed assets, run BuildMediaUnCompressed.bat.\n\nThese scripts are examples of how to use [TextureConverter.exe](#textureconverterexe).\n\n# Running\n\n- RunDirectStorage.bat  - Runs DirectStorage with whatever assets were last built (compressed or uncompressed).\n\n- RunNoDirectStorage.bat - Runs without DirectStorage using standard compressed assets that are decoded on CPU.\n\nThese batch files are samples of how to run the sample. They can be used as-is or modified. They are currently used by profile.bat, which is an example of how to [automate profiling](#automated-profiling) the sample.\n\n# Notes\n\n- Only textures are processed by DirectStorage in this sample at this time.\n\n- The profiling window shows the load time of each asset that has been streamed in. Multiple assets may have the same name if they are used twice in the scene.  \n\n- At the moment, if you intend to test DirectStorage with and without compression, then you will need to re-run the appropriate BuildMedia*.bat file to produce either the compressed or uncompressed assets. Each time these batch files are run, they overwrite the prior version of the content.\n\n- The TextureConverter.exe program compresses assets in formats and compression levels it supports. See the [command-line options for TextureConverter.exe](#textureconverterexe) for options. Run it to see available options. Examples for running it are located in BuildMediaCompressed.bat and BuildMediaUncompressed.bat.\n\n\n# Command-line Options \n\n## DirectStorage_DX12.exe\nCommand-line options are formatted as JSON and are case-sensative.\n\nExample:\n\n`{\"directstorage\": true, \"profile\": true, \"profileOutputPath\":\"DSOn.csv\", \"iotiming\":true}`\n\n\n### DirectStorage Options\n---\n\n#### __Direct Storage On/Off (directstorage)__\n`{\"directstorage:\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhether or not to use DirectStorage to load assets. Compression for assets is determined by the TextureConverter utility and the options it is run with before the demo is run with this option set to true.\n\n#### __Placed Resources (placedresources)__\n\n`{\"placedresources\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhen true, placed resources will be used for DirectStorage loaded assets. When false, committed resources will be used instead.\n\n#### __Staging Buffer Size (stagingbuffersize)__    \n\n`{\"stagingbuffersize\":\u003csize in bytes\u003e}`\n\nDefault: 192MiB\n\nStaging buffer size should be set commensurate to assets, compression technique, compression ratio, performance goals, and memory constraints. The default is currently 192MiB to accommodate systems with a small amount of VRAM while still being capable of loading all sample assets in this demo. For a system with significant VRAM \u003e= 8GiB, a reasonable staging buffer size would be \u003e= 512MiB.\n\nExample: `{\"stagingbuffersize\":536870912}`\n\n#### __Queue Length (queuelength)__\n\nDefault: 128\n\nSet queue length to a reasonable size to allow auto-submit to prevent delaying DirectStorage requests from being submitted to the API. Auto-submit triggers at half queue length. A program could set queue length very large and call submit manually at its own defined rate, and should call submit once it knows outstanding requests are all submitted to the queue. If queue length is set too small, and the queue's capacity is reached, it will cause any subsequent direct storage requests to block until there's room in the queue for the new request.\n\n#### __Allow Cancellation (allowcancellation)__\n\n`{\"allowcancellation\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhen true, if the camera exits the streaming volume of an asset before the asset is loaded, then all DirectStorage requests associated with that asset's DirectStorage requests will attempt cancellation. When false, DirectStorage requests associated with an asset will continue, even after the camera has exited the streaming volume, and the asset hasn't fully loaded yet.\n\nThis option can conserve bandwidth, processing power, and increase the probability that other more recent load requests complete sooner. It can prevent the application from \"falling behind\" or processing data that is no longer relevant in the current view.\n\n#### __Disable GPU Decompression (disablegpudecompression)__\n\n`{\"disablegpudecompression\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhen true, and when using compressed assets with GPU-compatible compression algorithms, force DirectStorage to utilize the CPU instead of the GPU. This can be of benefit with very low-end GPUs if GPU decompression is slower than CPU decompression.\n\n\n#### __Disable MetaCommand (disablemetacommand)__\n\n`{\"disablemetacommand\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhen true, DirectStorage will use the reference implementation for GPU decompression rather than allow the device driver to select a hardware-specific optimized variant.\n\n### Workload Options\n---\n#### __Mandelbrot Iterations(mandelbrotiterations)__\n\n`{\"mandelbrotiterations\":\u003c0,500000\u003e}`\n\nDefault: 0\n\nGenerates a Mandelbrot fractal each frame to simulate an increased workload on the GPU. Higher values increase the workload. This is useful for measuring how GPU decompression interacts with greater compute or graphics workloads.\n\n### Profiling Options\n---\n#### __Profile (profile)__\n\n`{\"profile\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nEnabling profiling has very little impact on performance. This option saves out a CSV file that includes many run-time characteristics (load/streaming/run-time) of the application. The overhead is very low, and output is only saved when the application successfully closes.\n\n#### __Profile Output Path (profileOutputPath)__\n\n`{\"profileOutputPath\":\"\u003cfile path\u003e\"}`\n\nDefault: \"profilerOutput.csv\"\n\nName of the CSV file when the profile option is set to true. Useful for scripting several runs in with different configurations and options.\n\nExample: `{\"profileOutputPath\":\"DSOn.csv\"}`\n\n#### __Profile Seconds(profileseconds)__\n\n`{\"profileseconds\": \u003cseconds\u003e}`\n\nDefault: 0\n\nHow long to allow the program to execute before exiting. This is useful for scripting.\n\nExample: `{\"profileseconds\": 500}`\n\n#### __I/O Timing (iotiming)__\n\n`{\"iotiming\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhen using DirectStorage, attempt to time the time of first enqueue to final processing of DirectStorage from all enqueued assets through the pipeline. May not be completely accurate. This option does not enable io timing for assets loaded when DirectStorage is not enabled or for any other assets using traditional I/O routines.\n\n### Camera Options\n---\n#### __Camera Speed (cameraspeed)__\n\n`{\"cameraspeed\":\u003c[0.0,20.0]\u003e}`\n\nDefault: 1.0\n\nSpeed of camera movement through the bounding volumes when autopilot is enabled.\n\n#### __Camera Velocity Non-Linear (cameravelocitynonlinear)__\n\n`{\"cameravelocitynonlinear\":\u003ctrue|false\u003e}`\n\nDefault: false\n\nWhether to use ease-in/ease-out camera movement rather than constant velocity. For profiling, linear camera is more predictable for comparing results.\n\n\n## TextureConverter.exe\n\n---\n```\nUsage: TextureConverter.exe -configFile=\u003cpath to DirectStorageSample.json\u003e -compressionFormat=\u003cCompression Format\u003e [-compressionLevel=\u003cValid Compression Level\u003e] [-compressionExhaustive=\u003cfalse|true\u003e]\nCompression Formats:\n        none\n        gdeflate\n\nCompression Level:\n        default (balance between compression ratio and compression performance)\n        fastest (fastest to compress, but potentially lower compression ratio)\n        best (highest compression ratio)\n\nCompression Exhaustive:\n        false (use the compressionLevel and compressionFormat specified -- default)\n        true (Use the compression format and compression level with the best compression ratio. compressionLevel and compressionFormat specified are ignored)\n```\n\nExample 1 (Pre-process without compression): `bin\\TextureConverter.exe -configFile=bin\\DirectStorageSample.json -compressionFormat=none`\n\nExample 2 (Pre-process with default compression): `bin\\TextureConverter.exe -configFile=bin\\DirectStorageSample.json -compressionFormat=gdeflate -compressionLevel=default`\n\nExample 3 (Pre-process trying to find best compression level and format. *This is very slow*): `bin\\TextureConverter.exe -configFile=bin\\DirectStorageSample.json -compressionExhaustive`\n\n# Controls Window (F1)\n\n![Controls Window](images/controlswindowsmall.png)\n\n### Autopilot Checkbox\n\n- Checked - Camera movement is automatic.\n- Unchecked - Camera movement is free with WASD+Mouse Look.\n\n### Play Checkbox\n- Checked - Animations, not including camera, enabled.\n- Unchecked - Animations, not including camera, disabled.\n\n### Speed Slider\n\nControls speed of animations and camera movement.\n\n### Artificial Workload (Mandelbrot Iterations)\nAdjusts the number of Mandelbrot iterations for generating a Mandelbrot fractal used as an artificial workload to simulate increased rendering or compute workloads as the sample's own rendering workload may not be substantial enough to see the impact of GPU decompression on other GPU workloads.\n\n### Other Options\nOther controls are useful to change postprocessing, debug modes, and presentation modes.\n\n# Legend Window\nThe loading status of an asset is marked by the color of the bounding volume which the camera enters.\n\n![Legend Window](images/legendonly.png)\n## Run-time Asset States\nUnloaded - The asset glTF file is not loaded. Memory for the asset has not been allocated.\n\nLoaded - The asset is fully loaded. No additional loading operations are required.\n\nLoading - The request to load the asset has been made due to the camera entering the bounding volume. The program has started making requests to load the glTF file, allocate memory, load shaders, vertex buffers, index buffers, and textures.\n\nUnloading - The request to unload the asset has been made due to the camera exiting the bounding volume.\n\nCancelling  - When the [allow cancellation](#allow-cancellation-allowcancellation) command-line option is supplied, this signals that the loading request has been cancelled before it could be completed and is in the process of cancelling.\n\n# Profiler Window (F2)\n\n![Profiler Window](images/profilerwindowsmall.png)\n\nContains basic information about the system along with recent GPU timings and Scene Loading Timings.\n\n## Scene Loading Timings\n\n### Scene Name\n\nThe glTF\u0026trade; scene or file which has been loaded and is visible. Multiple scenes may be loaded as the same time.\n\n### CPU Load Time\n\nThe time, as measured by the CPU performance counter, from the moment the camera is inside the streaming volume to the moment the scene is loaded and about to become visible. \n\nWhen DirectStorage is enabled, some data is loaded through standard file I/O routines. This sample loads texture data with DirectStorage and all other data (glTF file and vertex data) using standard Win32 file I/O.\n\n### IoTime (DirectStorage only)\n\nIoTime measures the CPU time from the moment the first texture request is made to DirectStorage until the time transfer to the GPU is complete. This includes any decompression on the GPU or CPU by the DirectStorage API. \n\nWhen not using placed resources, this also includes the time it takes to perform memory allocation from CreateCommittedResource. The impact of memory allocations can be eliminated by enabling [placed resources](#placed-resources-placedresources).\n\nDirectStorage request enqueue time is also included in this metric. If the queue capacity is too small, then it could take a considerable amount of time and distort the results for IoTime. To see the impact of queue capacity on the IoTime measurement, adjust [queue length](#)\n\n### DirectStorage\n\nOn/Off - Indicates whether DirectStorage is in use or not. See command-line option documentation for details on how to enable/disable [DirectStorage](#direct-storage-onoff-directstorage).\n\n### Placed Resources (DirectStorage only)\n\nOn/Off - Indicates whether placed resources were used for the glTF scene's textures. See command-line option documentation for details on how to enable/disable [placed resources](#placed-resources-placedresources).\n\n### Texture File Size (DirectStorage only)\n\nThis size of the texture data stored on disk. If compression is enabled, then it will be the pre-processed compressed size; otherwise, it will be the pre-processed uncompressed size.\n\n### Avg Disk Only Data Rate (DirectStorage only)\n\n$Texture File Size / IoTime$. Requires i/o timing enabled. See the command-line option for [i/o timing](#io-timing-iotiming) for details on how to enable/disable seeing this value.\n\n### Avg Amplified Data Rate (DirectStorage only)\n\n$Uncompressed Data Size / IoTime$. If the assets were not compressed, then the Avg Amplified Data Rate is equal to the Avg Disk Only Data Rate. If the assets were compressed, the IoTime may be shorter, and the data rate is then going to be faster. Requires i/o timing enabled. See the command-line option for [i/o timing](#io-timing-iotiming) for details on how to enable/disable seeing this value.\n\n# Built-in Profiling Output (CSV)\n\nBuilt-in profiling output is useful for comparing results as the command-line options and hardware configurations are changed.\n\nWhen using the [Profile option](#profile-profile), a CSV file is saved in the bin directory containing the same statistics as the [Profile Window](#profiler-window-f2). However, it also includes additional information:\n\n## Mean Frame Time Before Loading (microseconds)\n\nThis is the average frame time of the last 60 frames before the asset started loading.\n\n## Mean Frame Time During Loading (microseconds)\n\nThis is the average frame time of all frames from the time of the loading request until loading completes.\n\n## Frame Count\n\nThe number of frames from loading request until loading completion.\n\n\n# Automated Profiling\n\nProfiling can be automated through batch files for comparing hardware and software configurations. [Profile.bat](profile.bat) is provided as an example, and it can be run as-is after running BuildCode.bat.\n\nThe defaults for this script do the following:\n1. Run msinfo32.exe and dxdiag.exe and place their output into text files in the bin folder named with timestamps.\n2. Pre-process assets *without* compression enabled.\n3. Run the sample without DirectStorage enabled twice.\n4. Run the sample with DirectStorage enabled twice.\n5. Pre-process assets *with* compression enabled.\n6. Run the sample with DirectStorage enabled twice. Now, with compressed assets.\n7. Run the sample with DirectStorage enabled twice, but using the CPU for decompression rather than the GPU.\n\nThe script overrides the default staging buffer size, sets the run-time of each run to 500 seconds, sets the presentation mode to 0 (windowed), and the camera speed to half.\n\nAs configured, the script takes a bit over $8(runs) * 500(seconds)=66.67 minutes$ to complete. The results are stored in timestamped and named CSV files. These files can be opened or processed by your choice of program as they are [RFC 4180-compliant](!https://datatracker.ietf.org/doc/html/rfc4180).\n\n\n# Performance Discussion\n\nFor a more thorough discussion about DirectStorage, its benefits and performance, see our GDC talk about it at https://www.youtube.com/watch?v=LvYUmVtOMRU.\n\n## Overall Loading Performance\n\nLoading performance is greatly influenced by several factors. For most current high-end systems, when only considering hardware, the bottlenecks occur at the storage device or the decompression device. However, software must keep these devices fed with data in order to realize these constraints.\n\nMost modern storage devices have difficulty saturating the bus interface to the GPU as GPUs are usually attached to the bus with greater throughput than storage devices. For example, the storage device may be attached to the bus with a throughput of ~7.8GiB/s while the GPU is connected to the bus with a potential ~31GiB/s.\n\nGiven that the storage device is frequently the first location of limitation, keeping the data in compressed form enables the transfer of more data than would be possible if the data were uncompressed at the expense of the computation of decompression later on.\n\n## Choice of Decompression Location\nDecompression performance depends on the resources available on the chosen decompression device.\n\n### CPU Decompression\n\nDecompressing the data on the CPU uses CPU cycles and requires increased PCIe bus usage between the CPU and GPU. CPU decompression is the current norm for asset decompression. However, on most systems, the CPU is already busy with other workloads, processing the scene, generating rendering command-lists, game code, collision, etc. Streaming in assets seamlessly causes interruptions in processing other places, and could be difficult to schedule if there aren't enough resources available.\n\n### GPU Decompression\n\nThe transfer from CPU to GPU will take less time as the data will be in compressed form. Decompressing data on the GPU requires more GPU cycles which could interrupt rendering frame rate. However, on most current mid-range and high-end systems, the difference in frame rate during decompression may have a lower impact than using the CPU for decompression. Furthermore, bandwidth from CPU to GPU is conserved. Decompressing data where it will be used increases overall efficiency. Also, because the data can be broken into several streams of data that may be decompressed in parallel, we can exploit the parallelism of the GPU to decompress the data.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpuopen-librariesandsdks%2Fdirectstoragesample","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpuopen-librariesandsdks%2Fdirectstoragesample","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpuopen-librariesandsdks%2Fdirectstoragesample/lists"}