{"id":18421992,"url":"https://github.com/spcl/fuzzyflow-artifact","last_synced_at":"2025-04-13T12:11:23.294Z","repository":{"id":185069159,"uuid":"627303791","full_name":"spcl/fuzzyflow-artifact","owner":"spcl","description":"Computational artifacts for the FuzzyFlow publication","archived":false,"fork":false,"pushed_at":"2023-07-31T14:51:32.000Z","size":1372,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-02-10T00:57:55.493Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spcl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-13T07:32:57.000Z","updated_at":"2024-10-02T14:26:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"9c04b0bd-cbf3-4788-aac6-fcbac2469303","html_url":"https://github.com/spcl/fuzzyflow-artifact","commit_stats":null,"previous_names":["spcl/fuzzyflow-artifact"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Ffuzzyflow-artifact","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Ffuzzyflow-artifact/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Ffuzzyflow-artifact/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Ffuzzyflow-artifact/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spcl","download_url":"https://codeload.github.com/spcl/fuzzyflow-artifact/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248710445,"owners_count":21149190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T04:27:38.593Z","updated_at":"2025-04-13T12:11:23.273Z","avatar_url":"https://github.com/spcl.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Computational Artifact for FuzzyFlow\nThis repository contains everythin necessary to reproduce the results reported\nin the paper **\"FuzzyFlow: Leveraging Dataflow To Find and Squash Program\nOptimization Bugs\"**.\n\nThe artifact can be run containerized with all necessary dependencies and\nenvironment setup through a provided Docker image.\nYou can find setup instructions and a guide on how to reproduce results in the\nsections [Setup](#setup) and [Reproducing Results](#reproducing-results) below.\nThe experiment workflows themselves are described in\n[Experiment Workflows](#experiment-workflows).\n\n## Experiment Workflows\n### Workflow 1 (Paper Section 6.1)\nThis workflow reproduces the case study 'Minimizing Input Configurations'.\nIn this case study we test a series of loop vectorization optimizations\n(built-in optimizations in the DaCe framework) on the encoder layer from the\nnatural language model (Transformer) BERT. The input parameters used in the\npaper are as follows:\n- N = 1024\n- B = 8\n- H = 16\n- SM = 512\n- emb = 4096\n- P = 64\n\nWe report the following insights in the paper:\n1. The full application takes **12.1 seconds to run** with the listed input\nparameters while accelerating BLAS operations with Intel MKL (median over 10 runs).\n2. For a specific instance of the loop vectorization optimization, FuzzyFlow is\nable to **reduce the input configuration by 75%**.\n3. With the extracted and minimized cutout, AFL++ is able to run an average of\n**43.7 fuzzing trials per second** (tps).\n4. Considering the original application runtime of 12.1 seconds, which would lead\nto 0.083 tps, this corresponds to around **528 times faster testing**.\n\nA single Bash script in the provided artifact can be used to reproduce each of\nthese reported numbers. The script first runs the full application 10 times and\nreports the median runtime (1). The script then extracts and minimizes a cutout\nfor the reported vectorization optimization, reporting the input configuration\nsize before and after minimization together with a number indicating by how many\npercent the configuration was reduced (2). Finally, the script uses AFL++ to\nfuzz the extracted cutout and reports the number of fuzzing trials per second\nachieved (3 and 4).\n\n### Workflow 2 (Paper Section 6.2)\nThis workflow reproduces the case study 'From Multi-Node to Single-Node'. In\nthis case study, we demonstrate how using the cutout extraction method employed by\nFuzzyFlow, testing program transformations in a distributed, multi-node setting,\nthe resulting test cases can effectively reduce tests to single-node problems,\neliminating the need for multiple nodes and communication. Specifically, we\nextract a test case for an optimization of the Sampled Dense-Dense Matrix\nMultiplication (SDDMM) performed by forward propagation in a vanilla attention\ngraph neural network, designed to run in a distributed setting using MPI. Since\nthe optimization does not make any changes to MPI communication calls, the\nresulting test case can be run without communication on a single rank/node.\n\nA Bash script in the provided artifact automatically extracts a test case for\nthe discussed optimization and generates the corresponding C++ code. This can\nbe used to verify that the generated C++ code does not contain any MPI\ncommunication and may be run on a single node, as shown in the (simplified)\ncode from Figure 6 in the paper.\n\n### Workflow 3 (Paper Section 6.3)\nThis workflow reproduces the case study 'NPBench'. In this case study we use a\nset of micro-benchmarks (52) from the benchmark suite [NPBench](https://github.com/spcl/npbench)\nto test all of the built-in optimizations that the DaCe framework can apply to\nthose programs. Some types of transformations that require special hardware like\nFPGAs were omitted. We use FuzzyFlow to test a total of **3,280 transformation instances**\non the entire benchmark set. We report **6 transformations** from the DaCe\nframework that contain bugs according to the resulting failing test cases.\n\nA Bash script in the provided artifact can be used to run FuzzyFlow on the\ntested benchmarks and DaCe transformations. This generates all 3,280 resulting\ntest cases and tests them with differential fuzzing. All failing test cases are\nreported and through performing a manual triage the 6 transformation bugs\nreported in Table 2 of Section 6.3 can be found and reconstructed.\n\n### Workflow 4 (Paper Section 6.4)\nThis workflow reproduces the case study 'Optimizing Weather Forecasts'. In this\ncase study we test 2 custom transformations and one built-in optimization from\nthe DaCe framework on a real-world application. The custom transformations were\ntaken from a group of engineers that were in the process of optimizing the cloud\nmicrophysics scheme (CLOUDSC) of the European Centre for Medium-Range Weather\nForecasts' (ECMWF) Integrated Forecasting System (IFS). The engineers wrote\ncustom transformations to automatically extract GPU kernels and to perform loop\nunrolling. We use FuzzyFlow to test each instance of these custom transformations\non the CLOUDSC application - specifically on different versions of CLOUDSC\ntaken from different stages of their optimization workflow. Similarly, we test\neach instance of a built-in optimization from the DaCe framework that tries to\neliminiate intermediate writes on CLOUDSC.\n\nWe report the following insights in the paper:\n1. 62 instances of the GPU kernel extraction transformation were tested,\n**48 of which** changed program semantics.\n2. Testing a single instance of a GPU kernel extraction transformation took only\n**43 seconds**\n3. Most semantics altering transformation instances of the GPU kernel extraction\nwere uncovered after only 1 or 2 fuzzing trials.\n4. 19 instances of the loop unrolling transformation were tested, **one of which**\nchanged program semantics.\n5. 136 instances of the built-in write elimination transformation were tested,\n**one of which** changed program semantics.\n\nA Bash script in the provided artifact can be used to run FuzzyFlow on CLOUDSC\nfor all three of the listed transformations, reproducing each of these insights.\nThe script first tests the GPU kernel extraction transformation on CLOUDSC,\nextracting a minimal test case and demonstrating that finding a single bug only\ntakes 43 seconds (2). The script then tests each instance of the loop unrolling\ntransformation a version of CLOUDSC, demonstrating that from all tested 19\ninstances, the failing test case can be found (after manual triage) (4).\nFinally, the script tests each instance of the write elimination transformation\non two versions of CLOUDSC. This demonstrates that all 136 instances are quickly\ntested individually, each with a minimal test case. Out of all tested instances,\na few test cases fail testing and after performing triage, the bug can be\nidentified and reconstructed (5).\n\n## Setup\nTo build and deploy the provided Docker image, run through the following steps:\n1. Clonse the repository and navigate into it with your terminal. Ensure all\n    submodules are cloned by running `git submodule update --init --recursive`.\n2. Build the Docker image by running `docker build . -t fuzzyflow`.\n3. Launch the image in an Docker container with an interactive shell using:\n    `docker run --rm -it fuzzyflow`. (`--rm` removes the container again after\n    exiting. If you wish for it to persist, omit this option.)\n\n## Reproducing Results\nThe results in the paper may be reproduced by running the four provided\n[experiment workflows](#experiment-workflows).\nEach workflow can be launched through a single Bash script:\n1. To run workflow 1, reproducing Paper Section 6.1, run `./run_workflow_01.sh`\ninside the artifact's Docker container.\n2. To run workflow 2, reproducing Paper Section 6.2, run `./run_workflow_02.sh`\ninside the artifact's Docker container.\n3. To run workflow 3, reproducing Paper Section 6.3, run `./run_workflow_03.sh`\ninside the artifact's Docker container.\n4. To run workflow 4, reproducing Paper Section 6.4, run `./run_workflow_04.sh`\ninside the artifact's Docker container.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspcl%2Ffuzzyflow-artifact","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspcl%2Ffuzzyflow-artifact","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspcl%2Ffuzzyflow-artifact/lists"}