{"id":21822015,"url":"https://github.com/cea-hpc/pdwfs","last_synced_at":"2025-04-14T03:52:06.135Z","repository":{"id":64305922,"uuid":"175089510","full_name":"cea-hpc/pdwfs","owner":"cea-hpc","description":"A simple Redis-backed distributed virtual filesystem for co-execution of HPC and data analytics workloads","archived":false,"fork":false,"pushed_at":"2020-01-23T13:39:14.000Z","size":8880,"stargazers_count":14,"open_issues_count":4,"forks_count":1,"subscribers_count":10,"default_branch":"develop","last_synced_at":"2025-04-14T03:11:53.038Z","etag":null,"topics":["c","cea","go","golang","hpc","in-situ","in-transit","redis","workflow"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cea-hpc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-11T21:53:47.000Z","updated_at":"2024-02-23T19:58:13.000Z","dependencies_parsed_at":"2023-01-15T10:30:17.106Z","dependency_job_id":null,"html_url":"https://github.com/cea-hpc/pdwfs","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cea-hpc%2Fpdwfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cea-hpc%2Fpdwfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cea-hpc%2Fpdwfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cea-hpc%2Fpdwfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cea-hpc","download_url":"https://codeload.github.com/cea-hpc/pdwfs/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248819354,"owners_count":21166474,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","cea","go","golang","hpc","in-situ","in-transit","redis","workflow"],"created_at":"2024-11-27T17:12:20.904Z","updated_at":"2025-04-14T03:52:06.108Z","avatar_url":"https://github.com/cea-hpc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pdwfs\n\n[![Build Status](https://travis-ci.com/cea-hpc/pdwfs.svg?branch=develop)](https://travis-ci.com/cea-hpc/pdwfs)\n\npdwfs (we like to pronounce it \"*padawan-f-s*\", see [below](#padawan-project)) is a preload library implementing a simplified distributed in-memory filesystem in user space suitable for intercepting *bulk* I/O workloads typical of HPC simulations. It is using [Redis](https://redis.io) as the backend memory store.\n\npdwfs objective is to provide a user-friendly and lightweight software-defined infrastructure to execute HPC simulation workflows without writing/reading any intermediate data to/from a (parallel) filesystem. Instead, data is staged in memory. This type of approach is known as *in transit* or *loosely-coupled in situ* and is further explained in  a [section](#in-situ-and-in-transit-hpc-workflows) below.\n\npdwfs is written in [Go](https://golang.org) and C and runs on Linux systems only (we provide a Dockerfile for testing and development on other systems).\n\n## Content\n\n- [Dependencies](##Dependencies)\n- [Installation](##Installation)\n- [Testing](##Testing)\n- [Quick start](##Quick-start)\n- [Running pdwfs with SLURM](##Running-pdwfs-with-slurm)\n- [How does it work ?](##How-does-it-work-?)\n- [Performance](##Performance)\n- [Validation](##Validation)\n- [Examples](##Examples)\n- [Known limitations and troubleshooting](##Known-limitations-and-troubleshooting)\n- [PaDaWAn project](##PaDaWAn-project)\n- [In situ and in transit HPC workflows](##In-situ-and-in-transit-HPC-workflows)\n- [License](##License)\n- [Acknowledgements](##Acknowledgements)\n\n## Dependencies\n\npdwfs only dependencies are:\n\nBuild:\n- Go version ≥ 1.11\n\nRuntime (and testing):\n- Redis (version ≥ 5.0.3 is recommended)\n\n\n## Installation\n\n### From binary distribution\n\nA binary distribution is available for Linux system and x86_64 architecture in the [releases](http://github.com/cea-hpc/pdwfs/releases) page.\n\nThe following steps will install pdwfs and make it available in your PATH.\n\n```bash\n$ wget -O pdwfs.tar.gz https://github.com/cea-hpc/pdwfs/releases/download/v0.2.1/pdwfs-v0.2.1-linux-amd64.tar.gz\n$ mkdir /usr/local/pdwfs\n$ tar xf pdwfs.tar.gz --strip-component=1 -C /usr/local/pdwfs\n$ export PATH=\"/usr/local/pdwfs/bin:$PATH\" \n```\n\n### From source\nA Go distribution (version ≥ 1.11) is required to build pdwfs from source (instructions can be found on the [Go download page](https://golang.org/dl/)).\n\nTo build pdwfs develop branch (default branch):\n```bash\n$ git clone https://github.com/cea-hpc/pdwfs\n$ cd pdwfs\n$ make PREFIX=/usr/local/pdwfs install\n```\n\n### Using Spack\n\npdwfs and its dependencies (Redis and Go) can be installed with the package manager [Spack](https://spack.io).\n\nNOTE: at the time of this writing, the latest Spack release (v0.12.1) does not have the Redis package, it is only available in the develop branch. Still, the Redis package python file can easily be copy-pasted in your Spack installation.\n\nA Spack package for pdwfs is not yet available in Spack upstream repository, but is available [here](https://github.com/cea-hpc/pdwfs/releases/download/v0.2.1/pdwfs-spack.py).\n\nTo add pdwfs package to your Spack installation, you can proceed as follows:\n\n```bash\n$ export DIR=$SPACK_ROOT/var/spack/repos/builtin/pdwfs\n$ mkdir $DIR\n$ wget -O $DIR/package.py https://github.com/cea-hpc/pdwfs/releases/download/v0.2.1/pdwfs-spack.py\n$ spack spec pdwfs  # check everything is OK\n```\n\n## Testing\n\nTo run the test suite, you will need a Redis installation and make sure ```redis-server``` and ```redis-cli``` binaries are available in your PATH. Then:\n\n```bash\n$ make test\n```\n\nTo allow development and testing on a different OS (e.g. MacOS), we provide a development Dockerfile based on an official CentOS image from DockerHub. To build and run the container:\n\n```bash\n$ make -C docker run\n```\n\nThe working directory in the container is a mounted volume on the pdwfs repo on your host, so to build pdwfs inside the container, just use the Makefile as previously described (ie. ```make test```).\n\nNOTE: if you encounter permission denied issue when building pdwfs in the container that's probably because the non-root user and group IDs set in the Dockerfile do not match your UID and GID. Change the UID and GID values to yours in the Dockerfile and re-run the above command.\n\n## Quick start\n\nFirst, start a default Redis instance in the background.\n\n```bash\n$ redis-server --daemonize yes\n``` \n\nThen, assuming your simulation will write its data into the ```output``` directory, simply wrap the execution command of your simulation with pdwfs command-line script like this:\n\n```bash\n$ pdwfs -p output/ -- your_simulation_command\n```\nThat's it ! pdwfs will transparently intercept low-level I/O calls (open, write, read, ...) on any file/directory within the output/ directory and send data to Redis, no data will be written on disk.\n\nTo process the simulation data, just run your processing tool the same way:\n\n```bash\n$ pdwfs -p output/ -- your_processing_command\n```\n\nTo see the data staged within Redis (keys only) and check the memory used (and to give you a hint at how nice Redis is):\n\n```bash\n$ redis-cli keys *\n...\n$ redis-cli info memory\n...\n```\n\nFinally, to stop Redis (and discard all data staged in memory !):\n\n```bash\n$ redis-cli shutdown\n```\n\n## Running pdwfs with SLURM\n\npdwfs comes with a specialized CLI tool called ```pdwfs-slurm``` that simplifies the deployment of Redis instances in a SLURM job.\n\n```pdwfs-slurm``` has two subcommands: ```init``` and ```finalize```. \n\nThe following script illustrates the use of pdwfs and ```pdwfs-slurm``` to run the I/O benchmarking tool [ior](https://github.com/hpc/ior) (the script is available in examples/ior_slurm). The script assumes that pdwfs, Redis and ior are installed and available in the PATH.\n\nior_pdwfs.sh:\n```bash\n#!/bin/bash\n# SBATCH --job-name=pdwfs\n# SBATCH --nodes=12\n# SBATCH --exclusive\n\n# Initialize the Redis instances:\n# - 32 instances distributed on 4 nodes (8 per node)\n# - bind Redis servers to ib0 network interface\npdwfs-slurm init -N 4 -n 8 -i ib0\n\n# pdwfs-slurm produces a session file with some environment variables to source\nsource pdwfs.session\n\n# ior command will use MPI-IO in collective mode with data blocks of 100 MBytes\nIOR_CMD=\"ior -a MPIIO -c -t 100m -b 100m -o $SCRATCHDIR/testFile\"\n\n# pdwfs command will forward all I/O in $SCRATCHDIR in Redis instances\nWITH_PDWFS=\"pdwfs -p $SCRATCHDIR\"\n\n# Execute ior benchmark on 128 tasks\nsrun -N 8 --ntasks-per-node 16 $WITH_PDWFS $IOR_CMD\n\n# gracefully shuts down Redis instances\npdwfs-slurm finalize\n\n# pdwfs-slurm uses srun in background to execute Redis instances\n# wait for background srun to complete\nwait\n```\n\nThis script can be run with SLURM ```sbatch``` or with ```salloc``` as follows:\n```bash\n$ salloc -N 12 --exclusive ./ior_pdwfs.sh\n```\n\n## How does it work ?\n\npdwfs uses LD_PRELOAD to intercept a set of I/O-related function calls provided by the C standard library (libc).\n\nThe pdwfs CLI script execute the user command passed in argument with the LD_PRELOAD environment variable set to the installation path of the pdwfs.so shared library.\n\nCurrently about 90 libc I/O calls are intercepted but only 50 of them are currently implemented by pdwfs. If your program uses an I/O call intercepted but not implemented, it will raise an error. In this case, please file an [issue](https://github.com/cea-hpc/pdwfs/issues) (or send a pull request!).\n\nThe category of calls currently implemented are:\n- basic POSIX and C standard I/O calls (open, close, read, write, lseek, access, unlink, stat, fopen, fread, fprintf, ...)\n- additional I/O calls used by MPI-IO and Parallel HDF5 libraries (writev, pwrite, pwritev, readv, pread, preadv, statfs, ...)\n\n## Performance\n\nTo address the challenge of being competitive with parallel filesystems, an initial set of design choices and trade-offs have been made:\n- drasticaly limiting the amount of metadata and metadata manipulations by opting not to implement typical filesystem features such as linking, renaming, access rights, timestamping, etc.\n- no central metadata server, metadata are distributed accross Redis instances,\n- selecting the widely used database Redis to benefit from its mix of performance, simplicity and flexibility,\n- files are stripped accross multiple Redis instances with a predefined layout (by configuration), no metadata query for read/write,\n- being a simple software-defined infrastructure in user space allows adjustement and configuration on a per-simulation or per-workflow basis for increased efficiency.\n\nWith this set of choices, we expect our infrastructure to be horizontally scalable (adding more Redis instances to accomodate higher loads) and to accomodate certain I/O loads that are known to be detrimental for parallel filesystem (many files).\n\nOn the other hand, a few factors are known for impacting performance versus parallel filesystems:\n- Redis uses TCP communications while parallel filesystems rely on lower-level communication protocols like RDMA,\n- intercepting I/O calls and the use of CGO (system to call Go from C) adds some overhead to the calls,\n\nObvisouly, proper benchmarking at scale will have to be performed to assess pdwfs performances. Yet, considering these design choices and our past experience with PaDaWAn in designing in transit infrastructure, we are hopefull that we will get decent performances.\n\n\n## Validation\n\nTest cases have been successfully run so far with the followig codes and tools:\n- [IOR](https://github.com/hpc/ior) benchmark with POSIX, parallel HDF5 and MPI-IO methods  (OpenMPI v2),\n- [HydroC](https://github.com/HydroBench/Hydro) a 2D structured hydrodynamic mini-app using POSIX calls to produce VTK files,\n- [ParaView](https://www.paraview.org/in-situ/) VTK file reader.\n\n\n## Examples\nWe provide a set of Dockerfiles to test on a laptop the codes and tools described in the Validation section.\n\n- **Example 1**: HydroC + ParaView + FFmpeg workflow\n\nCheck the README in the example/HydroC_ParaView directory or just go ahead and type:\n```\n$ make -C examples/HydroC_ParaView run\n```\nYou can go grab some coffee, building the container takes a while... \n\n- **Example 2**: IOR benchmark\n\nAgain, check the README in the corresponding directory or go ahead and type:\n```\n$ make -C examples/IOR_benchmark run\n```\nYep, you can go grab a second coffee...\n\n## Known limitations and troubleshooting\n\n- Works only for dynamically linked executables,\n- Most core or shell utilities for file manipulations (e.g. ls, rm, redirections) will not work through pdwfs as they require particular libc calls not yet implemented,\n- Issues with Intel Fortran compiler have been observed, for the moment pdwfs is more reliable with GNU Fortran compiler\n- In Fortran, opened files should be explicitely closed in the program (at exit closing operations currently hang the process)\n\n\n## PaDaWAn project\n\npdwfs is a component of the PaDaWAn project (for Parallel Data Workflow for Analysis), a [CEA](http://www.cea.fr) project that aims at providing building blocks of a lightweight and *least*-intrusive software infrastructure to facilitate *in transit* execution of HPC simulation workflows. \n\nThe foundational work for this project was an initial version of pdwfs entierly written in Python and presented in the paper below:\n\n- *PaDaWAn: a Python Infrastructure for Loosely-Coupled In Situ Workflows*, J. Capul, S. Morais, J-B. Lekien, ISAV@SC (2018).\n\n## In situ and in transit HPC workflows\nWithin the HPC community, in situ data processing is getting quite some interests as a potential enabler for future exascale-era simulations. \n\nThe original in situ approach, also called tightly-coupled in situ, consists in executing data processing routines within the same address space as the simulation and sharing the resources with it. It requires the simulation to use a dedicated API and to link against a library embedding a processing runtime. Notable in situ frameworks are ParaView [Catalyst](https://www.paraview.org/in-situ/), VisIt [LibSim](https://wci.llnl.gov/simulation/computer-codes/visit). [SENSEI](http://sensei-insitu.org) provides a common API that can map to various in situ processing backends.\n\nThe loosely-coupled flavor of in situ, or in transit, relies on separate resources from the simulation to stage and/or process data. It requires a dedicated distributed infrastructure to extract data from the simulation and send it to a staging area or directly to consumers. Compared to the tightly-coupled in situ approach, it offers greater  flexibility to adjust the resources needed by each application in the workflow (not bound to efficiently use the same resources as the simulation). It can also accommodate a larger variety of workflows, in particular those requiring memory space for data windowing (e.g. statistics, time integration).\n\nThis latter approach, loosely-coupled in situ or in transit, is at the core of pdwfs. \n\n\n## License\n\npdwfs is distributed under the Apache License (Version 2.0).\n\n## Acknowledgements\n\nThis work was conducted in collaboration with [CINES](http://www.cines.fr).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcea-hpc%2Fpdwfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcea-hpc%2Fpdwfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcea-hpc%2Fpdwfs/lists"}