{"id":13481838,"url":"https://github.com/lsds/Saber","last_synced_at":"2025-03-27T12:31:58.067Z","repository":{"id":43807891,"uuid":"61813979","full_name":"lsds/Saber","owner":"lsds","description":"Window-Based Hybrid CPU/GPU Stream Processing Engine","archived":false,"fork":false,"pushed_at":"2022-11-16T09:24:56.000Z","size":70890,"stargazers_count":38,"open_issues_count":10,"forks_count":13,"subscribers_count":5,"default_branch":"sigmod16-reproducibility","last_synced_at":"2024-10-30T15:51:12.269Z","etag":null,"topics":["gpu","high-throughput","hybrid","multicore","multicore-cpu","saber","sliding-windows","stream","stream-processing","streaming","streaming-data"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lsds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-23T14:59:25.000Z","updated_at":"2024-03-31T14:17:58.000Z","dependencies_parsed_at":"2023-01-22T00:49:46.373Z","dependency_job_id":null,"html_url":"https://github.com/lsds/Saber","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsds%2FSaber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsds%2FSaber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsds%2FSaber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lsds%2FSaber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lsds","download_url":"https://codeload.github.com/lsds/Saber/tar.gz/refs/heads/sigmod16-reproducibility","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245845100,"owners_count":20681835,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu","high-throughput","hybrid","multicore","multicore-cpu","saber","sliding-windows","stream","stream-processing","streaming","streaming-data"],"created_at":"2024-07-31T17:00:56.491Z","updated_at":"2025-03-27T12:31:53.054Z","avatar_url":"https://github.com/lsds.png","language":"Java","readme":"# Readme for reproducibility submission of paper ID 195\n\n### Paper title \n\nSABER: Window-based hybrid stream processing for heterogeneous architectures\n\n### Authors\n\nAlexandros Koliousis, Matthias Weildich, Raul Castro Fernandez, Alexander L. Wolf, Paolo Costa, and Peter Pietzuch \n\n### Permissions\n\nOwner of artifact grants ACM permission to serve the artifact to users of the ACM Digital Library.\n\n## A) Source code info\n\nThe Saber source code is available to download from:\n\n[https://github.com/lsds/saber/tree/sigmod16-reproducibility](https://github.com/lsds/saber/tree/sigmod16-reproducibility)\n\n### Programming Language \u0026 Compiler Info\n\nSaber has been implemented in Java and C. The Java code is compiled and packaged using Apache Maven (3.3.1) and the Java SDK (1.7.0.79). The C code is compiled and packaged using GNU make (3.81) and gcc (4.8.4).\n\nThe CPU and GPU query operators have been implemented in Java and OpenCL, respectively. The GPU OpenCL operators are compiled _just-in-time_, when an operator is instantiated by a Saber application.\n\n### Packages \u0026 Libraries Needed\n\nThe system has been mainly tested on **Ubuntu Linux 14.04**. The `prepare-software.sh` script uses APT to handle the installation of these dependencies.\n\nEssential commands and their corresponding Ubuntu packages are: `javac` and `java` (openjdk-7-jdk), `mvn` (maven), `make` (make), `gcc` (gcc), and `python` (python).\n\nOptional commands and their packages are `epstopdf` (texlive-font-utils), and `gnuplot` (gnuplot). These are used to replot figures from any reproduced results.\n\n#### The libOpenCL.so library\n\nSaber requires libOpenCL.so to be present on the OS library path (`LD_LIBRARY_PATH`). In our installation, it is located under `/usr/local/cuda/lib64/`. The OpenCL headers are located under `/usr/local/cuda/include`.\n\n## B) Hardware Info\n\nWe performed our experiments on an HP Z840 Workstation.\n\n### Processors\n\nOur machine has 2 Intel Xeon CPU E5-2640 v3 @ 2.60GHz. There are 8 CPU cores per socket. Hyper-threading is disabled. The L1d (L1i), L2, and L3 cache sizes are 32 KB, 256 KB, and 20 MB, respectively.\n\n### Memory \n\nOur machine has a total of 64 GB of DDR4 SDRAM, spread equally across 8 CPU DIMM slots. Clock frequency is 2,133 MHz.\n\n### GPU\n\nThe GPU used in our experiments is an NVIDIA Quadro K5200 with 2,304 CUDA cores and 8,123 MB of RAM, attached to the host via PCIe 3.0 x16. The GPU has **two copy engines**, allowing for concurrent copy and kernel execution.\n\n```\nDevice 0: \"Quadro K5200\"  \n  CUDA Driver Version / Runtime Version          8.0 / 7.0  \n  CUDA Capability Major/Minor version number:    3.5  \n  Total amount of global memory:                 8123 MBytes (8517124096 bytes)  \n  (12) Multiprocessors, (192) CUDA Cores/MP:     2304 CUDA Cores  \n  GPU Max Clock rate:                            771 MHz (0.77 GHz)  \n  Memory Clock rate:                             3004 Mhz  \n  Memory Bus Width:                              256-bit  \n  L2 Cache Size:                                 1048576 bytes  \n  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)  \n  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers  \n  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers  \n  Total amount of constant memory:               65536 bytes  \n  Total amount of shared memory per block:       49152 bytes  \n  Total number of registers available per block: 65536  \n  Warp size:                                     32  \n  Maximum number of threads per multiprocessor:  2048  \n  Maximum number of threads per block:           1024  \n  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)  \n  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)  \n  Maximum memory pitch:                          2147483647 bytes  \n  Texture alignment:                             512 bytes  \n  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)  \n  Run time limit on kernels:                     No  \n  Integrated GPU sharing Host Memory:            No  \n  Support host page-locked memory mapping:       Yes  \n  Alignment requirement for Surfaces:            Yes  \n  Device has ECC support:                        Disabled  \n  Device supports Unified Addressing (UVA):      Yes  \n  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0  \n```\n\n## C) Experimentation Info\n\n### Installing the system\n\nThe `prepare-software.sh` script will guide you through the installation and compilation process of our system.\n\n```\n$ git clone http://github.com/lsds/saber.git saber.git\n$ cd saber.git\n$ git checkout --track origin/sigmod16-reproducibility\n$ export SABER_HOME=`pwd`\n$ ./scripts/prepare-software.sh\n```\n\n### Executing the experiments\n\nTo execute all available experiments and recompile the paper, run:\n\n```\n$ ./scripts/run-experiments.sh\n```\n\nTo reproduce specific figures, use the `--figure` (or `-f`) flag. For example, the command:\n\n```\n$ ./scripts/run-experiments.sh -f 11b -f 14\n```\n\nreproduces Figure 11(b) and Figure 14 from the paper.\n\nUnless the `--recompile-paper` (or `-r`) flag is set to `false`, the script uses results from any reproduced experiments to **replot figures** (say, Figure 11(b) and Figure 14 from the example above). Figures that have not been reproduced are replotted using the original results from the paper.\n\n### The saber.conf file\n\nEdit `scripts/saber.conf` and revisit Saber's default configuration parameters if necessary.\n\n#### Maven repository\n\n`MVN_REPOSITORY` is the local folder where Saber's Java dependencies are downloaded and stored automatically by Maven. The default location on Linux is `$HOME/.m2`. The variable is used to construct the Java class path when running Saber appications from the command line.\n\n#### JVM configuration\n\n`SABER_JVM_MS` and `SABER_JVM_MX` set the minimum and maximum Java heap size (in GB) of a Saber application. In our experiments, both values are set to 48 GB (out of 64 GB available on the machine), resulting in a **fixed heap size**.\n\n`SABER_JVM_NEWRATIO` sets the proportion of the heap dedicated to the young generation, where all new Java objects are allocated. The default value of `2` dedicates a third of the heap to the young generation.\n\n`SABER_JVM_SURVIVORRATIO` sets the ratio between Eden and Survivor space in the young generation. The default value is `16`.\n\nBased on the default configuration of the JVM, the Eden space of a Saber application is approximately 14 GB. In our experiments, the Eden space is never filled and thus no minor (_Stop the world_) garbage collection events occur. Saber reuses all major objects allocated (e.g., intermediate result buffers, tasks, etc.).\n\n#### Saber system configuration\n\nVariables in **saber.conf** prefixed with `SABER_CONF` configure the Saber runtime. Each of them also corresponds to a command-line argument available to all Saber applications:\n\n######--execution-mode `cpu`|`gpu`|`hybrid`\n\nSets the execution mode to either CPU-only, GPU-only or Hybrid, respectively. In the latter case, both processors execute query tasks opportunistically. The default execution mode is `cpu`.\n\nIt sets the `SABER_CONF_EXECUTIONMODE` variable.\n\n######--number-of-worker-threads _N_\n\nSets the number of CPU worker threads. The default value is `1`. In GPU-only execution mode, the value must be `1`. **CPU worker threads are pinned to physical cores**. The first thread is pinned to core id 1, the second to core id 2, and so on.\n\nIt sets the `SABER_CONF_NUMWORKERTHREADS` variable.\n\n######--number-of-result-slots _N_\n\nSets the number of intermediate query result slots. The default value is `1024`. \n\nIt sets the `SABER_CONF_NUMRESULTSLOTS` variable.\n\n######--scheduling-policy `fcfs`|`static`|`hls`\n\nSets the scheduling policy to either First-Come-First-Served (FCFS), Static or Heterogeneous Look-ahead Scheduling (HLS), respectively. The default value is `fcfs`.\n\nIt sets the `SABER_CONF_SCHEDULINGPOLICY` variable.\n\n######--switch-threshold _N_\n\nUsed by the HLS scheduling algorithm (`--scheduling-policy hls`) to allow the non-preferred processor of a query to execute some of that query's tasks. The default value is `10`, allowing the non-preferred processor to execute 1 every 10 tasks executed on the preferred one.\n\nIt sets the `SABER_CONF_SWITCHTHRESHOLD` variable.\n\n######--number-of-partial-windows _N_\n\nSets the maximum number of window fragments in a query task. The default value is `65536`.\n\nIt sets the `SABER_CONF_NUMPARTIALWINDOWS` variable.\n\n######--circular-buffer-size _N_\n\nSets the circular buffer size, in bytes. The default value is `1073741824`, i.e. 1 GB. \n\t\nIt sets the `SABER_CONF_CIRCULARBUFFERSIZE` variable.\n\n######--intermediate-buffer-size _N_\n\nSets the intermediate result buffer size, in bytes, The default value is `1048576`, i.e. 1 MB.\n\nIt sets the `SABER_CONF_INTERMEDIATEBUFFERSIZE` variable.\n\n######--hash-table-size _N_\n\nHash table size (in bytes): hash tables hold partial window aggregate results (default is 1048576, i.e. 1MB). \n\nIt sets the `SABER_CONF_HASHTABLESIZE` variable.\n\n######--throughput-monitor-interval _N_\n\nSets the query throughput matrix update interval, in msec. The default value is `1000` i.e. 1 sec.\n\nIt sets the `SABER_CONF_THROUGHPUTMONITORINTERVAL` variable. \n\n######--performance-monitor-interval _N_\n\nSets the performance monitor interval, in msec. The default value is `1000`, i.e. 1 sec. Controls how often Saber prints on standard output performance statistics such as throughput and latency. \n\nIt sets the `SABER_CONF_PERFORMANCEMONITORINTERVAL` variable.\n\n######--pipeline-depth _N_\n\nSets the GPU pipeline depth - the number of query tasks scheduled on the GPU before the result of the first one is returned. The default value is `4`. \n\nIt sets the `SABER_CONF_PIPELINEDEPTH` variable.\n\n######--enable-latency-measurements `true`|`false`\n\nDetermines whether Saber should measure task latency or not. Default value is `false`. \n\nIt sets the `SABER_CONF_MEASURELATENCY` variable.\n\n######--launch-web-server `true`|`false`\n\nDetermines whether Saber should launch the _Saber Workbench_ back-end server (see our DEBS 2016 demo for details). Default value is `false`.\n\nIt sets the `SABER_CONF_LAUNCHWEBSERVER` variable. \n\n######--experiment-duration _N_\n\nSets the duration after which no more performance statistics will be recorded by the system, in _performance monitor interval_ units. For example, if `--performance-monitor-interval 500` and `--experiment-duration 20` then the experiment duration is 10 sec. The default value is `0`, i.e. no limit specified.\n\nIt sets the `SABER_CONF_EXPERIMENTDURATION` variable.\n\n#### The run.sh command\n\n######--class _Canonical class name_\n\nRuns the specified Saber application class (e.g. `uk.ac.imperial.lsds.saber.experiments.microbenchmarks.TestProjection`).\n\n######--mode `foreground`|`background`\n\nRuns the Saber application specified by `--class` either in the foreground or the background. In background mode, standard output and standard error are redirected to log files in `$SABER_LOGDIR` and the application runs for a fixed duration.\n\n######--alias _string_\n\nSets a Saber application alias to manage its execution in background mode. For example, if alias is `test` then stdout and stderr are redirected to `$SABER_LOGDIR/test.out` and `$SABER_LOGDIR/test.err`, respectively.\n\n######--duration _N_\n\nRuns the Saber application in background mode for _N_ seconds and then exits gracefully.\n\n######-- [class arguments]\n\nAnything command-line argument after `--` is considered a Saber application argument. For example, `-- --execution-mode cpu --number-of-worker-threads 16` will run the application is CPU-only execution mode with 16 worker threads.\n\n### Operator microbenchmarks\n\nApart from the system configuration command-line arguments, all microbenchmarks also share the following:\n\n######--batch-size _N_\n\nSets the query task size, in bytes. The default value is `1048576`, i.e. 1 MB.\n\n######--window-type `row`|`range`, --window-size _N_, and --window-slide _M_\n\nSets the window definition for a particular query operator. Windows can be row-based or time-based, tumbling (_N_ = _M_) or sliding (_N_ \u003e _M_). \n\n######--input-attributes _N_\n\nSets the number of attributes in the input stream. The default value is `6` and depending on the microbenchmark running this will result in 32-byte tuples.\n\n######--tuples-per-insert _N_\n\nSets the number of tuples pushed to the input stream buffer. Default value is `32768`.\n\n#### The projection.sh command\n\n######--projected-attributes _N_\n\nSets the number of projected attributes. \n\n######--expression-depth _N_\n\nThe first attribute (\"_1\") is projected as the expression 3 x \"_1\" / 2. If the expression depth is 2 then the expression is 3 x (3 x _1 / 2) / 2; and so on.\n#### The selection.sh command\n\n######--comparisons _N_\n\nNumber of comparisons evaluated on the first attribute of a tuple.\n\n######--selectivity _N_\n\nThe selectivity of the selection query (between 1 and 100%). \n\n#### The aggregation.sh command\n\n######--aggregate-expression _string_\n\nAn aggregate expression can be one of `cnt`, `sum`, `avg`, `min`, `max` or combinations of those separated by comma (without spaces), e.g. `cnt,avg,min,max`.\n\n#### The theta-join.sh command\n\nThe theta-join microbenchmark is similar to selection, only that it operates on two streams. Each stream can be configured using the suffix `-of-first-stream` or `-of-second-stream` in command-line arguments such as window type, size, slide, and number of input attributes. E.g. `--window-type-of-first-stream row`.\n\n","funding_links":[],"categories":["Table of Contents"],"sub_categories":["Streaming Engine"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flsds%2FSaber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flsds%2FSaber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flsds%2FSaber/lists"}