{"id":15284342,"url":"https://github.com/alex1on/information-systems-ntua","last_synced_at":"2026-04-13T00:45:05.001Z","repository":{"id":257300056,"uuid":"741581455","full_name":"alex1on/Information-Systems-NTUA","owner":"alex1on","description":"Distributed execution of SQL queries over Trino","archived":false,"fork":false,"pushed_at":"2024-09-15T18:33:27.000Z","size":7438,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-28T23:23:28.252Z","etag":null,"topics":["cassandra","distributed-queries","postgresql","redis","sql-engine","tpc-ds-benchmark","trino"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alex1on.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-10T17:32:55.000Z","updated_at":"2024-09-15T18:36:40.000Z","dependencies_parsed_at":"2024-09-15T20:48:56.752Z","dependency_job_id":"0abeb492-643d-40c3-a144-e8291eca85f4","html_url":"https://github.com/alex1on/Information-Systems-NTUA","commit_stats":null,"previous_names":["alex1on/information-systems-ntua"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex1on%2FInformation-Systems-NTUA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex1on%2FInformation-Systems-NTUA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex1on%2FInformation-Systems-NTUA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alex1on%2FInformation-Systems-NTUA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alex1on","download_url":"https://codeload.github.com/alex1on/Information-Systems-NTUA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245138306,"owners_count":20566915,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","distributed-queries","postgresql","redis","sql-engine","tpc-ds-benchmark","trino"],"created_at":"2024-09-30T14:53:55.991Z","updated_at":"2026-04-13T00:45:04.935Z","avatar_url":"https://github.com/alex1on.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed Execution of SQL Queries over Trino (Information Systems - NTUA)\n\n## Overview \nThis project, undertaken for the Information Systems course at ECE NTUA, explores the performance of Trino, a distributed SQL query engine, across various data distribution strategies, worker configurations, and query complexities. Using PostgreSQL, Cassandra, and Redis as data sources, we analyze Trino's efficiency by measuring query latency and optimizer plans under different scenarios. The findings provide insights and recommendations for optimizing data distribution and performance in distributed systems.\n## Bookmarks\n\n- [Trino cluster](#trino-cluster-setup)\n- [Trino CLI](#trino-cli-setup)\n- [Set up Databases](#database-environment)\n- [How to connect Databases with Trino server](#database-connection-with-trino-server)\n- [TPC-DS](#working-with-the-tpc-ds-benchmark-suite)\n- [Loading TPC-DS Data](#loading-tpc-ds-data-to-the-databases)\n- [Benchmarks](#benchmarks)\n- [Figures](#figures)\n\n## Set up\n### Trino cluster setup:\n1. Install Java 17.03 or later:\n```console\n$ sudo apt install openjdk-17-jdk openjdk-17-jre\n```\n2. Ensure you have python installed 2.6 or later:\n```console\n$ python3 --version\nPython 3.10.12\n```\n3. Download and unpack the latest Trino release binary:\n\u003e You can find the latest version at the [Maven Central Repository](https://repo.maven.apache.org/maven2/io/trino/trino-server).\n```console\n$ wget https://repo.maven.apache.org/maven2/io/trino/trino-server/435/trino-server-435.tar.gz\n$ tar xvzf trino-server-435.tar.gz\n```\n4. Create an `etc` directory inside the installation directory:\n```console\n$ cd ./trino-server-435/\n$ mkdir etc/\n```\n5. Now you need to create the necessary Trino configuration files inside the `/etc` directory we previously created:\n    - jvm.config\n    - config.properties\n    - node.properties\n    - log.properties\n\nTo create and provide the correct configuration for the cluster run the following:\n```console\n$ touch jvm.config config.properties node.properties log.properties\n```\nAdd the following lines to each of the configuration files:\n```txt\n# jvm.config content\n-server\n# Change this to 76-85% of the total memory of your node \n-Xmx6G\n-XX:InitialRAMPercentage=80\n-XX:MaxRAMPercentage=80\n-XX:G1HeapRegionSize=32M\n-XX:+ExplicitGCInvokesConcurrent\n-XX:+ExitOnOutOfMemoryError\n-XX:+HeapDumpOnOutOfMemoryError\n-XX:-OmitStackTraceInFastThrow\n-XX:ReservedCodeCacheSize=512M\n-XX:PerMethodRecompilationCutoff=10000\n-XX:PerBytecodeRecompilationCutoff=10000\n-Djdk.attach.allowAttachSelf=true\n-Djdk.nio.maxCachedBufferSize=2000000\n-XX:+UnlockDiagnosticVMOptions\n-Dfile.encoding=UTF-8\n-XX:+UseAESCTRIntrinsics\n# Change this to the number of your cpu cores of your node \n-XX:GCLockerRetryAllocationCount=4\n```\n\n```txt\n# config.properties content - for the coordinator \ncoordinator=true\nnode-scheduler.include-coordinator=true\nhttp-server.http.port=8080\n# change localhost with coordinator's ip address\ndiscovery.uri=http://localhost:8080\n```\n```txt\n# config.properties content - for the workers \ncoordinator=false\nhttp-server.http.port=8080\n# change localhost with coordinator's ip address\ndiscovery.uri=http://localhost:8080\n```\n\n```txt\n# node.properties content\n# The name of your cluster \nnode.id=\u003cUUID\u003e\nnode.environment=development \n```\n\u003eYou can create a new `uuid` by running the Linux command `uuidgen` in your terminal.\n \n```txt\n# log.properties content\nio.trino=INFO\n```\n6. You are now ready to start the cluster. At all nodes (while in the default installation directory) run:\n```console\n$ bin/launcher start\n```\nTo verify that Trino started you can run:\n```console\n$ bin/launcher status\nRunning as 4716\n```\n\n### Trino CLI setup:\n1. Download the Trino CLI executable:\n```console\n$ wget https://repo1.maven.org/maven2/io/trino/trino-cli/435/trino-cli-435-executable.jar\n```\n2. Make it an executable:\n```console\n$ mv trino-cli-435-executable.jar trino\n$ chmod +x trino\n```\n3. Verify installation:\n```console\n$ ./trino --version\nTrino CLI 435\n```\n\n## Database environment\nInstall the following DBMSs one in each worker node of your Trino Cluster:\n\n  - [PostgreSQL](#postgresql-setup)\n  - [Cassandra](#cassandra-setup)\n  - [Redis](#redis-setup)\n\n### PostgreSQL Setup:\n1. Install PostgreSQL:\n```console\n$ sudo apt install postgresql postgresql-contrib\n```\n2. PostgreSQL creates a default user with the installation named `postgres`. To access the Postgres switch to that user and run:\n```console\n$ sudo -i -u postgres\n$ psql\npsql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))\nType \"help\" for help.\n\npostgres=#\n```\nYou now have PostgreSQL working in your node. \n\n### Cassandra Setup:\nFor Cassandra we follow the [Debian packages installation guide](https://cassandra.apache.org/doc/stable/cassandra/getting_started/installing.html#installing-the-debian-packages).\n1. Add the Apache repository of Cassandra to the file `cassandra.sources.list`. We use the latest major version 5.0:\n```console\n$ echo \"deb https://debian.cassandra.apache.org 50x main\" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list \n```\n2. Add the Apache Cassandra repository keys to the list of trusted keys on the server:\n```console\n$ curl https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -\n```\n3. Update the packages:\n```console\n$ sudo apt-get update\n```\n4. Install Cassandra with APT:\n```console\n$ sudo apt-get install cassandra\n```\nTo check the Cassandra installation run:\n```console\n$ nodetool status\nDatacenter: datacenter1\n=======================\nStatus=Up/Down\n|/ State=Normal/Leaving/Joining/Moving\n--  Address      Load        Tokens  Owns (effective)  Host ID                               Rack\nUN  192.168.1.2  203.77 KiB  16      100.0%            f6939e82-88d9-4cfa-a4f8-512b990ac76e  rack1\n```\nCassandra is available in your node.\n\n\n### Redis setup:\n1. Download and install the Redis GPG key:\n```console\n$ curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n```\n2. Add the Redis repository to the package manager:\n```console\n$ echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n```\n3. Update the packages:\n```console\n$ sudo apt-get update\n```\n4. Install Redis with APT:\n```console\n$ sudo apt-get install redis\n```  \n\n## Database connection with Trino server\n\n- [PostgreSQL](#postgresql)\n- [Cassandra](#cassandra)\n- [Redis](#redis)\n\n### PostgreSQL\n1. Make PostgreSQL accessible from all the cluster. The cluster is in a LAN so we can just expose PostgreSQL to the cluster LAN. In the Postgres configuration file `/etc/postgresql/14/main/pg_hba.conf` add the IP addresses of the nodes that we want to connect to the PostgreSQL server. Under the `# IPv4 local connection:` add the following:\n```txt\n# IPv4 local connections:\nhost    all             all             127.0.0.1/32            scram-sha-256\nhost    all             all             node1-ip/32             md5\nhost    all             all             node2-ip/32             md5\nhost    all             all             node3-ip/32             md5\nhost    all             all             node4-ip/32             md5\n```\n\nTo apply the changes restart the PostgreSQL service:\n```console\n$ service postgresql restart\n```\n\n2. Add a user in the PostgreSQL database that will be used instead of the default user. As in the installation connect to the PostgreSQL shell as follows:\n\n```console\n$ sudo -i -u postgres\n$ psql\npsql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))\nType \"help\" for help.\n\npostgres=#\n```\n\nThen create a user (ROLE) by running the following command:\n```console\npostgres=# create role your_username with password 'your_password';\n```\n\nWhen creating a ROLE in PostgreSQL you also have to create database with the same name. While being logged in as `postgres` user create a database with the same name as your ROLE name (`your_username`):\n\n```console\npostgres=# CREATE DATABASE trino WITH OWNER 'trino' TEMPLATE template0 ENCODING 'LATIN1' LC_COLLATE='en_US.ISO-8859-1' LC_CTYPE='en_US.ISO-8859-1';\n```\n\u003e The benchmark data we are going to use are from the TPC-DS benchmark suite. The data require `LATIN1` (`ISO-8859-1`) encoding to be loaded properly. You can use the default `UTF-8` if you choose your own data to load. \n\n\u003e **Important!** You will need to add to locale settings on your OS the `ISO-8859-1`. You can check if its already available by running the command `locale -a`. If it doesn't exist on your available locals you can install it by running the `sudo dpkg-reconfigure locales` and selecting the `en_US.ISO-8859-1` in the prompt that opens. \n\n\u003e You should also create a local user (on your machine) by running the `create user --interactive` and putting as a username the same username you specified in your PostgreSQL Role.\n\nYou can now login to PostgreSQL by running the following command:\n```console\n$ psql -h your_node_ip -U you_username -d your_database -W\npsql (14.10 (Ubuntu 14.10-0ubuntu0.22.04.1))\nType \"help\" for help.\n\nyour_username=#\n```\n\n3. Add the PostgreSQL to all the Trino Server nodes. Create a file named `postgres.properties` inside the `/etc/catalog/` directory of the Trino server installation (if the `catalog` directory does not exist, create it) with the following attributes:\n```txt\nconnector.name=postgresql\nconnection-url=jdbc:postgresql://your_node_ip:5432/your_database\nconnection-user=your_usename\nconnection-password=your_password\n```\n\n4. Verify that the connector works properly by querying the Catalogs inside the Trino server. Entering the `Trino CLI` you can run the `SHOW CATALOGS;` command. PostgreSQL and its data should appear there. \n\u003e After the addition of the PostgreSQL connector a Trino server restart might be needed.\n\n\n\n\n### Cassandra\n1. Following the installation guide for Cassandra it makes the Cassandra server accessible only from localhost. In our cluster it has to be accessible by all the nodes inside our LAN. To achieve that we have to change the Cassandra configuration. In the `cassandra.yaml` configuration file (located in `/etc/cassandra/cassandra.yaml`) we have to make the following changes. Change the seeds from `- seeds: \"localhost:7000\"` to `- seeds: \"your-node-ip:7000\"`. Change the listen and rpc addresses as follows. Listen address from `listen_address: localhost` to `listen_address: your-node-ip` and rpc address from `rpc_address: localhost` to `rpc_address: your-node-ip`.\n\n2. Add a \"user\" (or ROLES as per Cassandra documentation) that will be used to connect to Cassandra from the Trino server we have created. Again at the `cassandra.yaml` configuration we have to enable `PasswordAuthenticator` instead of the default `AllowAllAuthenticator` and `CassandraAuthorizer` instead of the default `AllowAllAuthorizer`.  \n```yaml\nauthenticator:\n  class_name : org.apache.cassandra.auth.PasswordAuthenticator\nauthorizer: CassandraAuthorizer\n```\nRestart the Cassandra service:\n```console\n$ sudo service cassandra restart\n```\n\nThen connect to the Cassandra server as follows:\n\u003e As we previously changed the Cassandra server ip address you have to provide the ip address you previously set as `your-node-ip`\n```console\n$ cqlsh -u cassandra 192.168.1.2\nPassword:\nWARNING: cqlsh was built against 5.0-beta1, but this server is 5.0.  All features may not work!\nConnected to Test Cluster at 192.168.1.2:9042\n[cqlsh 6.2.0 | Cassandra 5.0 | CQL spec 3.4.7 | Native protocol v5]\nUse HELP for help.\ncassandra@cqlsh\u003e\n```\nTo create a ROLE (user) from inside the cqlsh terminal you can run the following command:\n```console\ncassandra@cqlsh\u003e CREATE ROLE your_username WITH PASSWORD = 'your_password' AND SUPERUSER = true AND LOGIN = true  ;\n```\nThe above credentials will be used by the Trino connector to access the Cassandra database server. \n\nTo create a KEYSPACE (database) via cqlsh (Cassandra CLI) run the following command:\n```console\nCREATE KEYSPACE IF NOT EXISTS tpcds WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1} AND durable_writes = true;\n```\nYou may want to grant permissions. You can do so, using the following command:\n```console\nGRANT ALL PERMISSIONS ON KEYSPACE tpcds TO trino;\n```\nWe will use tpcds keyspace under the user trino.\n\n3. Add the Cassandra Trino connector to all the cluster nodes. Create a file inside the Trino Server installation directory at the `/etc/catalog` (if the catalog directory does not exist you also have to create it) named `cassandra.properties`. Add the following lines inside the file:\n```txt\nconnector.name=cassandra\ncassandra.contact-points=your_cassandra_node_ip\ncassandra.load-policy.dc-aware.local-dc=datacenter_name\ncassandra.username=your_username\ncassandra.password=your_password\n```\n\u003e You can find the `datacenter_name` in the output of the `nodetool status` terminal command.\n\n4. Verify that the connector works properly by querying the Catalogs inside the Trino server. Entering the `Trino CLI` you can run the `SHOW CATALOGS;` command. Cassandra and its data should appear there. \n\u003e After the addition of the Cassandra connector a Trino server restart might be needed.\n\n\n### Redis\n1. Access Redis Configuration file using the following command:\n```console\n$ sudo vim /etc/redis/redis.conf\n```\n2. Enable Password Authentication by adding the following line in `redis.conf`:\n```txt\nrequirepass \u003cyour_password\u003e\n```\n3. Make Redis listen to a specific (your node's) IP Address. In order to achieve this, add the following line in `redis.conf`:\n```txt\nbind \u003cnode's 4 IP address\u003e\n```\nTo apply the changes restart the Redis service:\n```console\n$ sudo service redis restart\n```\n4. Run Redis CLI by running:\n```console\n$ redis-cli -h \u003cnode's IP address\u003e -a \u003cyour_password\u003e\n```\n5. Add the Redis Trino connector to all the cluster nodes. Create a file inside the Trino Server installation directory at the `/etc/catalog` (if the catalog directory does not exist you also have to create it) named `redis.properties`. Add the following lines inside the file:\n```txt\nconnector.name=redis\nredis.table-names=your_list_of_table_names_seperated_with_comma\nredis.default-schema=tpcds\nredis.nodes=your_nodes'_ip_address:6379\nredis.key-prefix-schema-table=true\nredis.key-delimiter=.\nredis.table-description-dir=/home/user/schemas\nredis.hide-internal-columns=true\nredis.user=your_username\nredis.password=your_password\n```\n\u003e The default location of the `Table Description Files` is the path `/home/user/schemas` as they are created automatically a python script [here](#schema-creation-2).\n\n6. Verify that the connector works properly by querying the Catalogs inside the Trino server. Entering the `Trino CLI` you can run the `SHOW CATALOGS;` command. Redis and its data should appear there. \n\u003e After the addition of the Redis connector a Trino server restart might be needed.\n\n## Working with the TPC-DS benchmark suite\n\n- [Set up](#set-up-the-tpc-ds-benchmark-suite)\n- [Generate benchmark data](#generate-benchmark-data)\n- [Import TPC-DS data to databases](#database-initialization-with-tpc-ds-benchmark)\n\n### Set up the TPC-DS benchmark suite\n\n1. Download the TPC-DS source code from their [website](https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp)\n\n2. Unzip the source code:\n```console\n$ unzip TPC-DS-Tool.zip\n```\n\n3. Edit the `makefile` in the `/DSGen-software-code-3.2.0rc1/tools` directory and specify the `OS` of your machine in the `OS=` line of the `makefile`.\n\n4. Compile the code. To successfully compile it without errors you need to install an older version of the `gcc` compiler. Install it by running `sudo apt install gcc-9`.\n\u003e You might also need to install the following packages: `flex`, `bison`, `byacc`\nRun make:\n```console\n$ make CC=ggc-9\n```\n\n### Generate benchmark data\n\n1. In the `/DSGen-software-code-3.2.0rc1/tools` directory to generate benchmark data run the following command:\n```console\n$ ./dsdgen -scale \u003csize\u003e -dir \u003csave_directory\u003e\n```\n\u003e Specify the data sample size with the `size` parameter. The amount of data is in GBs.\n\n2. Clean up the generated data. Due to the format of the generated data, by the TPC-DS, not being directly compatible with the PostgreSQL and Cassandra `COPY` command, we created a script that fixes this issue. In the root directory of our project run the following script:\n\n```console\n$ ./utils/clean_tpc_data.sh\n```\n\n\u003e Note: change `datadir=\"../../tpc_data\"` to the directory you specified when generating the data in step 1.\n\n## Loading TPC-DS data to the databases\n\n- [PostgreSQL](#load-data-to-postgresql)\n- [Cassandra](#load-data-to-cassandra)\n- [Redis](#load-data-to-redis)\n\nTo set up the database schema and associations, the TPC-DS benchmark provides two essential files located in `DSGen-software-code-3.2.0rc1/tools`:\n1. `tpcds.sql`: Defines the schema with table definitions.\n2. `tpcds_ri.sql`: Specifies associations between tables by setting foreign key constraints.\n\nMost of the scripts referenced below retrieve authentication and connection details from a `.env` file structured as follows:\n```.env\nPOSTGRESQL_USER=\u003cyour_user\u003e\nPOSTGRESQL_PASSWORD=\u003cyour_password\u003e\nPOSTGRESQL_HOST=\u003cpostgres_host\u003e\nPOSTGRESQL_DATABASE=\u003cpostgres_db\u003e\nPOSTGRESQL_SCHEMA=\u003cschema_name\u003e\n\nCASSANDRA_USER=\u003cyour_user\u003e\nCASSANDRA_PASSWORD=\u003cyour_password\u003e\nCASSANDRA_HOST=\u003ccassandra_host\u003e\nCASSANDRA_KEYSPACE=\u003ckeyspace_name\u003e\n\nREDIS_DB=0\nREDIS_PASSWORD=\u003cyour_password\u003e\nREDIS_HOST=\u003credis_host\u003e\n\nTRINO_HOST=\u003ccoordinator_host\u003e\nTRINO_PORT=8080\n```\n\n### Load Data to PostgreSQL\n\n#### Schema creation:\nSince both are SQL files, they can be used directly without any preprocessing. For this purpose `Databases/PostgreSQL/create_pg_schema.py` is created which executes both, `tpcds.sql` \u0026 `tpcds_ri.sql`.  \n\n#### Data Loading:\nThe loading of data is accomplished through the `Databases/PostgreSQL/load_data_pg.py` script, which performs the following steps:\n1. Establishes a connection with PostgreSQL database using information from the `.env` file.\n2. Reads each `.dat` file under the `tpc_data` directory, where each file corresponds to a table and contains data for that table. The files use '|' as a delimiter.\n3. Utilizes the `COPY` command to efficiently load the data into the database.\n\nSo after running the following commands under `Databases/PostgreSQL/` dir:\n```console\n$ python create_pg_schema.py\n$ python load_data_pg.py\n```\nThe database schema should be properly defined, and the PostgreSQL database should be populated with the corresponding data.\n\nThe `load_data_pg.py` script accepts an optional `--partition` parameter, allowing selective loading of tables based on partition strategy. You can specify a partition by running:\n```console\n$ python load_data_pg.py --partition 1\n```\nor\n```console\n$ python load_data_pg.py --partition 2\n```\nThis will load only the tables defined in the specified partition (1 or 2). If no partition is provided, all tables are loaded by default.\n\n### Load Data to Cassandra\n\n#### Schema creation:\nAs Cassandra lacks the concept of foreign keys and relational integrity, the file `DSGen-software-code-3.2.0rc1/tools/tpcds_ri.sql` is not utilized, and foreign key constraints are not considered. \nWe (correctly) assume that the data adheres to these constraints, and there is no intention to insert additional data or modify existing records.\n\nSince Cassandra does not support SQL, the schema is created using the file `Databases/Cassandra/schema.cql`, written in CQL (Cassandra Query Language). This file, after necessary modifications to fit Cassandra, defines a keyspace named `tpcds` and contains the same table definitions as `DSGen-software-code-3.2.0rc1/tools/tpcds.sql`. `Cassandra/create_schema_cql.sh` executes this file.\n\n#### Data loading:\nData loading in Cassandra follows a similar process to PostgreSQL. The script `Databases/Cassandra/load_data_cass.py` is utilized, employing the `COPY` command to efficiently load the data into the Cassandra database.\n\nUpon executing the following commands:\n```console\n$ python Databases/Cassandra/utils/create_schema_cql.py\n$ python Databases/Cassandra/load_data_cass.py\n```\nThe Cassandra database should have its schema properly established, and data should be successfully loaded.\n\nSimilar to `load_data_pg.py`, `load_data_cass.py` script accepts two parameters:\n- `--all_tables=true/false`: Specifies whether to load all tables. The default is `false`.\n- `--partition (1 or 2)`: Defines which partition to load (1 or 2).\n\n### Load Data to Redis\n\n#### Schema creation:\nRedis, being a key-value database without built-in support for structured schemas, necessitates an unconventional approach for querying data using Trino with SQL queries. To enable this, we generate JSON schema definition files, describing the structure of each \"table\" in a manner that Trino can interpret as if they were structured tables.\n\nThe schema creation and data loading involve the following files:\n- `Databases/Redis/utils/tables.py`: Defines arrays/lists with essential information about table definitions, including table names, primary keys, table column names, and data types for each column.\n- `Databases/Redis/utils/json_schema.py`: Generates the JSON schema definition file for each table based on the information provided by tables.py.\n\n\u003e The `json_schema.py` creates the schemas as the default location in the directory `/home/user/schemas`. This has to be the same as the directory you have specified in the `redis.properties` connector file of Trino. \n\n#### Data loading:\nFor loading data into Redis, the following modules and scripts are utilized in conjunction with `Databases/Redis/utils/tables.py`:\n- `Databases/Redis/utils/redis_connection.py`: Establishes a connection with Redis using information from the `.env` file.\n- `Databases/Redis/utils/load_data_redis_helper.py`: A module responsible for loading data into Redis utilizing hashes.\n- `Databases/Redis/load_data_redis.py`: Employes the above modules in order to properly load data into Redis.\n\nUpon executing the following commands:\n```console\n$ ./create_trino_json_table_definitions.sh\n$ ./load_data_redis.sh\n```\nData is successfully inserted into Redis, and the JSON files are structured in a way that allows Trino to query the Redis data in a structured manner.\n\nSimilar to the previous scripts, `load_data_redis.py` accepts the following parameters:\n- `--table \u003ctable_name\u003e`: Specifies the name of the table to load into Redis.\n- `--partition [1/2]`: Loads data for all tables within the specified partition.\n- `--cleanup`: Cleans up the data file after processing. By default, no cleanup is performed.\n- `batch_processing`: Enables or disables batch processing. The default is `true`.\n\n#### Note: \nGiven that Redis is an in-memory database and certain tables contain millions of records, not all records can fit in memory. Consequently, the insertion process focuses on a subset of the data for practical considerations.\n\n## Generating TPC-DS queries\n\nTo generate the complete query suite (1 - 99) run the following script while on the root project directory:\n\n```console\n$ ./utils/fix_tpcds.sh\n```\n\nIt outputs all the queries with a unique sequence number in the `queries/` directory.\n\nDue to different SQL dialects that Trino and TPC-DS use we also need to make some adjustments in the queries. \n\nRun the following script to make the necessary changes:\n\n```console\n$ python ./utils/fix_queries.py\n```\n\nTo validate if the queries are now compatible with the Trino SQL dialect run the validation script:\n\n```console\n$ ./validate_queries.sh\n```\n\nIt outputs on a txt file the validity of each query (true or false).\n\n## Benchmarks\n\n- [No Distribution](#no-distribution)\n- [Fact Table ER-Based Distribution](#fact-table-er-based-distribution)\n- [Optimized Fact and Dimension Consolidation Distribution](#optimized-fact-and-dimension-consolidation-distribution)\n\n### No Distribution\n\nThe `Benchmarks/run_queries_custom.sh` script executes this initial benchmark. You can modify the `QUERIES` variable within the script to specify the queries you'd like to test. The script runs each query twice for each database (PostgreSQL, Cassandra, and Redis where applicable) using Trino, and stores the latency results under the `Benchmarks/query_results` directory.\n\n### Fact Table ER-Based Distribution\n\nThe `Benchmarks/run_queries_distributed.sh` script handles this benchmark by executing the queries defined in the `QUERIES` variable in a distributed manner. Each query is run twice, and the results are saved in the `Benchmarks/query_results directory`. The script accepts an optional parameter: `partition [1/2]`, allowing you to specify whether to run queries for strategy 1 or strategy 2, located in the `../../queries_strategy1` and `../../queries_strategy2` directories, respectively.\n\n### Optimized Fact and Dimension Consolidation Distribution\n\nThis benchmark is also executed using the `Benchmarks/run_queries_distributed.sh` script, following the same procedure as described above.\n\n\n## Figures\n\nThe scripts for generating figures are located in the `Benchmarks/figure_generators` directory, while the generated figures are stored in `Benchmarks/figure_generators/figures/`. The figure generators are as follows:\n\n- `no_dist_fig_gen.py`: Generates `group1.png`, `group2.png`, and `group3.png`, which compare PostgreSQL, Cassandra, and Redis (only in group 3) for queries in group 1, group 2, and group 3 based on the no-distribution benchmark.\n- `no_dist_vs_dist1_fig_gen.py`: Produces the `no_dist_vs_dist1.png` figure, comparing the no-distribution benchmark to the first distribution benchmark.\n- `dist1_vs_dist2_fig_gen.py`: Creates the `dist1_vs_dist2.png` figure, comparing the performance of the first and second distribution strategies.\n- `dist_workers_fig_gen.py`: Generates `dist_queries_over_workers.png`, which shows the performance of the final distribution strategy across different numbers of Trino workers (1, 2, and 3).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falex1on%2Finformation-systems-ntua","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falex1on%2Finformation-systems-ntua","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falex1on%2Finformation-systems-ntua/lists"}