https://github.com/smatiolids/feature-store-perf

Feature Store Performance test
https://github.com/smatiolids/feature-store-perf

Last synced: 7 days ago
JSON representation

Feature Store Performance test

• Host: GitHub
• URL: https://github.com/smatiolids/feature-store-perf
• Owner: smatiolids
• Created: 2024-11-04T17:28:05.000Z (10 days ago)
• Default Branch: main
• Last Pushed: 2024-11-04T17:39:27.000Z (10 days ago)
• Last Synced: 2024-11-04T18:34:38.563Z (10 days ago)
• Language: Python
• Size: 0 Bytes
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Feature Store - Astra DB Testing with PySpark & Locust

## Setup 

### Virtual Environment

```bash

python -m venv venv

```

### Activate Virtual Environment

```bash

source venv/bin/activate

```

### Install Requirements

```bash

pip install -r requirements.txt

```

# Loading the features with Python

```bash

python load/load_features.py

```

# Running Pyspark

## Loading the features with Pyspark

```bash

./pyspark/run_pyspark.sh

```

## Detailed command

```bash

$SPARK_HOME/bin/spark-submit \

--packages com.datastax.spark:spark-cassandra-connector_2.12:3.3.0 \

--files .zip  \

--conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions \

load.py 

```

# Running Locust

```bash

locust -f locustfile.py

```