{"id":20911517,"url":"https://github.com/anant/example-cassandra-spark-elasticsearch","last_synced_at":"2026-04-13T03:47:39.313Z","repository":{"id":112529271,"uuid":"379981964","full_name":"Anant/example-cassandra-spark-elasticsearch","owner":"Anant","description":null,"archived":false,"fork":false,"pushed_at":"2021-06-24T17:26:20.000Z","size":5,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-19T15:38:55.275Z","etag":null,"topics":["cassandra","datastax","docker","elasticsearch","scala","spark","spark-sql"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Anant.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-24T16:06:49.000Z","updated_at":"2022-08-03T19:18:18.000Z","dependencies_parsed_at":"2023-05-15T16:45:18.635Z","dependency_job_id":null,"html_url":"https://github.com/Anant/example-cassandra-spark-elasticsearch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anant%2Fexample-cassandra-spark-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anant%2Fexample-cassandra-spark-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anant%2Fexample-cassandra-spark-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anant%2Fexample-cassandra-spark-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Anant","download_url":"https://codeload.github.com/Anant/example-cassandra-spark-elasticsearch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243301986,"owners_count":20269433,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","datastax","docker","elasticsearch","scala","spark","spark-sql"],"created_at":"2024-11-18T14:22:14.505Z","updated_at":"2025-11-04T12:01:28.814Z","avatar_url":"https://github.com/Anant.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cassandra, Spark and Elasticsearch\n\nThis project is primarily a spark job written in Scala and built with SBT in the form of a fat jar with SBT assembly. It uses DSE Cassandra with Analytics (for Spark) and Elasticsearch, both running in separate docker containers but in the same docker-compose network, and performs three tasks located in three scala class files. \n\nOnce a fat jar is built, it is submitted to spark (with spark-submit) with a different class name corresponding to the three scala classes located in the code and performs three different tasks: \n1. Reading a .CSV file into a SparkSQL Dataframe and saving it to Cassandra\n2. Loading data from a Cassandra table into a SparkSQL Dataframe and saving that data into Elasticsearch\n3. Loading data from Elasticsearch into a SparkSQL Dataframe\n\n- - - \n\n## Software involved\n- docker (docker-compose)\n- Elasticsearch (7.13.0)\n- Scala (2.11.12)\n- DSE Server (6.7.7)\n- Apache Spark, SparkSQL (2.2.3)\n\n## Requirements\n- docker, docker-compose\n- sbt\n\n## Table of Contents\n1. [Run containers with docker-compose](#1-run-docker-containers)\n2. [Setup Cassandra Table](#2-setup-cassandra-table)\n3. [Perform first job (Read CSV, save to Cassandra)](#3-run-first-job)\n4. [Perform second job (Read from Cassandra, save to ES)](#4-run-second-job)\n5. [Perform third job (Read from ES)](#5-run-third-job)\n\n## 1. Run Docker Containers\nMake sure you are in the root folder of the repository. Run the following command: \n```bash\ndocker-compose up -d\n```\nAfter a minute or two, run the following command to make sure that both containers are up (both elasticsearch and dse server): \n```bash\ndocker ps -a\n```\n\n## 2. Setup Cassandra Table\nUse the following command to setup the test Cassandra table: \n```bash\ndocker-compose exec dse1 cqlsh -f /app/test-data/keyspace.cql\n```\nAdditionally, the fat jar needs to be built. Execute the following command in the root directory of the project: \n```bash\nsbt assembly\n```\n\n## 3. Run First Job\nThis first job will read data.csv (located in /test-data/) into a SparkSQL Dataframe and then save it to DSE Cassandra. \n```bash\ndocker-compose exec dse1 dse spark-submit \\\n    --master dse://dse1 \\\n    --jars /jars/test-project-name-assembly-0.1.jar \\\n    --class \"cassandraScalaEs.LoadIntoCass\" \\\n    --conf \"spark.driver.extraJavaOptions=-Dlogback.configurationFile=/app/src/test/resources/logback.xml\" \\\n    --files /app/test-data/data.csv \\\n    /jars/test-project-name-assembly-0.1.jar\n```\n\nTo test that the data is saved into Cassandra, see Second Job. \n\n## 4. Run Second Job\nThis second job will read data from DSE Cassandra that was inserted in the first job into a SparkSQL Dataframe. Afterwards, it will save that data to Elasticsearch. \nIf the second job worked properly, then this step will run and the resulting data (being read from DSE Cassandra) will display in the console. \nAdditionally, right after the non-filtered data is shown, a filtered version of the dataframe will also show (only users with id \u003e 1). \n```bash\ndocker-compose exec dse1 dse spark-submit \\\n    --master dse://dse1 \\\n    --jars /jars/test-project-name-assembly-0.1.jar \\\n    --class \"cassandraScalaEs.CassToEs\" \\\n    --conf \"spark.driver.extraJavaOptions=-Dlogback.configurationFile=/app/src/test/resources/logback.xml\" \\\n    /jars/test-project-name-assembly-0.1.jar\n```\n\nTo test that data was written to Elasticsearch, open up a browser and navigate to the following url: \n```bash\nhttp://localhost:9200/usertestindex/_search\n```\nThis should show all of the data from the original data.csv file written into the index \"usertestindex\" in Elasticsearch. Each row from the data is an individual doc entry in this case. \n\n## 5. Run Third Job\nThe third job reads from Elasticsearch's index that was created in the last job (testuserindex) and puts this data into a SparkSQL Dataframe. Afterwards, it displays the data in the console.\n```bash\ndocker-compose exec dse1 dse spark-submit \\\n    --master dse://dse1 \\\n    --jars /jars/test-project-name-assembly-0.1.jar \\\n    --class \"cassandraScalaEs.LoadFromEs\" \\\n    --conf \"spark.driver.extraJavaOptions=-Dlogback.configurationFile=/app/src/test/resources/logback.xml\" \\\n    /jars/test-project-name-assembly-0.1.jar\n```\n\nThis data is not filtered, but can be filtered with push-down operations (filter condition is automatically translated to a QueryDSL query which is then fed into elasticsearch by the elasticsearch spark connector, so that ES only gives back appropriate data)\n\nSee the following document for more information (Under Spark SQL Support section): \nhttps://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanant%2Fexample-cassandra-spark-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanant%2Fexample-cassandra-spark-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanant%2Fexample-cassandra-spark-elasticsearch/lists"}