{"id":21847003,"url":"https://github.com/duyet/spark-example-scala","last_synced_at":"2026-04-16T15:01:51.139Z","repository":{"id":75053868,"uuid":"604999659","full_name":"duyet/spark-example-scala","owner":"duyet","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-15T02:41:12.000Z","size":1774,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-26T12:13:06.389Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duyet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-22T08:23:28.000Z","updated_at":"2023-02-23T03:13:08.000Z","dependencies_parsed_at":"2023-12-15T03:38:12.050Z","dependency_job_id":"545f7e6b-d42e-4086-982e-a0d0db79fb65","html_url":"https://github.com/duyet/spark-example-scala","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fspark-example-scala","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fspark-example-scala/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fspark-example-scala/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fspark-example-scala/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duyet","download_url":"https://codeload.github.com/duyet/spark-example-scala/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244831124,"owners_count":20517634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-27T23:16:26.871Z","updated_at":"2026-04-16T15:01:46.083Z","avatar_url":"https://github.com/duyet.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Table of Contents\n- [Prerequisites](#prerequisites)\n- [Building Spark Apps](#build-spark-apps)\n- [Example 1: Spark Scala](#example-1-spark-scala)\n- [Example 2: Spark Scala Calling Rust Binary](#example-2-spark-scala-calling-rust-binary)\n\n# Prerequisites\n\nMake sure you have installed all of the following prerequisites on your development machine:\n\n- Scala + SBT\n\n  ```bash\n  brew install sbt\n  ```\n\n- Rust and Cargo\n\n  ```bash\n  curl https://sh.rustup.rs -sSf | sh\n  ```\n\n- Spark\n\n  ```bash\n  curl -o ~/Downloads/spark-3.3.2-bin-hadoop3.tgz https://dlcdn.apache.org/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz\n  tar zxvf ~/Downloads/spark-3.3.2-bin-hadoop3.tgz -C ~/Downloads\n  ```\n\n![](./.github/screenshots/download.png)\n\n`spark-submit` should available at `~/Downloads/spark-3.3.2-bin-hadoop3/bin/spark-submit`.\n\n\n# Build Spark Apps\n\nBuild all Spark Apps into single .jar file using\n\n```bash\nsbt package\n```\n\n![](./.github/screenshots/sbt-package.png)\n\n\n# Example 1: Spark Scala\n\nSimple SparkPI\n\n```bash\n~/Downloads/spark-3.3.2-bin-hadoop3/bin/spark-submit \\\n  --class \"example.SparkPI\" \\\n  --master \"local[*]\" \\\n  target/scala-2.12/spark-example_2.12-0.1.0-SNAPSHOT.jar 9999\n```\n\n![](./.github/screenshots/spark-submit.png)\n\n# Example 2: Spark Scala calling Rust binary\n\n1. Build Rust\n\n```bash\n(cd rust \u0026\u0026 cargo build --release \u0026\u0026 ls -lp target/release | grep -v /)\n```\n\nTest the Rust simple pipe processing:\n\n```bash\necho '{\"id\": \"duyet\", \"a\":1, \"b\": 2}' | ./rust/target/release/process-simple-line\n# {\"id\":\"duyet\",\"result\":3}\n```\n\n![](./.github/screenshots/rust-build.png)\n\n\n2. Submit Spark + Rust as the following:\n\n```bash\n~/Downloads/spark-3.3.2-bin-hadoop3/bin/spark-submit \\\n  --class \"example.SparkRustSimpleLine\" \\\n  --master \"local[*]\" \\\n  --files rust/target/release/process-simple-line \\\n  target/scala-2.12/spark-example_2.12-0.1.0-SNAPSHOT.jar\n```\n\n![](./.github/screenshots/spark-rust-result.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduyet%2Fspark-example-scala","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fduyet%2Fspark-example-scala","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduyet%2Fspark-example-scala/lists"}