{"id":22636116,"url":"https://github.com/47degrees/spark-sftp-test","last_synced_at":"2025-07-17T19:35:02.563Z","repository":{"id":53908212,"uuid":"212177344","full_name":"47degrees/spark-sftp-test","owner":"47degrees","description":"Testing different methods to read from SFTP from Spark","archived":false,"fork":false,"pushed_at":"2021-03-11T17:47:07.000Z","size":100,"stargazers_count":3,"open_issues_count":3,"forks_count":2,"subscribers_count":47,"default_branch":"master","last_synced_at":"2025-04-11T21:11:38.543Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/47degrees.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2019-10-01T19:03:23.000Z","updated_at":"2022-02-24T15:58:28.000Z","dependencies_parsed_at":"2022-08-13T04:00:55.052Z","dependency_job_id":null,"html_url":"https://github.com/47degrees/spark-sftp-test","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/47degrees%2Fspark-sftp-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/47degrees%2Fspark-sftp-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/47degrees%2Fspark-sftp-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/47degrees%2Fspark-sftp-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/47degrees","download_url":"https://codeload.github.com/47degrees/spark-sftp-test/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248480431,"owners_count":21110937,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-09T03:18:20.631Z","updated_at":"2025-04-11T21:11:48.538Z","avatar_url":"https://github.com/47degrees.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"## sparksftpTest is a project to showcase an ETL pipeline: SFTP - Hive - SFTP with Spark\n\nThe process can be summarized in\n\n-   Ingestion of the Data (CSV) from an SFTP server.\n-   Processed by Spark and stored in Parquet/Hive, original data + the ones with tranformations\n-   Storing the results on the SFTP server.\n\nThere are currently two implementations:\n\n- Spark with a SFTP connector\n- Spark with Hadoop's SFTP FileSystem Support\n\n### Spark with a SFTP connector\n\nThis is one of the prototypes developed that works and is based on the springml connector https://github.com/springml/spark-sftp\n\nThis solution is currently the fastest\n[ReadingSFTPConnectorApp](/src/main/scala/org/fortysevendeg/sparksftp/ReadingSFTPConnectorApp.scala)\n\n### Spark with Hadoop's SFTP FileSystem Support\n\n[ReadingSFTPHadoopApp](/src/main/scala/org/fortysevendeg/sparksftp/ReadingSFTPHadoopApp.scala)\n\n\nWith this solution we used the native support by Hadoop, although the implementation in the current releases contain\nsome bugs, the patches that exists have not been merged into the Hadoop released libraries, so we have included\nthe patched version in our code.\n\n\nIssues about native support of SFTP in Hadoop:\nhttps://issues.apache.org/jira/browse/HADOOP-5732\nhttps://issues.apache.org/jira/browse/HADOOP-14444\n\n### How to run the programs\n\n#### Local:\n\nTo run them locally with \"sbt run\" you would need to add a setting to the SparkConf `.set(\"spark.master\", \"local[*]\"`\n\n#### On your local Spark instance through spark-submit\n\n``sbt assembly`` to build the .jar\n\n```\nPATH-TO-SPARK/bin/spark-submit  --class org.fortysevendeg.sparksftp.ReadingSFTPHadoopApp --driver-class-path (/PATH-TO-HADOOP/bin/hadoop classpath) ./target/scala-2.11/sparksftpTest-assembly-0.0.1.jar --conf 'spark.executor.extraJavaOptions=-Dspark.executorEnv.SFTP_USER=XXXX' --files sftp.conf --driver-java-options =\"-Dspark.executorEnv.SFTP_USER=XXX\"\n```\n\n### On your Google Cloud Dataproc instance for Spark.\n\nYou can use the scripts `gcloudsubmit1.sh` and `gcloudsubmit2.sh` to see examples for submitting the spark jobs to a Google Cloud Dataproc cluster.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F47degrees%2Fspark-sftp-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F47degrees%2Fspark-sftp-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F47degrees%2Fspark-sftp-test/lists"}