{"id":33179090,"url":"https://github.com/homeaway/datapull","last_synced_at":"2026-01-12T09:30:29.269Z","repository":{"id":37601966,"uuid":"183268854","full_name":"homeaway/datapull","owner":"homeaway","description":"Cloud based Data Platform based on Apache Spark","archived":false,"fork":false,"pushed_at":"2025-10-07T14:29:22.000Z","size":14446,"stargazers_count":27,"open_issues_count":25,"forks_count":38,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-10-07T16:28:27.537Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://homeaway.github.io/datapull","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/homeaway.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-04-24T16:40:09.000Z","updated_at":"2025-10-07T14:29:27.000Z","dependencies_parsed_at":"2024-06-10T03:44:27.428Z","dependency_job_id":"52576ed8-eafe-48d9-ac6c-5dc29b834805","html_url":"https://github.com/homeaway/datapull","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/homeaway/datapull","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homeaway%2Fdatapull","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homeaway%2Fdatapull/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homeaway%2Fdatapull/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homeaway%2Fdatapull/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/homeaway","download_url":"https://codeload.github.com/homeaway/datapull/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/homeaway%2Fdatapull/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28337723,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T03:00:36.823Z","updated_at":"2026-01-12T09:30:29.263Z","avatar_url":"https://github.com/homeaway.png","language":"Java","funding_links":[],"categories":["大数据"],"sub_categories":[],"readme":"# DataPull #\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"222\" height=\"207\" src=\"./docs/docs/media/logo_transparent.png\"\u003e\n\u003c/p\u003e\nDataPull is a self-service Distributed ETL tool to join and transform data from heterogeneous datastores. It provides users an easy and consistent way to move data from one datastore to another. Supported datastores include, but are not limited to, SQLServer, MySql, Postgres, Cassandra, MongoDB, and Kafka.\n\n## Features\n1. JSON configuration-driven data movement - no Java/Scala knowledge needed\n1. Join and transform data among heterogeneous datastores (including NoSQL datastores) using ANSI SQL\n1. Deploys on Amazon AWS EMR and Fargate; but can run on any Spark cluster\n1. Picks up datastore credentials stored in [Hashicorp Vault](https://www.vaultproject.io), [Amazon Secrets Manager](https://aws.amazon.com/secrets-manager/)\n1. Execution logs and migration history configurable to Amazon AWS Cloudwatch, S3\n1. Use built-in cron scheduler, or call REST API from external schedulers\n\n... and many more features documented [here](./core/src/main/resources/Samples/Input_Json_Specification.json)\n\n## Run DataPull locally\n\u003e Note: DataPull consists of two services, an API written in Java Spring Boot, and a Spark app written in Scala. Although Scala apps can run on JDK 11, per [official docs](https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html#jdk-11-compatibility-notes) it is recommended that Java 8 be used for compiling Scala code. The effort to upgrade to OpenJDK 11+ is tracked [here](https://github.com/homeaway/datapull/issues/2)\n\n### Build and execute within a Dockerised Spark environment\n\u003e Pre-requisite: Docker Desktop\n* Clone this repo locally and check out the master branch\n  ```shell script\n  git clone git@github.com:homeaway/datapull.git\n  ```\n* Build a local docker image for running spark as a dockerised server\n  ```shell script\n  cd ./datapull\n  docker build -f ./core/docker_spark_server/Dockerfile -t expedia/spark2.4.8-scala2.11-hadoop2.10.1 ./core/docker_spark_server\n  ```\n* Build the Scala JAR from within the `core` folder\n  ```shell script\n  cd ./core\n  cp ../master_application_config-dev.yml ./src/main/resources/application.yml\n  docker run \\\n     -e MAVEN_OPTS=\"-Xmx1024M -Xss128M -XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=1024M -XX:+CMSClassUnloadingEnabled\" \\\n     --rm \\\n     -v \"${PWD}\":/usr/src/mymaven \\\n     -v \"${HOME}/.m2\":/root/.m2 \\\n     -w /usr/src/mymaven \\\n     maven:3.6.3-jdk-8 mvn clean install\n  ```\n* Execute a sample JSON input file [Input_Sample_filesystem-to-filesystem.json](core/src/main/resources/Samples/Input_Sample_filesystem-to-filesystem.json) that moves data from a CSV file [HelloWorld.csv](core/src/main/resources/SampleData/HelloWorld/HelloWorld.csv) to a folder of json files named SampleData_Json.  \n  ```\n  docker run \\\n     -v $(pwd):/core \\\n     -w /core \\\n     -it \\\n     --rm \\\n     expedia/spark2.4.8-scala2.11-hadoop2.10.1 spark-submit \\\n        --packages org.apache.spark:spark-sql_2.11:2.4.8,org.apache.spark:spark-avro_2.11:2.4.8,org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.8 \\\n        --deploy-mode client \\\n        --class core.DataPull \\\n        target/DataMigrationFramework-1.0-SNAPSHOT-jar-with-dependencies.jar src/main/resources/Samples/Input_Sample_filesystem-to-filesystem.json local\n  ```\n* Open the relative path `target/classes/SampleData_Json` to find the result of the DataPull i.e. the data from `target/classes/SampleData/HelloWorld/HelloWorld.csv` transformed into JSON.\n\n\u003e Pro-tip: The folder `target/classes/SampleData_Json` is created by the docker spark container, so you will not be able to delete it until you take ownership of it by running `sudo chown -R $(whoami):$(whoami) .`\n\n### Build and debug within an IDE (IntelliJ) ###\t\n\u003e Pre-requisite: IntelliJ with Scala plugin configured. Check out this [Help page](https://docs.scala-lang.org/getting-started-intellij-track/getting-started-with-scala-in-intellij.html) if this plugin is not installed.\t\n* Clone this repo locally and check out the master branch\t\n* Open the folder [core](core) in IntelliJ IDE.\t\n* When prompted, add this project as a maven project.\t\n* By default, this source code is designed to execute a sample JSON input file [Input_Sample_filesystem-to-filesystem.json](core/src/main/resources/Input_Sample_filesystem-to-filesystem.json) that moves data from a CSV file [HelloWorld.csv](core/src/main/resources/SampleData/HelloWorld/HelloWorld.csv) to a folder of json files named SampleData_Json.\t\n* Go to File \u003e Project Structure... , and choose 1.8 (java version) as the Project SDK\t\n* Go to Run \u003e Edit Configurations... , and do the following\t\n    * Create an Application configuration (use the + sign on the top left corner of the modal window)\t\n    * Set the Name to Debug\t\n    * Set the Main Class as Core.DataPull\t\n    * Use classpath of module Core.DataPull\t\n    * Set JRE to 1.8\t\n    * Click Apply and then OK\t\n* Click Run \u003e Debug 'Debug' to start the debug execution\t\n* Open the relative path target/classes/SampleData_Json to find the result of the DataPull i.e. the data from target/classes/SampleData/HelloWorld/HelloWorld.csv transformed into JSON.\n\n## Deploy DataPull to Amazon AWS\n\nDeploying DataPull to Amazon AWS, involves\n- installing the DataPull API and Spark JAR in AWS Fargate, using [this runbook](https://homeaway.github.io/datapull/install_on_aws/)\n- running DataPulls in AWS EMR, using [this runbook](https://homeaway.github.io/datapull/emr_runbook/)\n\n## Manual Tests\n\nPlease follow instructions in [manual-tests/README.md](manual-tests/README.md)\n\n## Contribute to this project\n\n### Bugs/Feature Requests\n\nPlease create an issue in this git repo, using the bug report or feature request templates.\n\n### Documentation\n\nDataPull documentation is available at https://homeaway.github.io/datapull/ . To update this documentation, please do the following steps...\n\n1. Fork the [DataPull](https://github.com/homeaway/datapull) repo\n\n1. In terminal from the root of the repo, run \n    1. if Docker is installed, run \n    ```\n    docker run --rm -it -p 8000:8000 -v ${PWD}/docs:/docs squidfunk/mkdocs-material\n    ```\n    2. or, if MkDocs and Material for MkDocs are installed, run \n    ```\n    cd docs\n    mkdocs serve\n    ```\n\n1. Open http://127.0.0.1:8000 to see a preview of the documentation site. You can edit the documentation by following https://www.mkdocs.org/#getting-started\n\n1. Once you're done updating the documentation, please commit and push your updates to your forked repo. \n\n1. In terminal from the root of the forked repo, run one of the following command blocks, to update and push your `gh-pages` branch.\n    1. if Docker is installed, run \n    ```\n    docker run --rm -it -v ~/.ssh:/root/.ssh -v ${PWD}:/docs squidfunk/mkdocs-material gh-deploy --config-file /docs/docs/mkdocs.yml\n    ```\n    2. or, if MkDocs and Material for MkDocs are installed, run \n    ```\n    cd docs\n    mkdocs gh-deploy\n    ```\n\n1. Create 2 PRs (one for forked repo branch that you updated, another for `gh-pages` branch) and we'll review and approve them.\n\nThanks again, for helping make DataPull better!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhomeaway%2Fdatapull","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhomeaway%2Fdatapull","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhomeaway%2Fdatapull/lists"}