{"id":13571323,"url":"https://github.com/NVIDIA/spark-rapids","last_synced_at":"2025-04-04T08:30:57.501Z","repository":{"id":36952656,"uuid":"264043501","full_name":"NVIDIA/spark-rapids","owner":"NVIDIA","description":"Spark RAPIDS plugin - accelerate Apache Spark with GPUs","archived":false,"fork":false,"pushed_at":"2025-03-21T09:43:59.000Z","size":59602,"stargazers_count":874,"open_issues_count":1587,"forks_count":244,"subscribers_count":44,"default_branch":"branch-25.04","last_synced_at":"2025-03-21T18:01:54.887Z","etag":null,"topics":["big-data","gpu","rapids","spark"],"latest_commit_sha":null,"homepage":"https://nvidia.github.io/spark-rapids","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":"docs/supported_ops.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-14T22:56:44.000Z","updated_at":"2025-03-21T09:43:44.000Z","dependencies_parsed_at":"2023-09-27T02:42:40.785Z","dependency_job_id":"5e87b256-9870-4905-b565-b69b66608287","html_url":"https://github.com/NVIDIA/spark-rapids","commit_stats":{"total_commits":5207,"total_committers":80,"mean_commits":65.0875,"dds":0.8736316497023238,"last_synced_commit":"4df6d601150fbe678237dcd54585ca26870a480a"},"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fspark-rapids","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fspark-rapids/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fspark-rapids/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA%2Fspark-rapids/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA","download_url":"https://codeload.github.com/NVIDIA/spark-rapids/tar.gz/refs/heads/branch-25.04","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246988773,"owners_count":20865296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","gpu","rapids","spark"],"created_at":"2024-08-01T14:01:01.016Z","updated_at":"2025-04-04T08:30:52.492Z","avatar_url":"https://github.com/NVIDIA.png","language":"Scala","funding_links":[],"categories":["Scala","HarmonyOS","大数据"],"sub_categories":["Windows Manager"],"readme":"# RAPIDS Accelerator For Apache Spark\nNOTE: For the latest stable [README.md](https://github.com/nvidia/spark-rapids/blob/main/README.md) ensure you are on the main branch.\n\nThe RAPIDS Accelerator for Apache Spark provides a set of plugins for\n[Apache Spark](https://spark.apache.org) that leverage GPUs to accelerate processing \nvia the [RAPIDS](https://rapids.ai) libraries.\n\nDocumentation on the current release can be found [here](https://nvidia.github.io/spark-rapids/).\n\nTo get started and try the plugin out use the [getting started guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/overview.html).\n\n## Compatibility\n\nThe SQL plugin tries to produce results that are bit for bit identical with Apache Spark.\nOperator compatibility is documented [here](./docs/compatibility.md)\n\n## Tuning\n\nTo get started tuning your job and get the most performance out of it please start with the\n[tuning guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/tuning-guide.html).\n\n## Configuration\n\nThe plugin has a set of Spark configs that control its behavior and are documented\n[here](docs/configs.md).\n\n## Issues \u0026 Questions\n\nWe use github to track bugs, feature requests, and answer questions. File an\n[issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose) for a bug or feature request. Ask\nor answer a question on the [discussion board](https://github.com/NVIDIA/spark-rapids/discussions).\n\n## Download\n\nThe jar files for the most recent release can be retrieved from the [download](https://nvidia.github.io/spark-rapids/docs/download.html)\npage.\n\n## Building From Source\n\nSee the [build instructions in the contributing guide](CONTRIBUTING.md#building-from-source).\n\n## Testing\n\nTests are described [here](tests/README.md).\n\n## Integration\nThe RAPIDS Accelerator For Apache Spark does provide some APIs for doing zero copy data\ntransfer into other GPU enabled applications.  It is described\n[here](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/ml-integration.html).\n\nCurrently, we are working with XGBoost to try to provide this integration out of the box.\n\nYou may need to disable RMM caching when exporting data to an ML library as that library\nwill likely want to use all of the GPU's memory and if it is not aware of RMM it will not have\naccess to any of the memory that RMM is holding.\n\n## Qualification and Profiling tools\n\nThe Qualification and Profiling tools have been moved to\n[nvidia/spark-rapids-tools](https://github.com/NVIDIA/spark-rapids-tools) repo.\n\nPlease refer to [Qualification tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/qualification/overview.html)\nand [Profiling tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/profiling/overview.html)\nfor more details on how to use the tools.\n\n## Dependency for External Projects\n\nIf you need to develop some functionality on top of RAPIDS Accelerator For Apache Spark (we currently\nlimit support to GPU-accelerated UDFs) we recommend you declare our distribution artifact\nas a `provided` dependency.\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.nvidia\u003c/groupId\u003e\n    \u003cartifactId\u003erapids-4-spark_2.12\u003c/artifactId\u003e\n    \u003cversion\u003e24.12.0-SNAPSHOT\u003c/version\u003e\n    \u003cscope\u003eprovided\u003c/scope\u003e\n\u003c/dependency\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVIDIA%2Fspark-rapids","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNVIDIA%2Fspark-rapids","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVIDIA%2Fspark-rapids/lists"}