{"id":13571376,"url":"https://github.com/alibaba/SparkCube","last_synced_at":"2025-04-04T08:30:59.089Z","repository":{"id":37313430,"uuid":"247653020","full_name":"alibaba/SparkCube","owner":"alibaba","description":"SparkCube is an open-source project for extremely fast OLAP data analysis. SparkCube is an extension of Apache Spark.","archived":false,"fork":false,"pushed_at":"2023-03-06T14:16:24.000Z","size":158,"stargazers_count":130,"open_issues_count":6,"forks_count":52,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-11-05T04:34:03.478Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alibaba.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-16T08:51:09.000Z","updated_at":"2024-11-05T00:07:48.000Z","dependencies_parsed_at":"2024-11-05T04:31:20.531Z","dependency_job_id":"018f4357-c75f-48c0-8c50-832a3f3b63e1","html_url":"https://github.com/alibaba/SparkCube","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FSparkCube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FSparkCube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FSparkCube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alibaba%2FSparkCube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alibaba","download_url":"https://codeload.github.com/alibaba/SparkCube/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247147013,"owners_count":20891608,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T14:01:01.499Z","updated_at":"2025-04-04T08:30:58.483Z","avatar_url":"https://github.com/alibaba.png","language":"Scala","funding_links":[],"categories":["Scala","数据科学"],"sub_categories":[],"readme":"# SparkCube ![](https://api.travis-ci.org/alibaba/SparkCube.svg?branch=master)\n\nSparkCube is an open-source project for extremely fast OLAP data analysis. SparkCube is an extension of [Apache Spark](http://spark.apache.org).\n\n## Build from source\n\n```\nmvn -DskipTests package\n```\n\nThe default Spark version used is 2.4.4.\n\n## Run tests\n\n```\nmvn test\n```\n\n## Use with Apache Spark\n\nThere are several configs you should add to your Spark configuration.\n\n| config | value | comment |  |\n| ---- | ---- | ---- | ---- |\n| spark.sql.extensions | com.alibaba.sparkcube.SparkCube | Add extension. | Required |\n| spark.sql.cache.tab.display | true | To show web UI in the certain application, typically Spark Thriftserver. | Required |\n| spark.sql.cache.useDatabase | db1,db2,dbn | A list of database names separated by comma. Only tables and views from these databases will be considered for cube building. | Required |\n| spark.sql.cache.cacheByPartition | true/false | To store cache by partition. | Optional |\n| spark.driver.extraClassPath |  /path/to/this/jar | For web UI resources. | Required |\n\nWith the configurations above set in your Spark thriftserver, you should be able to see \"Cube Management\" Tab from the UI of Spark Thriftserver after any `SELECT` command is run. Then you can create/delete/build cubes from this web page.\n\nAfter you have created appropriate cube, you can query the cube from any spark-sql client using Spark SQL. Note that the cube can be created against table or view, so you can join tables as view to create a complex cube.\n\nIf you want a more detailed tutorial for cube creating/building/dropping etc., please refer to\nhttps://help.aliyun.com/document_detail/149293.html\n\n## Learning materials\n\n(Slides)\n\nhttps://www.slidestalk.com/AliSpark/SparkRelationalCache78971\n\nhttps://www.slidestalk.com/AliSpark/SparkRelationalCache2019_57927\n\n(Blogs)\n\nhttps://yq.aliyun.com/articles/703046\n\nhttps://yq.aliyun.com/articles/703154\n\nhttps://yq.aliyun.com/articles/713746\n\nhttps://yq.aliyun.com/articles/725413\n\n(Blogs In English)\n\nhttps://community.alibabacloud.com/blog/rewriting-the-execution-plan-in-the-emr-spark-relational-cache_595267\n\nhttps://www.alibabacloud.com/blog/use-emr-spark-relational-cache-to-synchronize-data-across-clusters_595301\n\nhttps://www.alibabacloud.com/blog/using-data-preorganization-for-faster-queries-in-spark-on-emr_595599\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falibaba%2FSparkCube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falibaba%2FSparkCube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falibaba%2FSparkCube/lists"}