{"id":20572003,"url":"https://github.com/netease/kyuubi-arctic-playground","last_synced_at":"2025-03-06T10:20:40.819Z","repository":{"id":63168620,"uuid":"563810734","full_name":"NetEase/kyuubi-arctic-playground","owner":"NetEase","description":"Playbook of Kyuubi and Arctic Demo","archived":false,"fork":false,"pushed_at":"2022-11-20T06:47:27.000Z","size":12323,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-01-16T21:15:46.740Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NetEase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-09T11:41:57.000Z","updated_at":"2023-10-13T10:06:35.000Z","dependencies_parsed_at":"2022-11-14T09:46:44.962Z","dependency_job_id":null,"html_url":"https://github.com/NetEase/kyuubi-arctic-playground","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetEase%2Fkyuubi-arctic-playground","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetEase%2Fkyuubi-arctic-playground/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetEase%2Fkyuubi-arctic-playground/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetEase%2Fkyuubi-arctic-playground/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NetEase","download_url":"https://codeload.github.com/NetEase/kyuubi-arctic-playground/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242187807,"owners_count":20086224,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T05:18:01.431Z","updated_at":"2025-03-06T10:20:40.795Z","avatar_url":"https://github.com/NetEase.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"Playground\n===\n\n![Kyuubi Arctic Playgound](/img/overall.png)\n\n## Quick Start\n\n### Setup\n\n1. Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/);\n2. Use `docker compose up -d` to run compose services as daemon;\n\n![Docker Compose Services](/img/services.png)\n\n### Access Service\n\nMinIO: http://localhost:9001 (minio/minio_minio)\n\n![MinIO](/img/minio.png)\n\nMySQL localhost:3306 (root/mysql)\n\n![MySQL](/img/mysql.png)\n\nArctic Meta Service (AMS) Dashboard: http://localhost:1630 (admin/admin)\n\n![AMS](/img/ams.png)\n\nSpark UI: http://localhost:4040 (available after Spark application launching by Kyuubi, port may be 4041, 4042... if you launch more than one Spark applications)\n\n![Spark](/img/spark.png)\n\nFlink UI: http://localhost:8081 (available after starting CDC service)\n\n![Flink](/img/flink.png)\n\n### Play\n\n- Generate static data to MySQL\n   ```\n   docker exec -it mysql-datagen \\\n     java -jar lakehouse-benchmark.jar \\\n     -b tpcc,chbenchmark \\\n     -c config/mysql/sample_chbenchmark_config.xml \\\n     --create=true --load=true\n   ```\n\n- Start CDC service\n  ```\n  docker exec -it mysql-arctic-cdc \\\n    java -cp lakehouse-benchmark-ingestion-1.0-SNAPSHOT.jar \\\n    com.netease.arctic.benchmark.ingestion.MainRunner \\\n    -confDir /opt/lakehouse_benchmark_ingestion/conf \\\n    -sinkType arctic -sinkDatabase arctic_db\n  ```\n\n- Generate TPCC data to MySQL\n  ```\n  java -jar lakehouse-benchmark.jar \\\n    -b tpcc,chbenchmark -c config/mysql/sample_chbenchmark_config.xml \\\n    --execute=true -s 5\n  ```\n\n- Connect using `beeline`\n  ```\n  docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/'\n  ```\n\n- Connect using DBeaver\nAdd a Kyuubi datasource with\n  - connection url `jdbc:hive2://0.0.0.0:10009/`\n  - username: `anonymous`\n  - password: `\u003cempty\u003e`\n\n- Using built-in dataset\n  Kyuubi supply some built-in dataset, After the Kyuubi starts, you can run the following command to load the different datasets:\n\n  - For loading TPC-H tiny dataset to spark_catalog.tpch_tiny, run \n  ```\n  docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/' -f /opt/load_data/load-dataset-tpch-tiny.sql\n  ```\n\n- Query across catalogs.\n\n  Go [Kyuubi Docs](https://kyuubi.readthedocs.io/en/latest/connector/index.html) to check supported connectors.\n\n  ```\n  select\n      n_name,\n      sum(l_extendedprice * (1 - l_discount)) as revenue\n  from\n      tpch.tiny.customer,\n      tpch.tiny.orders,\n      tpch.tiny.lineitem,\n      arctic.arctic_db.supplier,\n      tpch.tiny.nation,\n      tpch.tiny.region\n  where\n      c_custkey = o_custkey\n      and l_orderkey = o_orderkey\n      and l_suppkey = su_suppkey\n      and n_regionkey = r_regionkey\n  group by\n      n_name\n  order by\n      revenue desc\n  ```\n\n### Shutdown\n\n1. Stop compose services by `docker compose down` assuming they are running as daemon;\n\n## How to build images\n\nWe have uploaded pre-build images to docker hub, and you may want to tune and build by yourself.\n\n### Build\n\n1. Build images `./build-image.sh`;\n2. Optional to use `buildx` to build and publish cross-platform images `BUILDX=1 ./build-image.sh`;\n\n## Links\n\n- [Kyuubi Online Try Me](https://try.kyuubi.cloud/)\n- [Kyuubi Playground w/ Iceberg](https://github.com/apache/incubator-kyuubi/tree/master/docker/playground)\n- [Kyuubi Playground w/ ClickHouse](https://github.com/housepower/spark-clickhouse-connector/tree/master/docker)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetease%2Fkyuubi-arctic-playground","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetease%2Fkyuubi-arctic-playground","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetease%2Fkyuubi-arctic-playground/lists"}