{"id":13458733,"url":"https://github.com/apache/kylin","last_synced_at":"2025-11-12T21:36:20.333Z","repository":{"id":25311773,"uuid":"28738447","full_name":"apache/kylin","owner":"apache","description":"Apache Kylin","archived":false,"fork":false,"pushed_at":"2025-10-20T06:00:44.000Z","size":573602,"stargazers_count":3759,"open_issues_count":80,"forks_count":1521,"subscribers_count":256,"default_branch":"kylin5","last_synced_at":"2025-11-12T21:36:04.567Z","etag":null,"topics":["kylin"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-01-03T08:00:05.000Z","updated_at":"2025-11-11T01:22:46.000Z","dependencies_parsed_at":"2024-01-23T17:27:58.503Z","dependency_job_id":"9bb7de97-870c-4f20-bebf-4b9e89511c13","html_url":"https://github.com/apache/kylin","commit_stats":{"total_commits":842,"total_committers":85,"mean_commits":9.905882352941177,"dds":0.8729216152019003,"last_synced_commit":"10d98b74d5f9f9e098e5582e7dadf6536d8837a6"},"previous_names":[],"tags_count":79,"template":false,"template_full_name":null,"purl":"pkg:github/apache/kylin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fkylin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fkylin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fkylin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fkylin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/kylin/tar.gz/refs/heads/kylin5","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fkylin/sbom","scorecard":{"id":201487,"data":{"date":"2025-08-11","repo":{"name":"github.com/apache/kylin","commit":"7fe50b2d5833ad4b8a09d6a27027563dedce8287"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.9,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":8,"reason":"Found 24/30 approved changesets -- score normalized to 8","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":8,"reason":"2 out of the last 2 releases have a total of 2 signed artifacts.","details":["Info: signed release artifact: apache-kylin-5.0.2-bin.tar.gz.asc: https://github.com/apache/kylin/releases/tag/kylin-5.0.2","Info: signed release artifact: apache-kylin-5.0.0-bin.tar.gz.asc: https://github.com/apache/kylin/releases/tag/kylin-5.0.0","Warn: release artifact kylin-5.0.2 does not have provenance: https://api.github.com/repos/apache/kylin/releases/208569479","Warn: release artifact kylin-5.0.0 does not have provenance: https://api.github.com/repos/apache/kylin/releases/177316805"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'kylin5'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: github.com/apache/.github/.github/SECURITY.md:1","Info: Found linked content: github.com/apache/.github/.github/SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: github.com/apache/.github/.github/SECURITY.md:1","Info: Found text in security policy: github.com/apache/.github/.github/SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Binary-Artifacts","score":1,"reason":"binaries present in source code","details":["Warn: binary detected: src/core-common/src/main/resources/async-profiler-lib/linux64/libasyncProfiler.so:1","Warn: binary detected: src/core-common/src/main/resources/async-profiler-lib/macOS/libasyncProfiler.so:1","Warn: binary detected: src/examples/test_case_data/localmeta/jars/custom_parser.jar:1","Warn: binary detected: src/examples/test_case_data/localmeta/lib/libasyncProfiler-linux-arm64.so:1","Warn: binary detected: src/examples/test_case_data/localmeta/lib/libasyncProfiler-linux-x64.so:1","Warn: binary detected: src/examples/test_case_data/localmeta/lib/libasyncProfiler-mac.so:1","Warn: binary detected: src/spark-project/spark-common/src/main/resources/async-profiler-lib/libasyncProfiler-linux-arm64.so:1","Warn: binary detected: src/spark-project/spark-common/src/main/resources/async-profiler-lib/libasyncProfiler-linux-x64.so:1","Warn: binary detected: src/spark-project/spark-common/src/main/resources/async-profiler-lib/libasyncProfiler-mac.so:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 25 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: dev-support/release-manager/release-pipeline-docker/release-machine/Dockerfile_1:24: pin your Docker image by updating openjdk:8-slim to openjdk:8-slim@sha256:19578a1e13b7a1e4cab9b227fb7b5d80e14665cf4024c6407d72ba89842a97ed","Warn: containerImage not pinned by hash: dev-support/release-manager/release-pipeline-docker/release-machine/Dockerfile_2:24: pin your Docker image by updating apachekylin/release-machine:5.0-base to apachekylin/release-machine:5.0-base@sha256:0363b082fdb48ec7d6fbec4da57fdefd3f86d9b565913e24a18702efad30032a","Warn: containerImage not pinned by hash: dev-support/release-manager/standalone-docker/all-in-one/Dockerfile:19: pin your Docker image by updating ubuntu:22.04 to ubuntu:22.04@sha256:1aa979d85661c488ce030ac292876cf6ed04535d3a237e49f61542d8e5de5ae0","Warn: containerImage not pinned by hash: dev-support/release-manager/standalone-docker/all-in-one/dev-docker/Dockerfile_kylin_dev:20: pin your Docker image by updating apachekylin/apache-kylin-standalone:5.x-base-dev-only to apachekylin/apache-kylin-standalone:5.x-base-dev-only@sha256:82cb123cf25e702aaa53b808aedf2f3397367bf7ebd011579e263e35bad453b0","Warn: downloadThenRun not pinned by hash: dev-support/release-manager/release-pipeline-docker/release-machine/Dockerfile_1:63-65","Warn: downloadThenRun not pinned by hash: dev-support/release-manager/release-pipeline-docker/release-machine/Dockerfile_1:68","Warn: npmCommand not pinned by hash: build/release/build.sh:32","Warn: npmCommand not pinned by hash: dev-support/contributor/local/local.sh:99","Info:   0 out of   2 downloadThenRun dependencies pinned","Info:   0 out of   2 npmCommand dependencies pinned","Info:   0 out of   4 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":9,"reason":"1 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-j288-q9x7-2f5v"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T22:55:00.404Z","repository_id":25311773,"created_at":"2025-08-16T22:55:00.404Z","updated_at":"2025-08-16T22:55:00.404Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284115869,"owners_count":26949957,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-12T02:00:06.336Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kylin"],"created_at":"2024-07-31T09:00:56.114Z","updated_at":"2025-11-12T21:36:20.317Z","avatar_url":"https://github.com/apache.png","language":"Java","readme":"\u003cdiv align=\"left\"\u003e\n\n# Apache Kylin\n\n[![License](https://img.shields.io/badge/license-Apache%202-ECA634.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)\n[![GitHub release](https://img.shields.io/github/release/apache/kylin.svg)](https://github.com/apache/kylin/releases)\n[![Commits](https://img.shields.io/github/commit-activity/m/apache/kylin)](https://github.com/apache/kylin/commits/kylin5/)\n[![docs](https://img.shields.io/badge/Docs-Overview-blue.svg)](https://kylin.apache.org/docs/overview)\n\n\u003cdiv\u003e\n\n[![Website](\u003chttps://img.shields.io/badge/-Goto_Official_Website-153E7B?style=for-the-badge\u003e)](https://kylin.apache.org/)\n[![Download](\u003chttps://img.shields.io/badge/-Download_Kylin-ECA634?style=for-the-badge\u003e)](https://kylin.apache.org/docs/download)\n\n\u003c/div\u003e\n\u003c/div\u003e\n\n---\nApache Kylin is a leading open source OLAP engine for Big Data capable for sub-second query latency on trillions of records. Since being created and open sourced by eBay in 2014, and graduated to Top Level Project of Apache Software Foundation in 2015.\nKylin has quickly been adopted by thousands of organizations world widely as their critical analytics application for Big Data.\n\nKylin has following key strengths:\n\n- High qerformance, high concurrency, sub-second query latency\n- Unified big data warehouse architecture\n- Seamless integration with BI tools\n- Comprehensive and enterprise-ready capabilities\n\n![overview](https://svn.apache.org/repos/asf/kylin/site/img/homepage/home_arc.svg)\n\n\n## What's New in Kylin 5.0\n\n### 📊 1. Internal Table\nKylin now support internal table, which is designed for flexible query and lakehouse scenarios.\n\n### 🦁 2. Model \u0026 Index Recommendation\n\nWith recommendation engine, you don't have to be an expert of modeling. Kylin now can auto modeling and optimizing indexes from you query history.\nYou can also create model by importing sql text.\n\n### 👾 3. Native Compute Engine\n\nStart from version 5.0, Kylin has integrated Gluten-ClickHouse Backend(incubating in apache software foundation) as native compute engine. And use Gluten mergetree as the default storage format of internal table.\nWhich can bring 2~4x performance improvement compared with vanilla spark. Both model and internal table queries can get benefits from the Gluten integration.\n\n### 🧜🏻‍♀️ 4. Streaming Data Source\n\nKylin now support Apache Kafka as streaming data source of model building. Users can create a fusion model to implement streaming-batch hybrid analysis.\n\n## Significant Change\n\n### 🤖1. Metadata Refactory\nIn Kylin 5.0, we have refactored the metadata storage structure and the transaction process, removed the project lock and Epoch mechanism. This has significantly improved transaction interface performance and system concurrency capabilities.\n\nTo upgrade from 5.0 alpha, beta, follow the [Metadata Migration Guide](https://kylin.apache.org/docs/operations/system-operation/cli_tool/metadata_operation#migration)\n\nThe metadata migration tool for upgrading from Kylin 4.0 is not tested, please contact kylin user or dev mailing list for help.\n\n### Other Optimizations and Improvements\nPlease refer to [Release Notes](https://kylin.apache.org/docs/release_notes/) for more details.\n\n## Quick Start\n\n### 🐳 Play Kylin in Docker\n\nTo explore new features in Kylin 5 on a laptop, we recommend pulling the Docker image and checking the [Apache Kylin Standalone Image on Docker Hub](https://hub.docker.com/r/apachekylin/apache-kylin-standalone) (For amd64 platform).\n\n```shell\ndocker run -d \\\n    --name Kylin5-Machine \\\n    --hostname localhost \\\n    -e TZ=UTC \\\n    -m 10G \\\n    -p 7070:7070 \\\n    -p 8088:8088 \\\n    -p 9870:9870 \\\n    -p 8032:8032 \\\n    -p 8042:8042 \\\n    -p 2181:2181 \\\n    apachekylin/apache-kylin-standalone:5.0.0-GA\n```\n\n\n---\n### Introduction\n\nKylin utilizes multidimensional modeling theory to build star or snowflake schemas based on tables, making it a powerful tool for large-scale data analysis. The **model** is Kylin's core component, consisting of three key aspects: *model design*, *index design*, and *data loading*. By carefully designing the model, optimizing indexes, and pre-computed data, queries executed on Kylin can avoid scanning the entire dataset, potentially reducing response times to mere seconds, even for petabyte-scale data.\n\n+ **Model design** refers to establishing relationships between data tables to enable fast extraction of key information from multidimensional data. The core elements of model design are computed columns, dimensions, measures, and join relations.\n\n+ **Index design** refers to creating indexes (CUBEs) within the model to precompute query results, thereby reducing query response time. Well-designed indexes not only improve query performance but also help minimize the storage and data-loading costs associated with precomputation.\n\n+ **Data loading** refers to the process of importing data into the model, enabling queries to utilize the pre-built indexes rather than scanning the entire dataset. This allows for faster query responses by leveraging the model's optimized structure.\n\n\n\n### Core Concepts\n\n- **Dimension**: A perspective of viewing data, which can be used to describe object attributes or characteristics, for example, product category.\n\n- **Measure**: An aggregated sum, which is usually a continuous value, for example, product sales.\n\n- **Pre-computation**: The process of aggregating data based on model dimension combinations and of storing the results as indexes to accelerate data query.\n\n- **Index**: Also called CUBE, which is used to accelerate data query. Indexes are divided into:\n    - **Aggregate Index**: An aggregated combination of multiple dimensions and measures, and can be used to answer aggregate queries such as total sales for a given year.\n    - **Table Index**: A multilevel index in a wide table and can be used to answer detailed queries such as the last 100 transactions of a certain user.\n\n\n---\n\n### Why Use Kylin\n\n+ **Low Query Latency vs. Large Volume**\n\n  When analyzing massive data, there are some techniques to speed up computing and storage, but they cannot change the time complexity of query, that is, query latency and data volume are linearly dependent.\n\n  If it takes 1 minute to query 100 million entries of data records, querying 10 billion data entries will take about 1 hour and 40 minutes. When companies want to analyze all business data piled up over the years or to add complexity to query, say, with more dimensions, queries will be running extremely slow or even time out.\n\n  ![Response Time vs. Data Volume](https://kylin.apache.org/assets/images/volume_per_time-5c703faf474c956998a461b67470e86c.png)\n\n+ **Pre-computation vs. Runtime Computation**\n\n  Pre-computation and runtime computation are two approaches to calculating results in data processing and analytics. **Pre-computation** involves calculating and storing results in advance, so they can be quickly retrieved when a query is run. In contrast, **runtime computation** dynamically computes results during query execution, processing raw data and applying aggregations, filters, or transformations as needed for each query.\n\n  Kylin primarily focuses on **pre-computation** to enhance query performance. However, we also offer advanced features that partially support runtime computation. For more details, please refer to [Table Snapshot](https://kylin.apache.org/docs/model/snapshot/), [Runtime Join](https://kylin.apache.org/docs/model/features/runtime_join), and [Internal Table](https://kylin.apache.org/docs/internaltable/intro).\n\n\n+ **Manual Modeling vs. Recommendation**\n\n  Before Kylin 5.0, model design had to be done manually, which was a tedious process requiring extensive knowledge of multidimensional modeling. However, this changed with the introduction of Kylin 5.0. We now offer a new approach to model design, called **recommendation**, which allows models to be created by importing SQL, along with an automatic way to remove unnecessary indexes. Additionally, the system can leverage query history to generate index recommendations, further optimizing query performance. For more details, please refer to [Recommendation](https://kylin.apache.org/docs/model/rec/intro).\n\n\n+ **Batch Data vs. Streaming Data**\n\n  In the OLAP field, data has traditionally been processed in batches. However, this is changing as more companies are now required to handle both batch and streaming data to meet their business objectives. The ability to process data in real-time has become increasingly critical for applications such as real-time analytics, monitoring, and event-driven decision-making.\n\n  To address these evolving needs, we have introduced support for streaming data in the new version. This allows users to efficiently process and analyze data as it is generated, complementing the traditional batch processing capabilities. For more details, please refer to [Streaming](https://kylin.apache.org/docs/model/streaming/intro).\n","funding_links":[],"categories":["Java","Uncategorized","Data Warehouse","大数据"],"sub_categories":["Uncategorized","Data Table Format"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fkylin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fkylin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fkylin/lists"}