{"id":20226876,"url":"https://github.com/qihoo360/xsql","last_synced_at":"2025-04-09T19:20:12.418Z","repository":{"id":37334630,"uuid":"205325240","full_name":"Qihoo360/XSQL","owner":"Qihoo360","description":"Unified SQL Analytics Engine Based on SparkSQL","archived":false,"fork":false,"pushed_at":"2022-12-05T23:44:38.000Z","size":15991,"stargazers_count":210,"open_issues_count":30,"forks_count":62,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-04-09T19:20:03.830Z","etag":null,"topics":["datasource","elasticsearch","federation","hive","spark","sql"],"latest_commit_sha":null,"homepage":"https://qihoo360.github.io/XSQL/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Qihoo360.png","metadata":{"files":{"readme":"README-CN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-30T07:07:05.000Z","updated_at":"2024-10-22T03:14:48.000Z","dependencies_parsed_at":"2023-01-23T05:31:03.599Z","dependency_job_id":null,"html_url":"https://github.com/Qihoo360/XSQL","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2FXSQL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2FXSQL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2FXSQL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2FXSQL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Qihoo360","download_url":"https://codeload.github.com/Qihoo360/XSQL/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248094991,"owners_count":21046770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasource","elasticsearch","federation","hive","spark","sql"],"created_at":"2024-11-14T07:20:57.175Z","updated_at":"2025-04-09T19:20:12.388Z","avatar_url":"https://github.com/Qihoo360.png","language":"Scala","readme":"![XSQL-logo](./sql/xsql/docs/docs/images/XSQL-300200.png)\n\n[English](https://github.com/Qihoo360/XSQL/blob/master/README.md) | [中文](https://github.com/Qihoo360/XSQL/blob/master/README-CN.md)\n\nXSQL是一款低门槛、更稳定的跨数据源查询引擎。1）XSQL提供了使用标准SQL读写NoSQL数据库的解决方案，能够大幅降低学习曲线，节省人力成本。 2）XSQL通过对SQL的优化以及一系列监控插件，为用户任务的稳定运行保驾护航。\n\n[https://qihoo360.github.io/XSQL/](https://qihoo360.github.io/XSQL/)\n\n## 功能特性\n\n- 内置8种数据源，包括：Hive、Mysql、EleasticSearch、Mongo、Kafka、Hbase、Redis、Druid等。\n- XSQL采用数据源（DataSource）、数据库（Database）、数据表（Table）的三层元信息，为异构数据源提供了统一视图，进而实现了跨数据源的数据关联\n- SQL Everything，将程序与数据源具体版本解耦，程序迁移能力得到加强\n- 对DDL、DML、可下推查询，延迟与Yarn的交互及资源申请，进而提升效率并节省资源。\n- 相比很多开源分布式查询引擎，XSQL替换了Spark SQL，因而只需要一次SQL解析，避免多次解析带来的时延。\n- XSQL允许用户将聚合、过滤、投影等操作下推至数据源计算引擎，相比DataSet API更容易实现毫秒级响应。\n- XSQL借鉴了业内优秀的开源项目，放弃元数据的中心化，因此避免了数据同步、数据不一致，数据延迟等不利因素。XSQL也因此在部署上更加轻量、简便。\n- XSQL对元数据的缓存有两种级别，既能减少对底层数据源的压力，也提升了XSQL的执行效率。\n- XSQL可以按照用户需要，设置元数据白名单来避免缓存多余的元信息，进一步提升执行效率。\n- 可适配到Spark 2.x任意版本，解压即可运行，不需要引入额外依赖。且与原生SparkSQL隔离运行，不影响现有程序运行\n\n## 编译\u0026部署\u0026运行指南\n\n### 编译环境依赖\n\n- jdk 1.8+\n\n### 编译步骤：\n\n1. 源码下载\n\n   ```\n   git clone https://github.com/Qihoo360/XSQL\n   ```\n\n2. 执行编译命令\n\n   如果你已经有现成的Spark，可以将XSQL编译为Spark插件：\n\n   ```\n   XSQL/build-plugin.sh\n   ```\n   编译成功后源码根目录将出现xsql-0.6.0-plugin-spark-2.4.3.tgz\n\n   如果你没有现成的Spark，那么你可以编译完整版的XSQL（内置Spark）：\n\n   ```\n   XSQL/build.sh\n   ```\n   编译成功后源码根目录将出现xsql-0.6.0-bin-spark-2.4.3.tgz\n\n### 运行环境依赖\n\n- jdk 1.8+\n\n- hadoop 2.7.2+\n\n- spark 2.4.x\n\n### 部署步骤：\n\n1. 解压到客户端\n\n   完整版解压：\n\n   ```shell\n   tar xvf xsql-0.6.0-bin-spark-2.4.3.tgz -C \"软件目录\"\n   ```\n\n   插件版解压：\n\n   ```shell\n   tar xvf xsql-0.6.0-plugin-spark-2.4.3.tgz -C \"已存在的SPARK_HOME\"\n   ```\n\n2. 配置数据源\n\n   这里给出一个简单的例子：\n\n   ```\n   spark.xsql.datasources                     default\n   spark.xsql.default.database                real_database\n   spark.xsql.datasource.default.type         mysql\n   spark.xsql.datasource.default.url          jdbc:mysql://127.0.0.1:2336\n   spark.xsql.datasource.default.user         real_username\n   spark.xsql.datasource.default.password     real_password\n   spark.xsql.datasource.default.version      5.6.19\n   ```\n\n### 运行示例\n\n1. 从命令行工具开始\n\n   ```shell\n   $SPARK_HOME/bin/spark-xsql\n   spark-xsql\u003e show datasources;\n   ```\n\n2. 从Scala API开始\n\n   ```scala\n   var spark = SparkSession\n     .builder()\n     .enableXSQLSupport()\n     .getOrCreate()\n   spark.sql(\"show datasources\")\n   ```\n\n\n## FAQ\n\n[访问更多数据源](https://qihoo360.github.io/XSQL/datasources/common/)\n\n[高级配置](https://qihoo360.github.io/XSQL/tutorial/configuration/)\n\n[特殊语法](https://qihoo360.github.io/XSQL/tutorial/syntax/)\n\n## 联系我们\n\nMail: XSQL开发者组（xsql_dev@groups.163.com）、XSQL用户组（xsql_user@groups.163.com）\n\nQQ群: XSQL用户群（838910008）\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqihoo360%2Fxsql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqihoo360%2Fxsql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqihoo360%2Fxsql/lists"}