{"id":13488115,"url":"https://github.com/huangfox/dpkb","last_synced_at":"2025-03-27T23:32:53.649Z","repository":{"id":51227960,"uuid":"349008268","full_name":"huangfox/dpkb","owner":"huangfox","description":"大数据相关内容汇总，包括分布式存储引擎、分布式计算引擎、数仓建设等。关键词：Hadoop、HBase、ES、Kudu、Hive、Presto、Spark、Flink、Kylin、ClickHouse","archived":false,"fork":false,"pushed_at":"2024-12-05T06:46:16.000Z","size":82,"stargazers_count":223,"open_issues_count":0,"forks_count":58,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-12-05T07:30:26.071Z","etag":null,"topics":["flink","hadoop","hbase","hive","presto","spark"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huangfox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-03-18T09:12:20.000Z","updated_at":"2024-12-05T06:46:20.000Z","dependencies_parsed_at":"2024-01-14T03:51:13.178Z","dependency_job_id":"de5e4b39-2cde-47a7-a967-6a5a5222c223","html_url":"https://github.com/huangfox/dpkb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huangfox%2Fdpkb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huangfox%2Fdpkb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huangfox%2Fdpkb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huangfox%2Fdpkb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huangfox","download_url":"https://codeload.github.com/huangfox/dpkb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245944036,"owners_count":20697945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flink","hadoop","hbase","hive","presto","spark"],"created_at":"2024-07-31T18:01:09.877Z","updated_at":"2025-03-27T23:32:48.618Z","avatar_url":"https://github.com/huangfox.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# DPKB\n\n大数据相关知识库，主要包括：\n* 数据存储层、数据库（HDFS、Hive、HBase、Kudu、Doris、StarRocks、ClickHouse、TiDB等）\n* 数据处理层、OLAP引擎（Spark、Flink、Presto、Trino等）\n* 数据湖（IceBerg、Hudi、Delta等）\n* 大数据开发、应用（主要包括ETL、调度、数仓、数据应用等，例如Seatunnel、Dolphinscheduler等）\n* 数据治理（元数据管理、数据模型、数据标准、数据质量、数据安全等）\n\n持续更新中（2023-11）\n\n\n\n\n\n## 一、数据存储层、数据库（HDFS、Hive、HBase、Kudu、Doris、StarRocks、ClickHouse、TiDB等）\n\n### ▶ HDFS\n\n\n\n\n### ▶ Hive\n#### 1）官网、社区、博客\n- [Hive 官网](https://hive.apache.org/)\n\n\n#### 2）专栏\n- [Hive 教程](columns/hive/hive教程.md)\n\n\n#### 3）大厂实践\n- [HiveCube 在有赞的实践](https://tech.youzan.com/cube/)    2019-11\n- [Hive Metastore Federation 在滴滴的实践](https://blog.didiyun.com/index.php/2019/03/25/hive-metastore-federation/)    2019-03\n\n\n\n\n### ▶ HBase\n#### 1）官网、社区、博客\n- [HBase 官网](https://hbase.apache.org/)\n- [hbasefly](http://hbasefly.com/) \n\n\n#### 2）专栏\n\n\n#### 3）大厂实践\n\n\n#### 4）其他\n- [HBase Bulkload 实践探讨](https://tech.youzan.com/hbase-bulkloadshi-practice/)    2019-12\n\n\n\n\n### ▶ Kudu\n#### 1）官网、社区、博客\n- [Kudu 官网](https://kudu.apache.org/)\n\n\n#### 2）专栏\n- [Kudu 原理 论文](columns/kudu/Kudu原理论文.md)\n- [网易云Kudu技术专栏](columns/kudu/网易云Kudu技术文章.md)\n\n\n#### 3）大厂实践\n- [Apache Kudu 在网易的实践](https://www.infoq.cn/article/kgwyqb5wer5wl8cquweq)   2021-08\n- [Apache Kudu 在网易实时数仓的实践](https://www.infoq.cn/article/QETxjyIu5tAJTZ9ksMdu)    2020-02\n- [Kudu架构介绍及其在小米的应用实践](https://www.modb.pro/db/119708)    2017-06\n\n\n#### 4) 其他\n- [我是如何成为Apache Kudu committer \u0026 PMC 的？](https://cloud.tencent.com/developer/article/1450749)    2019-06\n\n\n\n\n### ▶ Doris\n#### 1）官网、社区、博客\n- [Doris 官网](https://doris.apache.org/)\n- [Doris github](https://github.com/apache/doris)\n- [Doris 论坛](https://github.com/apache/incubator-doris/discussions)\n\n\n#### 2）专栏\n- [Doris全面解析](columns/doris/Doris全面解析.md)\n- [Doris最佳实践](columns/doris/Doris最佳实践.md)\n\n\n#### 3）案例实践\n- [Apache Doris在美团外卖数仓中的应用实践](https://tech.meituan.com/2020/04/09/doris-in-meituan-waimai.html)    2020-04\n- [Apache Doris 在韵达物流领域的应用实践](https://mp.weixin.qq.com/s/Z_PhWk92ctZ7slz4SrVZ9Q)    2021-07\n- [Apache Doris 在蜀海供应链的实践](https://mp.weixin.qq.com/s/SHuE-KCsIyh6jfo0DqLD6w)    2021-07\n- [京东物流基于 Doris 的亿级数据自助探索应用](https://mp.weixin.qq.com/s/qVFa40yMg0_N9Lsb10ACQA)    2021-07\n- [Doris on ES在快手商业化的最佳实践](https://mp.weixin.qq.com/s/5Pc5ewVFWPgauG4hNLH9xw)    2021-08\n- [基于Doris的有道精品课数据中台建设实践](https://mp.weixin.qq.com/s/Gz-au9CHJ4lHrs5MkzeAJg)    2020-12\n- [美团外卖实时数仓建设实践](https://mp.weixin.qq.com/s/-JPWqa_-at7F5hZ0zekVSQ)    2020-10\n- [Doris在作业帮实时数仓中的应用\u0026实践](https://mp.weixin.qq.com/s/hjbMM8CbElO04VLN5cfJtQ)    2020-09\n- [基于Apache Doris的小米增长分析平台实践](https://mp.weixin.qq.com/s/WeNAItPJ4b7fsqW4kf0dSA)    2020-08\n- [Apache Doris在京东双十一大促中的实践](https://mp.weixin.qq.com/s/8XnwJXm4kzq56SvElwL6kA)    2020-03\n- [Apache Doris 在百度商业大规模微服务全链路监控的实践](https://mp.weixin.qq.com/s/k7CcCdHPTK1ZTDs_qKgh5w)    2020-02\n\n\n\n\n### ▶ StarRocks\n#### 1）官网、社区、博客\n- [StarRocks](https://www.starrocks.com/zh-CN/index)\n- [StarRocks文档](https://docs.starrocks.com/zh-cn/main/introduction/StarRocks_intro)\n- [编程小梦 康凯森](https://blog.bcmeng.com/)\n\n\n#### 2) 专栏\n- [StarRocks技术内幕](columns/starrocks/StarRocks技术内幕.md)\n\n\n\n\n### ▶ ClickHouse\n#### 1）官网、社区、博客\n- [ClickHouse 官网](https://clickhouse.com/)\n\n\n#### 2）专栏\n\n\n#### 3）大厂实践\n- [ClickHouse 在有赞的实践之路](https://tech.youzan.com/clickhouse-zai-you-zan-de-shi-jian-zhi-lu/)    2021-01\n\n\n#### 4）其他\n\n\n\n\n## 二、数据处理层、OLAP引擎（Spark、Flink、Presto、Trino等）\n\n### ▶ Spark\n#### 1）官网、社区、博客\n- [Spark 官网](https://spark.apache.org/)\n\n\n#### 2）专栏\n- [Apache Spark 的设计与实现](columns/spark/Apache%20Spark的设计与实现.md)\n\n\n#### 3）大厂实践\n- [SparkSQL 在有赞的实践](https://tech.youzan.com/sparksql-in-youzan/)    2019-01\n- [SparkSQL 在有赞大数据的实践（二）](https://tech.youzan.com/sparksql-in-youzan-2/)    2020-01\n\n\n\n\n### ▶ Flink\n#### 1）官网、社区、博客\n- [Flink 官网](https://flink.apache.org/)\n- [Flink Confluence](https://cwiki.apache.org/confluence/display/FLINK/)\n- [Flink Blog](https://flink.apache.org/blog/)\n- [Ververica Blog](https://www.ververica.com/blog?hsLang=en) \n- [Ververica 中文](https://ververica.cn/developers-resources/)\n- [Flink 知识图谱](https://ververica.cn/wp-content/uploads/2020/03/Apache-Flink-Stateful-Computations-over-Data-Streams.pdf)\n- [Jark's Blog - WuChong - 云邪](http://wuchong.me/)\n\n\n#### 2）专栏\n- [Flink 架构、源码分析专栏](columns/flink/Flink架构、源码分析专栏.md)\n- [Flink 实战系列](columns/flink/Flink实战系列.md)\n- [Flink 开源项目汇总](columns/flink/Flink开源项目汇总.md)\n##### 教程\n- [Flink SQL Cookbook - Ververica](https://github.com/ververica/flink-sql-cookbook/)\n- [Flink 零基础入门](columns/flink/Flink零基础入门.md)\n- [Flink 进阶教程](columns/flink/Flink进阶教程.md)\n- [Apache Flink 漫谈系列](columns/flink/Apache%20Flink%20漫谈系列.md)\n- [Flink 相关论文](columns/flink/Flink%20相关论文.md)\n\n\n#### 3）大厂实践\n- [flink-forward-asia-hackathon-2021](https://github.com/flink-china/flink-forward-asia-hackathon-2021/issues)\n\n\n\n\n### ▶ Presto、Trino\n#### 1）官网、社区、博客\n- [PrestoDB 官网](https://prestodb.io/)\n- [Trino 官网](https://trino.io/)     原PrestoSql\n- [Google Presto Group](https://groups.google.com/g/presto-users)\n- [Presto 知乎专栏](https://www.zhihu.com/column/presto-cn)\n- [若飞-技术博客](http://armsword.com/archives/)\n\n\n#### 2）专栏\n- [Presto 架构、源码分析专栏](columns/presto/Presto架构、源码分析专栏.md)\n- [Presto 最佳实践、调优、踩坑专栏](columns/presto/Presto最佳实践、调优、踩坑专栏.md)\n- [Presto 资料汇总、会议资讯专栏](columns/presto/Presto资料汇总、会议资讯专栏.md)\n\n\n#### 3）大厂实践\n- [Presto 在车好多的实践](https://mp.weixin.qq.com/s/Bmqv54sVZgTqQ82I_RfmsA)    2020-12\n- [Presto 在滴滴的探索与实践](https://zhuanlan.zhihu.com/p/266162270)    2020-10\n- [Presto 在有赞的实践之路](https://tech.youzan.com/presto-zai-you-zan-de-shi-jian-zhi-lu/)    2020-04\n- [PrestoCon 2020：云原生数据湖分析DLA的Presto实践](https://zhuanlan.zhihu.com/p/260784762)    2020-03\n- [携程 Presto 技术演进之路](https://zhuanlan.zhihu.com/p/41538472)    2018-08\n- [Presto 实现原理和美团的使用实践](https://tech.meituan.com/2014/06/16/presto.html)    2014-06\n- [Presto 高性能引擎在美图的实践](https://zhuanlan.zhihu.com/p/408957032)    2021-09\n\n\n\n\n## 三、数据湖（IceBerg、Hudi、Delta等）\n\n- [一文看懂：什么是数据库、数据湖、数据仓库、湖仓一体、智能湖仓？](https://www.smartcity.team/consultingskills/experience/shujukuyushujuhu/#comments)    2021-08\n\n\n### ▶ Iceberg\n#### 1）官网、社区、博客\n- [Iceberg 官网](https://iceberg.apache.org/)\n\n\n#### 2）应用\n- [数据湖 Iceberg | 实时数据仓库的发展、架构和趋势](https://mp.weixin.qq.com/s?__biz=MzIwNTUxNTI1Ng==\u0026mid=2247485623\u0026idx=1\u0026sn=9f03a36dbfc06c712b6132faabaa1dfd\u0026chksm=972ef820a05971360311fd69c686e4b420222cfa639a1bcb5648bece4c3d886ae8f981712d8c\u0026scene=21#wechat_redirect)    2021-03\n- [数据湖 Iceberg | Apache Iceberg 快速入门](https://mp.weixin.qq.com/s?__biz=MzIwNTUxNTI1Ng==\u0026mid=2247485637\u0026idx=1\u0026sn=0489f233e3bda2bcef221c9532bb001e\u0026chksm=972ef852a0597144538b7807948443a27e58f99ba33d17a7bcb12ccb8b382fd1d712d6e80cbc\u0026cur_album_id=1746684202856579076\u0026scene=190#rd)    2021-03\n- [数据湖 Iceberg | 如何正确使用 Iceberg](https://mp.weixin.qq.com/s?__biz=MzIwNTUxNTI1Ng==\u0026mid=2247485644\u0026idx=1\u0026sn=b2194d8f3c1e7cf7e8e8d9296b9025e2\u0026chksm=972ef85ba059714dc69472e3860497389f2ca4503d2cddeedd348695b5c314da49aad0278978\u0026cur_album_id=1746684202856579076\u0026scene=190#rd)    2021-04\n- [数据湖 Iceberg | 在网易云音乐的实践](https://mp.weixin.qq.com/s?__biz=MzIwNTUxNTI1Ng==\u0026mid=2247485718\u0026idx=1\u0026sn=34347ac54e97877e4401ad37f1d15577\u0026chksm=972ef981a059709724b7abab56786ef047a68f31fd829031d2214fa4994b9ec0f1b04e25318c\u0026cur_album_id=1746684202856579076\u0026scene=190#rd)    2021-04\n\n\n\n\n### ▶ Hudi\n#### 1）官网、社区、博客\n- [Hudi 官网](https://hudi.apache.org/)\n\n#### 2）应用\n- [Flink CDC + Hudi + Hive + Presto 构建实时数据湖最佳实践](https://mp.weixin.qq.com/s/079VeDeIM_MQPyiiDX2l_w)\n \n\n\n\n### ▶ Delta\n\n\n\n\n## 四、大数据开发、应用（主要包括ETL、调度、数仓、数据应用等，例如Seatunnel、Dolphinscheduler等）\n\n### ▶ Seatunnel\n\n\n\n### ▶ DolphinScheduler\n\n\n\n### ▶ 大数据架构\n- [SQL on Hadoop 在快手大数据平台的实践与优化](https://www.infoq.cn/article/BN9cJjg1t-QSWE6fqkoR)    2019-06\n- [携程机票大数据架构最佳实践](https://dbaplus.cn/news-73-1420-1.html)    2017-08\n- [火山引擎DataLeap一站式数据治理解决方案及平台架构](https://www.cnblogs.com/bytedata/p/17745908.html)    2023-10\n\n\n\n### ▶ 数仓相关\n- [有赞数据仓库实践之路](https://tech.youzan.com/dw-in-youzan/)    2020-03\n- [OneData 建设探索之路：SaaS 收银运营数仓建设](https://tech.meituan.com/2019/10/17/meituan-saas-data-warehouse.html)    2019-10\n- [面向AI技术的工程架构实践 | 贝壳一站式大数据开发平台实践](https://www.infoq.cn/article/mmnwzdlcyjg83qm0tgqm)    2020-11\n\n\n\n\n### ▶ 报表平台\n- [有赞 BI 平台实现原理](https://tech.youzan.com/principle-on-bi-platform/)    2021-01\n\n\n\n\n## 五、数据治理（元数据管理、数据模型、数据标准、数据质量、数据安全等）\n\n### ▶ 数据治理\n- [美团配送数据治理实践](https://tech.meituan.com/2020/03/12/delivery-data-governance.html)    2020-03\n- [全链路数据治理在网易严选的实践](https://www.infoq.cn/article/FOV6aEWRGNOfhD91YVcr)    2020-10\n- [数据资产、数据治理 - 有赞](https://tech.youzan.com/shu-ju-zi-chan-zan-zhi-zhi-li/)    2019-11\n\n\n\n\n### ▶ 元数据管理\n- [字节跳动构建Data Catalog数据目录系统的实践](https://www.cnblogs.com/bytedata/p/16189474.html)    2022-04\n- [有赞数据仓库元数据系统实践](https://tech.youzan.com/youzan-metadata/)    2018-08\n- [饿了么元数据管理实践之路](https://dbaplus.cn/news-73-2143-1.html)    2018-07\n- [数据治理方案技术调研 Atlas VS Datahub VS Amundsen](https://cloud.tencent.com/developer/article/1746714)    2020-11\n- [数据资产治理-元数据采集那点事 - 有赞](https://tech.youzan.com/zi-chan-zhi-li-yuan-shu-ju-cai-ji-na-dian-shi/)    2020-12\n- [来看看字节跳动内部的数据血缘用例与设计](https://segmentfault.com/a/1190000041452770)    2022-02\n- [携程数据血缘构建及应用](https://mp.weixin.qq.com/s/LGK3YPZCe6oPTf48QaAIqA)    2021-09\n- [Datahub](https://datahubproject.io/)    A Metadata Platform for the Modern Data Stack\n\n\n\n\n### ▶ 数据标准\n- [有赞指标库实践](https://tech.youzan.com/you-zan-zhi-biao-ku-shi-jian/)    2020-03\n\n\n\n\n### ▶ 数据安全\n- [浅谈有赞大数据安全体系](https://tech.youzan.com/you-zan-da-shu-ju-an-quan-ti-xi-jian-she-shi-jian/)    2021-01\n\n\n\n## 六、机器学习、AI\n\n### ▶ 机器学习平台\n- [机器学习平台建设指南](https://mp.weixin.qq.com/s/HEg_6Gly2WMrcPD5Ao2n6g)    2021-04\n- [一站式机器学习平台建设实践](https://mp.weixin.qq.com/s/ZDRD0vAxkSqe4UeXi9avKQ)    2020-02\n- [汽车之家机器学习平台的架构与实践](https://blog.csdn.net/hellozhxy/article/details/107210015)    2020-07\n- [微博推荐算法实践与机器学习平台演进](https://blog.csdn.net/m0_37586850/article/details/116465255)    2021-05\n- [爱奇艺机器学习平台的建设实践](https://mp.weixin.qq.com/s/Np4w7RC2JFlB7ZGIduu71w)    2020-11\n- [爱奇艺一站式机器学习平台Deepthought的建设与初探](https://mp.weixin.qq.com/s?__biz=MzI0MjczMjM2NA==\u0026mid=2247487206\u0026idx=1\u0026sn=c8db1e12378376722a1521f409149d44\u0026chksm=e97692c5de011bd3f1b42a8112cd04c24907cb101ac5474b0054c95941ff5c4769a42d496f3a\u0026scene=21#wechat_redirect)    2020-06\n- [一站式机器学习平台在 vivo AI 的实践](https://www.infoq.cn/article/THlkStomYLRgXL2hzm8w)    2020-02\n- [再见，Yarn！滴滴机器学习平台架构演进](https://mp.weixin.qq.com/s/iTfHv8EFx4O4G1sNxsuMkg)    2019-03\n- [网易严选机器学习平台建设实践](https://www.6aiq.com/article/1661745581086)    2022\n- [Sunfish-有赞智能平台实践](https://tech.youzan.com/sunfish/)    2020-06\n\n\n\n\n## 七、资源汇总\n\n### ▶ 大厂技术博客\n- [美团技术团队](https://tech.meituan.com/)\n- [有赞技术团队](https://tech.youzan.com/)\n- [滴滴云博客](https://blog.didiyun.com/)\n\n\n\n### ▶ 大数据相关网站\n- [dbaplus](https://dbaplus.cn/)\n\n\n\n### ▶ 相关开源项目\n- [数仓相关开源项目汇总](columns/opensource/数仓相关开源项目汇总.md)\n\n\n\n### ▶ 相关论文\n- [raft 中文翻译](https://github.com/maemual/raft-zh_cn/blob/master/raft-zh_cn.md)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuangfox%2Fdpkb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuangfox%2Fdpkb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuangfox%2Fdpkb/lists"}