{"id":14965806,"url":"https://github.com/water8394/bigdata-interview","last_synced_at":"2025-05-15T23:05:13.931Z","repository":{"id":41180038,"uuid":"201843208","full_name":"water8394/BigData-Interview","owner":"water8394","description":":dart: :star2:[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结","archived":false,"fork":false,"pushed_at":"2021-08-30T09:16:28.000Z","size":6911,"stargazers_count":1613,"open_issues_count":5,"forks_count":446,"subscribers_count":52,"default_branch":"master","last_synced_at":"2025-05-15T23:05:04.406Z","etag":null,"topics":["bigdata","flink","hadoop","hbase","hdfs","interview","interview-questions","kafka","mapreduce","spark","yarn"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/water8394.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-12T02:27:22.000Z","updated_at":"2025-05-14T08:26:54.000Z","dependencies_parsed_at":"2022-07-10T16:17:20.727Z","dependency_job_id":null,"html_url":"https://github.com/water8394/BigData-Interview","commit_stats":null,"previous_names":["water8394/bigdata-interview"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/water8394%2FBigData-Interview","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/water8394%2FBigData-Interview/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/water8394%2FBigData-Interview/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/water8394%2FBigData-Interview/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/water8394","download_url":"https://codeload.github.com/water8394/BigData-Interview/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","flink","hadoop","hbase","hdfs","interview","interview-questions","kafka","mapreduce","spark","yarn"],"created_at":"2024-09-24T13:35:20.101Z","updated_at":"2025-05-15T23:05:08.910Z","avatar_url":"https://github.com/water8394.png","language":null,"readme":"###### 发一条内推消息，社招、实习生欢迎投递\n###### 字节跳动校招内推码: 3UWFMRS \n###### 投递链接: https://jobs.toutiao.com/s/eFCv3cB\n###### 社招：https://job.toutiao.com/s/eFCPHb6\n--- \n\n#### 大数据面试题汇总与答案分享\n\n------\n\n\u003ctable\u003e\n    \u003ctr\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/hadoop.jpg\"\u003e\u003c/th\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/hive.jpg\"\u003e\u003c/th\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/spark.jpg\"\u003e\u003c/th\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/flink.png\"\u003e\u003c/th\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/hbase.png\"\u003e\u003c/th\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/kafka.png\"\u003e\u003c/th\u003e\n     \u003cth\u003e\u003cimg width=\"50px\" src=\"./pictures/zookeeper.jpg\"\u003e\u003c/th\u003e\n    \u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#一hadoop\"\u003eHadoop\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#二hive\"\u003eHive\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#三spark\"\u003eSpark\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#四flink\"\u003eFlink\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#五hbase\"\u003eHBase\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#六kafka\"\u003eKafka\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"#七zookeeper\"\u003eZookeeper\u003c/a\u003e\u003c/td\u003e\n\u003c/tr\u003e\n    \u003c/table\u003e\n\n\n## 一、Hadoop\n\n1. [HDFS架构](./docs/HDFS架构.md)\n\n2. [Yarn架构](./docs/Yarn架构.md)\n\n3. [MapReduce过程](./docs/MapReduce过程.md)\n\n4. [Yarn 调度MapReduce](./docs/Yarn调度MapReduce.md)\n\n5. [hdfs写流程](./docs/hdfs写流程.md)\n\n6. [hdfs读流程](./docs/hdfs读流程.md)\n\n7. [hdfs创建一个文件的流程](./docs/hdfs创建一个文件的流程.md)\n\n8. [hadoop1.x 和hadoop 2.x 的区别](./docs/hadoop1.x和hadoop2.x的区别.md)\n\n9. [hadoop1.x的缺点](./docs/hadoop1.x的缺点.md)\n\n10. [hadoop HA介绍](./docs/hadoopHA介绍.md)\n\n11. [hadoop的常用配置文件有哪些,自己实际改过哪些?](./docs/hadoop的常用配置文件有哪些.md)\n\n12. [小文件过多会有什么危害,如何避免?](./docs/小文件过多会有什么危害.md)\n\n13. [启动hadoop集群会分别启动哪些进程,各自的作用](./docs/启动hadoop集群会分别启动哪些进程.md)\n\n14. 讲一下环形缓冲区的概念\n\n    \n\n## 二、Hive\n\n1. [hive 内部表和外部表的区别](./docs/hive内部表和外部表的区别.md)\n\n2. [hive中 sort by / order by / cluster by / distribute by 的区别](./docs/hive四种排序方式的区别.md)\n\n3. [hive的metastore的三种模式](./docs/hive的metastore的三种模式.md)\n\n4. [hive 中 join都有哪些](./docs/hive中join都有哪些.md)\n\n5. [Impala 和 hive 的查询有哪些区别](./docs/Impala和hive的查询有哪些区别.md)\n\n6. [Hive中大表join小表的优化方法](./docs/Hive中大表join小表的优化方法.md)\n\n7. [Hive Sql 是怎样解析成MR job的?](./docs/HiveToMR.md)\n\n8. [Hive UDF简单介绍](./docs/HiveUDF简单介绍.md)\n\n9. [SQL题: 按照学生科目分组, 取每个科目的TopN](./docs/按照学生科目取每个科目的TopN.md)\n\n10. [SQL题: 获取每个用户的前1/4次的数据](./docs/获取每个用户的前1/4次的数据.md)\n\n    \n\n## 三、Spark\n\n1. [讲一下spark 的运行架构](./docs/讲一下spark的运行架构.md)\n2. [一个spark程序的执行流程](./docs/一个spark程序的执行流程.md)\n3. [spark的shuffle介绍](./docs/spark的shuffle介绍.md)\n4. [Spark的 partitioner 都有哪些?](./docs/Spark的partitioner都有哪些.md)\n5. [spark 有哪几种join](./docs/spark有哪几种join.md)\n6. [RDD有哪些特点](./docs/RDD有哪些特点.md)\n7. [讲一下宽依赖和窄依赖](./docs/讲一下宽依赖和窄依赖.md)\n8. [Spark中的算子都有哪些](./docs/Spark中的算子都有哪些.md)\n9. [RDD的缓存级别都有哪些](./docs/RDD的缓存级别都有哪些.md)\n10. [RDD 懒加载是什么意思](./docs/RDD懒加载是什么意思.md)\n11. [讲一下spark的几种部署方式](./docs/讲一下spark的几种部署方式.md)\n12. [spark on yarn 模式下的 cluster模式和 client模式有什么区别](./docs/spark中cluster模式和client模式有什么区别.md)\n13. [spark运行原理,从提交一个jar到最后返回结果,整个过程](./docs/spark从提交一个jar到最后返回结果.md)\n14. [spark的stage是如何划分的](./docs/spark的stage是如何划分的.md)\n15. [spark的rpc: spark2.0为什么放弃了akka 而用netty?](./docs/spark2.0为什么放弃了akka而用netty.md)\n16. [spark的各种HA,  master/worker/executor/driver/task的ha](./docs/spark的各种HA.md)\n17. [spark的内存管理机制,spark 1.6前后分析对比, spark2.0 做出来哪些优化](./docs/spark的内存管理机制.md)\n18. [讲一下spark 中的广播变量](./docs/spark中的广播变量.md)\n19. [什么是数据倾斜,怎样去处理数据倾斜](./docs/怎样去处理数据倾斜.md)\n20. [分析一下一段spark代码中哪些部分在Driver端执行,哪些部分在Worker端执行](./docs/分析一下一段spark代码中哪些部分在Driver端执行.md)\n\n## 四、Flink\n\n1. [讲一下flink的运行架构](./docs/讲一下flink的运行架构.md)\n\n2. [讲一下flink的作业执行流程](./docs/讲一下flink的作业执行流程.md)\n\n3. [flink具体是如何实现exactly once 语义](./docs/flink具体是如何实现exactlyonce语义.md)\n\n4. [flink 的 window 实现机制](./docs/flink的window实现机制.md)\n\n5. [flink的window分类](./docs/flink的window分类.md)\n\n6. [flink 的 state 是存储在哪里的](./docs/flink的state是存储在哪里的.md)\n\n7. [flink是如何实现反压的](./docs/flink是如何实现反压的.md)\n\n8. [flink的部署模式都有哪些](./docs/flink的部署模式都有哪些.md)\n\n9. [讲一下flink on yarn的部署](./docs/讲一下flinkonyarn的部署.md)\n\n10. [flink中的时间概念 , eventTime 和 processTime的区别](./docs/flink中的时间概念.md)\n\n11. [flink中的session Window怎样使用](./docs/flink中的sessionWindow怎样使用.md)\n\n    \n\n\n## 五、HBase\n\n1. [讲一下 Hbase 架构](./docs/讲一下Hbase架构.md)\n2. [hbase 如何设计 rowkey](./docs/hbase如何设计rowkey.md)\n3. [讲一下hbase的存储结构,这样的存储结构有什么优缺点](./docs/讲一下hbase的存储结构.md)\n4. [hbase的HA实现,zookeeper在其中的作用](./docs/hbase的HA实现.md)\n5. [HMaster宕机的时候,哪些操作还能正常工作](./docs/HMaster宕机.md)\n6. [讲一下hbase的写数据的流程](./docs/讲一下hbase的写数据的流程.md)\n7. [讲一下hbase读数据的流程](./docs/讲一下hbase读数据的流程.md)\n\n## 六、Kafka\n\n1. [讲一下 kafka 的架构](./docs/讲一下kafka的架构.md)\n\n2. [kafka 与其他消息组件对比?](./docs/kafka与其他消息组件对比.md)\n\n3. [kafka 实现高吞吐的原理](./docs/kafka实现高吞吐的原理.md)\n\n4. [kafka怎样保证不重复消费](./docs/kafka怎样保证不重复消费.md)\n\n5. [kafka怎样保证不丢失消息](./docs/kafka怎样保证不丢失消息.md)\n\n6. [kafka 与 spark streaming 集成,如何保证 exactly once 语义](./docs/kafka与sparkstreaming集成.md)\n\n7. [ack 有哪几种, 生产中怎样选择?](./docs/ack有哪几种.md)\n\n8. [如何通过 offset 寻找数据](./docs/如何通过offset寻找数据.md)\n\n9. [如何清理过期数据](./docs/如何清理过期数据.md)\n\n10. [1条message中包含哪些信息](./docs/1条message中包含哪些信息.md)\n\n11. [讲一下zookeeper在kafka中的作用](./docs/讲一下zookeeper在kafka中的作用.md)\n\n12. [kafka 可以脱离 zookeeper 单独使用吗](./docs/kafka可以脱离zookeeper单独使用吗.md)\n\n13. [kafka有几种数据保留策略](./docs/kafka有几种数据保留策略.md)\n\n14. [kafka同时设置了7天和10G清除数据,到第5天的时候消息到达了10G,这个时候kafka如何处理?](./docs/kafka同时设置了7天和10G清除数据.md)\n\n    \n\n## 七、Zookeeper\n\n1. [zookeeper是什么,都有哪些功能](./docs/zookeeper是什么.md)\n2. [zk 有几种部署模式](./docs/zk有几种部署模式.md)\n3. [zk 是怎样保证主从节点的状态同步](./docs/zk是怎样保证主从节点的状态同步.md)\n4. [说一下 zk 的通知机制](./docs/说一下zk的通知机制.md)\n5. [zk 的分布式锁实现方式](./docs/zk的分布式锁实现方式.md)\n6. [zk 采用的哪种分布式一致性协议? 还有哪些分布式一致性协议](./docs/大数据生态圈还有哪些选举协议.md)\n7. [讲一下leader 选举过程](./docs/讲一下leader选举过程.md)\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwater8394%2Fbigdata-interview","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwater8394%2Fbigdata-interview","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwater8394%2Fbigdata-interview/lists"}