{"id":20673008,"url":"https://github.com/relph1119/bigdata-learning","last_synced_at":"2025-04-19T19:11:20.272Z","repository":{"id":106542523,"uuid":"594600283","full_name":"Relph1119/bigdata-learning","owner":"Relph1119","description":"大数据学习笔记，在线阅读地址：https://relph1119.github.io/bigdata-learning","archived":false,"fork":false,"pushed_at":"2024-04-19T06:28:16.000Z","size":2754,"stargazers_count":5,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T12:05:10.282Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Relph1119.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-01-29T03:19:38.000Z","updated_at":"2024-10-17T08:45:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"0eb39daf-d01d-4a00-a0e1-8cc4588be7c6","html_url":"https://github.com/Relph1119/bigdata-learning","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relph1119%2Fbigdata-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relph1119%2Fbigdata-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relph1119%2Fbigdata-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Relph1119%2Fbigdata-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Relph1119","download_url":"https://codeload.github.com/Relph1119/bigdata-learning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249773531,"owners_count":21323487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T20:39:46.922Z","updated_at":"2025-04-19T19:11:20.234Z","avatar_url":"https://github.com/Relph1119.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 大数据学习笔记\n\n主要记录大数据学习的相关笔记，包括Hadoop、Flume、Hive、Scala、Spark、Kafka、Redis、Flink、ClickHouse、Doris等各个组件的理论，并通过代码实验，了解组件的使用。\n\n## 在线阅读地址\n\n在线阅读地址：https://relph1119.github.io/bigdata-learning\n\n## 环境安装\n\n- OpenJDK Java版本：1.8.0_352\n- Scala版本：2.12.11 \n- Ubuntu版本：20.04\n- Vbox版本：6.1.28 r147628 (Qt5.6.2)\n- 虚拟机配置：显存大小50MB，内存大小4GB，硬盘大小（动态）50GB\n\n### 环境准备\n\n1. [Vbox配置Ubuntu的内外网访问](https://www.bilibili.com/video/av635603180/?vd_source=f4026a4ceb494a56ed0e12df39ea2d37)：主要使用NAT和Host-Only保证内外网的访问。\n2. 关闭Ubuntu防火墙\n   ```shell\n   sudo apt-get install ufw\n   ufw disable\n   ```\n2. 在hosts文件中配置域名：bigdata01 {仅主机(Host-Only)网络的IP地址}\n    - 查看`VirtualBox Host-Only Ethernet Adapter`网卡的IP设置，笔者的电脑设置为192.168.56.1\n    - 查看Vbox上仅主机(Host-Only)网络的IP地址，笔者的电脑设置为192.168.56.101，所有虚拟机的对外访问地址就是这个地址。\n3. 配置环境变量，打开`/etc/profile`，在文件末尾添加以下内容：\n    ```shell\n    export HADOOP_HOME=/data/soft/hadoop-3.2.0\n    export HIVE_HOME=/data/soft/apache-hive-3.1.2-bin\n    export SPARK_HOME=/data/soft/spark-3.1.3-bin-hadoop3.2\n    export HADOOP_CLASSPATH=`${HADOOP_HOME}/bin/hadoop classpath`\n    export PATH=.:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$SPARK_HOME/bin:$PATH\n    ```\n\n### 大数据组件版本\n\n- Hadoop版本：3.2.0\n- Flume版本：1.9.0\n- Hive版本：3.1.2\n- MySQL版本：8.0.32-0buntu0.20.04.1 (Ubuntu)\n- Spark版本：3.1.3-bin-hadoop3.2\n- Zookeeper版本：3.5.8\n- Kafka版本：kafka_2.12-2.4.1\n- Redis版本：5.0.9\n- Flink版本：1.11.1\n- ClickHouse版本：20.2.1\n- Doris版本：1.2.7\n- Nacos版本：2.3.2\n\n### 快速启动大数据组件\n\n- 启动Hadoop\n```shell\nstart-all.sh\nmapred --daemon start historyserver\n```\n\n- 设置MySQL开机自启动\n```shell\nsystemctl enable mysql.service\n```\n\n- 启动Hive\n```shell\nhiveserver2 \u0026\n```\n\n- 启动Spark HistoryServer\n```shell\ncd /data/soft/spark-3.1.3-bin-hadoop3.2\nsbin/start-history-server.sh\n```\n\n- 启动Zookeeper\n```shell\ncd /data/soft/apache-zookeeper-3.5.8-bin\nbin/zkServer.sh start\n```\n\n- 启动Kafka\n```shell\ncd /data/soft/kafka_2.12-2.4.1\nbin/kafka-server-start.sh -daemon config/server.properties\n```\n\n- 启动Redis\n```shell\ncd /data/soft/redis-5.0.9/\nredis-server redis.conf\n```\n\n- 启动Flink日志进程\n```shell\ncd /data/soft/flink-1.11.1\nbin/historyserver.sh start\n```\n\n- 启动ClickHouse\n```shell\nsudo /etc/init.d/clickhouse-server start\n```\n\n- 启动Doris\n```shell\ncd /data/soft/apache-doris-1.2.7-bin-x64/fe\n./bin/start_fe.sh --daemon\nsysctl -w vm.max_map_count=2000000\nulimit -n 65536\ncd /data/soft/apache-doris-1.2.7-bin-x64/be\n./bin/start_be.sh --daemon\n```\n\n- 启动Nacos\n```shell\ncd /data/soft/nacos/bin\nstartup.sh -m standalone\n```\n\n### 大数据组件默认端口\n\n- Hadoop的HDFS webui界面：http://bigdata01:9870\n- Hadoop的YARN webui界面：http://bigdata01:8088\n- HDFS端口：9000\n- MySQL端口：3306\n- Hive端口：10000\n- Spark History Server界面：http://bigdata01:18080/\n- Zookeeper端口：2181\n- Kafka端口：9092\n- Redis端口：6379\n- ClickHouse端口：8123\n- Doris端口：9030\n- Doris元数据页面：http://192.168.56.101:8030/ ，用户名`root`，密码`root`\n- Nacos端口：18848\n\n### 本地启动docsify\n```shell\ndocsify serve ./docs\n```\n\n## 学习注意事项\n1. **建议**从第01周第5章开始学习，可以用1.75倍的速度看视频\n2. 第06周第4章内容，可以不用学习CDH和HDP的部署安装\n3. 第07周第2章内容，由于机器不够，没有进行采集日志上传至HDFS的案例实验\n4. 修改了db_spark的依赖库，使用对应Hadoop和Scala版本的库，并添加了log4j的配置文件，删除了红色的Log日志\n5. 第12周前3章内容，可以重点听，后面代码实战内容可以快速观看，由于需要数据接口校验码，无法获取数据进行案例实战\n6. 第13周主要学习第3章内容，其他内容可快速观看\n7. 第17周第2章的Watermark理论部分有缺少，可以查看这篇文章[带你理解并使用flink中的WaterMark机制](https://blog.csdn.net/Chenftli/article/details/124274118)\n8. 第18、19周的项目实战内容可以快速观看，由于需要数据接口校验码，无法获取数据进行案例实战\n\n## 学习资料\n\n【1】大数据体系课-慕课网2019年课程：学习注意事项提到的内容是来源于本资料的。  \n【2】《ClickHouse性能之巅：从架构设计解读性能之谜》：第12章的内容来源于本书。  \n【3】《Doris实时数仓实战》：第13、14章的内容来源于本书。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frelph1119%2Fbigdata-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frelph1119%2Fbigdata-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frelph1119%2Fbigdata-learning/lists"}