{"id":19263023,"url":"https://github.com/datawhalechina/juicy-bigdata","last_synced_at":"2025-04-09T08:08:06.148Z","repository":{"id":38682161,"uuid":"429291596","full_name":"datawhalechina/juicy-bigdata","owner":"datawhalechina","description":"🎉🎉🐳 Datawhale大数据处理导论教程 | 大数据技术方向的开篇课程🎉🎉","archived":false,"fork":false,"pushed_at":"2023-05-06T12:45:52.000Z","size":28704,"stargazers_count":311,"open_issues_count":0,"forks_count":43,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-02T07:08:52.553Z","etag":null,"topics":["bigdata","hadoop","hbase","hdfs","hive","mapreduce","spark"],"latest_commit_sha":null,"homepage":"https://datawhalechina.github.io/juicy-bigdata/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datawhalechina.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-18T04:07:06.000Z","updated_at":"2025-04-01T11:37:49.000Z","dependencies_parsed_at":"2024-11-09T19:45:09.668Z","dependency_job_id":null,"html_url":"https://github.com/datawhalechina/juicy-bigdata","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datawhalechina%2Fjuicy-bigdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datawhalechina%2Fjuicy-bigdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datawhalechina%2Fjuicy-bigdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datawhalechina%2Fjuicy-bigdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datawhalechina","download_url":"https://codeload.github.com/datawhalechina/juicy-bigdata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247999859,"owners_count":21031046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","hadoop","hbase","hdfs","hive","mapreduce","spark"],"created_at":"2024-11-09T19:34:33.160Z","updated_at":"2025-04-09T08:08:06.126Z","avatar_url":"https://github.com/datawhalechina.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=center\u003e\n\u003cimg src=\"images/logo.png\" width=\"400\"\u003e\n\u003c/div\u003e\n\n# 妙趣横生大数据 Juicy Big Data\n\n[![](https://img.shields.io/github/watchers/datawhalechina/juicy-bigdata.svg?style=flat)](https://github.com/datawhalechina/juicy-bigdata/watchers)\n[![](https://img.shields.io/github/stars/datawhalechina/juicy-bigdata.svg?style=flat)](https://github.com/datawhalechina/juicy-bigdata/stargazers)\n[![](https://img.shields.io/github/forks/datawhalechina/juicy-bigdata.svg?style=flat)](https://github.com/datawhalechina/juicy-bigdata/network/members)\n[![](https://img.shields.io/github/issues-pr-closed-raw/datawhalechina/juicy-bigdata.svg?style=flat)](https://github.com/datawhalechina/juicy-bigdata/issues)\n![](https://img.shields.io/github/repo-size/datawhalechina/juicy-bigdata.svg?style=flat)\n\n本项目《妙趣横生大数据》是Datawhale:whale:大数据技术相关内容的导论课程。\n\n当今的时代处处充斥着大数据的影子，大数据技术也为信息技术发展带来了重大的变革，并深刻影响着人们生活的方方面面。而我们旨在带领大家走进大数据时代的浪潮中，理解并能够使用目前主流的大数据处理架构Hadoop解决相应的问题，从小白到实践者。\n\n## 食用方法\n\n:whale: 推荐使用 [**Big Data Github在线阅读**](https://datawhalechina.github.io/juicy-bigdata/) 进行学习。\n\n**项目结构：**\n\n\u003cpre\u003e\n├─docs---------------------------------------------章节主要内容（理论+实验）  \n├─experiments--------------------------------------每个章节配套的补充小实验  \n└─resources----------------------------------------相关资源  \n\u003c/pre\u003e\n\n以下是目录，还等什么，学就完事了～\n\n## 大数据处理技术导论目录\n\n* [第一章：大数据概述](https://datawhalechina.github.io/juicy-bigdata/#/ch01-bigdata-introduction)\n* [第二章：Hadoop](https://datawhalechina.github.io/juicy-bigdata/#/ch02-Hadoop)\n* [第三章：HDFS](https://datawhalechina.github.io/juicy-bigdata/#/ch03-HDFS)\n* [第四章：HBase](https://datawhalechina.github.io/juicy-bigdata/#/ch04-HBase)\n* [第五章：MapReduce](https://datawhalechina.github.io/juicy-bigdata/#/ch05-MapReduce)\n* [第六章：期中大作业](https://datawhalechina.github.io/juicy-bigdata/#/ch06-Homework01)\n* [第七章：Hive](https://datawhalechina.github.io/juicy-bigdata/#/ch07-Hive)\n* [第八章：Spark](https://datawhalechina.github.io/juicy-bigdata/#/ch08-Spark)\n* [第九章：大数据处理技术总结](https://datawhalechina.github.io/juicy-bigdata/#/ch09-bigdata-summary)\n* [第十章：期末大作业](https://datawhalechina.github.io/juicy-bigdata/#/ch10-Homework02)\n\n\u003e 第二章以后的章节都配置有实验内容，使用Linux操作系统进行编程实战。\n\n## 课程收获\n\n通过本教程的学习，你将能够学习到：\n\n- 了解并描述大数据技术的应用场景，真实世界的大数据问题和方法的例子。\n- 解释大数据4V的特性以及各种特性如何影响到数据的收集、监控、存储、分析和报告的方方面面。\n- 识别大数据问题，并能够将问题转化为数据科学问题。\n- 总结HDFS文件系统和MapReduce编程模型的特点和意义，以及它们与大数据的关系。\n- 识别和利用数据科学生命周期和相关数据流中的各种组件，如HBase，Hive等。\n\n## 参考教程\n\n1. 《大数据处理技术原理与应用 第三版》——林子雨\n2. Big Data US SanDiego\n3. 《Hadoop权威指南》\n4. 《Hive编程指南》\n5. 《维度建模权威指南(第3版)》\n6. 《大数据处理之道》\n7. 《Spark快速大数据分析》\n8. 牛客网部分面试题\n\n## 环境安装\n\n- Java 8\n- Hadoop 3.3.1\n- HBase 2.3.5 or 2.4.8\n- Mysql 8.0.32\n- Hive 2.3.9\n- Spark 3.2.0\n\n**安装包下载地址**：https://datawhale.feishu.cn/drive/folder/fldcnvODsgRWbyqVW9ApavEVEJg   密码: hO38\n\n## 致谢\n\n感谢以下Datawhale成员对项目推进作出的贡献(排名不分先后)：\n\n\u003ctable align=\"center\" style=\"width:80%;\"\u003e\n  \u003ccaption\u003e\u003cb\u003e贡献者名单\u003c/b\u003e\u003c/caption\u003e\n\u003cthead\u003e\n  \u003ctr\u003e\n    \u003cth\u003e成员\u003c/th\u003e\n    \u003cth\u003e个人简介及贡献\u003c/th\u003e\n    \u003cth\u003e个人主页\u003c/th\u003e\n  \u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e沈豪\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e复旦大学网安博士，项目负责人，参与前五章内容构建\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\n        \u003ca href=\"https://www.zhihu.com/people/shenhao-63\"\u003e知乎\u003c/a\u003e,\n        \u003ca href=\"https://github.com/shenhao-stu\"\u003eGithub\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e王洲烽\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e国防科技大学计算机研究生，Datawhale成员，主要贡献者\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\n        \u003ca href=\"https://blog.csdn.net/wangzhouf\"\u003eCSDN\u003c/a\u003e,\n        \u003ca href=\"https://github.com/wzfer\"\u003eGithub\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e蒋志政\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e电子科技大学计算机研究生，主要贡献者\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\n        \u003ca href=\"https://github.com/gezelligheid0314\"\u003eGithub\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e  \n  \u003ctr\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e王嘉鹏\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e小米大数据开发工程师，Datawhale成员，主要贡献者\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\n        \u003ca href=\"https://blog.csdn.net/qq_29027865\"\u003eCSDN\u003c/a\u003e,\n        \u003ca href=\"https://github.com/ditingdapeng\"\u003eGithub\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e刘洋\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e中科院数学与系统科学研究院研究生，Datawhale成员，主要贡献者\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\n        \u003ca href=\"https://www.zhihu.com/people/ming-ren-19-34\"\u003e知乎\u003c/a\u003e,\n        \u003ca href=\"https://github.com/liu-yang-maker\"\u003eGithub\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e胡锐锋\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cspan style=\"font-weight:normal;font-style:normal;text-decoration:none\"\u003e大数据平台研发工程师，Datawhale成员，主要贡献者\u003c/span\u003e\u003c/td\u003e\n    \u003ctd\u003e\n        \u003ca href=\"https://github.com/Relph1119\"\u003eGithub\u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e \n最后，也感谢伊小雪、毛自翔、萌弟、边圣陶参与本课程的内部评审！\n\n\n## 关注我们\n\u003cdiv align=center\u003e\n\u003cp\u003e扫描下方二维码关注公众号：Datawhale\u003c/p\u003e\n\u003cimg src=\"images/datawhale_qrcode.jpeg\" width = \"180\" height = \"180\"\u003e\n\u003c/div\u003e\n\u0026emsp;\u0026emsp;Datawhale是一个专注AI领域的开源组织，以“for the learner，和学习者一起成长”为愿景，构建对学习者最有价值的开源学习社区。关注我们，一起学习成长。\n\n## LICENSE\n\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"知识共享许可协议\" style=\"border-width:0\" src=\"https://img.shields.io/badge/license-CC%20BY--NC--SA%204.0-lightgrey\" /\u003e\u003c/a\u003e\u003cbr /\u003e本作品采用\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议\u003c/a\u003e进行许可。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatawhalechina%2Fjuicy-bigdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatawhalechina%2Fjuicy-bigdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatawhalechina%2Fjuicy-bigdata/lists"}