{"id":31635110,"url":"https://github.com/jianxi-erin/bigdata-machinelearning-lab","last_synced_at":"2026-05-03T22:32:12.349Z","repository":{"id":315576215,"uuid":"1059972548","full_name":"jianxi-Erin/BigData-MachineLearning-Lab","owner":"jianxi-Erin","description":"本项目是一个综合性的大数据与机器学习实验平台，包含两个主要任务，每个任务涵盖三个关键技术模块：大数据处理、数据分析和机器学习。项目基于真实的竞赛设计，提供完整的数据处理模拟和建模实践。","archived":false,"fork":false,"pushed_at":"2025-09-19T10:29:45.000Z","size":25029,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-19T12:35:37.068Z","etag":null,"topics":["data-analysis","data-visualization","hadoop","machine-learning","python","spark","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jianxi-Erin.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-19T07:53:47.000Z","updated_at":"2025-09-19T10:33:00.000Z","dependencies_parsed_at":"2025-09-19T12:49:13.699Z","dependency_job_id":null,"html_url":"https://github.com/jianxi-Erin/BigData-MachineLearning-Lab","commit_stats":null,"previous_names":["jianxi-erin/bigdata-machinelearning-lab"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jianxi-Erin/BigData-MachineLearning-Lab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianxi-Erin%2FBigData-MachineLearning-Lab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianxi-Erin%2FBigData-MachineLearning-Lab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianxi-Erin%2FBigData-MachineLearning-Lab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianxi-Erin%2FBigData-MachineLearning-Lab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jianxi-Erin","download_url":"https://codeload.github.com/jianxi-Erin/BigData-MachineLearning-Lab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jianxi-Erin%2FBigData-MachineLearning-Lab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278703580,"owners_count":26031205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","data-visualization","hadoop","machine-learning","python","spark","sql"],"created_at":"2025-10-07T00:48:16.699Z","updated_at":"2025-10-07T00:48:20.415Z","avatar_url":"https://github.com/jianxi-Erin.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 大数据与机器学习综合项目\n\n## 🎯 项目概述\n\n本项目是一个综合性的大数据与机器学习实验平台，包含两个主要任务，每个任务涵盖三个关键技术模块：大数据处理、数据分析和机器学习。项目基于真实的竞赛题目设计，提供完整的数据处理流水线和建模实践。\n\n## 📁 项目结构\n\n```\nBigData-MachineLearning-Lab/\n├── data/                    # 数据存储目录\n├── img/                     # 图像资源\n├── question/                # 任务书\n│   ├── task1/               # 任务1：三个技术模块\n│   │   ├── moduleA          # 模块A：大数据运维与应用\n│   │   ├── module_B         # 模块B：电影数据分析\n│   │   └── module_C         # 模块C：信用卡欺诈检测\n│   └── task2/               # 任务2：三个技术模块\n│       ├── module_A         # 模块A：游戏运营数据分析\n│       ├── module_B         # 模块B：奥林匹克数据分析\n│       └── module_C         # 模块C：橙汁销售数据预测\n├── Answer1.ipynb        # 任务一源码\n├── Answer2.ipynb        # 任务二源码\n└── README.md              # 项目说明文档\n```\n\n\n\n\n\n## 🏆 Task 1 详细说明\n\n### 模块A：[大数据运维与应用 (30分)](./question/task1/moduleA任务书.docx)\n**竞赛题目**：大数据运维与应用\n\n**技术栈**：\n\n- Hadoop生态系统 (HDFS, MapReduce, YARN)\n- Sqoop数据迁移工具\n- Hive数据仓库\n- Flume日志收集\n- Spark分布式计算\n\n**核心任务**：\n1. **Hadoop集群部署与配置**\n   - 配置hosts文件和三节点集群\n   - 修改HDFS数据备份数量\n   - 启动完整Hadoop服务\n\n2. **HDFS文件操作**\n   - 目录创建和权限管理\n   - 文件上传和同步\n\n3. **Sqoop数据迁移**\n   - MySQL与Hive数据双向同步\n   - 关系数据库与大数据平台集成\n\n4. **Hive数据分析**\n   - 复杂数据类型处理（Map, Array, Struct）\n   - 外部表创建和查询优化\n   - 产品评分分析\n\n5. **Flume实时监控**\n   - 文件监控和HDFS自动同步\n\n6. **Spark分布式计算**\n   - 电影评分数据分析\n   - 时间段划分和统计计算\n\n### 模块B：[数据分析 (30分)](./question/task1/moduleB任务书.docx)\n**竞赛题目**：电影数据分析\n\n**数据集**：\n- data/tmdb_5000_credits.csv\n- 'data/tmdb_5000_movies.csv\n\n**数据处理流程**：\n1. **数据清洗**\n   - 删除冗余字段\n   - 处理缺失值\n   - 特征工程（收益计算）\n\n2. **字段解析**\n   - JSON格式解析（genres, keywords, production_companies等）\n   - 导演信息提取\n   - 时间字段格式化\n\n**可视化分析**：\n- 📊 关键词词云分析\n- 📈 电影时长分布直方图\n- 🌹 产地分布玫瑰图\n- 🎯 导演票房排名条形图\n- 🌪️ 原创vs改编电影旋风图\n- 🥧 电影类型占比饼图\n- 📊 类型趋势折线图\n- 💰 收益能力混合图表\n- 🎬 类型Top5象形条形图\n\n### 模块C：[机器学习 (40分)](./question/task1/moduleC任务书.docx)\n**竞赛题目**：信用卡欺诈检测\n**数据集**：\n- data/credit_card_fraud.csv\n**数据特征**：\n- V1-V28: PCA变换后的主成分（保密特征）\n- Amount: 交易金额\n- Class: 欺诈标签（0:正常, 1:欺诈）\n\n**技术挑战**：\n- ⚖️ 极端数据不平衡（欺诈案例占少数）\n- 🔍 欺诈模式隐藏在海量正常交易中\n- ⚡ 需要快速适应变化的欺诈模式\n\n**建模方法**：\n1. **数据预处理**\n   - 特征标准化\n   - 处理不平衡数据（SMOTE, 欠采样等）\n\n2. **模型选择**\n   - 传统机器学习分类器\n   - 集成学习方法\n   - 深度学习模型\n\n3. **评估指标**\n   - F1分数（精确率和召回率的调和平均）\n   - 混淆矩阵分析\n   - ROC曲线评估\n\n## 🎮 Task 2 详细说明\n\n### 模块A：[大数据 (30分)](./question/task2/moduleA任务书.docx)\n**竞赛题目**：电影数据\n**竞赛题目**：游戏运营数据分析\n\n**数据表**：game_data（游戏用户行为数据）\n\n**核心指标分析**：\n1. **用户基础分析**\n   - 注册用户量随时间分布\n   - 最大注册日统计\n\n2. **营收指标**\n   - ARPU（每用户平均收入）= 总付费金额 / 活跃人数\n   - 活跃用户定义：日均在线时间 ≥ 30分钟\n\n3. **转化分析**\n   - 付费转化率 = 付费用户 / 总用户数 × 100%\n   - 等级与付费转化率关系分析\n\n**技术实施**：\n- Hadoop集群配置（块大小优化）\n- HDFS文件操作\n- Sqoop数据迁移\n- Hive数据分析查询\n- Spark集群部署和Hive集成\n\n### 模块B：[数据分析 (30分)](./question/task2/moduleB任务书.docx)\n**竞赛题目**：奥林匹克数据分析\n\n**数据集**：\n- data/athlete_events.csv (271,116条记录，15个字段)\n- data/noc_regions.csv (国家奥委会地区信息)\n\n**分析内容**：\n1. **国家参与分析**\n   - 每届奥运会都参加的国家\n   - 中国参与次数统计\n\n2. **奖牌分析**\n   - 2008年中国获奖人数\n   - 金牌最多国家和选手\n\n3. **选手分析**\n   - 参与次数最多的选手\n   - 年龄最大的运动员（排除艺术比赛）\n\n**可视化展示**：\n- 🏃 运动项目词云图\n- 👥 参赛者性别比例环形图\n- 📊 历年平均年龄变化折线图\n- 🥇 金牌国家Top10条形图\n- 🏆 个人奖牌榜柱状图\n- 🏊 田泳项目占比折线图\n- 🏟️ 主场优势折线矩阵图\n- 🇨🇳 中国奥运成绩变迁史\n- 🎯 中国强势项目蓝丁格尔图\n\n### 模块C：[机器学习 (40分)](./question/task2/moduleC任务书.docx)\n**竞赛题目**：橙汁销售数据预测\n\n**数据集**：\n- data/orange_juice.csv\n**数据描述**：\n- 训练集: 845个样本，17个特征 + 1个标签\n- 测试集: 212个样本，17个特征\n- 标签: Purchase (0:CH, 1:MM)\n\n**特征工程**：\n- 价格特征: PriceCH, PriceMM, SalePriceCH, SalePriceMM\n- 折扣特征: DiscCH, DiscMM, PctDiscCH, PctDiscMM\n- 促销特征: SpecialCH, SpecialMM\n- 客户特征: LoyalCH (品牌忠诚度)\n- 商店特征: StoreID, Store7, STORE\n- 衍生特征: PriceDiff, ListPriceDiff\n\n**建模方法**：\n- 多种分类器选择（15+种模型）\n- 参数调优和模型集成\n- 准确率(accuracy_score)作为评估指标\n\n**业务价值**：\n- 优化商品配置和库存管理\n- 提升销售额和市场占有率\n- 实现数据驱动的零售决策\n\n## 🛠️ 技术栈\n\n### 大数据技术\n- **分布式存储**: HDFS\n- **资源管理**: YARN\n- **数据迁移**: Sqoop\n- **数据仓库**: Hive\n- **实时收集**: Flume\n- **分布式计算**: Spark\n\n### 数据分析\n- **编程语言**: Python, SQL\n- **数据处理**: Pandas, NumPy\n- **数据可视化**: Matplotlib, Seaborn, Pyecharts\n- **JSON解析**: JSON库, 正则表达式\n\n### 机器学习\n- **传统ML**: Scikit-learn (15+分类器)\n- **集成学习**: XGBoost, LightGBM\n- **评估指标**: F1-score, Accuracy\n- **数据平衡**: SMOTE, 欠采样\n\n\n## 📊 预期成果\n\n### 技术能力提升\n- 掌握完整的大数据生态系统部署和运维\n- 熟练进行复杂数据清洗和特征工程\n- 具备多维度数据可视化能力\n- 掌握不平衡数据集的机器学习建模\n\n### 项目交付物\n- 完整可运行的大数据集群\n- 详细的数据分析报告和可视化图表\n- 高性能的机器学习预测模型\n- 完整的技术文档和代码注释\n\n## 📝 项目特点\n\n1. **真实性**: 基于实际竞赛题目，解决真实业务问题\n2. **完整性**: 覆盖数据采集、处理、分析、建模全流程\n3. **技术深度**: 涉及多种大数据和机器学习技术栈\n4. **实践性**: 提供完整的环境配置和代码实现\n5. **可扩展性**: 模块化设计，便于扩展和定制\n\n**⭐ 如果这个项目对您有帮助，请给它一个Star！**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjianxi-erin%2Fbigdata-machinelearning-lab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjianxi-erin%2Fbigdata-machinelearning-lab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjianxi-erin%2Fbigdata-machinelearning-lab/lists"}