{"id":13466854,"url":"https://github.com/LuckyZXL2016/Movie_Recommend","last_synced_at":"2025-03-26T00:31:38.421Z","repository":{"id":37336422,"uuid":"130085392","full_name":"LuckyZXL2016/Movie_Recommend","owner":"LuckyZXL2016","description":"基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统","archived":false,"fork":false,"pushed_at":"2019-04-01T13:10:59.000Z","size":57779,"stargazers_count":2907,"open_issues_count":13,"forks_count":1048,"subscribers_count":106,"default_branch":"master","last_synced_at":"2025-03-24T06:15:29.871Z","etag":null,"topics":["hadoop","hive","mysql","nginx","scala","scrapy","spark-mllib","spark-streaming","ssm-maven"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LuckyZXL2016.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-18T15:42:28.000Z","updated_at":"2025-03-24T04:34:48.000Z","dependencies_parsed_at":"2022-07-08T05:15:34.897Z","dependency_job_id":null,"html_url":"https://github.com/LuckyZXL2016/Movie_Recommend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LuckyZXL2016%2FMovie_Recommend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LuckyZXL2016%2FMovie_Recommend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LuckyZXL2016%2FMovie_Recommend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LuckyZXL2016%2FMovie_Recommend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LuckyZXL2016","download_url":"https://codeload.github.com/LuckyZXL2016/Movie_Recommend/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245566099,"owners_count":20636391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","hive","mysql","nginx","scala","scrapy","spark-mllib","spark-streaming","ssm-maven"],"created_at":"2024-07-31T15:00:50.761Z","updated_at":"2025-03-26T00:31:38.414Z","avatar_url":"https://github.com/LuckyZXL2016.png","language":"Java","funding_links":[],"categories":["Java","实战项目","教程系列"],"sub_categories":["毕业设计实战项目","文件同步"],"readme":"# 基于Spark的电影推荐系统\n\n本次项目是基于大数据过滤引擎的电影推荐系统--“懂你”电影网站，包含了爬虫、电影网站（前端和后端）、后台管理系统以及推荐系统（Spark）。\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/introduce.png)\n\n\n## 一、爬虫\n**开发环境：** pycharm + python3.6\n\n**软件架构：** mysql + scrapy\n\n**运行环境：** 本次爬取的内容在外网，所以需先翻墙后才能成功运行。\n\n**项目架构：**\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/scrapy.png)\n\n\n## 二、电影网站\n**开发环境：** IntelliJ IDEA + maven + git + linux + powerdesigner\n\n**软件架构：** mysql + mybatis + spring + springmvc\n\n**项目描述：** 懂你电影推荐网站是一个基于SSM框架的web项目，类似当前比较流行的豆瓣。用户可以在网站上浏览电影信息和查询电影，并且网站会根据用户的浏览记录给用户进行实时的电影推荐。现已将网站部署在 http://115.159.204.68 网站上，感兴趣的朋友可以自行查看。Git的安装与IDEA和github的集成可以参考[博客](https://blog.csdn.net/u011254180/article/details/79857684)。\n\n**项目架构：**\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/ssm.png)\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/web_1.png)\n\n\n## 三、后台管理系统\n\n**开发环境：** IntelliJ IDEA + maven + git + linux + powerdesigner\n\n**软件架构：** mysql + mybatis + spring + springmvc + easyui\n\n**项目描述：** 后台管理系统主要对用户信息和电影信息进行管理，如添加删除电影信息和完成用户信息的完善。其中为了更好地保存电影的图片信息，搭建了图片服务器，关于图片服务器FastDFS的搭建可参考[博客](https://blog.csdn.net/u011254180/article/details/79453775)。后台系统也布置在服务器上，感兴趣的朋友可以通过地址 http://115.159.204.68:8080/ 访问，为大家提供的测试账号为 test，密码为88888888。\n\n**项目架构：**\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/admin_1.png)\n\n\n## 四、推荐系统（Spark）\n\n**开发环境：** IntelliJ IDEA + maven + git + linux\n\n**软件架构：** hadoop + zookeeper + flume + kafka + nginx + spark + hive + mysql\n\n**项目描述：** 通过在电影网站系统埋点，获取到用户的点击事件（如用户喜欢哪部电影或对某部电影的评分）并将信息传至推荐系统，推荐系统根据该信息做出相应的处理，将推荐结果存入到mysql数据库中，web前端通过查询数据库将推荐的电影展示给用户。推荐流程如下：\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/recommend_1.png)\n\n**项目架构：**\n\n![image](https://github.com/ZzXxL1994/Movie_Recommend/blob/master/zxl_picture/recommend_2.png)\n\n**具体步骤：**\n\n**1.服务器规划（linux镜像为centos6）：**\n\n- spark1（ip 192.168.13.134），分配8G内存，4核\n- spark2（ip 192.168.13.135），分配6G内存，4核\n- spark3（ip 192.168.13.136），分配6G内存，4核\n\n\n**2.电影数据集，[地址](https://grouplens.org/datasets/movielens/)：** 本次下载的为1m大小的数据集\n\n\n**3.环境的搭建：**\n\n1）[hdfs搭建](http://blog.csdn.net/u011254180/article/details/79377808)\n\n- spark1上搭建namenode,secondary namenode,datanode\n- spark2上搭建datanode\n- spark3上搭建datanode\n\n2）[yarn搭建](http://blog.csdn.net/u011254180/article/details/79377808)\n\n- spark1上搭建resourcemanager,nodemanager\n- spark2上搭建nodemanager\n- spark3上搭建nodemanager\n\n3）[mysql搭建](http://blog.csdn.net/u011254180/article/details/77893585),在spark2上搭建\n\n4）[hive搭建](http://blog.csdn.net/u011254180/article/details/78068897),在spark1上搭建\n\n5）[spark集群搭建](http://blog.csdn.net/u011254180/article/details/79381172)，搭建standalone模式，spark1为master，其他为worker\n\n\n**4.数据的清洗：** （上传数据至hdfs中，[hdfs操作](http://blog.csdn.net/u011254180/article/details/79399422)）\n\n1）启动 hdfs：  [root@spark1 ~]# start-dfs.sh \n\n2）启动 yarn：  [root@spark1 ~]# start-yarn.sh \n\n3）启动 mysql： [root@spark2 ~]# service mysqld start\n\n4）启动 hive：  [root@spark1 ~]# hive --service metastore\n\n5）启动 spark集群： [root@spark1 spark-1.6.1]# ./sbin/start-all.sh\n\n6）代码(com.zxl.datacleaner.ETL)打包上传（[spark-sql与hive集成](http://blog.csdn.net/u011254180/article/details/79395227)）\n\n- 代码位于 package com.zxl.datacleaner.ETL，打包为 ETL.jar\n- 运行代码 spark-submit --class com.zxl.datacleaner.ETL --total-executor-cores 2 --executor-memory 2g lib/ETL.jar\n- 成功于hive中建表\n\n\n**5.数据的加工，** 根据ALS算法对数据建立模型([ALS论文](https://github.com/ZzXxL1994/Machine-Learning-Papers/tree/master/ALS))\n\n1）启动 hdfs：  [root@spark1 ~]# start-dfs.sh\n\n2）启动 yarn：  [root@spark1 ~]# start-yarn.sh\n\n3）启动 mysql： [root@spark2 ~]# service mysqld start\n\n4）启动 hive：  [root@spark1 ~]# hive --service metastore\n\n5）启动 spark集群： [root@spark1 spark-1.6.1]# ./sbin/start-all.sh\n\n6）代码(com.zxl.datacleaner.RatingData)打包上传，测试建立模型\n\n\n**6.建立模型，** 根据RMSE(均方根误差)选取较好的模型\n\n1）启动上述的服务\n\n2）代码(com.zxl.ml.ModelTraining)打包上传，建立模型\n\n注：com.zxl.ml.ModelTraining2中代码训练单个模型，其中参数 rank=50, iteration = 10, lambda = 0.01\n\n- 代码位于 package com.zxl.ml.ModelTraining，打包为 Spark_Movie.jar\n- 运行代码 spark-submit --class com.zxl.ml.ModelTraining lib/Spark_Movie.jar\n\n\n**7.产生推荐结果**\n\n1）启动上述的服务\n\n2）代码(com.zxl.ml.Recommender)打包上传，产生推荐结果\n\n\n**8.数据入库，** 存储为所有用户推荐的电影结果，mysql中存入的格式为(userid, movieid,rating)\n\n1）启动上述的服务\n\n2）代码(com.zxl.ml.RecommendForAllUsers)打包上传，数据入库\n\n- 运行代码 spark-submit --class com.zxl.ml.RecommendForAllUsers --jars lib/mysql-connector-java-5.1.35-bin.jar lib/Spark_Movie.jar\n\n\n**9.实时数据的发送**\n\n1）[安装nginx](https://blog.csdn.net/u011254180/article/details/77897663)，用来接收电影网站上用户的点击信息，写入本地文件\n\n2）[安装flume](https://blog.csdn.net/u011254180/article/details/80000763)，实时监控本地文件，将数据发送至kafka消息队列中\n\n\n**10.实时数据的接收处理** ，如果打包到服务器运行错误，也可在本地IDEA上运行\n\n1）[安装zookeeper](http://blog.csdn.net/u011254180/article/details/79480234)\n\n2）[安装kafka](http://blog.csdn.net/u011254180/article/details/79481088)，用来接收发送数据\n\n3）启动上述的服务\n\n4）启动zookeeper：  [root@spark1 soft]# zkServer.sh start\n\n4）启动flume：[root@spark1 flume]# bin/flume-ng agent -c ./conf/ -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -n a1\n\n5）启动kafka：  [root@spark1 kafka_2.11-0.10.1.0]# bin/kafka-server-start.sh config/server.properties\n\n6）代码(com.zxl.datacleaner.PopularMovies2)运行，用于为没有登录或新用户推荐，默认推荐观看最多的5部电影\n\n7）代码运行(需指定jar包 kafka-clients-0.10.1.0.jar)\n\n- spark-submit --class com.zxl.streaming.SparkDrStreamALS --total-executor-cores 2 --executor-memory 1g --jars lib/kafka-clients-0.10.1.0.jar lib/Spark_Movie.jar\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLuckyZXL2016%2FMovie_Recommend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLuckyZXL2016%2FMovie_Recommend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLuckyZXL2016%2FMovie_Recommend/lists"}