{"id":9243870,"url":"https://github.com/Shicoder/Deep_Rec","last_synced_at":"2025-08-17T08:33:31.508Z","repository":{"id":202166108,"uuid":"187350257","full_name":"Shicoder/Deep_Rec","owner":"Shicoder","description":"存放推荐算法相关代码、文档、资料","archived":false,"fork":false,"pushed_at":"2020-12-24T04:00:09.000Z","size":164,"stargazers_count":216,"open_issues_count":3,"forks_count":78,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-08-25T22:45:40.798Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shicoder.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-05-18T11:34:18.000Z","updated_at":"2024-08-24T05:24:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"81af6514-e724-41b3-a666-f97c8b4217fe","html_url":"https://github.com/Shicoder/Deep_Rec","commit_stats":null,"previous_names":["shicoder/deep_rec"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shicoder%2FDeep_Rec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shicoder%2FDeep_Rec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shicoder%2FDeep_Rec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shicoder%2FDeep_Rec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shicoder","download_url":"https://codeload.github.com/Shicoder/Deep_Rec/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230108744,"owners_count":18174538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-05-08T00:11:42.755Z","updated_at":"2024-12-17T11:30:31.858Z","avatar_url":"https://github.com/Shicoder.png","language":"Python","readme":"## 本库用于存放推荐相关算法代码和文档\n#### DeepRank/   \n\n#### Overview  \n\n**Deep_Rank是一套从数据生成到模型生成的完整框架。对输入特征，特征工程，核心模型，模型输出分别进行了封装。\n模型更新迭代的时候可以针对性的修改指定模块，也方便后续模型的切换，特征的迭代，以及输入数据在输入，特征工程，保存模型，线上服务各个模块的统一配置。本库对一些经典的ctr预估模型进行了复现，方便自己在工作中迭代优化模型。**  \n\n#### Data Read\n利用 `tfrecords/` 下的代码可以很方便的利用spark集群将数仓经过ETL的hive表训练数据转化成`frecords`格式，并存储HDFS上。\n      \n#### Config Setup\n##### model_schema.json   \n配置训练数据中的字段，可以只选择模型需要的字段；  \n*格式：*\n  \n    \"user_id\":     \n    {  \n            \"feature_type\": \"fixed_len\",  \n            \"value_type\": \"int\",  \n            \"default_value\": 0  \n    }  \n    \n    字段名为key;  \n    feature_type：tensorflow特征类型，当前所有模型使用的是fixed_len;  \n    value_type：特征值的类型，当前支持{'string', 'int', 'double'},  \n                可以在train_model.py#19 函数get_input_schema_spec中添加/修改;    \n    default_value: 该特征值的默认填充值，类型需要和value_type一致；          \n\n##### model_feature.json   \n \n    配置模型中特征的形式，使用tensorflow FeatureColumns API实现；    \n    具体支持的FeatureColumns包括：  \n    \"categorical_column_with_hash_bucket\"  \n    \"numeric_column\"  \n    \"embedding_column\"  \n    \"categorical_column_with_vocabulary_list\"  \n    \"categorical_column_with_identity\"  \n    \"indicator_column\"  \n    \"bucketized_column\"  \n    \"shared_embedding_columns\"  \n    \"crossed_column\"  \n    \"NumericColumnV2\" (这个是修改的原始numeric_column，用来自适应学习缺失值，具体思路可以看源码)  \n    配置格式：  \n    {  \n      \"name\": \"IndicatorColumn\",  \n      \"parameters\":   \n      {  \n        \"input_tensor\": \"u_sc_gender_hashbucket\",  \n        \"output_tensor\": \"u_sc_gender_indicator\",  \n        \"wide_or_deep\": \"deep\"  \n      }  \n    }  \n    input_tensor：定义输入名  \n    output_tensor：定义输出名  \n    wide_or_deep：用于wd，deepfm的wide侧和deep侧区分;    \n                  如果只是中间特征，不作为最后的输出则不填;   \n                  其他模型默认只取\"deep\"值特征;  \n                  din，dien等模型，用于attention的sequence特征和目标id不需要在model_feature.json进行配置;    \n        \n#### Model Implementation\n\n##### model_brain.py  \n*本文件存放模型核心代码，所有代码继承`BaseModel`类，底层使用tensorflow的FeatureColumns实现统一的embedding层。上层利用tensorlfow的Layers实现核心模型结构，最后把模型封装成Esitmator，输出统一的estimator_spec接口。*  \n\n###### model brain\n*已经实现好的模型有以下几个：*  \n  \n[`DNN`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/dnn_demo.py) 实现的是一个简单的embedding+MLP，方便调试整体代码，在`model_feature.json`中配置`wide_or_deep`参数值 `\"deep\"`;  \n  \n[`Deep Cross Network(DCN)` ](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/dcn.py)在`model_feature.json`中只需配置`deep特征`，croos层和deep层同用统一的embeddig层。    \n对应算法论文[[click here]](https://arxiv.org/abs/1708.05123)  \n   \n[`Wide and Deep Network(WD)`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/wide_deep.py)需要在`model_feature.json`配置wide侧和deep侧对应的特征。    \n对应算法论文[[click here]](https://arxiv.org/abs/1606.07792)   \n\n[`Deep Interest Network(DIN)`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/din.py)的上下文特征和基础画像特征需要配置`model_feature.json`中，统一使用`\"deep\"`特征,其中需要做`attention`的`sequence`和`sequence对应的目标id`不需要配置在`model_feature.json`中，直接配置在模型参数中。  \n在对应算法论文[[click here]](https://arxiv.org/abs/1706.06978)  \n\n[`Entire Space Multi-Task Model(ESMM)`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/esmm.py)也一样，默认只使用`\"deep\"`特征。 \n对应算法论文[[click here]](https://arxiv.org/abs/1804.07931)  \n  \n[`Deep Interest Evolution Network(DIEN)`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/dien.py)和DIN一样，`序列特征`和`序列对应目标id`需要在模型中写。  \n这里为了实现方便，序列的`负采样`部分没有按照原始论文的方式单独使用一份负采样的item数据集，而是直接使用`同一个batch`中的`其他sequence`作为当前的`负采样序列`。  \n对应算法论文[[click here]](https://arxiv.org/abs/1809.03672)   \n \n[`DeepFM`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/deepFM.py)的wide侧放线性特征，deep侧放的是`统一的MLP和FM的特征`，所有这里必须保证deep特征全部是embedding特征，并且embedding维度`全部一致`。这里后续会优化一下，对一些数值特征做优化，方便扩展。  \n对应算法论文[[click here]](https://arxiv.org/abs/1703.04247)   \n\n[`xDeepFM`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/xDeepFM.py)的参数配置和deepFM类似，wide侧放线性特征，deep侧放的是`统一的MLP和CIN特征`，所有这里必须保证deep特征全部是embedding特征，并且embedding维度`全部一致`。  \n对应算法论文[[click here]](https://arxiv.org/pdf/1803.05170.pdf)  \n\n[`DSSM`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/dssm.py)需要在wide和deep侧分别放置user info和item info。  \n对应算法论文[[click here]](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/cikm2013_DSSM_fullversion.pdf)   \n\n[`youtube_net`](https://github.com/Shicoder/Deep_Rec/blob/master/Deep_Rank/dssm.py)本文是youtube提出的排序模型 \"Recommending What Video to Watch Next: A Multitask Ranking System\"，不知道叫啥，所以直接叫youtube_net了。  \n对应算法论文[[click here]](https://dl.acm.org/citation.cfm?id=3346997)    \n    \n*后续利用空闲时间和节假日会持续添加新算法*\n\n#### Training\n\n直接运行主函数 \n  \n    `python train_model.py`\n\n\n##### 备注\n\n*部分实现参考了:*   \n  x-deeplearning的xdl-algorithm-solution: [[click here]](https://github.com/alibaba/x-deeplearning/tree/master/xdl-algorithm-solution)   \n  脚本配置方式：[[click here]](https://github.com/zhaoxin4data/atlas/tree/master/deeplearning/uciflowwd_train/config)  \n  esmm实现:[[click here]](https://github.com/yangxudong/deeplearning/tree/master/esmm)   \n  tensorflow自带的分类器: [[click here]](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/estimator/canned)  \n  xDeepFM实现:[click_here](https://github.com/Leavingseason/xDeepFM)  \n  MMoE实现：[click_here](https://github.com/drawbridge/keras-mmoe)\n  \n  *代码实现的问题：* \n    \n  *`代码实现为了方便调试，DIN，DIEN中的所有的序列特征都只用了id，可自行添加类目序列，产品词序列等边界信息`*  \n   \n#### tfrecords/\n存放tfrecords文件的生成和使用代码\n利用spark读取数据保存成tfrecods这里提供两个思路。  \n1.如果集群上已经安装了tensorflow，可以用本文件下的代码进行处理。  \n2.对于没有安装tensorflow的集群，可以参考:[spark-tensorflow-connector](https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-connector) 来将数据转换成tfrecords。这个更快。  \n  \n                            -------------------------以下已经删除或弃用--------------------------------\n#### wd_model/(已删除，合并到DeepRank)  \n所以这里我自己实现了一个wide\u0026deep模型。支持连续特征，类别特征，embedding特征，特征交叉等常见操作，也支持list特征以及各种数据类型。能很方便的将生成的模型export，以及支持分布式训练。  \n修改模型的核心代码可以很方便的扩展到其他DNN模型，现在暂时只支持Wide，DNN，WD三类。   \n\n#### esmm_model/(已删除，合并到DeepRank)  \n实现esmm模型\n#### esmm_model/esmm_v2(已删除，合并到DeepRank)  \n  便于迭代模型，以及之前脚本代码写的有点乱，部分同学反馈代码不是很优雅。所以针对性的进行了优化。  \n  \n##### 运行方式(弃用) \n- 配置config/inputs.json，配置好输入的字段，以及需要做的操作，以及相对应的操作  \n- 配置好对应的路径  \n- 运行train_wd_model.py即可。    \n\n##### config/(弃用)   \n存放模型配置文件  \n\n#### utils/\n\n\n\n","funding_links":[],"categories":["Recommendation, Advertisement \u0026 Ranking"],"sub_categories":["Others"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShicoder%2FDeep_Rec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FShicoder%2FDeep_Rec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShicoder%2FDeep_Rec/lists"}