{"id":19020760,"url":"https://github.com/librauee/bytedanceai","last_synced_at":"2025-04-23T07:05:45.217Z","repository":{"id":107434519,"uuid":"434877048","full_name":"librauee/ByteDanceAI","owner":"librauee","description":"2021字节跳动安全AI挑战赛 6th","archived":false,"fork":false,"pushed_at":"2021-12-06T10:30:06.000Z","size":6,"stargazers_count":16,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-23T07:05:39.481Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://security.bytedance.com/fe/ai-challenge#/challenge#rankList","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/librauee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-04T11:01:19.000Z","updated_at":"2024-05-13T06:30:40.000Z","dependencies_parsed_at":"2023-05-17T11:30:28.768Z","dependency_job_id":null,"html_url":"https://github.com/librauee/ByteDanceAI","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/librauee%2FByteDanceAI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/librauee%2FByteDanceAI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/librauee%2FByteDanceAI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/librauee%2FByteDanceAI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/librauee","download_url":"https://codeload.github.com/librauee/ByteDanceAI/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250386752,"owners_count":21422028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T20:18:36.410Z","updated_at":"2025-04-23T07:05:45.176Z","avatar_url":"https://github.com/librauee.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 字节跳动安全AI挑战赛-把杰泥牛逼打在公屏上-Writeup \u0026 Reproduce\n\n## **1. 环境依赖**\n\n- Python 3.7.0\n- numpy 1.19.5\n- pandas 0.24.2\n- scikit-learn 0.24.2\n- tqdm 4.46.1\n- gensim 3.8.3\n- lightgbm 3.3.1\n  \n## **2. 目录结构**\n\n```\n./\n├── README.md\n├── requirements.txt, python package requirements \n├── init.sh, script for installing package requirements\n├── train.sh, script for preparing training data and training models, including pretrained models\n├── code\n│   ├── train.py main code\n├── data\n│   ├── data file\n├── submission\n│   ├── submission file\n```\n\n## **3. 运行流程**\n\n- 将数据集放至data目录下\n- 安装环境：sh init.sh\n- 模型训练、预测：sh train.sh\n\n## **4. 模型及特征**\n- 模型：[lightgbm](http://papers.nips.cc/paper/6907-a-highly-efficient-gradient-boosting-decision-tree.pdf)\n- 特征：\n    - 用户侧特征：\n       - 账户本身的基础特征\n       - 账户本身的特征计数统计\n       - 粉丝量、关注量、发帖量、被点赞量 除法交叉\n       - 登录时间、注册时间 减法交叉\n\n    - 请求侧特征：\n      - 点赞、关注基础特征\n      - 机型、ip、app_version、app_channel统计\n      - 聚合特征1，在每个请求ip下有多少不同的用户，用户请求的所有ip的用户数的均值和方差\n      - 聚合特征2， 用户请求时间的均值方差等等\n      - w2v特征， 每个用户的请求ip序列、机型序列等等建模\n\n## **5. 方案说明**\n\n- 本次方案采用单模型LGB模型进行训练预测，主要从两张表中提取特征，用户基础信息表可以对账户本身的基础特征进行刻画，用户请求表是用户请求行为的记录，可针对此表刻画用户的行为形象，采用的方案是以每一条请求记录作为一条数据进行建模，最终对于每个用户取请求行为的预测概率最大值作为预测结果；\n\n- 特征方案也从上述两个方面展开，基于账户本身基础特征，可以做这些类别特征的计数统计（'user_freq_ip', 'user_profile', 'user_name', 'user_register_time', 'user_least_login_time',  'user_register_type', 'user_register_app', 'user_least_login_app'这些数值列在数据中出现了多少次，用value_counts()计算）、对于粉丝量等数值特征可以做除法的交叉（user_post_like_num/user_post_num， user_post_like_num/user_follow_num， user_post_like_num/user_fans_num，user_post_num/user_follow_num，user_post_num/user_fans_num，user_follow_num， user_fans_num）、登录时间和注册时间特征可以做减法交叉（user_least_login_time-user_register_time），基于请求行为，我们可以对机型、ip、app_version、app_channel做频数统计（'request_model_id', 'request_ip','request_device_type', 'request_app_version'， 'request_app_channel' 这些数值列在数据中出现了多少次,用value_counts()计算）对于用户的请求行为序列，我们可以构建w2v特征(把用户请求行为序列看成句子，行为看作词，训练Word2Vec模型得到每个行为的表征)，对于用户的请求时间，我们可以计算请求时间的均值方差、请求时间间隔的统计特征等等；\n          \n- 模型训练采用传统分层五折交叉验证，本方案未对模型进行参数调参以及输出后处理。\n\n## **6. 一些尝试**\n\n- 本方案的解题思路与题目所要求的可能并不匹配，尝试使用一些半监督学习的方法，包括伪标签、标签传播等来更加充分利用无标签样本，但是几乎都没有收益；\n- 如何更好地利用无标签样本可能是本题的解题关键，期待其他选手的分享。\n\n## **7. 算法性能**\n\n- 资源配置：cpu i7 16G内存\n- 总计耗时：约为60分钟\n\n## **8. 相关文献**\n* Ke G, Meng Q, Finley T, et al. Lightgbm: A highly efficient gradient boosting decision tree[J]. Advances in neural information processing systems, 2017, 30: 3146-3154.\n* Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space[J]. arXiv preprint arXiv:1301.3781, 2013.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibrauee%2Fbytedanceai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flibrauee%2Fbytedanceai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibrauee%2Fbytedanceai/lists"}