{"id":13778992,"url":"https://github.com/mindawei/aliyun-safe-match","last_synced_at":"2025-04-23T12:14:10.284Z","repository":{"id":215749587,"uuid":"95944655","full_name":"mindawei/aliyun-safe-match","owner":"mindawei","description":"Webshell和钓鱼网站检测（阿里云安全算法挑战赛 第29名）","archived":false,"fork":false,"pushed_at":"2018-02-02T04:39:04.000Z","size":4371,"stargazers_count":60,"open_issues_count":1,"forks_count":23,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-23T12:13:50.077Z","etag":null,"topics":["fish","java","maven","regular-expression","safe","webshell"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mindawei.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-07-01T05:41:07.000Z","updated_at":"2024-09-03T09:40:11.000Z","dependencies_parsed_at":"2024-01-15T08:06:51.743Z","dependency_job_id":null,"html_url":"https://github.com/mindawei/aliyun-safe-match","commit_stats":null,"previous_names":["mindawei/aliyun-safe-match"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindawei%2Faliyun-safe-match","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindawei%2Faliyun-safe-match/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindawei%2Faliyun-safe-match/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mindawei%2Faliyun-safe-match/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mindawei","download_url":"https://codeload.github.com/mindawei/aliyun-safe-match/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250430595,"owners_count":21429324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fish","java","maven","regular-expression","safe","webshell"],"created_at":"2024-08-03T18:00:59.844Z","updated_at":"2025-04-23T12:14:10.259Z","avatar_url":"https://github.com/mindawei.png","language":"Java","funding_links":[],"categories":["\u003ca id=\"39e5bd43766abbdbc518390d86b3a0a5\"\u003e\u003c/a\u003eWebshell检测"],"sub_categories":[],"readme":"# 1 背景\n1. 该项目的代码是参加 \n[阿里云安全算法挑战赛](https://tianchi.aliyun.com/competition/information.htm?raceId=231585) 时写的。主要利用规则和正则表达式，排名第29。\n2. 为了更好地理解项目，对项目进行了重构，保留了处理的主体部分。\n\n# 2 QuickStart\n1. 需要配置maven环境。\n2. 下载并导入项目，将lib下的sun.misc.BASE64Decoder.jar添加项目中。\n3. 运行 [`src/test/java/safe/fish/TestFishFilter.java`](https://github.com/mindawei/aliyun-safe-match/blob/master/src/test/java/safe/fish/TestFishFilter.java) 可使用钓鱼网站检测功能。钓鱼网站检测功能接口如下。\n```\nFishFilter.isFish(String URL,String HTML)  // 静态方法，传入url和网页内容\n```\n4. 运行 [`src/test/java/safe/webshell/TestWebShellDetector.java`](https://github.com/mindawei/aliyun-safe-match/blob/master/src/test/java/safe/webshell/TestWebShellDetector.java) 可使用WebShell检测功能。WebShell检测功能接口如下。\n```\nWebShellDetector.isWebShell(String postData)  // 静态方法，传入网址后面带的参数或者上传内容\n```\n\n\n# 3 大致解题思路\n\n## 3.1 钓鱼网站判断\u003c/h2\u003e\n* 爬取“站长之家”的网站域名，建立白名单（中文关键字到网站的地址映射表）\u003cbr\u003e\n* 解析网页,获取关键字\u003cbr\u003e\n* 根据关键字查询白名单\u003cbr\u003e\n\n## 3.2 WebShell通信检测\u003c/h2\u003e\n* 列举恶意代码，如：eval call_user_func 等\u003cbr\u003e\n* 写出恶意代码的正则表达式\u003cbr\u003e\n* 进行正则表达式匹配\u003cbr\u003e\n\n\n# 4 项目结构\n* 测试文件夹 `test` 中的代码可以帮助理解一些代码的使用。\n* `data` 文件下存放了一些项目需要使用的数据。\n* 跟钓鱼网站相关的代码都在 `safe.fish` 包中，其中 `FishFilter.java` 是入口类。\n* 跟 WebShell检测相关的代码都在 ` safe.webshell` 包中，其中 `WebShellDetector.java` 是入口类。\n\n以下是 `src/main` 中的代码说明：\n```\n| - safe.fish 判断钓鱼网站的主体逻辑包\n\n| ----  | DomainConflictChecker.java 跟网页本身的一些链接冲突进行判断\n\n| ----  | FishFilter.java 判断钓鱼网站的入口类，主体逻辑\n\n| ----  | PageInfo.java 根据URL、HTML解析出的网页信息对象\n\n| ----  | PageKeyChecker.java 根据网页文本中的关键字进行检查\n\n| ----  | Result.java 检测结果枚举类\n\n| - safe.fish.utils  判断钓鱼网站的工具类\n\n| ----  | DataLoader.java 加载文件中的数据\n\n| ----  | TitleDivider.java 从网页标题中获取关键字\n\n| ----  | XnTool.java 解析xn--的网址中的中文\n\n| ----  | ZhushiUtil.java 去掉一些注释\n\n| - safe.fish.whitelist  中文到域名的映射包\n\n| ----  | ChineseDns.java 构建中文到域名的映射\n\n| ----  | SpiderOfChinaz.java 爬取站长之家的域名数据\n\n| - safe.webshell 判断webshell的主体逻辑包\n\n| ----  | Decoder.java 解码类\n\n| ----  | TextChecker.java 判断文本是否可以执行\n\n| ----  | WebShellDetector.java 对网址参数进行检测\n```\n\n以下是 `src/test` 中的代码说明：\n```\n| - safe.fish\n\n| ----  | TestFishFilter.java 测试钓鱼网站检测功能\n\n| - safe.fish.utils  \n\n| ----  | TestXnTool.java 测试解析xn--的网址中的中文\n\n| - safe.webshell \n\n| ----  | TestDecoder.java 测试解码功能\n\n| ----  | TestTextChecker.java 测试判断文本是否可以执行功能\n\n| ----  | TestWebShellDetector.java 测试 WebShell 检测功能\n```\n\n\n# 5 代码重构\n本次代码重构的方法如下所示。\n\n|问题 | 处理 |\n|--- | --- | \n| 注释掉的代码太多，舍不得扔掉代码，影响可读性 | 简单功能直接删去，复杂的功能可以重构独立到工具类中 |\n| 测试和源代码混合在一起 | 将测试代码移入到 `src/test` 中去 |\n| 代码中硬嵌手数据太多（比赛平台原因） | 移入到外部数据源中 |\n| 类和函数名过于简单随意，难以理解 | 根据类和函数功能，结合百度翻译等工具，取合适的名字 |\n| 一些状态无关、没有复用逻辑的类被设计成了单例，多次一举 | 改为静态函数 |\n| 钓鱼网站部分代码混乱，存在重复工作部分 | 将解析和判断分离，主要解析在创建 `PageInfo` 类时全部完成 |\n| 基于规则（比赛样本较少，也是不得已为之）的结构会导致：逻辑冲突、难以扩展等问题 | 样本充足时可使用机器学习训练的方法（重构中未实现，但已将部分特征集中在 `PageInfo` 类中）|\n| 文档不完善 | 完善文档 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindawei%2Faliyun-safe-match","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmindawei%2Faliyun-safe-match","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmindawei%2Faliyun-safe-match/lists"}