{"id":44689723,"url":"https://github.com/hooj0/sensitive-words-filter","last_synced_at":"2026-03-13T06:00:59.608Z","repository":{"id":44841973,"uuid":"126786818","full_name":"hooj0/sensitive-words-filter","owner":"hooj0","description":":page_facing_up:Sensitive word filtering items provide TTMP, DFA, DAT, hash bucket, and Tire algorithm support filtering. Supports text highlighting, filtering, verdicts, and alternative interface support.敏感词过滤项目，提供TTMP、DFA、DAT、hash bucket、Tire算法支持过滤。可以支持文本的高亮、过滤、判词、替换的接口支持。","archived":false,"fork":false,"pushed_at":"2023-02-22T07:12:58.000Z","size":264,"stargazers_count":176,"open_issues_count":3,"forks_count":46,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-10-20T20:14:29.826Z","etag":null,"topics":["dat","dfa","hash-bucket","hashing","sensitive","sensitive-words-filter","tire","ttmp","wordfilter"],"latest_commit_sha":null,"homepage":"https://hooj0.github.io/sensitive-words-filter/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hooj0.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-03-26T06:57:43.000Z","updated_at":"2023-10-20T20:14:30.477Z","dependencies_parsed_at":"2023-01-26T00:32:37.297Z","dependency_job_id":"df0037eb-5af7-4b2e-b0e5-9289259e15ca","html_url":"https://github.com/hooj0/sensitive-words-filter","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/hooj0/sensitive-words-filter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooj0%2Fsensitive-words-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooj0%2Fsensitive-words-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooj0%2Fsensitive-words-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooj0%2Fsensitive-words-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hooj0","download_url":"https://codeload.github.com/hooj0/sensitive-words-filter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hooj0%2Fsensitive-words-filter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30459760,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-13T03:55:51.346Z","status":"ssl_error","status_checked_at":"2026-03-13T03:55:33.055Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dat","dfa","hash-bucket","hashing","sensitive","sensitive-words-filter","tire","ttmp","wordfilter"],"created_at":"2026-02-15T07:00:30.141Z","updated_at":"2026-03-13T06:00:59.601Z","avatar_url":"https://github.com/hooj0.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":["自然语言处理"],"readme":"\n\n\n# 敏感词过滤、脱敏\n\n[TOC]\n\n## 1、简介\n`sensitive-words-filter` 是一个`JavaWeb`工程下的过滤器，可以过滤一些敏感的字符串，如：`色情`、`政治`、`暴力`、`人名`等特殊字符，防止发表一些不和谐的词条给系统带来运营成本。\n\n目前`sensitive-words-filter`提供了如下几种算法脱敏支持：\n\n* `DFA(\"dfa算法\", DfaFilter.class)` 综合性能比较高，表现突出，过滤效果好\n* `TIRE(\"tire树算法\", TireTreeFilter.class),` 大文本过滤效率稍低\n* `HASH_BUCKET(\"二级hash算法\", HashBucketFilter.class),` 综合性能中等，实现简单易懂\n* `DAT(\"双数组算法\", DatFilter.class),` 小文本过滤效果突出，实现超简单\n* `TTMP(\"ttmp算法\", TtmpFilter.class),` 综合性能突出，性能稍低占有内存大，但效率快，匹配有漏词情况\n* `SIMHASH(\"simhash算法\", SimHashFilter.class)` 可以了解，不推荐使用\n\n以上每一种算法都有自己的特点，以供选择。\n\n\n\n## 2、使用方法\n\n\u003e 提供两种方法，一种直接运行敏感词执行器`AbstractFilterExecutor`的实现类，该类的实现类是单例模式，提供常用的**判词、高亮、替换、查词**等接口；另一种就是工具类的接口`SensitiveWordsFilterUtils`,相对于执行器提供了**刷新缓存、刷新全部缓存**的接口\n\n\n\n下面以 `DatFilterExecutor`算法实现类为例演示\n\n### 2.1、初始化并添加敏感词\n\u003e 初始化是初始化缓存和数据存储根对象，init方法就是初始化根对象的，put方法是添加敏感词\n\n```java\nDfaFilterExecutor.getInstance().init();\nDfaFilterExecutor.getInstance().put(\"中国人\");\nDfaFilterExecutor.getInstance().put(\"中国男人\");\nDfaFilterExecutor.getInstance().put(\"中国人民\");\nDfaFilterExecutor.getInstance().put(\"人民\");\nDfaFilterExecutor.getInstance().put(\"中间\");\nDfaFilterExecutor.getInstance().put(\"女人\");\n\nDfaFilterExecutor.getInstance().put(\"一举\");\nDfaFilterExecutor.getInstance().put(\"一举成名\");\nDfaFilterExecutor.getInstance().put(\"一举成名走四方\");\nDfaFilterExecutor.getInstance().put(\"成名\");\nDfaFilterExecutor.getInstance().put(\"走四方\");\n```\n\n\n### 2.2、匹配敏感词\n\n\u003e 匹配敏感词会把符合的敏感词都找到，该方法有两个参数\n\n+ **接口说明：**\n\n```java\n/**\n* 返回匹配到的敏感词语\n* @author hoojo\n* @createDate 2018年2月9日 下午4:00:06\n* @param partMatch 是否部分匹配\n* @param content 被匹配的语句\n* @return 返回匹配的敏感词语集合\n*/\npublic Set\u003cString\u003e getWords(boolean partMatch, String content) throws RuntimeException;\n```\n\n+ **运行示例：**\n\n```java\nString content = \"我们中国人都是好人，在他们中间有男人和女人。中国男人很惨，中国人民长期被压迫。\";\nSystem.out.println(DfaFilterExecutor.getInstance().getWords(true, content));\nSystem.out.println(DfaFilterExecutor.getInstance().getWords(false, content));\n```\n\n+ **运行结果：**\n\n```\n[中国人, 女人, 中间, 中国男人]\n[中国人, 女人, 中间, 中国男人, 中国人民, 人民]\n```\n\u003e 从上面结果发现部分匹配和完整匹配的结果集不一样，部分匹配是匹配到敏感词后立即退出当前匹配；而完整匹配会把所有词都匹配出来，也就是把敏感词中的小词也匹配到。\n\n### 2.3、过滤敏感词\n\n\u003e 过滤敏感词主要是将匹配到的敏感词过滤掉，以某种字符串进行替换敏感词字符\n\n\n+ **接口说明：**\n\n```java\n/**\n* 过滤敏感词，并把敏感词替换为指定字符\n* @author hoojo\n* @createDate 2018年2月9日 下午4:38:12\n* @param partMatch 是否部分匹配\n* @param content 被匹配的语句\n* @param replaceChar 替换字符\n* @return 过滤后的字符串\n* @throws RuntimeException\n*/\npublic String filter(boolean partMatch, String content, char replaceChar) throws RuntimeException;\n```\n\n+ **运行示例：**\n\n```java\nSystem.out.println(DfaFilterExecutor.getInstance().filter(false, content, '*'));\n```\n\n\n+ **运行结果：**\n```\n我们***都是好人，在他们**有男人和**。****很惨，***民长期被压迫。\n```\n\n### 2.4、敏感词高亮\n\n\u003e 敏感词高亮就是将匹配到的敏感字符以HTML的tag进行替换，这样在前端显示的时候就会比较突出\n\n+ **接口介绍：**\n\n```java\n/**\n * html高亮敏感词\n * @author hoojo\n * @createDate 2018年2月9日 下午4:37:33\n * @param partMatch 是否部分匹配\n * @param content 被匹配的语句\n * @return 返回html高亮敏感词\n * @throws RuntimeException\n */\npublic String highlight(boolean partMatch, String content) throws RuntimeException;\n```\n\n+ **运行示例：**\n```java\nSystem.out.println(DfaFilterExecutor.getInstance().highlight(false, content));\n```\n\n+ **运行结果：**\n```html\n我们\u003cfont color='red'\u003e中国人\u003c/font\u003e都是好人，在他们\u003cfont color='red'\u003e中间\u003c/font\u003e有男人和\u003cfont color='red'\u003e女人\u003c/font\u003e。\u003cfont color='red'\u003e中国男人\u003c/font\u003e很惨，\u003cfont color='red'\u003e中国人\u003c/font\u003e民长期被压迫。\n```\n\n\n### 2.5、是否存在敏感词\n\n\u003e 判断一段文本是否包含敏感词，若包含立即返回true，否则 false\n\n+ **接口说明：**\n\n```java\n/**\n* 是否包含敏感字符\n* @author hoojo\n* @createDate 2018年2月9日 下午2:57:52\n* @param partMatch 是否支持匹配词语的一部分\n* @param content 被匹配内容\n* @return 是否包含敏感字符\n*/\npublic boolean contains(boolean partMatch, String content) throws RuntimeException;\n```\n\n+ **运行示例：**\n\n\n```java\nSystem.out.println(DfaFilterExecutor.getInstance().contains(true, content));\n```\n\n+ **运行结果：**\n\n```\ntrue\n```\n\n\n## 3、各算法实现测试说明\n\n\u003e 针对各算法进行测试，分别测试 匹配文本 344字符、5519字符、11.304.959字符\n\n+ **测试结果**\n\n| 算法接口                                     |       过滤字符数       |  耗时(毫秒)   |    内存消耗（KB）     |\n| :--------------------------------------- | :---------------: | :-------: | :-------------: |\n| `DFA(\"dfa算法\", DfaFilter.class)`          | 344/5519/11304959 |  5/7/241  | 3276/3276/42470 |\n| `TIRE(\"tire树算法\", TireTreeFilter.class) ` | 344/5519/11304959 | 1/9/12413 | 1638/1638/47934 |\n| `HASH_BUCKET(\"hash桶算法\")`                 | 344/5519/11304959 |  0/4/659  | 1638/1638/79269 |\n| `DAT(\"双数组算法\", DatFilter.class)`          | 344/5519/11304959 |  1/4/720  | 819/819/424066  |\n| `TTMP(\"ttmp算法\", TtmpFilter.class)`       | 344/5519/11304959 |  0/2/226  | 819/819/567125  |\n\n在小于5000字左右，各算法差距不大。但字符量大的情况下，差距明显。\n\n## 4、总结\n+ `TTMP`算法用的内存最多，但速度最快，但是存在一个漏词的问题。\n+ `DFA`算法表现良好，各方面都不错，比较实用，特别在大量文本情况下很稳定。\n+ `TIRE`算法在大量文本情况下，效率稍低。可以优化下查找速度。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhooj0%2Fsensitive-words-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhooj0%2Fsensitive-words-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhooj0%2Fsensitive-words-filter/lists"}