{"id":18373682,"url":"https://github.com/xxddpac/warmor","last_synced_at":"2025-04-06T19:32:25.249Z","repository":{"id":207525154,"uuid":"718906433","full_name":"xxddpac/wArmor","owner":"xxddpac","description":"🚀 An WAF Framework based on OpenResty","archived":false,"fork":false,"pushed_at":"2023-11-21T08:45:06.000Z","size":89325,"stargazers_count":115,"open_issues_count":2,"forks_count":25,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-22T06:11:53.992Z","etag":null,"topics":["golang","lua","openresty","security","waf"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xxddpac.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-15T03:00:45.000Z","updated_at":"2025-01-24T18:28:15.000Z","dependencies_parsed_at":"2024-11-06T00:21:30.547Z","dependency_job_id":null,"html_url":"https://github.com/xxddpac/wArmor","commit_stats":null,"previous_names":["xxddpac/warmor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxddpac%2FwArmor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxddpac%2FwArmor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxddpac%2FwArmor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxddpac%2FwArmor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xxddpac","download_url":"https://codeload.github.com/xxddpac/wArmor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247539341,"owners_count":20955295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","lua","openresty","security","waf"],"created_at":"2024-11-06T00:11:24.661Z","updated_at":"2025-04-06T19:32:20.239Z","avatar_url":"https://github.com/xxddpac.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wArmor - An WAF Framework based on OpenResty\n\n[![Security Status](https://www.murphysec.com/platform3/v31/badge/1725063962646499328.svg)](https://www.murphysec.com/console/report/1725060297802076160/1725063962646499328)\n\n### 性能并发 【Performance concurrency】\n\n原生 `Nginx` 是基于事件驱动架构、异步非阻塞设计和轻量级进程的核心特性，实现了高性能和高并发的Web服务。\n`wArmor` 充分利用了 `Openresty` 官方提供的异步非阻塞模块，如 `resty.http` 、`resty.redis` 、`resty.mysql`\n和 `resty.logger.socket` 等，确保实现功能的同时也保证 `Nginx` 的高并发稳定性。\n\nThe native Nginx leverages an event-driven architecture, asynchronous non-blocking design, and lightweight processes as\ncore features, enabling high-performance and high-concurrency web services. wArmor maximizes the use of Openresty's\nofficially provided asynchronous non-blocking modules such as resty.http, resty.redis, resty.mysql, and\nresty.logger.socket, ensuring both functionality and the high-concurrency stability of Nginx.\n\n### 规则热更新 【Rule hot-reloading】\n\n将 `WAF` 规则单独抽出为一个 `API` 服务，可在平台上针对规则、配置进行变更并实时通知 `WAF` 而无需重启 `Nginx` 。\n\nExtracting the WAF rules into a separate API service allows for real-time modifications of rules and configurations on\nthe platform, providing instant notifications to the WAF without the need to restart Nginx.\n\n### 编排联动 【Orchestrating linkages】\n\n引入了灵活的 `IP黑白名单` 机制，可配合其他安全工具进行联动，`WAF`\n的主要任务是拦截恶意请求并记录相关日志，而其他工具负责检测和更新 `IP黑名单` 。这样的分工协同确保了高效的安全响应。\n另外 `IP黑白名单` 支持永久、临时封禁也同样可在平台上进行手动更新。\n\nAn adaptable IP blacklisting and whitelisting mechanism has been introduced, capable of collaborating with other\nsecurity tools. The primary role of the WAF is to intercept malicious requests and log relevant activities, while other\ntools handle the detection and updating of the IP blacklist. This division of tasks ensures an efficient security\nresponse. Additionally, the IP blacklisting and whitelisting feature supports permanent and temporary bans, allowing\nmanual updates via the platform.\n\n### 灰度验证(TODO) 【Gray verification】\n\n考虑到 `Nginx` 多集群部署的情况，后续也会提供选择性关闭热更新的选项，以便在特定 `Nginx`\n集群上进行规则更新。这使得灰度测试变得更加容易，确保规则的更新不会影响整个环境的正常运行，\n一旦在特定集群上确认规则的正常工作，随后即可批量下发规则到其余集群。\n\nConsidering the scenario of multiple-cluster deployment for Nginx, we'll be introducing an option to selectively disable\nhot-reloading. This allows for rule updates on specific Nginx clusters, making gray testing more straightforward. It\nensures that rule updates do not affect the normal operation of the entire environment. Once the rules are confirmed to\nbe functioning properly on a specific cluster, they can be uniformly deployed across the remaining clusters.\n\n### 规则引擎API文档 【rule engine API documentation】\n\n访问 `http://rule_engine_ip:9999//swagger/index.html#/` API文档，针对 `规则` 、 `配置` 、 `IP黑白名单`\n增删改查，某些枚举值含义参考枚举对应关系。\n\nAccess the API documentation at http://rule_engine_ip:9999//swagger/index.html#/. Perform CRUD operations for Rules,\nConfigurations, and IP Black/Whitelists. For certain enumerated values, refer to the corresponding relationship mapping.\n\n![api](rule_engine/images/api.png)\n\n## 规则引擎部署 【Rule Engine Deployment】\n\n### `mysql` 数据库 【mysql database】\n\n- 创建 `waf` 数据库 【CREATE DATABASE waf】\n\n```mysql\n    CREATE DATABASE waf CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;\n```\n\n- 新建 `config` 配置表 【CREATE TABLE config】\n\n```mysql\n    CREATE TABLE IF NOT EXISTS config\n    (\n        id         INT AUTO_INCREMENT PRIMARY KEY COMMENT '配置模式主键ID',\n        operator   VARCHAR(10) NOT NULL COMMENT '操作人',\n        mode       INT         NOT NULL COMMENT '配置模式',\n        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',\n        updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间'\n    )\n```\n\n- 新建 `ip` 黑白名单表 【CREATE TABLE ip】\n\n```mysql\n    CREATE TABLE IF NOT EXISTS ip\n    (\n        id          INT AUTO_INCREMENT PRIMARY KEY COMMENT '黑白名单主键ID',\n        operator    VARCHAR(10)  NOT NULL COMMENT '操作人',\n        comment     VARCHAR(255) NOT NULL COMMENT '备注信息',\n        ip_type     INT          NOT NULL COMMENT '黑白名单类型',\n        block_type  INT       DEFAULT 1 COMMENT '封禁类型',\n        ip_address  VARCHAR(15)  NOT NULL COMMENT '黑白名单IP',\n        expire_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT '黑名单封禁时间',\n        created_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',\n        updated_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间'\n    )\n```\n\n- 新建 `rule` 规则表 【CREATE TABLE rule】\n\n```mysql\n#主表\nCREATE TABLE IF NOT EXISTS rule\n(\n    id              INT AUTO_INCREMENT PRIMARY KEY COMMENT '规则主键ID',\n    operator        VARCHAR(10)  NOT NULL COMMENT '操作人',\n    rule_variable   INT          NOT NULL COMMENT '规则变量',\n    rule_type       INT          NOT NULL COMMENT '规则类型',\n    status          BOOLEAN      NOT NULL COMMENT '规则开启状态',\n    rule_action     INT          NOT NULL COMMENT '规则执行动作',\n    description     VARCHAR(255) NOT NULL COMMENT '规则描述',\n    severity        INT          NOT NULL COMMENT '风险级别',\n    rules_operation VARCHAR(3)   NOT NULL COMMENT '规则匹配条件(and/or)',\n    created_at      TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',\n    updated_at      TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间'\n)\n```\n\n```mysql\n#关联表\nCREATE TABLE rules\n(\n    id         INT AUTO_INCREMENT PRIMARY KEY COMMENT '规则匹配详情主键ID',\n    rule_id    INT          NOT NULL COMMENT '规则表外键',\n    rules      VARCHAR(255) NOT NULL COMMENT '规则详情',\n    FOREIGN KEY (rule_id) REFERENCES rule (id) ON DELETE CASCADE,\n    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',\n    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间'\n)\n```\n\n### 启动 `rule_engine` 【start rule_engine】\n\n`config core.config.toml` 作为规则引擎配置文件，需自行配置`mysql`、`redis` 相关信息\n\nConfigure the core.config.toml file as the rule engine configuration file, where you need to set up the mysql and redis\nrelated information manually.\n\n```bash\n# start rule_engine\n./bin/rule_engine -config core.config.toml\n```\n\n```bash\n# check ping\ncurl http://rule_engine_ip:9999/ping\n# pong\n```\n\n### 设定 `waf` 配置模式 【Set up the WAF configuration mode】\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d ' {\n                \"operator\":\"wArmor\",\n                \"mode\": 1\n            }' http://rule_engine_ip:9999/api/v1/config\n```\n\n### 创建 `路径遍历` 规则 【Create a Path Traversal rule】\n\n```bash\n curl -X POST -H \"Content-Type: application/json\" -d ' {\n                \"rule_variable\": 1,\n                \"operator\": \"wArmor\",\n                \"rule_type\": 4,\n                \"status\": true,\n                \"rules\": [\n                    \"(?:etc/\\\\W*passwd)\",\n                    \"(?:/|\\\\|\\\\\\\\)?(?:\\\\.\\\\./)+\"\n                ],\n                \"rule_action\": 2,\n                \"description\": \"当检测到路径遍历攻击时waf将重定向拦截此行为\",\n                \"rules_operation\": \"or\",\n                \"severity\": 3\n            }' http://rule_engine_ip:9999/api/v1/rule\n```\n\n### 创建 `敏感信息监控` 规则 【Create a Sensitive information monitoring rule】\n\n```bash\n curl -X POST -H \"Content-Type: application/json\" -d ' {\n                \"rule_variable\": 6,\n                \"operator\": \"wArmor\",\n                \"rule_type\": 9,\n                \"status\": true,\n                \"rules\": [\"You have an error in your SQL syntax\"],\n                \"rule_action\": 3,\n                \"description\": \"检测响应体是否包含原始sql错误信息而泄露表结构信息,开发者应处理该错误返回\",\n                \"rules_operation\": \"or\",\n                \"severity\": 2\n            }' http://rule_engine_ip:9999/api/v1/rule\n``````\n\n## WAF部署 【WAF deployment】\n\n### 安装 `libmaxminddb` 库 【install libmaxminddb】\n\n```bash\nwget -P /usr/local/src https://github.com/maxmind/libmaxminddb/releases/download/1.7.1/libmaxminddb-1.7.1.tar.gz\ntar -zxvf libmaxminddb-1.7.1.tar.gz\ncd libmaxminddb-1.7.1\n./configure\nmake \u0026\u0026 make install\necho /usr/local/lib \u003e\u003e /etc/ld.so.conf.d/local.conf\nldconfig\n```\n\n将 `GeoLite2-City.mmdb` 拷贝至 `/usr/local/share/GeoIP` 目录下。\n\nCopy GeoLite2-City.mmdb to the /usr/local/share/GeoIP directory\n\n### 安装 `Openresty` 【install Openresty】\n\n```bash\ncd /usr/local/src\nwget https://openresty.org/download/openresty-1.21.4.2.tar.gz\ntar zxf openresty-1.21.4.2.tar.gz\ncd openresty-1.21.4.2\n./configure --prefix=/usr/local/openresty \\\n--with-http_ssl_module \\\n--with-http_v2_module \\\n--with-http_realip_module \\\n--with-http_sub_module \\\n--with-http_stub_status_module \\\n--with-http_auth_request_module \\\n--with-http_secure_link_module \\\n--with-stream \\\n--with-stream_ssl_module \\\n--with-stream_realip_module \\\n--without-http_fastcgi_module \\\n--without-mail_pop3_module \\\n--without-mail_imap_module \\\n--without-mail_smtp_module\n\nmake \u0026\u0026 make install\n```\n\n`config.lua` 作为 `waf` 的配置文件，需自行配置`mysql`、`redis`、`syslog` 等信息，\n并将 `waf` 文件夹拷贝至 `/usr/local/openresty` 目录下。\n\nEdit the config.lua file as the configuration file for waf, requiring manual configuration of mysql, redis, syslog, and\nother related information. Copy the waf folder to the /usr/local/openresty directory\n\n编辑 `nginx` 配置文件 `/usr/local/openresty/nginx/conf/nginx.conf` ，在 `http` 模块下新增\n\nEdit the nginx configuration file located at /usr/local/openresty/nginx/conf/nginx.conf. Add the following within the\nhttp module\n\n```bash\nlua_shared_dict waf_rule 20m;\nlua_shared_dict waf_config 1m;\nlua_shared_dict waf_ip 20m;\n\nlua_package_path \"/usr/local/openresty/waf/?.lua;/usr/local/openresty/waf/lib/?.lua;;\";\ninit_worker_by_lua_file /usr/local/openresty/waf/worker.lua;\naccess_by_lua_file /usr/local/openresty/waf/waf.lua;\nbody_filter_by_lua_file /usr/local/openresty/waf/body_filter.lua;\nheader_filter_by_lua_file /usr/local/openresty/waf/header_filter.lua;\n```\n\n重启 `nginx` 服务\n\nRestart nginx\n\n```bash\n/usr/local/openresty/nginx/sbin/nginx -s reload\n```\n\n如果部署一切正常，访问 `http://nginx_ip/api/v1/rule?query=../../etc/passwd` 浏览器会出现拦截信息。\n\nIf the deployment is successful, accessing http://nginx_ip/api/v1/rule?query=../../etc/passwd will result in\ninterception messages in the browser\n\n![waf](rule_engine/images/waf.png)\n\n拦截日志会异步写入远程 `syslog-ng` 服务器。\n\nThe interception logs will be asynchronously written to a remote syslog-ng server\n\n- `路径遍历` 被拦截日志 【Logs for Path Traversal interceptions】\n\n```json\n{\n  \"program\": \"wArmor\",\n  \"rule_action\": \"重定向\",\n  \"city\": \"杭州\",\n  \"country\": \"中国\",\n  \"latitude\": \"30.2994\",\n  \"longitude\": \"120.1612\",\n  \"iso_code\": \"CN\",\n  \"request_server_name\": \"*******\",\n  \"request_uri\": \"/api/v1/rule?query=../../etc/passwd\",\n  \"request_ua\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36\",\n  \"timestamp\": \"2023-11-15 17:18:45\",\n  \"request_id\": \"a2deb9c0-d003-47c2-96f4-452c8f1d502e\",\n  \"request_ip\": \"********\",\n  \"severity\": \"中危\",\n  \"rule_id\": 1,\n  \"request_data\": \"\",\n  \"mode\": \"拦截模式\",\n  \"request_method\": \"GET\",\n  \"rule_type\": \"路径遍历\"\n}\n```\n\n- `敏感信息监控` 日志 【Logs for Sensitive Information Monitoring】\n\n```json\n{\n  \"rule_type\": \"敏感信息\",\n  \"rule_id\": 2,\n  \"response\": \"{\\\"code\\\":400,\\\"msg\\\":\\\"Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'rules VARCHAR(255) NOT NULL COMMENT '规则详情',    \\\\n        \\\\t   FOREIGN KEY (rule_id) RE' at line 4\\\",\\\"data\\\":null}\",\n  \"request_server_name\": \"*****\",\n  \"request_uri\": \"/api/v1/user/list\",\n  \"severity\": \"高危\",\n  \"timestamp\": \"2023-11-17 14:41:12\",\n  \"program\": \"wArmor\",\n  \"request_method\": \"GET\"\n}\n```\n\n## 架构设计 【Architectural design】\n\n![wArmor](rule_engine/images/wArmor.jpg)\n\n\n\n\u003e [!IMPORTANT]\n\u003e 当 **WAF** 和**规则引擎**都正常运行时，**规则引擎**每**10秒**会发送心跳以确保**WAF**订阅通道保持监听状态。\n\u003e\n\u003e When both the WAF and the rule engine are functioning properly, the rule engine sends a heartbeat every 10 seconds to\n\u003e ensure the WAF subscription channel remains in a listening state.\n\u003e\n\u003e 然而，如果**规则引擎**发生异常，**WAF**\n\u003e 在5分钟内未收到心跳信号，订阅通道将进入超时状态从而导致热更新功能失效。在此情况下，需要重新启动**规则引擎**并重启\n\u003e **Nginx**服务。\n\u003e\n\u003e However, if the rule engine encounters an issue and the WAF fails to receive a heartbeat signal within 5 minutes, the\n\u003e subscription channel will enter a timeout state, resulting in the failure of the hot update functionality. In such a\n\u003e scenario, it's necessary to restart the rule engine and subsequently reboot the Nginx service.\n\u003e\n\u003e 你也可以通过访问**waf/config.lua**文件来自定义**Redis**的`read_timeout`读取超时时间以应对这种极端情况，并设置更长的超时时间。\n\u003e\n\u003e You can also address this extreme situation by accessing the waf/config.lua file to customize the read_timeout for\n\u003e Redis, thereby setting a longer timeout duration to mitigate this issue.\n\n## 文末 【End】\n\n感谢 [bukaleyang/zhongkui-waf](https://github.com/bukaleyang/zhongkui-waf) 项目, `wArmor` 在其基础上进行了二次开发，并进行了性能和架构上的优化。\n\n- `WAF` 与规则解耦实现热更新,现在每次规则变动都无需重启线上的 `Nginx` 服务，规则的热更新提升了灵活性和维护效率。\n\n- 规则引入变量提高检查速度,`WAF` 知道要检查 `HTTP` 的哪个部分标识，而不必每次请求都遍历所有规则,提升了规则检查的速度。\n\n- 日志性能优化,使用非阻塞的 `resty.logger.socket` 实现日志的异步写入，在高并发情况下保证了稳定性，避免了阻塞对业务的影响。另外日志中增加了请求的唯一ID，\n  这在处理问题时非常有用，根据ID可以快速在海量日志中检索到对应的请求日志。\n\nThank you to the [bukaleyang/zhongkui-waf](https://github.com/bukaleyang/zhongkui-waf) project. wArmor underwent\nsecondary development based on it, focusing on\nperformance and architectural optimizations.\n\n- Decoupling WAF from rules for hot updates: Now, every rule change doesn't require a restart of the online Nginx\n  service,\n  enhancing flexibility and maintenance efficiency.\n\n- Rule variable introduction for improved inspection speed: WAF identifies which part of the HTTP to inspect,\n  significantly improving inspection speed without iterating through all rules for each request.\n\n- Performance-optimized logging: Asynchronous log writing using non-blocking resty.logger.socket ensures stability under\n  high concurrency, preventing blocking and its impact on business. Additionally, unique request IDs are added to the\n  logs, facilitating quick retrieval of corresponding request logs within extensive log data, particularly useful for\n  issue resolution.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxxddpac%2Fwarmor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxxddpac%2Fwarmor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxxddpac%2Fwarmor/lists"}