{"id":13841850,"url":"https://github.com/jerryting/ngxlua","last_synced_at":"2025-07-11T13:32:43.510Z","repository":{"id":218715096,"uuid":"115404195","full_name":"jerryting/ngxlua","owner":"jerryting","description":"nginx/openresty lua access limit 限流防爬","archived":false,"fork":false,"pushed_at":"2022-09-28T14:10:37.000Z","size":22,"stargazers_count":40,"open_issues_count":0,"forks_count":18,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-08-05T17:29:32.873Z","etag":null,"topics":["blackl","limit","lua","nginx","spider","useragent"],"latest_commit_sha":null,"homepage":null,"language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jerryting.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-12-26T08:48:54.000Z","updated_at":"2024-06-28T13:54:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"3f15d3b0-acac-43ba-9fa4-2d5955504a86","html_url":"https://github.com/jerryting/ngxlua","commit_stats":null,"previous_names":["jerryting/ngxlua"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerryting%2Fngxlua","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerryting%2Fngxlua/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerryting%2Fngxlua/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jerryting%2Fngxlua/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jerryting","download_url":"https://codeload.github.com/jerryting/ngxlua/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225729708,"owners_count":17515157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blackl","limit","lua","nginx","spider","useragent"],"created_at":"2024-08-04T17:01:22.816Z","updated_at":"2024-11-21T12:30:30.355Z","avatar_url":"https://github.com/jerryting.png","language":"Lua","readme":"\u003e 主要目的是防止爬虫、恶意访问等，爬虫本身基本防不住，所以尽量让爬虫爬取时对服务器产生压力控制在带宽和服务器性能允许的范围，并且做一个动态黑名单的规则，比如永久或一段时间内拉黑某个ip、某个用户id，禁止其再访问服务器，即保证后端服务器不受影响，也保证正常用户不受影响。\n\n\u003e web服务器使用openresty，用它主要是出于nginx_lua模块灵活和便捷的考虑，当然使用原生nginx也可以，在安装的时候带上lua支持模块，另外 春哥的openresty里 有个限流分流的模块 [lua-resty-limit-traffic](https://github.com/openresty/lua-resty-limit-traffic) 如果想做限流分流，服务降级，灰度等等 这个模块确实也是可以用的，但是如果涉及到用户级别的缓存，比如用户ID有效性，宕机缓存失效等，所以改用redis 做缓存，没有用官方提供的模块。\n\n\u003e 运行环境：centos + openresty + lua + redis (安装略)\n\n\u003e openresty/nginx配置：\n\n```\n对访问本机x-service的所有请求，请求到达时执行access_limit.lua脚本，进行限流检查\nlocation /x-service {\n            lua_code_cache on; #代码缓存，如果nginx -s reload 失效，最好执行以下 stop 再启动\n            access_by_lua_file /usr/local/openresty-1.13.6.1/mylua/access_limit.lua; # 限流脚本入口\n            proxy_pass http://server_ups/x-service-web;\n            proxy_set_header  X-Real-IP  $remote_addr;\n            proxy_set_header  Host       $http_host;\n        }\n```\n\n\u003e ***lua_utils.lua*** 提供获取客户端ip公共方法\n\n```\nlocal _M = {}\n\nfunction _M.new(self)\n\treturn self\nend\n--获取客户端IP\nfunction _M.get_clientip(self)\n\tlocal client_ip = ngx.req.get_headers()[\"X-Real-IP\"]\n\tif client_ip == nil then\n\t\tclient_ip = ngx.req.get_headers()[\"x_forwarded_for\"]\n\tend\n\n\tif client_ip == nil then\n\t\tclient_ip = ngx.var.remote_addr\n\tend\n\treturn client_ip\nend\n\nreturn _M\n```\n\u003e ***redispool.lua*** 提供redis连接池\n\n```\nlocal redis = require \"resty.redis\"\n\nlocal config = {\n\thost = \"127.0.0.1\",\n    port = 6379,\n    password = \"password\"\n}\n\nlocal _M = {}\n\n--获取redis连接\nfunction _M.new(self)\n    local red = redis:new()\n    red:set_timeout(1000) -- one second timeout\n    local res = red:connect(config['host'], config['port'])\n    if not res then\n        return nil\n    end\n    if config['password'] ~= nil then\n\t\tres = red:auth(config['password'])\n\t    if not res then\n\t        return nil\n\t    end\n    end\n    red.close = close\n    return red\nend\n--归还连接到连接池 以备复用\nfunction close(self)\n    self:set_keepalive(120000, 50) --50个连接，每个120秒保活\nend\n\nreturn _M\n```\n\u003e ***ip_limit.lua*** 全局IP限制：例如单IP对本机该服务的所有访问次数达到200次，则拉黑该IP，封禁24小时，24小时候自动解封\n(如果你觉得代理ip一抓一大把，那这个脚本确实没啥大用，你可以考虑限制ua、固定访问header等)\n\n```\npackage.path  = '/usr/local/openresty-1.13.6.1/mylua/?.lua;;' .. package.path\n\nlocal ttl_timespan_s = 86400  \t\t\t--封禁时长\nlocal ip_check_timespan_s = 86400    \t--检查步长\nlocal access_threshold_count = 200 \t\t--访问频率计数阈值\nlocal error_status = 403\n\nlocal key_prefix_isdeny = \"globalipfreq_isdeny_\"\nlocal key_prefix_stime = \"globalipfreq_stime_\"\nlocal key_prefix_count = \"globalipfreq_count_\"\n\nlocal _M = {}\n\nfunction _M.new(self)\n\treturn self\nend\n\nfunction _M.check_ip_freq(self)\n\tlocal utils = require \"lua_utils\"\n\tlocal client_ip = utils.get_clientip()\n\n\tlocal rds_key = client_ip\n\n\t--获取redis连接\n\tlocal redis = require \"redispool\"\n\tlocal rds = redis.new()\n\t--如果连接失败、redis丢失服务等 按无限制操作\n\tif not rds then\n\t\treturn 1\n\tend\n\t--查询ip是否在封禁段内，若在则返回http错误码\n\t--封禁时间段内不再对ip时间和计数做处理\n\tlocal is_deny , err = rds:get(key_prefix_isdeny..rds_key)\n\tif tonumber(is_deny) == 1 then\n\t\tngx.log(ngx.ERR,\"globalipfreq_deny \",\"ip: \"..rds_key..\" : deny timespan : \"..ttl_timespan_s)\n\t\trds:close()\n\t\treturn error_status\n\tend\n\n\tlocal start_time , err = rds:get(key_prefix_stime..rds_key)\n\tlocal count , err = rds:get(key_prefix_count..rds_key)\n\t--如果ip记录时间大于指定时间间隔或者记录时间不存在,则初始化记录时间、计数\n\t--如果IP访问的时间间隔小于约定的时间间隔，则ip计数正常加1，且如果ip计数大于约定阈值，则设置ip的封禁key为1,即将此IP拉黑 ,同时设置封禁IP的数据过期时间\n\tif start_time == ngx.null or (os.time() - start_time) \u003e= ip_check_timespan_s then\n\t\tres , err = rds:set(key_prefix_stime..rds_key , os.time())\n\t\trds:expire(key_prefix_stime..rds_key,ttl_timespan_s)\n\t\tres , err = rds:set(key_prefix_count..rds_key , 1)\n\t\trds:expire(key_prefix_count..rds_key,ttl_timespan_s)\n\t\trds:close()\n\t\treturn 1\n\telse\n\t\tif count == ngx.null then\n\t\t\trds:set(key_prefix_count..rds_key , 1)\n\t\t\trds:expire(key_prefix_count..rds_key,ttl_timespan_s)\n\t\telse\n\t\t\tcount = count + 1\n\t\t\tres , err = rds:incr(key_prefix_count..rds_key)\n\t\t\tif count \u003e= access_threshold_count then\n\t\t\t\tres , err = rds:set(key_prefix_isdeny..rds_key,1)\n\t\t\t\tres , err = rds:expire(key_prefix_isdeny..rds_key,ttl_timespan_s)\n\t\t\t\tngx.log(ngx.ERR,\"globalipfreq_deny \",\"ip: \"..rds_key..\" : deny timespan : \"..ttl_timespan_s)\n\t\t\t\trds:close()\n\t\t\t\treturn error_status\n\t\t\tend\n\t\tend\n\t\trds:close()\n\tend\nend\n\nreturn _M\n```\n\u003e ***uid_limit.lua*** 对用户ID限制: 针对业务接口比如/api/detail/接口的限制，比如一天内 同一个UID只能访问25次，超过25次立即封禁，限制服务24h，24h后自动解封\n(如果多个规则组合判断，能判断到确实是一个爬虫或者恶意访问就直接永久封禁，当然规则要兼顾很多地方，我这里采用一刀切的方式，后面有时间再慢慢补充，让整个规则更符合业务需要)\n\n```\npackage.path  = '/usr/local/openresty-1.13.6.1/mylua/?.lua;;' .. package.path\n\nlocal ttl_timespan_s = 86400  \t\t\t--封禁时长\nlocal uid_check_timespan_s = 86400    \t--检查步长\nlocal access_threshold_count = 25 \t\t--访问频率计数阈值\nlocal error_status = 403\n\nlocal key_prefix_isdeny = \"uidfreq_isdeny_\"\nlocal key_prefix_stime = \"uidfreq_stime_\"\nlocal key_prefix_count = \"uidfreq_count_\"\nlocal key_prefix_uid = \"UID_\"\n\nlocal _M = {}\n\nfunction _M.new(self)\n\treturn self\nend\n\nfunction _M.check_uid_freq(self)\n\t--只针对指定接口做限流,其他接口放行\n\tlocal uri = ngx.var.uri\n\tif not string.find(uri,\"/api/detail/\") then\n\t\treturn 1\n\tend\n\t--获取用户id\n\tlocal uid = ngx.var.arg_uid\n\tlocal rds_key = uid\n\tif not uid then\n\t\treturn error_status\n\tend\n\t--连接redis\n\tlocal redis = require \"redispool\"\n\tlocal rds = redis.new()\n\t--如果连接失败、redis丢失服务等 按无限制操作\n\tif not rds then\n\t\treturn 1\n\tend\n\t--UID有效性验证 非法用户直接拒绝访问\n\tlocal uiddata,err = rds:get(key_prefix_uid..uid) --用户的ID需要提前预热到redis，批量操作+ 动态增加的方式\n\tif uiddata == ngx.null then\n\t\treturn error_status\n\tend\n\t--查询是否在封禁段内，若在则返回错误码\n\t--因封禁时间会大于记录时间，故此处不对时间key和计数key做处理\n\tlocal is_deny , err = rds:get(key_prefix_isdeny..rds_key)\n\tif tonumber(is_deny) == 1 then\n\t\tngx.log(ngx.ERR,\"uidfreq_deny \",\"ip: \"..rds_key..\" : deny timespan : \"..ttl_timespan_s)\n\t\trds:close()\n\t\treturn error_status\n\tend\n\n\tlocal start_time , err = rds:get(key_prefix_stime..rds_key)\n\tlocal count , err = rds:get(key_prefix_count..rds_key)\n\t--如果记录时间大于指定时间间隔或者记录时间不存在,则重置记录时间、计数归1\n\t--如果访问的时间间隔小于约定的时间间隔，则计数正常+1，且如果计数大于约定阈值，则设置封禁标识为1,即将此ID拉黑\n\t--同时设置封禁的数据过期时间\n\tif start_time == ngx.null or (os.time() - start_time) \u003e= uid_check_timespan_s then\n\t\tres , err = rds:set(key_prefix_stime..rds_key , os.time())\n\t\trds:expire(key_prefix_stime..rds_key,ttl_timespan_s)\n\t\tres , err = rds:set(key_prefix_count..rds_key , 1)\n\t\trds:expire(key_prefix_count..rds_key,ttl_timespan_s)\n\t\trds:close()\n\t\treturn 1\n\telse\n\t\tif count == ngx.null then\n\t\t\trds:set(key_prefix_count..rds_key , 1)\n\t\t\trds:expire(key_prefix_count..rds_key,ttl_timespan_s)\n\t\telse\n\t\t\tcount = count + 1\n\t\t\tres , err = rds:incr(key_prefix_count..rds_key)\n\t\t\tngx.log(ngx.ERR,\"inc.....\",\"\"..uid)\n\t\t\tif count \u003e= access_threshold_count then\n\t\t\t\tres , err = rds:set(key_prefix_isdeny..rds_key,1)\n\t\t\t\tres , err = rds:expire(key_prefix_isdeny..rds_key,ttl_timespan_s)\n\t\t\t\tngx.log(ngx.ERR,\"uidfreq_deny \",\"ip: \"..rds_key..\" : deny timespan : \"..ttl_timespan_s)\n\t\t\t\trds:close()\n\t\t\t\treturn error_status\n\t\t\tend\n\t\tend\n\t\trds:close()\n\tend\nend\n\nreturn _M\n```\n\u003e ***access_limit.lua*** 入口\n\n```\npackage.path  = '/usr/local/openresty-1.13.6.1/mylua/?.lua;;' .. package.path\n--白名单IP ,放入redis也可以，如果图方便直接写也可以，毕竟白单不会太多\nlocal ipTable = \"11.25.17.18\"\n\nlocal utils = require \"lua_utils\"\nlocal clientip = utils.get_clientip()\n\nlocal uidLimit = require \"uid_limit\"\nlocal uidL = uidLimit:new()\n\nlocal ipLimit = require \"ip_limit\"\nlocal iplimit = ipLimit:new()\n\nif not string.find(ipTable,clientip) then --白单过滤\n\t--单IP单uid固定接口 组合限制\n\t--当某一uid被限流时尽量不影响该IP内的其他用户\n\tlocal ok1 = uidL.check_uid_freq()\n\tif ok1 and ok1 ~= 1 then\n\t\tngx.exit(ok1)\n\tend\n\t--单IP全局限流检查\n\tlocal ok = iplimit.check_ip_freq()\n\tif ok and ok ~= 1 then\n\t\tngx.exit(ok)\n\tend\nend\n```\nEnjoy!!!\n","funding_links":[],"categories":["Lua (24)","Lua"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjerryting%2Fngxlua","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjerryting%2Fngxlua","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjerryting%2Fngxlua/lists"}