{"id":13636552,"url":"https://github.com/Kong/lua-resty-worker-events","last_synced_at":"2025-04-19T08:32:35.325Z","repository":{"id":3624554,"uuid":"50317581","full_name":"Kong/lua-resty-worker-events","owner":"Kong","description":"Cross Worker Events for Nginx in Pure Lua","archived":false,"fork":false,"pushed_at":"2022-06-02T22:24:10.000Z","size":131,"stargazers_count":193,"open_issues_count":3,"forks_count":48,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-04-13T05:30:42.045Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kong.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-25T01:24:42.000Z","updated_at":"2024-12-17T10:03:30.000Z","dependencies_parsed_at":"2022-09-03T09:11:01.848Z","dependency_job_id":null,"html_url":"https://github.com/Kong/lua-resty-worker-events","commit_stats":null,"previous_names":["mashape/lua-resty-worker-events"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kong%2Flua-resty-worker-events","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kong%2Flua-resty-worker-events/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kong%2Flua-resty-worker-events/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kong%2Flua-resty-worker-events/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kong","download_url":"https://codeload.github.com/Kong/lua-resty-worker-events/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249650249,"owners_count":21305984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T00:01:02.638Z","updated_at":"2025-04-19T08:32:35.042Z","avatar_url":"https://github.com/Kong.png","language":"Lua","funding_links":[],"categories":["Libraries","Lua"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/Kong/lua-resty-worker-events.svg?branch=master)](https://travis-ci.org/Kong/lua-resty-worker-events)\n\nlua-resty-worker-events\n=======================\n\nInter process events for Nginx worker processes\n\nTable of Contents\n=================\n\n* [Name](#name)\n* [Status](#status)\n* [Synopsis](#synopsis)\n* [Description](#description)\n* [Troubleshooting](#troubleshooting)\n* [Methods](#methods)\n    * [configure](#configure)\n    * [configured](#configured)\n    * [event_list](#event_list)\n    * [poll](#poll)\n    * [post](#post)\n    * [post_local](#post_local)\n    * [register](#register)\n    * [register_weak](#register_weak)\n    * [unregister](#unregister)\n* [Installation](#installation)\n* [Bugs and Patches](#bugs-and-patches)\n* [Author](#author)\n* [Copyright and License](#copyright-and-license)\n* [History](#history)\n* [See Also](#see-also)\n\nStatus\n======\n\nThis library is production ready.\n\nSynopsis\n========\n\n```nginx\nhttp {\n    lua_package_path \"/path/to/lua-resty-worker-events/lib/?.lua;;\";\n\n    # the size depends on the number of event to handle:\n    lua_shared_dict process_events 1m;\n\n    init_worker_by_lua_block {\n        local ev = require \"resty.worker.events\"\n\n        local handler = function(data, event, source, pid)\n            print(\"received event; source=\",source,\n                  \", event=\",event,\n                  \", data=\", tostring(data),\n                  \", from process \",pid)\n        end\n\n        ev.register(handler)\n\n        local ok, err = ev.configure {\n            shm = \"process_events\", -- defined by \"lua_shared_dict\"\n            timeout = 2,            -- life time of unique event data in shm\n            interval = 1,           -- poll interval (seconds)\n\n            wait_interval = 0.010,  -- wait before retry fetching event data\n            wait_max = 0.5,         -- max wait time before discarding event\n            shm_retries = 999,      -- retries for shm fragmentation (no memory)\n        }\n        if not ok then\n            ngx.log(ngx.ERR, \"failed to start event system: \", err)\n            return\n        end\n    }\n\n    server {\n        ...\n\n        # example for polling:\n        location = /some/path {\n\n            default_type text/plain;\n            content_by_lua_block {\n                -- manually call `poll` to stay up to date, can be used instead,\n                -- or together with the timer interval. Polling is efficient,\n                -- so if staying up-to-date is important, this is preferred.\n                require(\"resty.worker.events\").poll()\n\n                -- do regular stuff here\n\n            }\n        }\n    }\n}\n```\n\nDescription\n===========\n\n[Back to TOC](#table-of-contents)\n\nThis module provides a way to send events to the other worker processes in an Nginx\nserver. Communication is through a shared memory zone where event data will be stored.\n\nThe order of events in all workers is __guaranteed__ to be the same.\n\nThe worker process will setup a timer to check for events in the background. The\nmodule follows a singleton pattern and hence runs once per worker. If staying\nup-to-date is important though, the interval can be set to a lesser frequency and a\ncall to [poll](#poll) upon each request received makes sure everything is handled\nas soon as possible.\n\nThe design allows for 3 usecases;\n\n1. broadcast an event to all workers processes, see [post](#post). In this case\nthe order of the events is guaranteed to be the same in all worker processes. Example;\na healthcheck running in one worker, but informing all workers of a failed\nupstream node.\n2. broadcast an event to the local worker only, see [post_local](#post_local).\n3. coalesce external events to a single action. Example; all workers watch\nexternal events indicating an in-memory cache needs to be refreshed. When\nreceiving it they all post it with a unique event hash (all workers generate the\nsame hash), see `unique` parameter of [post](#post). Now only 1 worker will\nreceive the event _only once_, so only one worker will hit the upstream\ndatabase to refresh the in-memory data.\n\nThis module itself will fire two events with `source=\"resty-worker-events\"`;\n * `event=\"started\"` when the module is first configured (note: the event handler must be\n   [registered](#register) before calling [configure](#configure) to be able to catch the event)\n * `event=\"stopping\"` when the worker process exits (based on a timer `premature` setting)\n\nSee [event_list](#event_list) for using events without hardcoded magic\nvalues/strings.\n\n\nTroubleshooting\n================\n\nTo properly size the shm, it is important to understand how it is being used.\nEvent data is stored in the shm to pass it to the other workers. As such there\nare 2 types of entries in the shm:\n\n1. events that are to be executed by only a single worker (see the\n   `unique` parameter of the `post` method). These entries get a `ttl` in the\n   shm and will hence expire.\n2. all other events (except local events which do not use the SHM). In these\n   cases there is no `ttl` set.\n\nThe result of the above is that the SHM will always be full! so that is not a\nmetric to investigate at.\n\nHow to prevent problems:\n\n* the SHM size must at least be a multiple of the maximum payload expected. It\n  must be able to cater for all the events that might be send within one\n  `interval` (see `configure`).\n* `no memory` errors *cannot* be resolved by making the SHM bigger. The only way\n  to resolve those is by increasing the `shm_retries` option passed to\n  `configure` (which already has a high default).\n  This is because the error is due to fragmentation and not a lack of memory.\n* the `waiting for event data timed out` error happens if event data gets\n  evicted before all the workers got to deal with it. This can happen if\n  there is a burst of (large-payload) events. To resolve these:\n\n    * try to avoid big event payloads\n    * use a smaller `interval`, so workers check for (and deal with) events\n      more frequently (see `interval` option as passed to `configure`)\n    * increase the SHM size, such that it can hold all the event data that\n      might be send within 1 interval.\n\n\nMethods\n=======\n\n[Back to TOC](#table-of-contents)\n\nconfigure\n---------\n`syntax: success, err = events.configure(opts)`\n\nWill initialize the event listener. This should typically be called from the\n`init_by_lua` handler, because it will make sure all workers start with the\nfirst event. In case of a reload of the system (starting new and stopping old\nworkers) past events will not be replayed. And because the order in which\nworkers reload cannot be guaranteed, also the event start cannot be guaranteed.\nSo if some sort of state is derived from the events you have to manage that\nstate separately.\n\nThe `opts` parameter is a Lua table with named options:\n\n* `shm`: (required) name of the shared memory to use. Event data will not expire, so\n  the module relies on the shm lru mechanism to evict old events from the shm. As such\n  the shm should probably not be used for other purposes.\n* `shm_retries`: (optional) number of retries when the shm returns \"no memory\" on posting\n  an event, default 999. Each time there is an insertion attempt and no memory is available\n  (either no space is available or the memory is available but fragmented), \"up to tens\"\n  of old entries are evicted. After that, if there's still no memory available, the\n  \"no memory\" error is returned. Retrying the insertion triggers the eviction phase\n  several times, increasing the memory available as well as the probability of finding a\n  large enough contiguous memory block available for the new event data.\n* `interval`: (optional) interval to poll for events (in seconds), default 1. Set to 0 to\n  disable polling.\n* `wait_interval`: (optional) interval between two tries when a new eventid is found, but the\n  data is not available yet (due to asynchronous behaviour of the worker processes)\n* `wait_max`: (optional) max time to wait for data when event id is found, before discarding\n  the event. This is a fail-safe setting in case something went wrong.\n* `timeout`: (optional) timeout of unique event data stored in shm (in seconds), default 2.\n  See the `unique` parameter of the [post](#post) method.\n\nThe return value will be `true`, or `nil` and an error message.\n\nThis method can be called repeatedly to update the settings, except for the `shm` value which\ncannot be changed after the initial configuration.\n\nNOTE: the `wait_interval` is executed using the `ngx.sleep` function. In contexts where this\nfunction is not available (eg. `init_worker`) it will execute a busy-wait to execute the delay.\n\n[Back to TOC](#table-of-contents)\n\nconfigured\n----------\n`syntax: is_already_configured = events.configured()`\n\nThe events module runs as a singelton per workerprocess. The `configured`\nfunction allows to check whether it is already up and running.\nA check before starting any dependencies is recommended;\n```lua\nlocal events = require \"resty.worker.events\"\n\nlocal initialization_of_my_module = function()\n    assert(events.configured(), \"Please configure the 'lua-resty-worker-events' \"..\n           \"module before using my_module\")\n\n    -- do initialization here\nend\n```\n\n[Back to TOC](#table-of-contents)\n\nevent_list\n----------\n`syntax: _M.events = events.event_list(sourcename, event1, event2, ...)`\n\nUtility function to generate event lists and prevent typos in\nmagic strings. Accessing a non-existing event on the returned table will result\nin an 'unknown event error'.\nThe first parameter `sourcename` is a unique name that identifies the event\nsource, which will be available as field `_source`. All following parameters\nare the named events generated by the event source.\n\nExample usage;\n```lua\nlocal ev = require \"resty.worker.events\"\n\n-- Event source example\n\nlocal events = ev.event_list(\n        \"my-module-event-source\", -- available as _M.events._source\n        \"started\",                -- available as _M.events.started\n        \"event2\"                  -- available as _M.events.event2\n    )\n\nlocal raise_event = function(event, data)\n    return ev.post(events._source, event, data)\nend\n\n-- Post my own 'started' event\nraise_event(events.started, nil) -- nil for clarity, no eventdata is passed\n\n-- define my module table\nlocal _M = {\n  events = events   -- export events table\n\n  -- implementation goes here\n}\nreturn _M\n\n\n\n-- Event client example;\nlocal mymod = require(\"some_module\")  -- module with an `events` table\n\n-- define a callback and use source modules events table\nlocal my_callback = function(data, event, source, pid)\n    if event == mymod.events.started then  -- 'started' is the event name\n\n        -- started event from the resty-worker-events module\n\n    elseif event == mymod.events.stoppping then  -- 'stopping' is the event name\n\n        -- the above will throw an error because of the typo in `stoppping`\n\n    end\nend\n\nev.register(my_callback, mymod.events._source)\n\n```\n\n[Back to TOC](#table-of-contents)\n\npoll\n----\n`syntax: success, err = events.poll()`\n\nWill poll for new events and handle them all (call the registered callbacks). The implementation is\nefficient, it will only check a single shared memory value and return immediately if no new events\nare available.\n\nThe return value will be `\"done\"` when it handled all events, `\"recursive\"` if it was\nalready in a polling-loop, or `nil + error` if something went wrong.\nThe `\"recursive\"` result simply means that an event-handler called `poll` again.\n\n[Back to TOC](#table-of-contents)\n\npost\n----\n`syntax: success, err = events.post(source, event, data, unique)`\n\nWill post a new event. `source` and `event` are both strings. `data` can be anything (including `nil`)\nas long as it is (de)serializable by the cjson module.\n\nIf the `unique` parameter is provided then only one worker will execute the event,\nthe other workers will ignore it. Also any follow up events with the same `unique`\nvalue will be ignored (for the `timeout` period specified to [configure](#configure)).\nThe process executing the event will not necessarily be the process posting the event.\n\nThe return value will be `true` when the event was successfully posted or\n`nil + error` in case of failure.\n\n*Note*: the worker process sending the event, will also receive the event! So if\nthe eventsource will also act upon the event, it should not do so from the event\nposting code, but only when receiving it.\n\n[Back to TOC](#table-of-contents)\n\npost_local\n----------\n`syntax: success, err = events.post_local(source, event, data)`\n\nThe same as [post](#post) except that the event will be local to the worker process,\nit will not be broadcasted to other workers. With this method, the `data` element\nwill not be jsonified.\n\nThe return value will be `true` when the event was successfully posted or\n`nil + error` in case of failure.\n\n[Back to TOC](#table-of-contents)\n\nregister\n--------\n`syntax: events.register(callback, source, event1, event2, ...)`\n\nWill register a callback function to receive events. If `source` and `event` are omitted, then the\ncallback will be executed on _every_ event, if `source` is provided, then only events with a\nmatching source will be passed. If (one or more) event name is given, then only when\nboth `source` and `event` match the callback is invoked.\n\nThe callback should have the following signature;\n\n`syntax: callback = function(data, event, source, pid)`\n\nThe parameters will be the same as the ones provided to [post](#post), except for the extra value\n`pid` which will be the pid of the originating worker process, or `nil` if it was a local event\nonly. Any return value from `callback` will be discarded.\n*Note:* `data` may be a reference type of data (eg. a Lua `table`  type). The same value is passed\nto all callbacks, _so do not change the value in your handler, unless you know what you are doing!_\n\nThe return value of `register` will be `true`, or it will throw an error if `callback` is not a\nfunction value.\n\n*WARNING*: event handlers must return quickly. If a handler takes more time than\nthe configured `timeout` value, events will be dropped!\n\n*Note*: to receive the process own `started` event, the handler must be registered before\ncalling [configure](#configure)\n\n[Back to TOC](#table-of-contents)\n\nregister_weak\n-------------\n`syntax: events.register_weak(callback, source, event1, event2, ...)`\n\nThis function is identical to `register`, with the exception that the module\nwill only hold _weak references_ to the `callback` function.\n\n[Back to TOC](#table-of-contents)\n\nunregister\n----------\n`syntax: events.unregister(callback, source, event1, event2, ...)`\n\nWill unregister the callback function and prevent it from receiving further events. The parameters\nwork exactly the same as with [register](#register).\n\nThe return value will be `true` if it was removed, `false` if it was not in the handlers list, or\nit will throw an error if `callback` is not a function value.\n\n[Back to TOC](#table-of-contents)\n\n\nInstallation\n============\n\nNothing special is required, install like any other pure Lua module. Just make\nsure its location is in the module search path.\n\n[Back to TOC](#table-of-contents)\n\n\nBugs and Patches\n================\n\nPlease report bugs or submit patches by creating a ticket on the [GitHub Issue Tracker](http://github.com/Kong/lua-resty-worker-events/issues),\n\n[Back to TOC](#table-of-contents)\n\nAuthor\n======\n\nThijs Schreijer \u003cthijs@konghq.com\u003e, Kong Inc.\n\n[Back to TOC](#table-of-contents)\n\nCopyright and License\n=====================\n\nThis module is licensed under the [Apache 2.0 license](https://opensource.org/licenses/Apache-2.0).\n\nCopyright (C) 2016-2020, by Thijs Schreijer, Kong Inc.\n\nAll rights reserved.\n\n[Back to TOC](#table-of-contents)\n\n\nHistory\n=======\n\n### Releasing new versions\n\n- make sure changelog below is up-to-date\n- update version number in the code\n- create a new rockspec in `./rockspecs`\n- commit with message `release x.x.x`\n- tag the commit as `x.x.x`\n- push commit and tags\n- upload to luarocks\n\n### unreleased\n\n- chore: remove redundant type checking of unique_timeout.\n- chore: add stacktrace to `post` errors for better debug information.\n\n### 2.0.1, 28-June-2021\n\n- fix: possible deadlock in the `init` phase\n\n### 2.0.0, 16-September-2020\n\n- BREAKING: the `post` function does not call `poll` anymore, making all events\n  asynchronous. When an immediate treatment to an event is needed an explicit\n  call to `poll` must be done.\n- BREAKING: the `post_local` function does not immediately execute the\n  event anymore, making all local events asynchronous. When an immediate\n  treatment to an event is needed an explicit call to `poll` must be done.\n- fix: prevent spinning at 100% CPU when during a reload the event-shm is\n  cleared\n- fix: improved logging in case of failure to write to shm (add payload size\n  for troubleshooting purposes)\n- fix: do not log the payload anymore, since it might expose sensitive data\n  through the logs\n- change: updated `shm_retries` default to 999\n- change: changed timer loop to a sleep-loop (performance)\n- fix: when re-configuring make sure callbacks table is initialized\n\n### 1.1.0, 23-Dec-2020 (maintenance release)\n\n- feature: the polling loop now runs forever, sleeping for 0.5 seconds between\n  runs, avoiding to create new timers on every step.\n\n### 1.0.0, 18-July-2019\n\n- BREAKING: the return values from `poll` (and hence also `post` and `post_local`)\n  changed to be more lua-ish, to be truthy when all is well.\n- feature: new option `shm_retries` to fix \"no memory\" errors caused by memory\n  fragmentation in the shm when posting events.\n- fix: fixed two typos in variable names (edge cases)\n\n### 0.3.3, 8-May-2018\n\n- fix: timeouts in init phases, by removing timeout setting, see issue #9\n\n### 0.3.2, 11-Apr-2018\n\n- change: add a stacktrace to handler errors\n- fix: failing error handler if value was non-serializable, see issue #5\n- fix: fix a test for the weak handlers\n\n\n[Back to TOC](#table-of-contents)\n\n\nSee Also\n========\n* OpenResty: http://openresty.org\n\n[Back to TOC](#table-of-contents)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKong%2Flua-resty-worker-events","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKong%2Flua-resty-worker-events","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKong%2Flua-resty-worker-events/lists"}