https://github.com/veeqo/activejob-uniqueness
Unique jobs for ActiveJob. Ensure the uniqueness of jobs in the queue.
https://github.com/veeqo/activejob-uniqueness
activejob distributed-locks lock rails redis uniqueness
Last synced: 14 days ago
JSON representation
Unique jobs for ActiveJob. Ensure the uniqueness of jobs in the queue.
- Host: GitHub
- URL: https://github.com/veeqo/activejob-uniqueness
- Owner: veeqo
- License: mit
- Created: 2020-07-02T19:18:42.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2024-12-07T14:12:32.000Z (6 months ago)
- Last Synced: 2025-04-13T12:41:39.372Z (about 1 month ago)
- Topics: activejob, distributed-locks, lock, rails, redis, uniqueness
- Language: Ruby
- Homepage: https://devs.veeqo.com/job-uniqueness-for-activejob/
- Size: 139 KB
- Stars: 286
- Watchers: 21
- Forks: 28
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Job uniqueness for ActiveJob
[](https://github.com/veeqo/activejob-uniqueness/actions/workflows/main.yml) [](https://badge.fury.io/rb/activejob-uniqueness)The gem allows to protect job uniqueness with next strategies:
| Strategy | The job is locked | The job is unlocked |
|-|-|-|
| `until_executing` | when **pushed** to the queue | when **processing starts** |
| `until_executed` | when **pushed** to the queue | when the job is **processed successfully** |
| `until_expired` | when **pushed** to the queue | when the lock is **expired** |
| `until_and_while_executing` | when **pushed** to the queue | when **processing starts**
a runtime lock is acquired to **prevent simultaneous jobs**
*has extra options: `runtime_lock_ttl`, `on_runtime_conflict`* |
| `while_executing` | when **processing starts** | when the job is **processed**
with any result including an error |Inspired by [SidekiqUniqueJobs](https://github.com/mhenrixon/sidekiq-unique-jobs), uses [Redlock](https://github.com/leandromoreira/redlock-rb) under the hood.
## Installation
Add the `activejob-uniqueness` gem to your Gemfile.
```ruby
gem 'activejob-uniqueness'
```If you want jobs unlocking for Sidekiq Web UI, require the patch explicitly. [**Queues cleanup becomes slower!**](#sidekiq-api-support)
```ruby
gem 'activejob-uniqueness', require: 'active_job/uniqueness/sidekiq_patch'
```And run `bundle install` command.
## Configuration
ActiveJob::Uniqueness is ready to work without any configuration. It will use `REDIS_URL` to connect to Redis instance.
To override the defaults, create an initializer `config/initializers/active_job_uniqueness.rb` using the following command:```sh
rails generate active_job:uniqueness:install
```This gem relies on `redlock` for it's Redis connection, that means **it will not inherit global configuration of `Sidekiq`**. To configure the connection, you can use `config.redlock_servers`, for example to disable SSL verification for Redis/Key-Value cloud providers:
```ruby
ActiveJob::Uniqueness.configure do |config|
config.redlock_servers = [
RedisClient.new(
url: ENV['REDIS_URL'],
ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
)
]
end
```## Usage
### Make the job to be unique
```ruby
class MyJob < ActiveJob::Base
# new jobs with the same args will raise error until existing one is executed
unique :until_executeddef perform(args)
# work
end
end
```### Tune uniqueness settings per job
```ruby
class MyJob < ActiveJob::Base
# new jobs with the same args will be logged within 3 hours or until existing one is being executing
unique :until_executing, lock_ttl: 3.hours, on_conflict: :logdef perform(args)
# work
end
end
```You can set defaults globally with [the configuration](#configuration)
### Control lock conflicts
```ruby
class MyJob < ActiveJob::Base
# Proc gets the job instance including its arguments
unique :until_executing, on_conflict: ->(job) { job.logger.info "Oops: #{job.arguments}" }def perform(args)
# work
end
end
```### Control redis connection errors
```ruby
class MyJob < ActiveJob::Base
# Proc gets the job instance including its arguments, and as keyword arguments the resource(lock key) `resource` and the original error `error`
unique :until_executing, on_redis_connection_error: ->(job, resource: _, error: _) { job.logger.info "Oops: #{job.arguments}" }def perform(args)
# work
end
end
```### Control lock key arguments
```ruby
class MyJob < ActiveJob::Base
unique :until_executeddef perform(foo, bar, baz)
# work
enddef lock_key_arguments
arguments.first(2) # baz is ignored
end
end
```### Control the lock key
```ruby
class MyJob < ActiveJob::Base
unique :until_executeddef perform(foo, bar, baz)
# work
enddef lock_key
'qux' # completely custom lock key
enddef runtime_lock_key
'quux' # completely custom runtime lock key for :until_and_while_executing
end
end
```### Unlock jobs manually
The selected strategy automatically unlocks jobs, but in some cases (e.g. the queue is purged) it is handy to unlock jobs manually.
```ruby
# Remove the lock for particular arguments:
MyJob.unlock!(foo: 'bar')
# or
ActiveJob::Uniqueness.unlock!(job_class_name: 'MyJob', arguments: [{foo: 'bar'}])# Remove all locks of MyJob
MyJob.unlock!
# or
ActiveJob::Uniqueness.unlock!(job_class_name: 'MyJob')# Remove all locks
ActiveJob::Uniqueness.unlock!
```## Test mode
Most probably you don't want jobs to be locked in tests. Add this line to your test suite (`rails_helper.rb`):
```ruby
ActiveJob::Uniqueness.test_mode!
```## Logging
ActiveJob::Uniqueness instruments `ActiveSupport::Notifications` with next events:
* `lock.active_job_uniqueness`
* `runtime_lock.active_job_uniqueness`
* `unlock.active_job_uniqueness`
* `runtime_unlock.active_job_uniqueness`
* `conflict.active_job_uniqueness`
* `runtime_conflict.active_job_uniqueness`And then writes to `ActiveJob::Base.logger`.
**ActiveJob prior to version `6.1` will always log `Enqueued MyJob (Job ID) ...` even if the callback chain is halted. [Details](https://github.com/rails/rails/pull/37830)**
## Testing
Run redis server (in separate console):
```
docker run --rm -p 6379:6379 redis
```Run tests with:
```sh
bundle
rake
```## Sidekiq API support
ActiveJob::Uniqueness supports Sidekiq API to unset job locks on queues cleanup (e.g. via Sidekiq Web UI). Starting Sidekiq 5.1 job death also triggers locks cleanup.
Take into account that **[big queues cleanup becomes much slower](https://github.com/veeqo/activejob-uniqueness/issues/16)** because each job is being unlocked individually. In order to activate Sidekiq API patch require it explicitly in your Gemfile:```ruby
gem 'activejob-uniqueness', require: 'active_job/uniqueness/sidekiq_patch'
```## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/veeqo/activejob-uniqueness.
## License
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
## About [Veeqo](https://www.veeqo.com)
At Veeqo, our team of Engineers is on a mission to create a world-class Inventory and Shipping platform, built to the highest standards in best coding practices. We are a growing team, looking for other passionate developers to [join us](https://veeqo-ltd.breezy.hr/) on our journey. If you're looking for a career working for one of the most exciting tech companies in ecommerce, we want to hear from you.
[Veeqo developers blog](https://devs.veeqo.com)