Sidekiq Unique Jobs

https://github.com/zoolutions/sidekiq-unique-jobs
Admin
Sidekiq Unique Jobs is a Sidekiq middleware gem that uses Redis locks to prevent duplicate jobs from...
Tokens:15,025
Snippets:162
Trust Score:6.2
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Example comparison output

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Displays the relative performance improvement between branches.

```text
lock with failure simulation:
   main:                        1.234k i/s
   improved-exception-handling: 2.345k i/s  <- ~90% improvement!
```

--------------------------------

### Example IPS output

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Shows sample output for iterations per second (IPS) measurements.

```text
lock with failure simulation    1.234k i/s
lock and unlock - normal flow   5.678k i/s
```

--------------------------------

### Example memory usage output

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Displays memory allocation and retention metrics for a set number of cycles.

```text
100 lock/unlock cycles
  allocated:    45.2 MB
  retained:     123 KB
```

--------------------------------

### Run tests with bundle exec rspec and appraisal

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/CLAUDE.md

Use these commands to run the test suite. The first command runs all tests, the second runs a specific file, and the third runs a specific line number. The appraisal commands run tests against different Sidekiq versions, with sidekiq-6.0 as an example.

```bash
# Run all tests
bundle exec rspec

# Run specific test file
bundle exec rspec spec/path/to/spec.rb

# Run specific test by line number
bundle exec rspec spec/path/to/spec.rb:42

# Run tests with specific appraisals (different Sidekiq versions)
bundle exec appraisal rspec
bundle exec appraisal sidekiq-6.0 rspec
```

--------------------------------

### Install sidekiq-unique-jobs gem

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Add the gem to your Gemfile. Requires Ruby >= 3.2, Sidekiq >= 8.0, and Redis >= 6.2.

```ruby
gem "sidekiq-unique-jobs", "~> 9.0"
```

--------------------------------

### Configure Sidekiq middleware for sidekiq-unique-jobs

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Configure the client and server middleware in your Sidekiq initializer. This setup is required for the gem to intercept duplicate jobs.

```ruby
# config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
end

Sidekiq.configure_server do |config|
  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end

  config.server_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  SidekiqUniqueJobs::Server.configure(config)
end
```

--------------------------------

### Release commands: rake release[1.2.3], rake release[1.2.3.alpha1], rake release[pre]

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/RELEASING.md

Use these commands to trigger a release. The rake task handles version bump, commit, and push locally, then runs `gh release create` to start the CI pipeline.

```bash
# Stable release
rake release[1.2.3]

# Pre-release
rake release[1.2.3.alpha1]

# Release current version as pre-release (no version bump)
rake release[pre]
```

--------------------------------

### Example DocsUI::Page subclass with Markdown and Code helper

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/docs/AGENTS.md

Shows the structure of a docs page: a class inheriting from DocsUI::Page, with title, eyebrow, lead, and content. Uses a single-quoted heredoc for Markdown and the DocsUI::Code helper for a code block.

```ruby
class Views::Docs::Pages::Guide < DocsUI::Page
  title "Guide"
  eyebrow "Getting started"

  def lead = "One sentence under the page title."

  def content
    DocsUI::Section("First steps", description: "What this covers.") do
      md <<~'MD'
        Prose as **Markdown** — lists, `inline code`, links, GFM tables, and
        fenced ```ruby``` blocks all render styled. Use Markdown `###` only for
        sub-headings *inside* a Section.
      MD

      DocsUI::Code(<<~RUBY, filename: "config/routes.rb")
        Rails.application.routes.draw { mount DocsKit::Engine, at: "/docs" }
      RUBY
    end
  end
end
```

--------------------------------

### Set lock type and conflict strategy with sidekiq_options

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Configure lock type and conflict strategy for a Sidekiq job. The first example uses :until_executed with :reject to prevent duplicates until the job completes. The second shows separate client and server conflict strategies with :until_and_while_executing.

```ruby
sidekiq_options lock: :until_executed,
               on_conflict: :reject

# Or different strategies for client (enqueue) and server (execute):
sidekiq_options lock: :until_and_while_executing,
               on_conflict: { client: :log, server: :reschedule }
```

--------------------------------

### Run comparison tool

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Executes the comparison tool to verify performance improvements between the main branch and a feature branch.

```bash
# Run the comparison tool
bin/compare_performance main improved-exception-handling
```

--------------------------------

### Run comprehensive benchmarks with bin/benchmark

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Runs the full suite of benchmarks covering lock acquisition, contention, digest generation, conflict resolution, concurrent locking, and memory usage.

```bash
bin/benchmark
```

--------------------------------

### Add New Benchmark to bin/benchmark_improvements

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Template for adding a new benchmark to bin/benchmark_improvements. Uses Benchmark.ips with a 5-second time and 2-second warmup, defines two test cases, and optionally calls x.compare! to show relative performance.

```ruby
puts "\n" + "=" * 80
puts "BENCHMARK N: Your Benchmark Name"
puts "Improvement: Description of what was improved"
puts "=" * 80

Benchmark.ips do |x|
  x.config(time: 5, warmup: 2)

  x.report("test case 1") do
    # Your benchmark code here
  end

  x.report("test case 2") do
    # Another test case
  end

  x.compare! # Optional: shows relative performance
end
```

--------------------------------

### Configure SidekiqUniqueJobs globally with config block

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Configure global SidekiqUniqueJobs settings in an initializer. Set lock TTL, timeout, prefix, conflict strategy, lock info, enabled flag, reaper options, and digest algorithm. Defaults are shown in comments; adjust them to change uniqueness behavior globally.

```ruby
SidekiqUniqueJobs.configure do |config|
  config.lock_ttl          = nil     # Lock expiration in seconds (nil = no expiry)
  config.lock_timeout      = 0       # How long to wait for a lock (0 = don't wait)
  config.lock_prefix       = "uniquejobs"
  config.on_conflict       = nil     # Global default conflict strategy
  config.lock_info         = false   # Store lock metadata (useful for debugging)
  config.enabled           = true    # Disable uniqueness globally
  config.reaper            = :ruby   # Orphaned lock cleanup (:ruby, :lua, true, :none, false)
  config.reaper_count      = 1000    # Max locks to reap per cycle
  config.reaper_interval   = 600     # Seconds between reaper runs
  config.reaper_timeout    = 10      # Max seconds per reaper run
  config.digest_algorithm  = :legacy # :legacy (MD5) or :modern (SHA3-256)
end
```

--------------------------------

### Run manual benchmarks

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Manually benchmarks the main branch and the current branch, saving results to text files for comparison using the diff command.

```bash
# Save current branch name
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)

# Benchmark main branch
git checkout main
bin/benchmark_improvements > results_main.txt

# Benchmark your branch
git checkout $CURRENT_BRANCH
bin/benchmark_improvements > results_current.txt

# Compare results
diff -y results_main.txt results_current.txt | less
```

--------------------------------

### Run orphan reaper benchmarks with bin/bench

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Runs focused benchmarks for the orphan reaper, comparing Ruby and Lua implementations.

```bash
bin/bench
```

--------------------------------

### Run all checks and release gem with bundle exec rake

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/CLAUDE.md

Use these commands for build and release. The first runs all checks (style, tests, documentation). The second releases a new gem version and is only for maintainers.

```bash
# Run all checks (style, tests, documentation)
bundle exec rake

# Release a new gem version (only for maintainers)
bundle exec rake release
```

--------------------------------

### Run code quality checks with rake rubocop, reek, style, and yard

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/CLAUDE.md

Use these commands for code quality checks. The first runs RuboCop, the second runs Reek, the third runs all style checks, and the fourth generates YARD documentation.

```bash
# Run rubocop linter
bundle exec rake rubocop

# Run reek (code smell detector)
bundle exec rake reek

# Run all style checks
bundle exec rake style

# Generate documentation
bundle exec rake yard
```

--------------------------------

### Scaffold a docs page with rails g docs_kit:page

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/docs/AGENTS.md

Run this command to generate a new docs page. It writes the page file and injects the required registry line into the Doc registry. Overrides include --slug, --view, --eyebrow, and --registry.

```bash
rails g docs_kit:page "Getting Started" --group=Guide
```

--------------------------------

### Run improvement-focused benchmarks with bin/benchmark_improvements

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Runs targeted benchmarks for specific performance improvements, including non-blocking lock acquisition, orphan reaper Lua optimization, lock TTL calculation, scripts class operations, thread-safe concurrent operations, callback exception handling, and blocking Redis operations. Displays detailed results with iterations per second (IPS).

```bash
bin/benchmark_improvements
```

--------------------------------

### Verify checksums: gh release download and sha256sum/sha512sum

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/RELEASING.md

Download the release assets and verify the gem's SHA-256 and SHA-512 checksums against the attached checksum files.

```bash
# Download release assets
gh release download vX.Y.Z --repo zoolutions/sidekiq-unique-jobs

# Verify checksums
sha256sum -c sidekiq-unique-jobs-X.Y.Z.gem.sha256
sha512sum -c sidekiq-unique-jobs-X.Y.Z.gem.sha512
```

--------------------------------

### CI/CD Benchmark Workflow

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

GitHub Actions workflow to run performance benchmarks on pull requests targeting the main branch. It sets up a Redis 7 service, Ruby 3.2 with bundler cache, runs the benchmark script, and compares performance against the main branch.

```yaml
# .github/workflows/benchmark.yml
name: Performance Benchmarks

on:
  pull_request:
    branches: [main]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7
        ports:
          - 6379:6379
    steps:
      - uses: actions/checkout@v3
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.2
          bundler-cache: true
      - name: Run benchmarks
        run: bin/benchmark_improvements
      - name: Compare with main
        run: |
          git fetch origin main
          bin/compare_performance origin/main HEAD
```

--------------------------------

### Verify docs changes with rspec, rubocop, and build:css

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/docs/AGENTS.md

Run these commands to ensure tests and lint pass, and to rebuild CSS if new classes were added. The build:css command is only needed when adding classes that the CSS scans.

```bash
bundle exec rspec && bundle exec rubocop   # tests + lint must pass
bun run build:css                          # if you added classes the CSS scans
```

--------------------------------

### Configure reflection API callbacks

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/UPGRADING.md

Define callbacks for various job lifecycle events using SidekiqUniqueJobs.reflect. Each callback receives a job_hash and optionally an exception. Use this to log, collect metrics, or handle failures. Not all callbacks are required; some are informational, others for metrics, and a few (failures, timeouts) may be of real interest.

```ruby
SidekiqUniqueJobs.reflect do |on|
  # Only raised when you have defined such a callback
  on.after_unlock_callback_failed do |job_hash|
    logger.warn(job_hash.merge(message: "Unlock callback failed"))
  end

  # This job is skipped because it is a duplicate
  on.duplicate do |job_hash|
    logger.warn(job_hash.merge(message: "Duplicate Job"))
  end

  # This means your code broke and we caught the execption to provide this reflection for you. It allows your to gather metrics and details about the error. Those details allow you to act on it as you see fit.
  on.execution_failed do |job_hash, exception = nil|
    message = "Execution failed"
    message = message + "(#{exception.message})" if exception
    logger.warn(job_hash.merge(message: message)
  end

  # Failed to acquire lock in a timely fashion
  on.lock_failed do |job_hash|
    logger.warn(job_hash.merge(message: "Lock failed"))
  end

  # In case you want to collect metrics
  on.locked do |job_hash|
    logger.debug(job_hash.merge(message: "Lock success"))
  end

  # When your conflict strategy is to reschedule and it failed
  on.reschedule_failed do |job_hash|
    logger.debug(job_hash.merge(message: "Reschedule failed"))
  end

  # When your conflict strategy is to reschedule and it was successful
  # Mostly for metrics I guess
  on.rescheduled do |job_hash|
    logger.debug(job_hash.merge(message: "Reschedule success"))
  end

  # You asked to wait for a lock to be achieved but we timed out waiting
  on.timeout do |job_hash|
    logger.warn(job_hash.merge(message: "Oh no! Timeout!! Timeout!!"))
  end

  # The current worker isn't part of this sidekiq servers workers
  on.unknown_sidekiq_worker do |job_hash|
    logger.warn(job_hash.merge(message: "WAT!? Why? What is this worker?"))
  end

  # Unlock failed! Not good
  on.unlock_failed do |job_hash|
    logger.warn(job_hash.merge(message: "Unlock failed"))
  end

  # Unlock was successful, perhaps mostly interesting for metrics
  on.unlocked do |job_hash|
    logger.warn(job_hash.merge(message: "Unlock success"))
  end
```

--------------------------------

### Test with Sidekiq::Testing.inline!

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Use Sidekiq::Testing.inline! to execute jobs immediately while still enforcing uniqueness.

```ruby
Sidekiq::Testing.inline! do
  # Jobs execute immediately, uniqueness still enforced
end
```

--------------------------------

### Verify Sigstore attestation: sigstore-cli verify-bundle

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/RELEASING.md

Verify the Sigstore attestation bundle for the gem. The certificate identity and OIDC issuer must match the release workflow and tag.

```bash
gem exec sigstore-cli verify-bundle \
  --bundle sidekiq-unique-jobs-X.Y.Z.gem.sigstore.json \
  --certificate-identity "https://github.com/zoolutions/sidekiq-unique-jobs/.github/workflows/release.yml@refs/tags/vX.Y.Z" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  sidekiq-unique-jobs-X.Y.Z.gem
```

--------------------------------

### Troubleshoot Redis Connection Issues

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Commands to troubleshoot Redis connection issues: verify Redis is running with redis-cli ping, and run benchmarks with a custom Redis URL via the REDIS_URL environment variable.

```bash
# Check Redis is running
redis-cli ping

# Use custom Redis URL
REDIS_URL=redis://localhost:6379/1 bin/benchmark_improvements
```

--------------------------------

### Local build verification: bundle exec rake build

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/RELEASING.md

Builds the gem locally with `--strict` mode and lists all packaged files, useful for verifying gem contents without publishing.

```bash
bundle exec rake build
```

--------------------------------

### Fix middleware order in config/initializers/sidekiq.rb

Source: https://github.com/zoolutions/sidekiq-unique-jobs/wiki/Using-sidekiq‐unique‐jobs-with-sidekiq‐batch

Configures the client middleware order in config/initializers/sidekiq.rb so that Sidekiq::Batch::Middleware::ClientMiddleware runs after SidekiqUniqueJobs::Client::Middleware. This ensures uniqueness filtering happens before the batch registers the job.

```ruby
Sidekiq.client_middleware do |chain|
  chain.insert_after SidekiqUniqueJobs::Client::Middleware,
                     Sidekiq::Batch::Middleware::ClientMiddleware
end
```

--------------------------------

### Compare performance between branches with bin/compare_performance

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Compares performance between two git branches. The first command compares the current branch with main, the second compares main with a specific branch, and the third uses custom base and comparison branches. The script checks out the base branch (default: main), runs full benchmarks, checks out the comparison branch (default: current branch), runs benchmarks again, returns to the original branch, and displays a summary comparison. Results are saved to tmp/benchmark_results/.

```bash
# Compare current branch with main
bin/compare_performance
```

```bash
# Compare specific branches
bin/compare_performance main improved-exception-handling
```

```bash
# Compare with custom base and comparison branches
bin/compare_performance v8.0.11 feature-branch
```

--------------------------------

### Benchmark Results Archive Structure

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/BENCHMARKING.md

Directory structure showing where benchmark results are saved. Files are gitignored but useful for historical comparison.

```text
tmp/benchmark_results/
├── results_main.txt
├── results_improved-exception-handling.txt
└── ...
```

--------------------------------

### Enable reliable fetch with SidekiqUniqueJobs::Fetch::Reliable

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Enable the optional reliable fetch strategy in v9. It provides atomic LMOVE, crash recovery, lock-aware acknowledgment, and lock-preserving requeue.

```ruby
Sidekiq.configure_server do |config|
  config[:fetch_class] = SidekiqUniqueJobs::Fetch::Reliable
end
```

--------------------------------

### Correct behavior after fixing middleware order: batch receives 1 JID

Source: https://github.com/zoolutions/sidekiq-unique-jobs/wiki/Using-sidekiq‐unique‐jobs-with-sidekiq‐batch

Shows the correct behavior after fixing the middleware order: the batch receives only the JID of the job that was actually enqueued, resulting in a batch count of 1. Use this to verify the fix.

```ruby
class MyWorker
  include Sidekiq::Worker
  sidekiq_options unique: :until_and_while_executing

  def perform
  end
end

batch = Sidekiq::Batch.new
jids = batch.jobs { 3.times.map { MyWorker.perform_async } }

jids.length
# => 1 (correct)
```

--------------------------------

### Enable uniqueness across queues

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Allow the same arguments on different queues to be considered duplicates by setting unique_across_queues to true.

```ruby
sidekiq_options lock: :until_executed,
               unique_across_queues: true  # Same args on different queues = duplicate
```

--------------------------------

### Set lock TTL with lock_ttl

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Set a TTL for the lock so it expires after a specified number of seconds. Here the lock expires after 1 hour.

```ruby
sidekiq_options lock: :until_expired,
               lock_ttl: 3600  # Lock expires after 1 hour
```

--------------------------------

### Reflect on lock lifecycle events with SidekiqUniqueJobs.reflect

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Use this block to observe lock lifecycle events without modifying behavior. It registers callbacks for locked, unlocked, lock_failed, and execution_failed events, logging each with the job class. This is the reflection system introduced in v9, replacing the removed changelog.

```ruby
SidekiqUniqueJobs.reflect do |on|
  on.locked { |job| logger.info("Locked: #{job['class']}") }
  on.unlocked { |job| logger.info("Unlocked: #{job['class']}") }
  on.lock_failed { |job| logger.warn("Lock failed: #{job['class']}") }
  on.execution_failed { |job| logger.error("Execution failed: #{job['class']}") }
end
```

--------------------------------

### Require sidekiq_unique_jobs/web for Web UI

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Add the Locks tab to the Sidekiq Web UI. This allows browsing, filtering, and deleting locks.

```ruby
require "sidekiq_unique_jobs/web"
```

--------------------------------

### Custom lock arguments with lock_args_method

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Customize uniqueness to use only the first argument by setting lock_args_method. The worker class, queue, and all arguments are used by default.

```ruby
class MyJob
  include Sidekiq::Job

  sidekiq_options lock: :until_executed,
                 lock_args_method: ->(args) { [args.first] }

  def perform(user_id, timestamp)
    # Only user_id determines uniqueness, timestamp is ignored
  end
end
```

--------------------------------

### Define a unique job with lock: :until_executed

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Define a job with the `lock: :until_executed` option to prevent duplicate executions for the same arguments. Duplicate jobs are silently dropped by default.

```ruby
class MyJob
  include Sidekiq::Job

  sidekiq_options lock: :until_executed

  def perform(user_id)
    # Only one job per user_id runs at a time
  end
end
```

--------------------------------

### Incorrect middleware order: batch receives 3 JIDs

Source: https://github.com/zoolutions/sidekiq-unique-jobs/wiki/Using-sidekiq‐unique‐jobs-with-sidekiq‐batch

Shows the incorrect behavior when middleware order is not adjusted: the batch collects three JIDs even though only one job is enqueued due to the uniqueness rule. Use this to reproduce the bug before fixing the middleware order.

```ruby
class MyWorker
  include Sidekiq::Worker
  sidekiq_options unique: :until_and_while_executing

  def perform
  end
end

batch = Sidekiq::Batch.new
jids = batch.jobs { 3.times.map { MyWorker.perform_async } }

jids.length
# => 3 (incorrect — only one job should be enqueued)
```

--------------------------------

### Disable uniqueness in tests

Source: https://github.com/zoolutions/sidekiq-unique-jobs/blob/main/README.md

Disable uniqueness enforcement in tests by setting config.enabled to false.

```ruby
SidekiqUniqueJobs.config.enabled = false
```
Tokens:
Raw