### Install Que Gem via Bundler Source: https://github.com/que-rb/que/blob/master/README.md Execute the bundle command after adding the gem to your Gemfile to install it. ```bash bundle ``` -------------------------------- ### Install Que Gem Manually Source: https://github.com/que-rb/que/blob/master/README.md Alternatively, install the Que gem directly using the gem install command. ```bash gem install que ``` -------------------------------- ### Start Postgres Database with Docker Compose Source: https://github.com/que-rb/que/blob/master/README.md Use Docker Compose to quickly start a Postgres database instance for testing purposes without needing a full Dockerized environment. ```bash docker compose up -d db ``` -------------------------------- ### Install Que and Create Schema Source: https://context7.com/que-rb/que/llms.txt Add the Que gem to your Gemfile and create the necessary database schema using an ActiveRecord migration. Explicitly pin the migration version for predictable upgrades. ```ruby # Gemfile gem 'que' ``` ```ruby # db/migrate/20240101000000_create_que_schema.rb class CreateQueSchema < ActiveRecord::Migration[6.0] def up # Always pin the version explicitly so future Que upgrades don't # silently change your schema during a deploy. Que.migrate!(version: 7) end def down Que.migrate!(version: 0) # removes Que entirely end end ``` -------------------------------- ### Configure Que with Plain PG Connection Source: https://github.com/que-rb/que/wiki/Advanced-Setup If not using an ORM, configure Que to use a direct PG connection. Ensure the 'pg' gem is installed. ```ruby require 'uri' require 'pg' uri = URI.parse(ENV['DATABASE_URL']) Que.connection = PG::Connection.open :host => uri.host, :user => uri.user, :password => uri.password, :port => uri.port || 5432, :dbname => uri.path[1..-1] ``` -------------------------------- ### Setup Git Pre-push Hook Source: https://github.com/que-rb/que/blob/master/README.md Run this command to set up the Git pre-push hook locally. This hook helps prevent breaking the build by verifying code before pushing. ```bash #!/bin/bash $(dirname $0)/../../auto/pre-push-hook > .git/hooks/pre-push chmod +x .git/hooks/pre-push ``` -------------------------------- ### Run Que Workers via Executable Source: https://github.com/que-rb/que/blob/master/docs/README.md Execute the `que` command to start a pool of workers. You can configure the number of workers using the `--worker-count` flag. ```shell # Run a pool of 6 workers: que # Or configure the number of workers: que --worker-count 10 ``` -------------------------------- ### Run Specs with Docker Source: https://github.com/que-rb/que/blob/master/README.md Execute the project's specifications within a Dockerized environment to avoid manual setup of Ruby, gems, and Postgres. ```bash ./auto/test ``` -------------------------------- ### Run Que Worker Process Source: https://context7.com/que-rb/que/llms.txt Start the Que worker process from the command line using `bundle exec que`. Customize worker count, priority distribution, queues to work, poll interval, log level, and specify an environment file if not using Rails. ```bash # Start with default settings (6 workers, priority distribution 10,30,50,any,any,any) bundle exec que ``` ```bash # Customise worker count bundle exec que --worker-count 10 ``` ```bash # Fine-grained priority distribution: 2 high-priority workers + 4 general workers bundle exec que --worker-priorities 10,30,any,any,any,any ``` ```bash # Work specific queues bundle exec que -q default -q credit_cards ``` ```bash # Set poll interval and log level bundle exec que --poll-interval 10 --log-level warn ``` ```bash # Pass a file to require when not using Rails bundle exec que ./config/environment.rb ``` -------------------------------- ### Run a Que Worker Source: https://github.com/que-rb/que/blob/master/README.md Start a separate worker process to process jobs. Use `bundle exec que`. Pass a file path to require if Que cannot automatically load your application (e.g., not using Rails). ```bash bundle exec que ``` -------------------------------- ### Set Que Logger Source: https://github.com/que-rb/que/wiki/Advanced-Setup Configure a logger for Que to use, for example, logging to standard output. ```ruby Que.logger = Logger.new(STDOUT) ``` -------------------------------- ### Define SQL Middleware Source: https://github.com/que-rb/que/blob/master/docs/README.md Demonstrates how to define custom middleware for wrapping SQL statements executed by Que. This can be used to instrument SQL queries, for example, by integrating with services like NewRelic. ```ruby Que.sql_middleware.push( -> (sql, params, &block) { Service.instrument(sql: sql, params: params) do block.call end nil # Still doesn't matter what's returned. } ) ``` -------------------------------- ### Define Job Middleware Source: https://github.com/que-rb/que/blob/master/docs/README.md Provides an example of how to define custom middleware for wrapping job executions. This middleware can be used for tasks like performance monitoring or custom logging. ```ruby Que.job_middleware.push( -> (job, &block) { # Do stuff with the job object - report on it, count time elapsed, etc. block.call nil # Doesn't matter what's returned. } ) ``` -------------------------------- ### Job with External Service Calls and Transactional Updates Source: https://github.com/que-rb/que/wiki/Writing-Reliable-Jobs This example shows a job that interacts with an external service before updating the database. It highlights the risk of duplicate charges if the job fails after the external call but before the transaction commits. ```ruby class ChargeCreditCard < Que::Job def run(user_id, credit_card_id) CreditCardProvider.charge(credit_card_id, :amount => "$10.00") ActiveRecord::Base.transaction do User.where(:id => user_id).update :charged_at => Time.now destroy end end end ``` -------------------------------- ### Get a Shell in Docker Environment Source: https://github.com/que-rb/que/blob/master/README.md Access a shell within the Dockerized development environment to interact with the project and its dependencies. ```bash ./auto/dev ``` -------------------------------- ### Conditionally Log Que Events Source: https://github.com/que-rb/que/blob/master/docs/README.md Use a proc with `Que.log_formatter=` to conditionally log events. If the proc returns `nil` or `false`, the event will not be logged. This example only logs `job_worked` and `job_unavailable` events. ```ruby Que.log_formatter = proc do |data| if [:job_worked, :job_unavailable].include?(data[:event]) JSON.dump(data) end end ``` -------------------------------- ### Get Job Statistics Source: https://github.com/que-rb/que/blob/master/docs/README.md Call `Que.job_stats` to retrieve aggregate data on job types currently in the queue, including counts, working jobs, errored jobs, highest error counts, and the oldest run time. ```ruby [ ... ] ``` -------------------------------- ### Using ConnectionPool with Que Source: https://github.com/que-rb/que/blob/master/docs/README.md Integrate Que with the ConnectionPool gem by initializing a ConnectionPool instance and assigning it to Que.connection. Ensure `gem 'connection_pool'` is in your Gemfile. The pool size directly impacts overhead. ```ruby require 'uri' require 'pg' require 'connection_pool' uri = URI.parse(ENV['DATABASE_URL']) Que.connection = ConnectionPool.new(size: 10) do PG::Connection.open( host: uri.host, user: uri.user, password: uri.password, port: uri.port || 5432, dbname: uri.path[1..-1] ) end ``` -------------------------------- ### Using Sequel with Que Source: https://github.com/que-rb/que/blob/master/docs/README.md Integrate Que with Sequel by connecting to the database and assigning the Sequel database instance to Que.connection. This allows transactional job management. ```ruby DB = Sequel.connect(ENV['DATABASE_URL']) Que.connection = DB ``` -------------------------------- ### Using Pond with Que Source: https://github.com/que-rb/que/blob/master/docs/README.md Integrate Que with the Pond gem for lazy connection establishment. Assign a Pond instance to Que.connection. Ensure `gem 'pond'` is in your Gemfile. Pond avoids the overhead of always-open connections. ```ruby require 'uri' require 'pg' require 'pond' uri = URI.parse(ENV['DATABASE_URL']) Que.connection = Pond.new(maximum_size: 10) do PG::Connection.open( host: uri.host, user: uri.user, password: uri.password, port: uri.port || 5432, dbname: uri.path[1..-1] ) end ``` -------------------------------- ### Run Specs Without Docker (Postgres) Source: https://github.com/que-rb/que/blob/master/README.md Run the project's specifications when Postgres is already running locally. Ensure the database, username, and password match the configuration. ```bash DATABASE_URL=postgres://que:que@localhost:5697/que-test bundle exec rake ``` -------------------------------- ### Configure Que with ConnectionPool for Multithreading Source: https://github.com/que-rb/que/wiki/Advanced-Setup For efficient multithreading, use the 'connection_pool' gem to manage PG connections. Add 'gem "connection_pool"' to your Gemfile. ```ruby require 'uri' require 'pg' require 'connection_pool' uri = URI.parse(ENV['DATABASE_URL']) pool = ConnectionPool.new :size => 10 do PG::Connection.open :host => uri.host, :user => uri.user, :password => uri.password, :port => uri.port || 5432, :dbname => uri.path[1..-1] end ``` -------------------------------- ### Que CLI Options Source: https://github.com/que-rb/que/blob/master/docs/README.md Command-line options for running the Que worker. Configure polling intervals, worker priorities, queue names, and connection details. ```bash usage: que [options] [file/to/require] ... -h, --help Show this help text. -i, --poll-interval [INTERVAL] Set maximum interval between polls for available jobs, in seconds (default: 5) -l, --log-level [LEVEL] Set level at which to log to STDOUT (debug, info, warn, error, fatal) (default: info) -p, --worker-priorities [LIST] List of priorities to assign to workers (default: 10,30,50,any,any,any) -q, --queue-name [NAME] Set a queue name to work jobs from. Can be passed multiple times. (default: the default queue only) -w, --worker-count [COUNT] Set number of workers in process (default: 6) -v, --version Print Que version and exit. --connection-url [URL] Set a custom database url to connect to for locking purposes. --log-internals Log verbosely about Que's internal state. Only recommended for debugging issues --maximum-buffer-size [SIZE] Set maximum number of jobs to be locked and held in this process awaiting a worker (default: 8) --wait-period [PERIOD] Set maximum interval between checks of the in-memory job queue, in milliseconds (default: 50) ``` -------------------------------- ### Query Que Jobs with Sequel Source: https://github.com/que-rb/que/wiki/Querying-the-Job-Table Use Sequel's dataset methods to query the `que_jobs` table directly. This is useful if you are already using Sequel in your project. ```ruby DB[:que_jobs].all ``` -------------------------------- ### Configure Que with Sequel Connection Source: https://github.com/que-rb/que/wiki/Advanced-Setup For Sequel users, provide Que with a specific database instance. This allows for safe transaction usage within jobs. ```ruby DB = Sequel.connect(ENV['DATABASE_URL']) Que.connection = DB ``` ```ruby class MyJob < Que::Job def run DB.transaction do # Make whatever changes you like to the database. destroy end end end ``` -------------------------------- ### Create Que Schema Migration Source: https://github.com/que-rb/que/blob/master/README.md Use this migration to set up the necessary schema for Que in your PostgreSQL database. Always specify the version you are migrating to. ```ruby class CreateQueSchema < ActiveRecord::Migration[6.0] def up # Whenever you use Que in a migration, always specify the version you're # migrating to. If you're unsure what the current version is, check the # changelog. Que.migrate!(version: 7) end def down # Migrate to version 0 to remove Que entirely. Que.migrate!(version: 0) end end ``` -------------------------------- ### Enqueue Job with Options Source: https://github.com/que-rb/que/blob/master/docs/README.md Shows how to enqueue a job with specific options such as run time and priority using the `job_options` hash. ```ruby ChargeCreditCard.enqueue(card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5 }) ``` -------------------------------- ### Que Job Options Reference Source: https://context7.com/que-rb/que/llms.txt Configure per-enqueue options for Que jobs using the `job_options:` keyword argument or class-level defaults. Options include `queue`, `priority`, and `run_at`. ```ruby class ProcessOrder < Que::Job self.queue = 'orders' # default queue name self.priority = 20 # lower = more important (max: 32767) self.run_at = proc { 5.seconds.from_now } # callable or Time def run(order_id) # ... destroy end end ``` -------------------------------- ### Integrating Que with ActiveJob Source: https://github.com/que-rb/que/blob/master/docs/README.md Include `Que::ActiveJob::JobExtensions` in your `ApplicationJob` subclass to enable Que's helper methods and support for keyword arguments in the `run()` method. These methods become no-ops if a non-Que adapter is used. ```ruby class ApplicationJob < ActiveJob::Base include Que::ActiveJob::JobExtensions end ``` -------------------------------- ### Define a Job Class with Que Source: https://context7.com/que-rb/que/llms.txt Define background jobs by inheriting from `Que::Job` and implementing the `run` method. Configure default queue, priority, and retry behavior using class-level settings. ```ruby # app/jobs/charge_credit_card.rb class ChargeCreditCard < Que::Job # Linux priority scale: lower number = higher importance (default: 100) self.priority = 10 # Delay execution by 1 minute after enqueue (can be a proc or a Time) self.run_at = proc { 1.minute.from_now } # Stop retrying after this many failures (default: 15) self.maximum_retry_count = 5 def run(credit_card_id, user_id:) user = User.find(user_id) card = CreditCard.find(credit_card_id) ActiveRecord::Base.transaction do user.update!(charged_at: Time.now) # Destroy the job inside the same transaction so that success and # job removal are atomic. If the transaction rolls back, the job # will be retried. destroy end end end ``` -------------------------------- ### Publish New Gem Version Source: https://github.com/que-rb/que/blob/master/README.md Commands to build and publish a new version of the Que gem to RubyGems. ```bash gem build -o que.gem && gem push que.gem ``` -------------------------------- ### Configure Que Connection (Non-Rails) Source: https://context7.com/que-rb/que/llms.txt Manually configure Que's database connection when not using Rails. Supports ActiveRecord, Sequel, and plain PG connection pools. ```ruby ActiveRecord::Base.establish_connection(ENV['DATABASE_URL']) require 'que' Que.connection = ActiveRecord ``` ```ruby DB = Sequel.connect(ENV['DATABASE_URL']) Que.connection = DB ``` ```ruby require 'uri'; require 'pg'; require 'connection_pool' uri = URI.parse(ENV['DATABASE_URL']) Que.connection = ConnectionPool.new(size: 10) do PG::Connection.open( host: uri.host, user: uri.user, password: uri.password, port: uri.port || 5432, dbname: uri.path[1..-1] ) end ``` ```ruby # Fully custom connection pool — proc must be reentrant and yield PG::Connection Que.connection_proc = proc do |&block| MyCustomPool.checkout do |conn| block.call(conn) end end ``` -------------------------------- ### Job and SQL Middleware Configuration Source: https://context7.com/que-rb/que/llms.txt Wrap job executions or SQL statements with custom middleware lambdas for tasks like performance monitoring or logging. ```ruby # Job middleware — instrument job timing Que.job_middleware.push( ->(job, &block) { start = Time.now block.call elapsed = Time.now - start StatsD.timing("que.job.#{job.que_attrs[:job_class]}", elapsed * 1000) nil } ) # SQL middleware — attach to NewRelic or similar APM Que.sql_middleware.push( ->(sql, params, &block) { NewRelic::Agent::MethodTracerHelpers.trace_execution_scoped("Que/SQL") do block.call end nil } ) ``` -------------------------------- ### Set Up Que Error Notifier Source: https://github.com/que-rb/que/blob/master/docs/README.md Integrate Que with an error notification system by providing a callable to `Que.error_notifier`. This callable receives the error and a hash representing the job row in the database. ```ruby Que.error_notifier = proc do |error, job| # Do whatever you want with the error object or job row here. Note that the # job passed is not the actual job object, but the hash representing the job # row in the database, which looks like: # { # :priority => 100, # :run_at => "2017-09-15T20:18:52.018101Z", # :id => 172340879, # :job_class => "TestJob", # :error_count => 0, # :last_error_message => nil, # :queue => "default", # :last_error_backtrace => nil, # :finished_at => nil, # :expired_at => nil, # :args => [], # :data => {} # } # This is done because the job may not have been able to be deserialized # properly, if the name of the job class was changed or the job class isn't # loaded for some reason. The job argument may also be nil, if there was a # connection failure or something similar. end ``` -------------------------------- ### ActiveJob Integration with Que Source: https://context7.com/que-rb/que/llms.txt Configure Rails to use Que as its ActiveJob backend. Optionally include `Que::ActiveJob::JobExtensions` for Que-specific helper methods within ActiveJobs. ```ruby # config/application.rb config.active_job.queue_adapter = :que ``` ```ruby # app/jobs/application_job.rb class ApplicationJob < ActiveJob::Base # Gives all ActiveJob jobs access to Que helper methods (destroy, finish, # expire, retry_in, error_count). These become no-ops with other adapters. include Que::ActiveJob::JobExtensions end ``` ```ruby # app/jobs/send_welcome_email_job.rb class SendWelcomeEmailJob < ApplicationJob queue_as :default def perform(user_id) user = User.find(user_id) UserMailer.welcome(user).deliver_now finish # keep job record instead of destroying it end end ``` ```ruby SendWelcomeEmailJob.perform_later(current_user.id) ``` -------------------------------- ### Run Manual Vacuum on que_jobs Table Source: https://github.com/que-rb/que/blob/master/docs/README.md Use this recurring job to manually vacuum the que_jobs table. Set a high priority to ensure it runs before other jobs. Adjust the INTERVAL based on your database's needs. ```ruby class ManualVacuumJob < CronJob self.priority = 1 # set this to the highest priority since it keeps the table healthy for other jobs INTERVAL = 300 def run(args) DB.run "VACUUM VERBOSE ANALYZE que_jobs" end end ``` -------------------------------- ### Integrate Que with Active Job Source: https://github.com/que-rb/que/blob/master/README.md Set the Active Job queue adapter to `:que` in `config/application.rb` or a specific environment file. ```ruby config.active_job.queue_adapter = :que ``` -------------------------------- ### Enqueue Job with Custom Class Source: https://github.com/que-rb/que/blob/master/docs/README.md Illustrates how to specify a custom job class when enqueuing a job using `Que.enqueue`. This allows enqueuing jobs without needing the job class to be defined in the enqueueing process. ```ruby Que.enqueue(current_user.id, job_options: { job_class: 'ProcessCreditCard' }) ``` -------------------------------- ### Add Que Gem to Gemfile Source: https://github.com/que-rb/que/blob/master/README.md Add the Que gem to your application's Gemfile to include it in your project. ```ruby gem 'que' ``` -------------------------------- ### Database Schema Migration Source: https://context7.com/que-rb/que/llms.txt Manage Que's database schema versions using migrations. Pin the target version when upgrading or downgrading. ```ruby # Upgrade to the latest schema (version 7 for Que 2.x) class UpdateQueToVersion7 < ActiveRecord::Migration[6.0] def up Que.migrate!(version: 7) end def down Que.migrate!(version: 6) end end # Check current schema version at any time Que.db_version # => 7 ``` -------------------------------- ### Que Worker CLI Options Source: https://github.com/que-rb/que/blob/master/README.md View available runtime options for the Que worker using `que -h`. Options include setting the poll interval. ```bash que -h ``` ```bash usage: que [options] [file/to/require] ... -h, --help Show this help text. -i, --poll-interval [INTERVAL] Set maximum interval between polls for available jobs, in seconds (default: 5) ... ``` -------------------------------- ### Migrate Que Database Schema Source: https://context7.com/que-rb/que/llms.txt Use `Que.migrate!(version: 0)` to completely remove the Que database schema, useful for resetting the queue entirely. ```ruby Que.migrate!(version: 0) ``` -------------------------------- ### Configure Rails Queues Source: https://github.com/que-rb/que/blob/master/README.md Configure Rails to send internal job types to the 'default' queue by adding settings to `config/application.rb`. Alternatively, tell Que to work multiple specific queues. ```ruby config.action_mailer.deliver_later_queue_name = :default config.action_mailbox.queues.incineration = :default config.action_mailbox.queues.routing = :default config.active_storage.queues.analysis = :default config.active_storage.queues.purge = :default ``` ```bash que -q default -q mailers -q action_mailbox_incineration -q action_mailbox_routing -q active_storage_analysis -q active_storage_purge ``` -------------------------------- ### Manually Migrate Que Schema Source: https://github.com/que-rb/que/blob/master/docs/README.md Perform a manual migration of the Que database schema to a specific version in a console session. You can also check the current schema version. ```ruby # Change schema to version 3. Que.migrate!(version: 3) # Check your current schema version. Que.db_version #=> 3 ``` -------------------------------- ### Schedule Periodic Vacuuming of Que Jobs Table Source: https://context7.com/que-rb/que/llms.txt Schedule a recurring job to vacuum the `que_jobs` table to prevent bloat due to high churn. Alternatively, use direct SQL for batch deletion of old finished jobs. ```ruby # A recurring vacuum job (using a cron-style scheduler like que-scheduler) class VacuumQueJobsJob < Que::Job self.priority = 1 # highest priority — keeps the table healthy for others def run ActiveRecord::Base.connection.execute('VACUUM VERBOSE ANALYZE que_jobs') destroy end end ``` ```sql # Or directly in SQL to batch-delete old finished jobs without triggering NOTIFY # BEGIN; # SET LOCAL que.skip_notify TO true; # DELETE FROM que_jobs WHERE finished_at < now() - interval '7 days'; # ``` -------------------------------- ### Enqueue Jobs in Bulk Source: https://github.com/que-rb/que/blob/master/docs/README.md Use the `Que.bulk_enqueue` block for enqueuing a large number of jobs efficiently. Jobs are inserted in a single query at the end of the block. Note that ActiveJob is not supported, and all jobs must use the same class and `job_options`. ```ruby Que.bulk_enqueue do MyJob.enqueue(user_id: 1) MyJob.enqueue(user_id: 2) # ... end ``` ```ruby Que.bulk_enqueue(notify: true) { ... } ``` -------------------------------- ### Customize Que Log Formatter Source: https://github.com/que-rb/que/blob/master/docs/README.md Provide a callable object (like a proc) to `Que.log_formatter=` to customize the format of log output. The proc should accept a hash of data and return a string. ```ruby Que.log_formatter = proc do |data| "Thread number #{data[:thread]} experienced a #{data[:event]}" end ``` -------------------------------- ### Custom Connection Pool Proc for Que Source: https://github.com/que-rb/que/blob/master/docs/README.md Define a proc for Que to use any in-process connection pool. The proc must accept a block and yield a PG::Connection instance. It must be reentrant and lock the connection during the block's execution. ```ruby Que.connection_proc = proc do |&block| DB.synchronize do |connection| block.call(connection) end end ``` -------------------------------- ### Configure Que Worker Pool Source: https://github.com/que-rb/que/wiki/Advanced-Setup Set Que to run in asynchronous mode and specify the number of worker threads. The optimal worker count depends on the environment and job types. ```ruby Que.mode = :async Que.worker_count = 8 ``` -------------------------------- ### Configure Que Logger Source: https://github.com/que-rb/que/blob/master/docs/README.md Customize the logger used by Que by assigning a `Logger` instance to `Que.logger`. This allows you to direct Que's logs to a specific output. ```ruby Que.logger = Logger.new(...) ``` -------------------------------- ### Configure Que with ActiveRecord Connection Source: https://github.com/que-rb/que/wiki/Advanced-Setup When not using Rails, explicitly set Que to use ActiveRecord's connection pool. ```ruby Que.connection = ActiveRecord ``` -------------------------------- ### Bulk Enqueueing with Que Source: https://context7.com/que-rb/que/llms.txt Use `Que.bulk_enqueue` to insert multiple jobs in a single SQL statement for maximum throughput, bypassing the per-row NOTIFY trigger. All jobs within a `bulk_enqueue` block must share the same class and job_options. Set `notify: true` to force LISTEN/NOTIFY for each job, ensuring immediate pickup by workers. ```ruby # All jobs in the block must share the same class and job_options. Que.bulk_enqueue(job_options: { priority: 50 }) do User.find_each do |user| DailyDigestEmail.enqueue(user_id: user.id) end end ``` ```ruby # Force LISTEN/NOTIFY to fire for each job (slower, but workers pick up # jobs immediately instead of waiting for the next poll cycle) Que.bulk_enqueue(notify: true) do MyJob.enqueue(user_id: 1) MyJob.enqueue(user_id: 2) end ``` -------------------------------- ### Sequel Migrations with Que Source: https://github.com/que-rb/que/blob/master/docs/README.md When using Sequel's migrator, ensure Que is required and its connection is set within the migration. This allows Que.migrate! to be called within the migration's up and down blocks. ```ruby require 'que' Sequel.migration do up do Que.connection = self Que.migrate!(version: 7) end down do Que.connection = self Que.migrate!(version: 0) end end ``` -------------------------------- ### Inspecting Job Statistics and Using ORM Models Source: https://context7.com/que-rb/que/llms.txt Query aggregate job statistics or use provided ORM models (ActiveRecord/Sequel) for custom queries on job data. ```ruby # Aggregate stats across all job classes Que.job_stats # => [ # { job_class: "ChargeCreditCard", count: 10, count_working: 4, # count_errored: 2, highest_error_count: 5, # oldest_run_at: 2024-01-01 12:00:00 }, # { job_class: "SendEmail", count: 1, count_working: 0, # count_errored: 0, highest_error_count: 0, # oldest_run_at: 2024-01-01 13:00:00 } # ] # ActiveRecord model for custom queries # app/models/que_job.rb require 'que/active_record/model' class QueJob < Que::ActiveRecord::Model; end QueJob.where(job_class: 'ChargeCreditCard').count # => 10 QueJob.finished.where('finished_at < ?', 7.days.ago) # old finished jobs QueJob.errored.order(:error_count).last # most-failed job QueJob.expired # jobs that won't retry # Sequel model require 'que/sequel/model' class QueJob < Que::Sequel::Model; end QueJob.not_finished.where(queue: 'credit_cards').all ``` -------------------------------- ### Define Sequel Model for Que Jobs Source: https://github.com/que-rb/que/blob/master/docs/README.md Define a model that inherits from `Que::Sequel::Model` to interact with Que jobs using Sequel. This allows you to use Sequel query methods on your Que jobs. ```ruby require 'que/sequel/model' class QueJob < Que::Sequel::Model end QueJob.finished # => # ``` -------------------------------- ### Job with HTTP Timeout Source: https://github.com/que-rb/que/blob/master/docs/README.md Demonstrates how to add a timeout to an HTTP request within a Que job. If the request exceeds the specified timeout, an error is raised, and Que will retry the job later. ```ruby class ScrapeStuff < Que::Job def run(url_to_scrape) result = YourHTTPLibrary.get(url_to_scrape, timeout: 5) ActiveRecord::Base.transaction do # Insert result... destroy end end end ``` -------------------------------- ### Define a Que Job Source: https://github.com/que-rb/que/blob/master/README.md Define a job class inheriting from Que::Job. Set default run_at and priority. The run method contains the job's logic. Use destroy to remove the job after successful execution within a transaction, or finish to leave it in the database for history. ```ruby class ChargeCreditCard < Que::Job self.run_at = proc { 1.minute.from_now } self.priority = 10 def run(credit_card_id, user_id:) user = User.find(user_id) card = CreditCard.find(credit_card_id) User.transaction do user.update charged_at: Time.now destroy end end end ``` -------------------------------- ### Job Resolution Control Methods Source: https://context7.com/que-rb/que/llms.txt Use instance methods like `destroy`, `finish`, `expire`, and `retry_in` within a job's `run` method to control its lifecycle and resolution. ```ruby class MyJob < Que::Job def run(order_id) order = Order.find(order_id) ActiveRecord::Base.transaction do order.process! # destroy – deletes the job row (default behavior if no resolve call made) # finish – marks finished_at but keeps the row for audit purposes # expire – marks expired_at; won't retry; stays in table # retry_in – schedules a retry; increments error_count if order.complete? finish # keep job row for history elsif order.invalid? expire # do not retry else retry_in(10.minutes) end end end # Override the default action when run() exits without any resolve call def default_resolve_action finish # keep finished jobs by default instead of destroying them end end ``` -------------------------------- ### Enqueue Job with Custom Options Source: https://context7.com/que-rb/que/llms.txt Override default job settings like queue name, priority, run time, and tags when enqueuing a job. ```ruby ProcessOrder.enqueue( order.id, job_options: { queue: 'priority_orders', priority: 1, run_at: Time.now + 3600, tags: ['vip'] # up to 5 tags, 100 chars each } ) ``` -------------------------------- ### Define ActiveRecord Model for Que Jobs Source: https://github.com/que-rb/que/wiki/Querying-the-Job-Table Define an ActiveRecord model to interact with the `que_jobs` table. This allows standard ActiveRecord querying methods. ```ruby class QueJob < ActiveRecord::Base end ``` ```ruby class MyJob < ActiveRecord::Base self.table_name = :que_jobs end ``` -------------------------------- ### Run Que Jobs Synchronously in Tests Source: https://context7.com/que-rb/que/llms.txt Set `Que::Job.run_synchronously = true` to execute jobs immediately in the current thread, simplifying testing by allowing direct assertion on side-effects. ```ruby # spec/rails_helper.rb or test/test_helper.rb Que::Job.run_synchronously = true ``` ```ruby # With run_synchronously, .enqueue immediately executes the job inline # so you can assert on side-effects without a worker process RSpec.describe ChargeCreditCard do it 'charges the card and updates the user' do user = create(:user) card = create(:credit_card, user: user) expect(PaymentService).to receive(:charge).with(card.id) ChargeCreditCard.enqueue(card.id, user_id: user.id) expect(user.reload.charged_at).not_to be_nil end end ``` ```ruby # Or leave run_synchronously off and assert on the database state it 'enqueues a job' do expect { ChargeCreditCard.enqueue(card.id, user_id: user.id) } .to change { QueJob.count }.by(1) job = QueJob.last expect(job.job_class).to eq('ChargeCreditCard') expect(job.kwargs[:user_id]).to eq(user.id) end ``` -------------------------------- ### Enqueue Jobs with Que Source: https://context7.com/que-rb/que/llms.txt Enqueue jobs using `JobClass.enqueue` which persists the job to the database. For atomicity, enqueue jobs within the same database transaction as related data changes. Job options like `run_at`, `priority`, `queue`, and `tags` can be overridden at enqueue time. ```ruby # Enqueue with positional and keyword args inside a transaction CreditCard.transaction do card = CreditCard.create!(params[:credit_card]) ChargeCreditCard.enqueue(card.id, user_id: current_user.id) end ``` ```ruby # Override job options at enqueue time ChargeCreditCard.enqueue( card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5, queue: 'credit_cards', tags: ['billing', 'high-value'] } ) ``` ```ruby # Enqueue without knowing the job class in the current process Que.enqueue(current_user.id, job_options: { job_class: 'ProcessCreditCard' }) ``` -------------------------------- ### Set Postgres Version for Docker Source: https://github.com/que-rb/que/blob/master/README.md Specify a different version of Postgres to be used when running with Docker Compose by setting the POSTGRES_VERSION environment variable. ```bash export POSTGRES_VERSION=12 ``` -------------------------------- ### Configure Job Retry Intervals Source: https://github.com/que-rb/que/blob/master/docs/README.md Set custom retry intervals for failed jobs. Options include a fixed interval, immediate retries (not recommended), or a callable that dynamically calculates the interval based on failure count. ```ruby class MyJob < Que::Job # Just retry a failed job every 5 seconds: self.retry_interval = 5 # Always retry this job immediately (not recommended, or transient # errors will spam your error reporting): self.retry_interval = 0 # Increase the delay by 30 seconds every time this job fails: self.retry_interval = proc { |count| count * 30 } end ``` -------------------------------- ### Idempotent Job for External Service Charges Source: https://github.com/que-rb/que/wiki/Writing-Reliable-Jobs Implement idempotency by checking for previous actions before executing them, such as verifying if a credit card has already been charged. This prevents duplicate charges even if the job is retried. ```ruby class ChargeCreditCard < Que::Job def run(user_id, credit_card_id) unless CreditCardProvider.check_for_previous_charge(credit_card_id) CreditCardProvider.charge(credit_card_id, :amount => "$10.00") end ActiveRecord::Base.transaction do User.where(:id => user_id).update_all :charged_at => Time.now destroy end end end ``` -------------------------------- ### Migrate Que Jobs Table Schema Source: https://github.com/que-rb/que/blob/master/docs/README.md Update the Que jobs table schema to a specific version using `Que.migrate!`. This is useful for ensuring compatibility during upgrades. ```ruby # Update the schema to version #7. Que.migrate!(version: 7) ``` -------------------------------- ### Enqueue Jobs to a Specific Queue Source: https://github.com/que-rb/que/blob/master/docs/README.md Set jobs to be enqueued in a specific queue, either by passing the :queue option during enqueueing or by defining a default queue for a job class. ```ruby ProcessCreditCard.enqueue(current_user.id, job_options: { queue: 'credit_cards' }) # Or: class ProcessCreditCard < Que::Job # Set a default queue for this job class; this can be overridden by # passing the :queue parameter to enqueue like above. self.queue = 'credit_cards' end ``` -------------------------------- ### Run Specs Multiple Times with Different Seeds Source: https://github.com/que-rb/que/blob/master/README.md Iterate through specs many times with varying seeds to detect hangs and race conditions. Note the seed number if a hang occurs for further debugging. ```bash for i in {1..1000}; do SEED=$i bundle exec rake; done ``` -------------------------------- ### Migrate Que Schema with ActiveRecord Source: https://github.com/que-rb/que/blob/master/docs/README.md Use this migration to update the Que database schema version when upgrading Que. Ensure your database schema stays consistent with your codebase. ```ruby class UpdateQue < ActiveRecord::Migration[5.0] def self.up Que.migrate!(version: 3) end def self.down Que.migrate!(version: 2) end end ``` -------------------------------- ### Clear All Queued Jobs Source: https://context7.com/que-rb/que/llms.txt Use `Que.clear!` to remove all jobs from the queue, which is particularly useful for test environments. ```ruby Que.clear! ``` -------------------------------- ### Run Que with a Single Worker for Thread-Unsafe Code Source: https://github.com/que-rb/que/blob/master/docs/README.md If your application code is not thread-safe, run Que with a single worker to prevent concurrent job processing. This ensures that no other jobs are processed while one is running. ```shell que --worker-count 1 ``` -------------------------------- ### Global Error Notifier Configuration Source: https://context7.com/que-rb/que/llms.txt Configure a global error notifier to capture and report job errors to external services like Sentry or Honeybadger. ```ruby # Global error notifier (Sentry, Honeybadger, etc.) Que.error_notifier = proc do |error, job| Sentry.capture_exception(error, extra: { job: job }) end ``` -------------------------------- ### Transactional Job Execution with Sequel Source: https://github.com/que-rb/que/blob/master/docs/README.md Perform database operations and job destruction within a Sequel transaction to ensure atomicity. This guarantees that either all changes, including job destruction, succeed or none do. ```ruby class MyJob < Que::Job def run(user_id:) # Do stuff. DB.transaction do # Make changes to the database. # Destroying this job will be protected by the same transaction. destroy end end end ``` ```ruby # Or, in your controller action: DB.transaction do @user = User.create(params[:user]) MyJob.enqueue user_id: @user.id end ``` -------------------------------- ### Enqueue a Que Job Source: https://github.com/que-rb/que/blob/master/README.md Enqueue jobs within a database transaction for consistency. Arguments are serialized to JSON, so use simple data types. Options for run_at and priority can be provided. ```ruby CreditCard.transaction do card = CreditCard.create(params[:credit_card]) ChargeCreditCard.enqueue(card.id, user_id: current_user.id) end ``` ```ruby ChargeCreditCard.enqueue(card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5 }) ``` -------------------------------- ### Update Widget Price Safely with Transactions Source: https://github.com/que-rb/que/blob/master/docs/README.md Ensures data consistency by performing database updates and job destruction within a single transaction. This guarantees that either both operations succeed or neither does, preventing partial updates and ensuring jobs are run only once. ```ruby class UpdateWidgetPrice < Que::Job def run(widget_id) widget = Widget[widget_id] price = ExternalService.get_widget_price(widget_id) ActiveRecord::Base.transaction do # Make changes to the database. widget.update price: price # Mark the job as destroyed, so it doesn't run again. destroy end end end ``` -------------------------------- ### Reset Que Jobs Table Schema Source: https://github.com/que-rb/que/blob/master/docs/README.md Completely remove Que's jobs table by migrating to version 0 using `Que.migrate!`. Use with caution as this will delete all queued jobs. ```ruby # Remove Que's jobs table entirely. Que.migrate!(version: 0) ``` -------------------------------- ### Run Que Workers on a Specific Queue Source: https://github.com/que-rb/que/blob/master/docs/README.md Specify a queue name when running Que workers to process jobs from a particular queue. This is useful for isolating tasks like credit card processing. ```shell que --queue-name credit_cards # The -q flag is equivalent, and either can be passed multiple times. que -q default -q credit_cards ``` -------------------------------- ### Route Jobs to Multiple Queues Source: https://context7.com/que-rb/que/llms.txt Assign different job types to specific queues for workload isolation. This can be done by setting a default queue on the job class or specifying it at enqueue time. ```ruby # Set a default queue on the job class class ProcessCreditCard < Que::Job self.queue = 'credit_cards' def run(user_id) # ... destroy end end ``` ```ruby # Or specify the queue at enqueue time ProcessCreditCard.enqueue(user.id, job_options: { queue: 'credit_cards' }) ``` ```bash # Start a worker process that only works the credit_cards queue # bundle exec que -q credit_cards ``` ```bash # Work multiple queues in the same process # bundle exec que -q default -q credit_cards ``` ```ruby # Route Rails internal job types so they are worked by default workers # config/application.rb config.action_mailer.deliver_later_queue_name = :default config.active_storage.queues.analysis = :default config.active_storage.queues.purge = :default ``` -------------------------------- ### Custom Retry Intervals for Jobs Source: https://context7.com/que-rb/que/llms.txt Define custom retry intervals for failed jobs. This can be a fixed number of seconds or a callable that dynamically calculates the interval based on the failure count. ```ruby class ResilientJob < Que::Job # Flat retry interval self.retry_interval = 30 # Or a callable for dynamic intervals self.retry_interval = proc { |count| count * 60 } def run(resource_id) ExternalAPI.process(resource_id) destroy end def handle_error(error) case error when RateLimitError retry_in(60) false # return false to suppress error notification when PermanentFailure expire # marks job expired, stops retrying when TransientError result = super # default exponential backoff + notifier error_count > 5 ? result : false # only notify after 5 failures else super end end end ``` -------------------------------- ### Charge Credit Card with Idempotent Design Source: https://github.com/que-rb/que/blob/master/docs/README.md Handles external service interactions that cannot be rolled back by making the job idempotent. It checks for previous charges before attempting to charge the credit card again, preventing duplicate charges if the job is retried. ```ruby class ChargeCreditCard < Que::Job def run(user_id, credit_card_id) unless CreditCardService.check_for_previous_charge(credit_card_id) CreditCardService.charge(credit_card_id, amount: "$10.00") end ActiveRecord::Base.transaction do User.where(id: user_id).update_all charged_at: Time.now destroy end end end ``` -------------------------------- ### Implement Custom Job Error Handling Source: https://github.com/que-rb/que/blob/master/docs/README.md Define a `handle_error` method within a Que::Job subclass to implement custom error handling logic. This allows for specific actions like retrying or expiring jobs based on error type. ```ruby class MyJob < Que::Job def run(*args) # Your code goes here. end def handle_error(error) case error when TemporaryError then retry_in 10.seconds when PermanentError then expire else super # Default (exponential backoff) behavior. end end end ``` -------------------------------- ### Customizing Que Logging Source: https://context7.com/que-rb/que/llms.txt Customize the logger, log formatter, and per-job log levels. By default, Que logs JSON to Rails' logger or STDOUT. ```ruby # Use a custom logger Que.logger = Logger.new('log/que.log') # Custom format (return nil/false to suppress a log line) Que.log_formatter = proc do |data| next unless [:job_worked, :job_errored].include?(data[:event]) JSON.dump(data.merge(app: 'my_service')) end # Per-job dynamic log level based on elapsed time class TimeSensitiveJob < Que::Job def run(*) RemoteAPI.call destroy end def log_level(elapsed) if elapsed > 60 then :warn elsif elapsed > 30 then :info else false # don't log fast completions at all end end end ``` -------------------------------- ### Re-run Expired Job Source: https://github.com/que-rb/que/blob/master/docs/README.md To re-run an expired job, clear its `error_count` and `expired_at` columns in the `que_jobs` table. ```sql UPDATE que_jobs SET error_count = 0, expired_at = NULL WHERE id = 172340879; ``` -------------------------------- ### Job Performing Non-Transactional Actions Source: https://github.com/que-rb/que/wiki/Writing-Reliable-Jobs For jobs that perform actions not tied to database transactions, like sending emails, omit the `destroy` call within a transaction. Que will automatically clean up unfinished jobs. ```ruby class SendVerificationEmail < Que::Job def run(email_address) Mailer.verification_email(email_address).deliver end end ```