### Install RQ using pip Source: https://github.com/frappe/rq/blob/v1.14/README.md Install the latest stable version of RQ from PyPI using pip. ```bash pip install rq ``` -------------------------------- ### Install and Run RQ Dashboard Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/monitoring.md Install the RQ dashboard using pip and run it to get a web-based monitor for RQ. ```console $ pip install rq-dashboard $ rq-dashboard ``` -------------------------------- ### Initialize and Start Vagrant Environment Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Initializes a Vagrant environment with a specified box and starts the virtual machine. This is the first step for both running tests and building docs. ```bash vagrant init ubuntu/trusty64 vagrant up ``` -------------------------------- ### Install Latest RQ from Git Source: https://github.com/frappe/rq/blob/v1.14/README.md Install the cutting-edge version of RQ directly from its GitHub repository. This version may be unstable. ```bash pip install git+https://github.com/rq/rq.git@master#egg=rq ``` -------------------------------- ### Start Redis Server Source: https://github.com/frappe/rq/blob/v1.14/README.md Before using RQ, ensure a Redis server is running. This command starts the Redis server. ```console redis-server ``` -------------------------------- ### RQ Configuration File Example Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/index.md An example configuration file for RQ, specifying connection details for Redis, including host, port, password, and SSL settings. ```python REDIS_HOST = "host" REDIS_PORT = port REDIS_PASSWORD = "password" REDIS_SSL = True REDIS_SSL_CA_CERTS = None REDIS_DB = 0 REDIS_SSL_CERT_REQS = None ``` -------------------------------- ### Install Documentation Dependencies in Vagrant Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Installs Ruby development headers and Node.js, followed by the Jekyll gem, which is necessary for building and serving the documentation. ```bash vagrant ssh -- "sudo apt-get -y install ruby-dev nodejs" vagrant ssh -- "sudo gem install jekyll" ``` -------------------------------- ### Serve Documentation with Jekyll Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/docs.md Run this command in your terminal to build and serve the documentation locally using Jekyll. Ensure Jekyll is installed. ```bash jekyll serve ``` -------------------------------- ### Serve Documentation in Vagrant Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Navigates to the project directory within the Vagrant environment and starts the Jekyll server. The documentation can then be accessed via a forwarded port. ```bash vagrant ssh -- "(cd /vagrant; jekyll serve)" ``` -------------------------------- ### Accessing and Inspecting StartedJobRegistry Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/job_registries.md Demonstrates how to get a StartedJobRegistry instance, retrieve its associated queue, count the number of jobs, and list job IDs. It also shows how to check if a job exists within the registry. ```python import time from redis import Redis from rq import Queue from rq.registry import StartedJobRegistry from somewhere import count_words_at_url redis = Redis() queue = Queue(connection=redis) job = queue.enqueue(count_words_at_url, 'http://nvie.com') # get StartedJobRegistry by queue registry = StartedJobRegistry(queue=queue) # or get StartedJobRegistry by queue name and connection registry2 = StartedJobRegistry(name='my_queue', connection=redis) # sleep for a moment while job is taken off the queue time.sleep(0.1) print('Queue associated with the registry: %s' % registry.get_queue()) print('Number of jobs in registry %s' % registry.count) # get the list of ids for the jobs in the registry print('IDs in registry %s' % registry.get_job_ids()) # test if a job is in the registry using the job instance or job id print('Job in registry %s' % (job in registry)) print('Job in registry %s' % (job.id in registry)) ``` -------------------------------- ### Retrieving Job IDs and Job Instances Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Demonstrates how to get a list of job IDs and job instances from a queue, and how to fetch a specific job by its ID. ```python # Retrieving jobs queued_job_ids = q.job_ids # Gets a list of job IDs from the queue queued_jobs = q.jobs # Gets a list of enqueued job instances job = q.fetch_job('my_id') # Returns job having ID "my_id" ``` -------------------------------- ### Install Test Dependencies in Vagrant Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Installs Redis server, Python development headers, and pip within the Vagrant environment, followed by Python packages required for testing. This prepares the environment for running tests. ```bash vagrant ssh -- "sudo apt-get -y install redis-server python-dev python-pip" vagrant ssh -- "sudo pip install --no-input redis hiredis mock" ``` -------------------------------- ### Start an RQ Worker Source: https://github.com/frappe/rq/blob/v1.14/README.md Launch an RQ worker process to execute jobs from the queue. The `--with-scheduler` flag enables scheduled jobs. ```console $ rq worker --with-scheduler *** Listening for work on default Got count_words_at_url('http://nvie.com') from default Job result = 818 *** Listening for work on default ``` -------------------------------- ### Start RQ Workers Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Start RQ workers to process jobs from specified queues (high, default, low). Workers continuously listen for new work. ```console $ rq worker high default low *** Listening for work on high, default, low Got send_newsletter('me@nvie.com') from default Job ended normally without result *** Listening for work on high, default, low ... ``` -------------------------------- ### Enqueue a Job to the Default Queue Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md This example demonstrates how to enqueue a function call to the default RQ queue using a Redis connection. It also shows how to retrieve the job's result after a delay. ```python from rq import Queue from redis import Redis from somewhere import count_words_at_url import time # Tell RQ what Redis connection to use redis_conn = Redis() q = Queue(connection=redis_conn) # no args implies the default queue # Delay execution of count_words_at_url('http://nvie.com') job = q.enqueue(count_words_at_url, 'http://nvie.com') print(job.result) # => None # Changed to job.return_value() in RQ >= 1.12.0 # Now, wait a while, until the worker is finished time.sleep(2) print(job.result) # => 889 # Changed to job.return_value() in RQ >= 1.12.0 ``` -------------------------------- ### Run all tests with Tox Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/testing.md Use tox to run tests across all supported Python versions locally. Ensure all required Python versions are installed. ```bash tox ``` -------------------------------- ### Enqueueing a function by string specification Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Example of how to enqueue a function using its string path, useful when the worker and submitter do not share codebases. ```python q.enqueue("my.math.lib.fibonacci", 5) ``` -------------------------------- ### Starting a Worker with a Custom Name Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Specify a custom name for a worker instance when starting it programmatically to override the default random generation. ```python from redis import Redis from rq import Queue, Worker redis = Redis() queue = Queue('queue_name') # Start a worker with a custom name worker = Worker([queue], connection=redis, name='foo') ``` -------------------------------- ### Sentinel Configuration for Fault-Tolerant Connections Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/connections.md Example configuration for Redis Sentinel, enabling fault-tolerant connections to Redis masters. This is useful for workers and RQ with automatic restart options. ```python SENTINEL: { 'INSTANCES':[('remote.host1.org', 26379), ('remote.host2.org', 26379), ('remote.host3.org', 26379)], 'MASTER_NAME': 'master', 'DB': 2, 'USERNAME': 'redis-user', 'PASSWORD': 'redis-secret', 'SOCKET_TIMEOUT': None, 'CONNECTION_KWARGS': { # Eventual addition Redis connection arguments 'ssl_ca_path': None, }, 'SENTINEL_KWARGS': { # Eventual Sentinels connections arguments 'username': 'sentinel-user', 'password': 'sentinel-secret', }, } ``` -------------------------------- ### Start a Worker Pool Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Launch multiple workers with a single command using `rq worker-pool`. This feature is in beta. Specify queues, number of workers, and other options like Redis URL, worker class, and logging level. ```shell rq worker-pool high default low -n 3 ``` -------------------------------- ### Start RQ Worker in Django Context Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/django.md Manually start an RQ worker that operates within your Django project's settings context. Ensure the DJANGO_SETTINGS_MODULE environment variable is set to your project's settings file. ```bash DJANGO_SETTINGS_MODULE=settings rq worker high default low ``` -------------------------------- ### Run Tests in Vagrant Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Navigates to the project directory within the Vagrant environment and executes the test runner script. This command should be run after dependencies are installed. ```bash vagrant ssh -- "(cd /vagrant; ./run_tests)" ``` -------------------------------- ### Enqueue a Job to a Specific Queue Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md To distribute work flexibly, jobs can be placed on named queues. This example enqueues a job to a queue named 'low'. ```python q = Queue('low', connection=redis_conn) q.enqueue(count_words_at_url, 'http://nvie.com') ``` -------------------------------- ### RQ Info CLI Command Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/index.md A command-line interface command to get information about RQ. It requires a configuration file specified by the --config flag. ```console rq info --config rq_conf ``` -------------------------------- ### Define Custom Job and Queue Classes in Python Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Example of defining custom Job and Queue classes in Python. The CustomQueue class is configured to use the CustomJob class. ```python from rq import Queue from rq.job import Job class CustomJob(Job): pass class CustomQueue(Queue): job_class = CustomJob queue = CustomQueue('default', connection=redis_conn) queue.enqueue(some_func) ``` -------------------------------- ### Enqueue Job Using String Reference Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md When the worker process cannot directly import the function, it can be enqueued as a string reference. This example enqueues 'my_package.my_module.my_func'. ```python q = Queue('low', connection=redis_conn) q.enqueue('my_package.my_module.my_func', 3, 4) ``` -------------------------------- ### Setting Custom Failure TTL for Enqueued Jobs Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Shows how to specify a custom time-to-live (TTL) in seconds for failed jobs when they are enqueued. This example sets the TTL to 5 minutes (300 seconds). ```python job = queue.enqueue(foo_job, failure_ttl=300) # 5 minutes in seconds ``` -------------------------------- ### Accessing Job Registries from Queue Objects Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/job_registries.md Shows how to directly access different job registries (started, deferred, finished, failed, scheduled) as attributes of a Queue object. This provides a convenient way to interact with job status. ```python from redis import Redis from rq import Queue redis = Redis() queue = Queue(connection=redis) queue.started_job_registry # Returns StartedJobRegistry queue.deferred_job_registry # Returns DeferredJobRegistry queue.finished_job_registry # Returns FinishedJobRegistry queue.failed_job_registry # Returns FailedJobRegistry queue.scheduled_job_registry # Returns ScheduledJobRegistry ``` -------------------------------- ### Custom Worker Script for Preloading Modules Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Use a custom worker script to import necessary modules before the worker loop starts, improving performance for jobs with lengthy setups or shared dependencies. ```python #!/usr/bin/env python from redis import Redis from rq import Worker # Preload libraries import library_that_you_want_preloaded # Provide the worker with the list of queues (str) to listen to. w = Worker(['default'], connection=Redis()) w.work() ``` -------------------------------- ### RQ Worker with configuration file Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Demonstrates how to run the rqworker using a configuration file instead of command-line options. ```bash rqworker -c settings ``` -------------------------------- ### Start RQ Workers in Burst Mode Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Start RQ workers in burst mode to process all available jobs in the specified queues and then quit. This is useful for batch processing or temporary scaling. ```console $ rq worker --burst high default low *** Listening for work on high, default, low Got send_newsletter('me@nvie.com') from default Job ended normally without result No more work, burst finished. Registering death. ``` -------------------------------- ### Create a job with custom arguments and keyword arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Demonstrates how to pass arguments and keyword arguments to a job function, especially when these conflict with RQ's own enqueue arguments like `ttl` or `description`. ```python job = Job.create(count_words_at_url, ttl=30, # This ttl will be used by RQ args=('http://nvie.com',), kwargs={ 'description': 'Function description', # This is passed on to count_words_at_url 'ttl': 15 # This is passed on to count_words_at_url function }) ``` -------------------------------- ### Create an RQ Queue Source: https://github.com/frappe/rq/blob/v1.14/README.md Initialize an RQ queue instance, connecting to a Redis server. Ensure Redis is running on the default host and port. ```python from redis import Redis from rq import Queue queue = Queue(connection=Redis()) ``` -------------------------------- ### Define an Asynchronous Function Source: https://github.com/frappe/rq/blob/v1.14/README.md Define a Python function that will be executed asynchronously. This example fetches a URL and counts words. ```python import requests def count_words_at_url(url): """Just an example function that's called async.""" resp = requests.get(url) return len(resp.text.split()) ``` -------------------------------- ### Using Different Connections for Multiple Queues Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/connections.md Demonstrates how to create and use distinct Redis connections for different RQ queues by passing unique connection objects to their constructors. ```python from rq import Queue from redis import Redis conn1 = Redis('localhost', 6379) conn2 = Redis('remote.host.org', 9836) q1 = Queue('foo', connection=conn1) q2 = Queue('bar', connection=conn2) ``` -------------------------------- ### Pushing and Popping Connections for Testing Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/connections.md Shows how to use `push_connection()` and `pop_connection()` to manage Redis connections, particularly useful for setting up unit tests when `with` statements are not feasible. ```python import unittest from rq import Queue from rq import push_connection, pop_connection class MyTest(unittest.TestCase): def setUp(self): push_connection(Redis()) def tearDown(self): pop_connection() def test_foo(self): """Any queues created here use local Redis.""" q = Queue() ``` -------------------------------- ### Create and enqueue a job directly Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Creates a job instance directly using `Job.create()` and then enqueues it. Useful for more control over job creation before enqueuing. ```python from rq.job import Job job = Job.create(count_words_at_url, 'http://nvie.com') print('Job id: %s' % job.id) q.enqueue_job(job) ``` -------------------------------- ### Enqueueing Functions with JSON Keyword Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Demonstrates enqueuing a function with a keyword argument provided as a JSON string. ```bash rq enqueue path.to.func 'key:=: { "json": "abc" }' ``` -------------------------------- ### Organize Workers by Queue Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/monitoring.md Use the `-R` or `--by-queue` flag with `rq info` to organize the output by queue instead of by worker. ```console $ rq info -R ... high: Bricktop.25458 (busy), Mickey.26421 (idle), Turkish.25812 (busy) low: Bricktop.25458 (busy) default: Bricktop.25458 (busy), Mickey.26421 (idle), Turkish.25812 (busy) failed: – 3 workers, 4 queues ``` -------------------------------- ### Enqueueing Functions with Keyword Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Shows how to enqueue a function with a keyword argument specified as plain text. ```bash rq enqueue path.to.func abc=def ``` -------------------------------- ### Enqueueing Functions with Plain Text Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Demonstrates how to enqueue a function with a plain text string argument. ```bash rq enqueue path.to.func abc ``` -------------------------------- ### Queue constructor with async bypass Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Shows how to instantiate a Queue with an optional async=False argument to bypass the worker for testing purposes. ```python Queue(async=False) ``` -------------------------------- ### Enqueueing Functions with File Path Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Demonstrates enqueuing a function where an argument is read from a file path. ```bash rq enqueue path.to.func @path/to/file ``` ```bash rq enqueue path.to.func key=@path/to/file ``` -------------------------------- ### Queue Initialization and Job Count Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Initializes a Redis connection and an RQ Queue, then retrieves the number of queued jobs. ```python from rq import Queue from redis import Redis redis_conn = Redis() q = Queue(connection=redis_conn) # Getting the number of jobs in the queue # Note: Only queued jobs are counted, not including deferred ones print(len(q)) ``` -------------------------------- ### Define a background task function Source: https://github.com/frappe/rq/blob/v1.14/docs/index.md Define a typical lengthy or blocking function that can be executed in the background. This example uses the requests library to count words at a given URL. ```python import requests def count_words_at_url(url): resp = requests.get(url) return len(resp.text.split()) ``` -------------------------------- ### Access Documentation Locally Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Specifies the local URL to access the documentation served from the Vagrant environment. This URL uses the host port configured for forwarding. ```text http://127.0.0.1:4001 ``` -------------------------------- ### Display All Queues and Workers Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/monitoring.md Use `rq info` to display all existing queues, active workers, and the total number of jobs. ```console $ rq info high |██████████████████████████ 20 low |██████████████ 12 default |█████████ 8 3 queues, 45 jobs total Bricktop.19233 idle: low Bricktop.19232 idle: high, default, low Bricktop.18349 idle: default 3 workers, 3 queues ``` -------------------------------- ### Get Job Position in Queue Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Determine a job's position within a queue using `job.get_position()` or `q.get_job_position(job)`. Note that these methods can be inefficient for very large queues. ```python from rq import Queue from redis import Redis from hello import say_hello redis_conn = Redis() q = Queue(connection=redis_conn) job = q.enqueue(say_hello) job2 = q.enqueue(say_hello) job2.get_position() # returns 1 q.get_job_position(job) # return 0 ``` -------------------------------- ### Shortcut for Latest Return Value Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/results.md Use `job.return_value()` as a convenient shortcut to get the return value of the latest successful job execution. Returns None if the latest result is not successful. ```python job = Job.fetch(id='my_id', connection=redis) print(job.return_value()) # Shortcut for job.latest_result().return_value ``` -------------------------------- ### Get Latest Job Result Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/results.md Fetch a job by its ID and retrieve its latest execution result. Check the result type and access the return value or exception string accordingly. ```python job = Job.fetch(id='my_id', connection=redis) result = job.latest_result() # returns Result(id=uid, type=SUCCESSFUL) if result == result.Type.SUCCESSFUL: print(result.return_value) else: print(result.exc_string) ``` -------------------------------- ### Enqueueing Functions with JSON Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Illustrates enqueuing a function with a JSON-formatted string argument. ```bash rq enqueue path.to.func ':{ "json": "abc" }' ``` -------------------------------- ### Emptying and Deleting a Queue Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Shows how to remove all jobs from a queue and how to delete the queue itself. ```python # Emptying a queue, this will delete all jobs in this queue q.empty() # Deleting a queue q.delete(delete_jobs=True) # Passing in `True` will remove all jobs in the queue # queue is now unusable. It can be recreated by enqueueing jobs to it. ``` -------------------------------- ### Run RQ Tasks in Unit Tests with SimpleWorker Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/testing.md Use SimpleWorker to avoid fork() issues with in-memory databases in testing frameworks. This snippet shows how to enqueue a job and run it immediately. ```python from redis import Redis from rq import SimpleWorker, Queue queue = Queue(connection=Redis()) queue.enqueue(my_long_running_job) worker = SimpleWorker([queue], connection=queue.connection) worker.work(burst=True) # Runs enqueued job # Check for result... ``` -------------------------------- ### Manual Redis Connection for RQ Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/index.md Provides an alternative method to manually establish a Redis connection if the `from_url` function fails to parse credentials. Details for host, password, port, and SSL settings can be found on the Redis add-on's settings page on Heroku. ```python conn = redis.Redis( host=host, password=password, port=port, ssl=True, ssl_cert_reqs=None ) ``` -------------------------------- ### Enqueueing Functions with JSON File Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Shows how to enqueue a function where a JSON argument is loaded from a file. ```bash rq enqueue path.to.func :@path/to/file.json ``` ```bash rq enqueue path.to.func key:=@path/to/file.json ``` -------------------------------- ### Configure RQ Worker with Sentry Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Integrate RQ with Sentry for runtime exception collection. The 'sync+' prefix is required for the SENTRY_DSN. ```python SENTRY_DSN = 'sync+http://public:secret@example.com/1' ``` -------------------------------- ### Requeuing All Failed Jobs via CLI Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Shows how to use the RQ CLI to requeue all jobs currently in a specified queue's failed job registry. This is a convenient way to clear the entire failed job list. ```console # This command will requeue all jobs in myqueue's failed job registry rq requeue --queue myqueue -u redis://localhost:6379 --all ``` -------------------------------- ### Passing Redis Connection to RQ Queue Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/connections.md Instantiate a Redis connection and pass it to the RQ Queue constructor. This is the recommended method for managing connections. ```python from redis import Redis from rq import Queue redis = Redis('localhost', 6789) q = Queue(connection=redis) ``` -------------------------------- ### Using Connection Context Manager (Deprecated) Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/connections.md Illustrates the deprecated use of the `Connection` context manager to temporarily override the default Redis connection for newly created RQ objects within the context. Note: This method is deprecated and explicit connection management is preferred. ```python from rq import Queue, Connection from redis import Redis with Connection(Redis('localhost', 6379)): q1 = Queue('foo') with Connection(Redis('remote.host.org', 9836)): q2 = Queue('bar') q3 = Queue('qux') assert q1.connection != q2.connection assert q2.connection != q3.connection assert q1.connection == q3.connection ``` -------------------------------- ### Run Jobs Instantly in Unit Tests with Fake Redis Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/testing.md Use `is_async=False` with a fake Redis instance to run jobs synchronously in the same thread without needing a separate Redis server. This is useful for testing. ```python from fakeredis import FakeStrictRedis from rq import Queue queue = Queue(is_async=False, connection=FakeStrictRedis()) job = queue.enqueue(my_long_running_job) assert job.is_finished ``` -------------------------------- ### Specify Worker Settings Module Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Use the -c option to specify the module from which RQ should read settings. ```console $ rq worker -c settings ``` -------------------------------- ### RQ Worker with explicit connection Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Illustrates how workers can accept explicit Redis connections, similar to Queues. ```python worker = Worker(connection=redis_conn) ``` -------------------------------- ### Enqueuing Jobs with Retries Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/exceptions.md Demonstrates how to configure job retries using the Retry object, specifying maximum retries and intervals between them. ```python from redis import Redis from rq import Retry, Queue from somewhere import my_func queue = Queue(connection=redis) # Retry up to 3 times, failed job will be requeued immediately queue.enqueue(my_func, retry=Retry(max=3)) # Retry up to 3 times, with 60 seconds interval in between executions queue.enqueue(my_func, retry=Retry(max=3, interval=60)) # Retry up to 3 times, with longer interval in between retries queue.enqueue(my_func, retry=Retry(max=3, interval=[10, 30, 60])) ``` -------------------------------- ### Create Queue or Job with Custom Serializer Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md When initializing a `Queue` or `Job` instance, you can specify a custom serializer by passing a class with `loads` and `dumps` methods. The default serializer is `pickle`. ```python from rq import Queue from rq.job import Job from rq.serializers import JSONSerializer job = Job(connection=connection, serializer=JSONSerializer) queue = Queue(connection=connection, serializer=JSONSerializer) ``` -------------------------------- ### Run RQ worker with scheduler Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/scheduling.md To enable the built-in scheduler, run the RQ worker with the `--with-scheduler` flag. ```console $ rq worker --with-scheduler ``` -------------------------------- ### Create a job with a predetermined ID directly Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Creates a job instance with a specific ID using `Job.create()` and then enqueues it. ```python # create a job with a predetermined id job = Job.create(count_words_at url, 'http://nvie.com', id='my_job_id') ``` -------------------------------- ### Registering Custom Handler and Disabling Default Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/exceptions.md This snippet demonstrates registering a custom exception handler and disabling the default one. ```python from exception_handlers import foo_handler w = Worker([q], exception_handlers=[foo_handler], disable_default_exception_handler=True) ``` -------------------------------- ### Configure Sentry via Config File Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/sentry.md Set the SENTRY_DSN variable in your RQ config file and run the worker with the -c flag. ```python SENTRY_DSN = 'https://my-dsn@sentry.io/123' ``` ```console rq worker -c my_settings ``` -------------------------------- ### Schedule jobs with RQ Source: https://github.com/frappe/rq/blob/v1.14/docs/index.md Schedule jobs to run at a specific time or after a delay. Use `enqueue_at` for a fixed time and `enqueue_in` for a relative delay. ```python # Schedule job to run at 9:15, October 10th job = queue.enqueue_at(datetime(2019, 10, 8, 9, 15), say_hello) # Schedule job to be run in 10 seconds job = queue.enqueue_in(timedelta(seconds=10), say_hello) ``` -------------------------------- ### Configure Port Forwarding in Vagrantfile Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/vagrant.md Adds a port forwarding rule to the Vagrantfile to make the documentation server accessible from the host machine. This allows local access to the served docs. ```ruby config.vm.network "forwarded_port", guest: 4000, host: 4001 ``` -------------------------------- ### Systemd Unit File for RQ Worker Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/systemd.md This is a systemd service unit file template for running an RQ worker. It defines the worker's description, dependencies, execution environment, and restart behavior. Customize paths and configuration as needed. ```systemd [Unit] Description=RQ Worker Number %i After=network.target [Service] Type=simple WorkingDirectory=/path/to/working_directory Environment=LANG=en_US.UTF-8 Environment=LC_ALL=en_US.UTF-8 Environment=LC_LANG=en_US.UTF-8 ExecStart=/path/to/rq worker -c config.py ExecReload=/bin/kill -s HUP $MAINPID ExecStop=/bin/kill -s TERM $MAINPID PrivateTmp=true Restart=always [Install] WantedBy=multi-user.target ``` -------------------------------- ### Setting Default Job Timeout for Queue Instances Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/results.md Demonstrates how to configure a default job timeout for all jobs enqueued on a specific queue instance. Individual jobs can still override this default. ```python # High prio jobs should end in 8 secs, while low prio # work may take up to 10 mins high = Queue('high', default_timeout=8) # 8 secs low = Queue('low', default_timeout=600) # 10 mins # Individual jobs can still override these defaults low.enqueue(really_really_slow, job_timeout=3600) # 1 hr ``` -------------------------------- ### Enqueueing Functions with Literal Eval Keyword Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Illustrates enqueuing a function with keyword arguments parsed using ast.literal_eval. ```bash rq enqueue path.to.func 'key%= (1, 2)' ``` ```bash rq enqueue path.to.func 'key%={ "foo": True }' ``` -------------------------------- ### Multiple Job Dependencies Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Chain a job to execute after multiple other jobs have completed successfully by providing a list to `depends_on`. ```python queue = Queue('low', connection=redis) foo_job = queue.enqueue(foo) bar_job = queue.enqueue(bar) baz_job = queue.enqueue(baz, depends_on=[foo_job, bar_job]) ``` -------------------------------- ### Enqueueing and Verifying a Failed Job Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Demonstrates how a job that raises an exception is automatically placed in the FailedJobRegistry. Asserts that the registry contains one failed job after worker execution. ```python from redis import Redis from rq import Queue from rq.job import Job def div_by_zero(x): return x / 0 connection = Redis() queue = Queue(connection=connection) job = queue.enqueue(div_by_zero, 1) registry = queue.failed_job_registry worker = Worker([queue]) worker.work(burst=True) assert len(registry) == 1 # Failed jobs are kept in FailedJobRegistry ``` -------------------------------- ### Requeuing Specific Failed Jobs via CLI Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Demonstrates using the RQ CLI to requeue specific failed jobs by their IDs from a named queue. Requires specifying the queue name and Redis connection URL. ```console # This will requeue foo_job_id and bar_job_id from myqueue's failed job registry rq requeue --queue myqueue -u redis://localhost:6379 foo_job_id bar_job_id ``` -------------------------------- ### Run tests with coverage and specific report format Source: https://github.com/frappe/rq/blob/v1.14/docs/contrib/testing.md Execute tests with coverage enabled and generate reports in a specified format (term, html, xml). This command also includes slow tests and shows test durations. ```bash RUN_SLOW_TESTS_TOO=1 pytest --cov=./ --cov-report={{report_format}} --durations=5 ``` -------------------------------- ### Configure Sentry via CLI Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/sentry.md Use the --sentry-dsn argument when invoking the rqworker script to send exceptions to Sentry. ```console rq worker --sentry-dsn https://my-dsn@sentry.io/123 ``` -------------------------------- ### RQ Worker with Sentry DSN Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Shows how to enable Sentry as the exception handler for rqworker using the --sentry-dsn flag. ```bash rqworker --sentry-dsn="http://public:secret@example.com/1" ``` -------------------------------- ### Calling a decorated job Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Illustrates how to call a function decorated with @job using the .delay() method. ```python from foo.bar import some_work some_work.delay(2, 3) ``` -------------------------------- ### Programmatically Requeuing Failed Jobs Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Illustrates how to retrieve job IDs from the FailedJobRegistry and requeue them back into their original queues. Verifies that the registry is empty after successful requeuing. ```python from redis import Redis from rq import Queue connection = Redis() queue = Queue(connection=connection) registry = queue.failed_job_registry # This is how to get jobs from FailedJobRegistry for job_id in registry.get_job_ids(): registry.requeue(job_id) # Puts job back in its original queue assert len(registry) == 0 # Registry will be empty when job is requeued ``` -------------------------------- ### Enqueueing Functions with Literal Eval Arguments Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Shows how to enqueue a function with arguments parsed using Python's ast.literal_eval. ```bash rq enqueue path.to.func %1, 2 ``` ```bash rq enqueue path.to.func %None ``` ```bash rq enqueue path.to.func %True ``` -------------------------------- ### CLI Enqueue Command Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Enqueue jobs from the command line using the `rq enqueue` command followed by the function name and its arguments. ```bash rq enqueue [OPTIONS] FUNCTION [ARGUMENTS] ``` -------------------------------- ### Specify Custom Job and Queue Classes Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/workers.md Use --job-class and --queue-class options to specify custom classes for jobs and queues. Ensure these classes are also used when enqueueing jobs. ```console $ rq worker --job-class 'custom.JobClass' --queue-class 'custom.QueueClass' ``` -------------------------------- ### Handling Job Failures with Dependencies Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Use the `Dependency` class to control how dependent jobs execute when their predecessors fail. `allow_failure=True` ensures the dependent job runs even if the dependency fails. `enqueue_at_front=True` prioritizes the dependent job. ```python from redis import Redis from rq.job import Dependency from rq import Queue queue = Queue(connection=Redis()) job_1 = queue.enqueue(div_by_zero) dependency = Dependency( jobs=[job_1], allow_failure=True, # allow_failure defaults to False enqueue_at_front=True # enqueue_at_front defaults to False ) job_2 = queue.enqueue(say_hello, depends_on=dependency) """ job_2 will execute even though its dependency (job_1) fails, and it will be enqueued at the front of the queue. """ ``` -------------------------------- ### Supervisor configuration for RQ worker Source: https://github.com/frappe/rq/blob/v1.14/docs/patterns/supervisor.md Standard supervisor configuration for running an RQ worker. Ensure the command points to the correct rq executable and includes a settings module for worker configuration. The directory should be where your source code is importable from. ```ini [program:myworker] ; Point the command to the specific rq command you want to run. ; If you use virtualenv, be sure to point it to ; /path/to/virtualenv/bin/rq ; Also, you probably want to include a settings module to configure this ; worker. For more info on that, see http://python-rq.org/docs/workers/ command=/path/to/rq worker -c mysettings high default low ; process_num is required if you specify >1 numprocs process_name=%(program_name)s-%(process_num)s ; If you want to run more than one worker instance, increase this numprocs=1 ; This is the directory from which RQ is ran. Be sure to point this to the ; directory where your source code is importable from directory=/path/to ; RQ requires the TERM signal to perform a warm shutdown. If RQ does not die ; within 10 seconds, supervisor will forcefully kill it stopsignal=TERM ; These are up to you autostart=true autorestart=true ``` -------------------------------- ### Query Specific Queues Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/monitoring.md Filter the output of `rq info` to display only specific queues by listing their names. ```console $ rq info high default high |██████████████████████████ 20 default |█████████ 8 2 queues, 28 jobs total Bricktop.19232 idle: high, default Bricktop.18349 idle: default 2 workers, 2 queues ``` -------------------------------- ### Basic Job Callbacks Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Register functions to be executed upon job completion or failure using `on_success` and `on_failure` arguments. ```python queue.enqueue(say_hello, on_success=report_success, on_failure=report_failure) ``` -------------------------------- ### Registering Custom Exception Handlers Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/exceptions.md Shows how to register custom exception handlers for RQ workers, providing a function that accepts job and exception details. ```python from exception_handlers import foo_handler, bar_handler w = Worker([q], exception_handlers=[foo_handler, bar_handler]) ``` -------------------------------- ### Configure job retries Source: https://github.com/frappe/rq/blob/v1.14/docs/index.md Set up automatic retries for failed jobs. You can specify the maximum number of retries and the intervals between them. ```python from rq import Retry # Retry up to 3 times, failed job will be requeued immediately queue.enqueue(say_hello, retry=Retry(max=3)) # Retry up to 3 times, with configurable intervals between retries queue.enqueue(say_hello, retry=Retry(max=3, interval=[10, 30, 60])) ``` -------------------------------- ### Accessing Job Status Source: https://github.com/frappe/rq/blob/v1.14/CHANGES.md Illustrates how to check the status of a job using the 'status' property or boolean accessor properties. ```python job.status job.is_queued job.is_finished job.is_failed ``` -------------------------------- ### Enable Interval Polling Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/monitoring.md Use the `--interval` flag with `rq info` to continuously update the statistics at a specified interval in seconds. ```console $ rq info --interval 1 ``` ```console $ rq info --interval 0.5 ``` -------------------------------- ### Registering Work Horse Killed Handler Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/exceptions.md Shows how to set a specific handler for unexpected workhorse terminations on an RQ worker. ```python from my_handlers import my_work_horse_killed_handler w = Worker([q], work_horse_killed_handler=my_work_horse_killed_handler) ``` -------------------------------- ### Bulk Job Enqueueing with Pipeline Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/index.md Enqueue multiple jobs using `enqueue_many` with an explicitly provided Redis pipeline. Ensure to execute the pipeline after enqueuing. ```python with q.connection.pipeline() as pipe: jobs = q.enqueue_many( [ Queue.prepare_data(count_words_at_url, ('http://nvie.com',), job_id='my_job_id'), Queue.prepare_data(count_words_at_url, ('http://nvie.com',), job_id='my_other_job_id'), ], pipeline=pipe ) pipe.execute() ``` -------------------------------- ### Retry Failed Jobs Immediately Source: https://github.com/frappe/rq/blob/v1.14/README.md Configure a job to automatically retry up to a maximum number of times if it fails. Retries occur immediately by default. ```python from rq import Retry # Retry up to 3 times, failed job will be requeued immediately queue.enqueue(say_hello, retry=Retry(max=3)) ``` -------------------------------- ### Schedule a Job for a Specific Time Source: https://github.com/frappe/rq/blob/v1.14/README.md Schedule a job to be executed at a precise date and time. Requires importing `datetime`. ```python # Schedule job to run at 9:15, October 10th job = queue.enqueue_at(datetime(2019, 10, 10, 9, 15), say_hello) ``` -------------------------------- ### Removing Jobs from a Registry Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/job_registries.md Illustrates how to remove jobs from a registry using the `remove()` method. It demonstrates removing a job ID and also removing a job ID while simultaneously deleting the job from storage. ```python from redis import Redis from rq import Queue from rq.registry import FailedJobRegistry redis = Redis() queue = Queue(connection=redis) registry = FailedJobRegistry(queue=queue) # This is how to remove a job from a registry for job_id in registry.get_job_ids(): registry.remove(job_id) # If you want to remove a job from a registry AND delete the job, # use `delete_job=True` for job_id in registry.get_job_ids(): registry.remove(job_id, delete_job=True) ``` -------------------------------- ### Custom Exception Handler with *exc_info Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/exceptions.md An alternative signature for a custom exception handler using *exc_info to capture exception details. ```python def my_handler(job, *exc_info): # do custom things here ``` -------------------------------- ### Set Job and Result Time-to-Live (TTL) Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/jobs.md Configure `result_ttl` to control how long successful job results are stored, and `ttl` to set the maximum time a job can remain queued before being discarded. ```python # When creating the job: job = Job.create(func=say_hello, result_ttl=600, # how long (in seconds) to keep the job (if successful) and its results ttl=43, # maximum queued time (in seconds) of the job before it's discarded. ) # or when queueing a new job: job = q.enqueue(count_words_at_url, 'http://nvie.com', result_ttl=600, # how long to keep the job (if successful) and its results ttl=43 # maximum queued time ) ``` -------------------------------- ### Iterate Through Multiple Job Results Source: https://github.com/frappe/rq/blob/v1.14/docs/docs/results.md Fetch a job by its ID and iterate through its historical execution results, printing the creation time and type for each. ```python job = Job.fetch(id='my_id', connection=redis) for result in job.results(): print(result.created_at, result.type) ``` -------------------------------- ### Schedule a Job for a Future Time Source: https://github.com/frappe/rq/blob/v1.14/README.md Schedule a job to be executed after a specified delay. Requires importing `timedelta`. ```python # Schedule job to run in 10 seconds job = queue.enqueue_in(timedelta(seconds=10), say_hello) ```