=============== LIBRARY RULES =============== From library maintainers: - The installed consumer command is huey_consumer; huey_consumer.py only exists in a source checkout. - Huey shuts down gracefully on SIGINT; SIGTERM interrupts running tasks. Supervisors must be configured with stopsignal=INT (supervisord) or KillSignal=SIGINT (systemd). - Task priorities on redis require PriorityRedisHuey (redis 5.0+); plain RedisHuey raises an error for prioritized tasks. SqliteHuey supports priorities natively. - For tests, use immediate=True - it executes tasks synchronously through the real execution path with in-memory storage. - When running multiple consumers of one queue, pass -n/--no-periodic to all but one, or periodic tasks run once per consumer. - Task functions are regular def functions; async def execution is not supported. Use huey.contrib.asyncio.aget_result to await results from async code. - Django: configure via settings.HUEY and run manage.py run_huey. The django.tasks backend is huey.contrib.djhuey.tasks_backend.HueyBackend. ### Huey Configuration: Basic Redis Setup Source: https://github.com/coleifer/huey/blob/master/docs/django.md Example configuration for a Huey instance using Redis running locally with four worker threads. ```python # Redis running locally with four worker threads. HUEY = { 'name': 'my-app', 'consumer': {'workers': 4, 'worker_type': 'thread'}, } ``` -------------------------------- ### Initializing shared resources with on_startup hook Source: https://github.com/coleifer/huey/blob/master/docs/shared_resources.md Example of using the `Huey.on_startup()` decorator to open a database connection when a worker starts, which can then be used by multiple tasks. ```python import peewee db = PostgresqlDatabase('my_app') @huey.on_startup() def open_db_connection(): # If for some reason the db connection appears to already be open, # close it first. if not db.is_closed(): db.close() db.connect() @huey.task() def run_query(n): db.execute_sql('select pg_sleep(%s)', (n,)) return n ``` -------------------------------- ### Install Huey from Source using Git Source: https://github.com/coleifer/huey/blob/master/docs/installation.md Clone the Huey repository and install the library directly from the source code. ```shell git clone https://github.com/coleifer/huey.git cd huey python setup.py install ``` -------------------------------- ### MiniHuey start() method Source: https://github.com/coleifer/huey/blob/master/docs/contrib.md Starts the MiniHuey scheduler in a new green thread. This is necessary for scheduling tasks in the future or running periodic tasks. It should typically be called when the application starts. ```APIDOC ## start() ### Description Start the scheduler in a new green thread. The scheduler is needed if you plan to schedule tasks for execution using the `schedule()` method, or if you want to run periodic tasks. Typically this method should be called when your application starts up. For example, a WSGI application might do something like: ```python # Always apply gevent monkey-patch before anything else! from gevent import monkey; monkey.patch_all() from my_app import app # flask/bottle/whatever WSGI app. from my_app import mini_huey # Start the scheduler. Returns immediately. mini_huey.start() # Run the WSGI server. from gevent.pywsgi import WSGIServer WSGIServer(('127.0.0.1', 8000), app).serve_forever() ``` ``` -------------------------------- ### on_startup Source: https://github.com/coleifer/huey/blob/master/docs/api.md A decorator for registering a function to be executed when a worker starts up. ```APIDOC ## on_startup(name=None) ### Description Register a startup hook. The callback will be executed whenever a worker comes online. Uncaught exceptions will be logged but will have no other effect on the overall operation of the worker. ### Parameters * **name** (*str*, optional) - Name for the hook. ### Returns A decorator used to wrap the actual on-startup function. ### Callback Signature The callback function must not accept any parameters. ### Usage Example ```python db_connection = None @huey.on_startup() def setup_db_connection(): global db_connection db_connection = psycopg2.connect(database='my_db') @huey.task() def write_data(rows): cursor = db_connection.cursor() # ... ``` ``` -------------------------------- ### Python Import Example Source: https://github.com/coleifer/huey/blob/master/docs/consumer.md Demonstrates the Python import path format used to reference a Huey instance within an application. ```python >>> from blog.main import huey ``` -------------------------------- ### Run Huey Consumer Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Starts the Huey consumer process. Specify the import path to your Huey instance. ```shell huey_consumer demo.huey ``` -------------------------------- ### Install Redis Dependency Source: https://github.com/coleifer/huey/blob/master/docs/installation.md Install the redis-py library if you plan to use Redis as your task storage backend. ```shell pip install redis ``` -------------------------------- ### Install Huey Base Package Source: https://github.com/coleifer/huey/blob/master/docs/installation.md Install the core Huey package using pip. This is the primary installation step. ```shell pip install huey ``` -------------------------------- ### Basic Huey Task Setup Source: https://github.com/coleifer/huey/blob/master/docs/api.md Demonstrates how to create a RedisHuey instance, define a task using a decorator, and a periodic task. Tasks are enqueued for execution by a consumer. ```python from huey import RedisHuey # Create a huey instance. huey = RedisHuey('my-app') @huey.task() def add_numbers(a, b): return a + b @huey.periodic_task(crontab(minute='0', hour='2')) def nightly_report(): generate_nightly_report() ``` -------------------------------- ### Run Huey Consumer Source: https://github.com/coleifer/huey/blob/master/docs/consumer.md Start the Huey consumer by providing the import path to your Huey instance and optionally specifying a log file. ```shell huey_consumer blog.main.huey --logfile=../logs/huey.log ``` -------------------------------- ### Example Usage Source: https://github.com/coleifer/huey/blob/master/docs/mini.md Demonstrates how to execute tasks asynchronously and retrieve their results. ```python # Executes the task asynchronously in a new greenlet. result = fetch_url('https://google.com/') # Wait for the task to finish. html = result.get() ``` -------------------------------- ### systemd Installation and Service Management Commands Source: https://github.com/coleifer/huey/blob/master/docs/deployment.md Commands to install the huey systemd service file and manage the consumer process. ```shell sudo cp huey.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now huey ``` -------------------------------- ### Run Huey Consumers Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md Command-line examples for running Huey consumers for specific queues with custom worker counts and kinds. ```shell huey_consumer myapp.queues.emails -w 8 -k greenlet huey_consumer myapp.queues.reports -w 2 -k process ``` -------------------------------- ### Install Postgres Dependency Source: https://github.com/coleifer/huey/blob/master/docs/installation.md Install the psycopg library (version 3.2+) if you intend to use PostgreSQL for task storage. ```shell pip install psycopg ``` -------------------------------- ### Starting the Scheduler in a WSGI Application Source: https://github.com/coleifer/huey/blob/master/docs/mini.md Illustrates how to start the MiniHuey scheduler within a WSGI application, typically called at application startup. ```python # Always apply gevent monkey-patch before anything else! from gevent import monkey; monkey.patch_all() from my_app import app # flask/bottle/whatever WSGI app. from my_app import mini_huey # Start the scheduler. Returns immediately. mini_huey.start() # Run the WSGI server. from gevent.pywsgi import WSGIServer WSGIServer(('127.0.0.1', 8000), app).serve_forever() ``` -------------------------------- ### Immediate Mode Task Execution Example Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Demonstrates synchronous task execution and revocation in immediate mode within the Python shell. No separate consumer or Redis server is needed when immediate mode is enabled. ```pycon >>> from huey import RedisHuey >>> huey = RedisHuey() >>> huey.immediate = True >>> @huey.task() ... def add(a, b): ... return a + b ... >>> result = add(1, 2) >>> result() 3 >>> add.revoke(revoke_once=True) # We can revoke tasks. >>> result = add(2, 3) >>> result() is None True >>> add(3, 4)() # No longer revoked, was restored automatically. 7 ``` -------------------------------- ### Run Tests from Source Source: https://github.com/coleifer/huey/blob/master/docs/installation.md Execute the test suite after installing Huey from the source repository. ```shell python setup.py test ``` -------------------------------- ### Run Huey Consumer with Greenlet Workers Source: https://github.com/coleifer/huey/blob/master/docs/consumer.md Starts the Huey consumer using the greenlet worker type with 16 workers. Ensure gevent is installed and understand its cooperative multitasking model before using this. ```shell huey_consumer main.huey -k greenlet -w 16 ``` -------------------------------- ### Run Huey Consumer (Multi-process) Source: https://github.com/coleifer/huey/blob/master/README.rst Start the Huey consumer with four worker processes using the 'process' kind. ```console $ huey_consumer my_app.huey -k process -w 4 ``` -------------------------------- ### Huey Schedule Shorthand Examples Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Demonstrates equivalent ways to schedule a task using delay (integer, timedelta) or ETA (datetime). ```python # These are all equivalent: add.schedule(args=(1, 2), delay=60) add.schedule(args=(1, 2), delay=timedelta(seconds=60)) add.schedule(args=(1, 2), eta=datetime.now() + timedelta(seconds=60)) ``` -------------------------------- ### Enable and Start Huey Systemd Services Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md Command to enable and start Huey consumers managed by systemd for specific queues. ```shell systemctl enable --now huey@emails huey@reports ``` -------------------------------- ### Configure Multiple Queues with SQLite Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md Example of configuring multiple Huey queues using the SQLite backend, ensuring distinct names for shared databases. ```python emails = SqliteHuey('emails', filename='/var/lib/myapp/huey.db') reports = SqliteHuey('reports', filename='/var/lib/myapp/huey.db') ``` -------------------------------- ### Signals and Immediate Mode Example Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Demonstrates how signals function in immediate mode for easier testing. ```python huey.immediate = True state = [] @huey.signal(SIGNAL_COMPLETE) def on_complete(signal, task): state.append(task.id) result = add(1, 2) # Executes immediately, fires signals. assert len(state) == 1 assert state[0] == result.id ``` -------------------------------- ### Pre and post execute hooks example Source: https://github.com/coleifer/huey/blob/master/docs/shared_resources.md Example demonstrating the usage of `Huey.pre_execute()` to cancel task execution and `Huey.post_execute()` to log task outcomes. ```python from huey import CancelExecution @huey.pre_execute() def pre_execute_hook(task): # Pre-execute hooks are passed the task that is about to be run. # This pre-execute task will cancel the execution of every task if the # current day is Sunday. if datetime.datetime.now().weekday() == 6: raise CancelExecution('No tasks on sunday!') @huey.post_execute() def post_execute_hook(task, task_value, exc): # Post-execute hooks are passed the task, the return value (if the task # succeeded), and the exception (if one occurred). if exc is not None: print('Task "%s" failed with error: %s!' % (task.id, exc)) ``` -------------------------------- ### Running the Huey Consumer Source: https://github.com/coleifer/huey/blob/master/docs/api.md Command to start the Huey consumer process with a specified number of worker threads. ```console $ huey_consumer demo.huey -w 4 ``` -------------------------------- ### Start MiniHuey Scheduler for WSGI Apps Source: https://github.com/coleifer/huey/blob/master/docs/contrib.md Apply gevent monkey-patching before anything else. Call mini_huey.start() when your application initializes to enable scheduled and periodic tasks. This method returns immediately. ```python # Always apply gevent monkey-patch before anything else! from gevent import monkey; monkey.patch_all() from my_app import app # flask/bottle/whatever WSGI app. from my_app import mini_huey # Start the scheduler. Returns immediately. mini_huey.start() # Run the WSGI server. from gevent.pywsgi import WSGIServer WSGIServer(('127.0.0.1', 8000), app).serve_forever() ``` -------------------------------- ### Run Huey Consumer with Django Source: https://github.com/coleifer/huey/blob/master/docs/django.md This is the basic command to start the Huey consumer within your Django project. It automatically discovers and loads task modules from your `INSTALLED_APPS`. ```shell ./manage.py run_huey ``` -------------------------------- ### Schedule Dynamic Periodic Tasks (Interactive) Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md These examples show how to invoke the `schedule_message` task to set up new periodic tasks with different schedules. The consumer must be running for these tasks to be registered and executed. ```python >>> from demo import schedule_message >>> schedule_message('I run every 5 minutes', '*/5') >>> schedule_message('I run between 0-15 and 30-45', '0-15,30-45') ``` -------------------------------- ### Huey Consumer Process Management Source: https://github.com/coleifer/huey/blob/master/docs/deployment.md Example of a Procfile for running a Huey consumer with gunicorn for web processes. Ensures tasks are handled gracefully during deploys. ```Procfile web: gunicorn my_app.wsgi worker: huey_consumer my_app.huey -w 4 -k thread ``` -------------------------------- ### Schedule a Task with Default and Overridden Priority Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Schedule a task to run in the future. The first example uses the default priority, while the second overrides it to 50. ```python # Uses priority=10, since that was the default we used when # declaring the send_email task: send_email.schedule(('foo@bar.com', 'subj', 'msg'), delay=60) # Override, specifying priority=50 for this task. send_email.schedule(('bar@foo.com', 'subj', 'msg'), delay=60, priority=50) ``` -------------------------------- ### Logrotate Configuration for Huey Source: https://github.com/coleifer/huey/blob/master/docs/deployment.md Example logrotate configuration for managing Huey's log files. Uses copytruncate to handle logs held open by the consumer. ```text # /etc/logrotate.d/huey /var/log/huey.log { weekly rotate 8 compress copytruncate missingok } ``` -------------------------------- ### Unit Testing Huey Tasks Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Provides examples for unit testing Huey tasks, including setting immediate mode, calling tasks locally, and testing for exceptions and side effects. ```python # Consider the following Huey instance and tasks: huey = RedisHuey(...) # Fake database to check side-effects. database = [] @huey.task() def add(a, b): return a + b @huey.periodic_task(crontab(...)) def run_reports(): global database database.append(True) # Simulate a side-effect. # The return values for periodic tasks are discarded when run by the # Huey consumer. A return value may be helpful for testing, however, so # in this example we will return something to demonstrate how to test # our periodic task. return 42 class TestMyTasks(unittest.TestCase): def setUp(self): # Make tasks run synchronously. huey.immediate = True def tearDown(self): huey.immediate = False def test_task(self): result_handle = add(3, 4) self.assertEqual(result_handle.get(), 7) # Alternatively, you can also: self.assertEqual(add.call_local(3, 4), 7) def test_task_exceptions(self): # We can also test exceptions. result_handle = add(3, None) # Exception logged, but not raised. with self.assertRaises(TaskException): result_handle.get() with self.assertRaises(TypeError): # Exception raised directly when using call_local(). add.call_local(3, None) def test_periodic_task(self): # We cannot use the result-handle from a periodic task, because the # results are always discarded by the consumer. In this case it is # necessary to use `.call_local()` if we want to check the return # value. self.assertEqual(run_reports.call_local(), 42) self.assertTrue(len(database), 1) # If our periodic task has a side-effect, however, we can call it # normally and check the side-effect happened. For example, if the # run_reports() periodic task wrote a row to a database, we could # do something like: run_reports() self.assertTrue(len(database), 2) ``` -------------------------------- ### Run Huey Consumer with Threads and Logfile Source: https://github.com/coleifer/huey/blob/master/docs/consumer.md Starts the Huey consumer with 8 worker threads and logs errors to a specified file. Use this for general-purpose task processing where I/O is common. ```shell huey_consumer my.app.huey -l /var/log/app.huey.log -w 8 -q ``` -------------------------------- ### Declare Multiple Huey Instances for Different Queues Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md This example shows how to configure multiple Huey instances, each connected to a Redis backend via a shared connection pool. This allows for routing tasks to different queues based on their purpose, such as emails or reports. ```python # myapp/queues.py from huey import RedisHuey from redis import ConnectionPool pool = ConnectionPool(host='localhost', port=6379, max_connections=20) emails = RedisHuey('myapp_emails', connection_pool=pool) reports = RedisHuey('myapp_reports', connection_pool=pool) ``` -------------------------------- ### Run Huey Consumer with Greenlets and Custom Scheduler Source: https://github.com/coleifer/huey/blob/master/docs/consumer.md Starts the Huey consumer with 50 greenlet workers, no health checking, and a scheduler granularity of 60 seconds. This is ideal for I/O-bound tasks when using gevent. ```shell huey_consumer my.app.huey -w 50 -k greenlet -C -s 60 ``` -------------------------------- ### Configure Huey via Django Settings Source: https://github.com/coleifer/huey/blob/master/docs/contrib.md Configure Huey and its consumer settings using the settings.HUEY dictionary in your Django settings.py. This example shows all supported options with their default values. ```python # settings.py HUEY = { 'huey_class': 'huey.RedisHuey', # Huey implementation to use. 'name': settings.DATABASES['default']['NAME'], # Use db name for huey. 'results': True, # Store return values of tasks. 'store_none': False, # If a task returns None, do not save to results. 'immediate': settings.DEBUG, # If DEBUG=True, run synchronously. 'utc': True, # Use UTC for all times internally. 'blocking': True, # Perform blocking pop rather than poll Redis. 'connection': { 'host': 'localhost', 'port': 6379, 'db': 0, 'connection_pool': None, # Definitely you should use pooling! # ... tons of other options, see redis-py for details. # huey-specific connection parameters. 'read_timeout': 1, # If not polling (blocking pop), use timeout. 'url': None, # Allow Redis config via a DSN. }, 'consumer': { 'workers': 1, 'worker_type': 'thread', 'initial_delay': 0.1, # Smallest polling interval, same as -d. 'backoff': 1.15, # Exponential backoff using this rate, -b. 'max_delay': 10.0, # Max possible polling interval, -m. 'scheduler_interval': 1, # Check schedule every second, -s. 'periodic': True, # Enable crontab feature. 'check_worker_health': True, # Enable worker health checks. 'health_check_interval': 1, # Check worker health every second. }, } ``` -------------------------------- ### Configuring Immediate Mode on Initialization Source: https://github.com/coleifer/huey/blob/master/docs/api.md Shows how to initialize Huey with immediate mode enabled directly in the constructor. ```python huey = RedisHuey(immediate=True) ``` -------------------------------- ### Run Huey Consumer Single-Threaded with Logging to Stdout Source: https://github.com/coleifer/huey/blob/master/docs/consumer.md Starts the Huey consumer in single-threaded mode, disabling periodic task support. Logs are directed to standard output. Use this for simple debugging or minimal setups. ```shell huey_consumer my.app.huey -v -n ``` -------------------------------- ### Huey Class Initialization Source: https://github.com/coleifer/huey/blob/master/docs/api.md Demonstrates how to initialize a Huey instance with various configuration options, including name, result storage, serialization, compression, and immediate mode. ```APIDOC ## Huey Class Initialization ### Description Initializes a Huey instance with various configuration options. ### Parameters * **name** (str) - The name of the task queue. * **results** (bool) - Whether to store task results. * **store_none** (bool) - Whether to store `None` in the result store. * **utc** (bool) - Use UTC internally for datetime operations. * **immediate** (bool) - Enable synchronous task execution for debugging and testing. * **serializer** (Serializer) - Serializer implementation for tasks and result data. * **compression** (bool) - Compress tasks and result data. * **use_zlib** (bool) - Use zlib for compression instead of gzip. * **immediate_use_memory** (bool) - Automatically switch to in-memory storage when immediate mode is enabled. * **storage_kwargs** - Arbitrary keyword arguments passed to the storage backend. ``` -------------------------------- ### Install Gevent for Greenlet Workers Source: https://github.com/coleifer/huey/blob/master/docs/installation.md Install gevent if you need to use the greenlet worker type for IO-bound tasks. ```shell pip install gevent ``` -------------------------------- ### Task Pipelines Source: https://github.com/coleifer/huey/blob/master/docs/api.md Demonstrates how to create a pipeline of tasks where the output of one task becomes the input for the next. ```APIDOC ## Task Pipelines ### Description Create task execution pipelines using the `.then()` method on a task. This allows chaining tasks together, where each subsequent task receives the result of the previous one. ### Example ```python @huey.task() def add(a, b): return a + b add_task = add.s(1, 2) # Represent task invocation. pipeline = (add_task .then(add, 3) # Call add() with previous result and 3. .then(add, 4) # etc... .then(add, 5)) results = huey.enqueue(pipeline) # Print results of above pipeline. print(results.get(blocking=True)) # Expected output: [3, 6, 10, 15] ``` ``` -------------------------------- ### Example Tasks Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Defines two example tasks, 'add' and 'flaky_task', to demonstrate signal usage. ```python @huey.task() def add(a, b): return a + b @huey.task(retries=2, retry_delay=10) def flaky_task(): if random.randint(0, 1) == 0: raise ValueError('uh-oh') return 'OK' ``` -------------------------------- ### Flask Home Template Source: https://github.com/coleifer/huey/blob/master/examples/flask_ex/templates/home.html This HTML template is used in the Flask example to display messages and provide a form to enqueue tasks. ```html {% if message %} #### {{ message }} {% endif %} Enter a number Run task ``` -------------------------------- ### Immediate Mode with Live Storage Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Configure immediate mode to use live storage by setting `immediate_use_memory=False` during instance creation. By default, immediate mode uses in-memory storage to prevent accidental writes to live storage. ```python huey = RedisHuey('my-app', immediate_use_memory=False) ``` -------------------------------- ### Immediate Mode Configuration Source: https://github.com/coleifer/huey/blob/master/docs/api.md Explains how to enable and disable immediate mode for synchronous task execution, useful for testing and debugging. ```APIDOC ## Immediate Mode ### Description Immediate mode causes task-decorated functions to execute synchronously by the caller, useful for development and testing. When enabled, Huey typically switches to in-memory storage unless `immediate_use_memory=False` is specified. ### Enabling Immediate Mode ```python huey = RedisHuey() # Enable immediate mode. Tasks executed synchronously. huey.immediate = True # Disable immediate mode. huey.immediate = False ``` ### Initializing with Immediate Mode ```python huey = RedisHuey(immediate=True) ``` ``` -------------------------------- ### RedisHuey Initialization Source: https://github.com/coleifer/huey/blob/master/docs/api.md Configuration options for the RedisHuey storage implementation. ```APIDOC ## RedisHuey ### Description Huey implementation that utilizes redis for queue and result storage. Requires redis-py. ### Parameters * **blocking** (*bool*) – Use blocking-pop when reading from the queue (as opposed to polling). Default is true. * **connection_pool** – a redis-py `ConnectionPool` instance. * **url** – url for Redis connection. * **host** – hostname of the Redis server. * **port** – port number. * **password** – password for Redis. * **db** (*int*) – Redis database to use (typically 0-15, default is 0). * **notify_result** (*bool*) – use a blocking-pop on a result-ready key to enable low-latency result reading. * **notify_result_ttl** (*int*) – TTL for result-ready key to automatically expire un-awaited results. ### Notes * Does not support task priorities. Use `PriorityRedisHuey` for that. * Uses a Redis LIST for pending tasks. * `notify_result=True` enables low-latency result fetching via a blocking pop on a dedicated result list. * `notify_result_ttl` controls the expiration of the result-readiness list. ``` -------------------------------- ### Building Task Pipelines with .then() Source: https://github.com/coleifer/huey/blob/master/docs/api.md Demonstrates how to create a sequence of dependent tasks using the .then() method on a task instance, forming a pipeline for sequential execution. ```python add_task = add.s(1, 2) # Represent task invocation. pipeline = (add_task .then(add, 3) # Call add() with previous result and 3. .then(add, 4) # etc... .then(add, 5)) results = huey.enqueue(pipeline) # Print results of above pipeline. print(results.get(blocking=True)) ``` -------------------------------- ### Example of a Task that May Fail Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Illustrates a task that might fail and be retried automatically. ```python result = flaky_task() try: result.get(blocking=True) except TaskException: result.reset() result.get(blocking=True) # Try again if first time fails. ``` -------------------------------- ### Simple Task Execution Example Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Demonstrates a simple task execution and retrieval of its result. ```python result = add(1, 2) result.get(blocking=True) ``` -------------------------------- ### Chaining Addition Tasks with then() Source: https://github.com/coleifer/huey/blob/master/docs/api.md Demonstrates creating a task pipeline by chaining multiple 'add' tasks. The result of each task is passed as input to the next. ```python add_task = add.s(1, 2) # Represent task invocation. pipeline = (add_task .then(add, 3) # Call add() with previous result and 3. .then(add, 4) # etc... .then(add, 5)) result_group = huey.enqueue(pipeline) print(result_group.get(blocking=True)) # [3, 6, 10, 15] ``` -------------------------------- ### Configure Huey for Testing with immediate=True Source: https://github.com/coleifer/huey/blob/master/docs/troubleshooting.md Set Huey to use immediate mode for testing. When immediate=True, Huey defaults to an in-memory storage layer and executes tasks synchronously. ```python test_mode = os.environ.get('TEST_MODE') # When immediate=True, Huey will default to using an in-memory # storage layer. huey = RedisHuey(immediate=test_mode) # Alternatively, you can set the `immediate` attribute: huey.immediate = True if test_mode else False ``` -------------------------------- ### Get Queue Statistics Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md Retrieve counts of pending, scheduled, and result tasks in the queue for monitoring purposes. ```python def get_queue_stats(): return { 'pending': huey.pending_count(), 'scheduled': huey.scheduled_count(), 'results': huey.result_count(), } ``` -------------------------------- ### Calling a Huey Task Source: https://github.com/coleifer/huey/blob/master/docs/api.md Example of how to call a Huey task asynchronously and retrieve its result by blocking until it's available. ```python >>> from demo import add_numbers >>> res = add_numbers(1, 2) >>> res(blocking=True) # Blocks until result is available. 3 ``` -------------------------------- ### Handling SIGNAL_INTERRUPTED Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Provides an example of a signal handler for `SIGNAL_INTERRUPTED` to re-enqueue tasks during consumer shutdown. ```python @huey.signal(SIGNAL_INTERRUPTED) def on_interrupted(signal, task, *args, **kwargs): # The consumer was shutdown before `task` finished executing. # Re-enqueue it. huey.enqueue(task) ``` -------------------------------- ### Registering Signal Handlers Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Examples of how to register signal handlers using the Huey.signal() decorator for different scenarios. ```python import huey from huey.api import SIGNAL_ERROR, SIGNAL_LOCKED, SIGNAL_CANCELED, SIGNAL_REVOKED, SIGNAL_COMPLETE # Assuming 'huey' is an instance of Huey huey = huey.Huey() @huey.signal() def all_signal_handler(signal, task, exc=None): # This handler will be called for every signal. print('%s - %s' % (signal, task.id)) @huey.signal(SIGNAL_ERROR, SIGNAL_LOCKED, SIGNAL_CANCELED, SIGNAL_REVOKED) def task_not_executed_handler(signal, task, exc=None): # This handler will be called for the 4 signals listed, which # correspond to error conditions. print('[%s] %s - not executed' % (signal, task.id)) @huey.signal(SIGNAL_COMPLETE) def task_success(signal, task): # This handler will be called for each task that completes successfully. pass ``` ```python # Disconnect the "task_success" signal handler. huey.disconnect_signal(task_success) # Disconnect the "task_not_executed_handler", but just from # handling SIGNAL_LOCKED. huey.disconnect_signal(task_not_executed_handler, SIGNAL_LOCKED) ``` -------------------------------- ### Initialize Huey with Redis URL from Environment Source: https://github.com/coleifer/huey/blob/master/docs/deployment.md Configure Huey to use Redis, reading the connection URL from an environment variable. Essential for dynamic environment configurations. ```Python import os from huey import RedisHuey huey = RedisHuey('my-app', url=os.environ['REDIS_URL']) ``` -------------------------------- ### Run Huey Consumer Source: https://github.com/coleifer/huey/blob/master/docs/imports.md This command shows how to run the Huey consumer, pointing it to the Huey instance in the main module. ```shell huey_consumer main.huey ``` -------------------------------- ### Signal Handler Error Resilience Example Source: https://github.com/coleifer/huey/blob/master/docs/signals.md Illustrates how Huey handles exceptions raised within signal handlers. ```python @huey.signal(SIGNAL_COMPLETE) def broken_handler(signal, task): raise ValueError('oops') @huey.signal(SIGNAL_COMPLETE) def working_handler(signal, task): # This will still be called, even if broken_handler raised. record_completion(task.id) ``` -------------------------------- ### Enable Immediate Mode Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Set the `immediate` attribute to `True` to enable immediate mode. This makes tasks run synchronously. ```python huey = RedisHuey('my-app') huey.immediate = True ``` -------------------------------- ### FileHuey Initialization Source: https://github.com/coleifer/huey/blob/master/docs/api.md Details the initialization of `FileHuey`, a Huey backend that uses the file system for storage. Note its limitations in high-throughput environments due to locking. ```APIDOC ## FileHuey Initialization ### Description Initializes a `FileHuey` instance, which uses the file-system for storage. This backend utilizes exclusive locks and is not recommended for high-throughput, highly-concurrent environments. ### Parameters * **path** (str) - The base path for storing Huey data (queue tasks, schedule, and results). * **levels** (int) - The number of directory levels for the results directory to manage file count. * **use_thread_lock** (bool) - Use `threading.Lock` instead of a lockfile for file-system operations. Enable only for greenlet or thread consumer models. ``` -------------------------------- ### get Source: https://github.com/coleifer/huey/blob/master/docs/api.md Read a value from the result-store at the given key. By default reads are destructive. To preserve the value for additional reads, specify `peek=True`. ```APIDOC ## get(key, peek=False) ### Description Read a value from the result-store at the given key. By default reads are destructive. To preserve the value for additional reads, specify `peek=True`. ### Parameters #### Path Parameters - **key** - Required - key to read - **peek** (bool) - Optional - non-destructive read. Defaults to False. ``` -------------------------------- ### Get the Huey Instance from Django Settings Source: https://github.com/coleifer/huey/blob/master/docs/django.md Access the underlying Huey instance from `djhuey` to interact with Huey APIs not directly exposed by the Django integration. ```python from huey.contrib.djhuey import HUEY as huey # E.g., get the underlying Storage instance. storage = huey.storage ``` -------------------------------- ### Define Huey Instance Source: https://github.com/coleifer/huey/blob/master/docs/imports.md This snippet shows how to define a Huey instance in a configuration file. ```python # config.py from huey import RedisHuey huey = RedisHuey('testing') ``` -------------------------------- ### Redis Sentinel Configuration Source: https://github.com/coleifer/huey/blob/master/docs/recipes.md Configure Huey to use Redis Sentinel for high availability. This setup ensures automatic failover without requiring a restart. ```python from redis.sentinel import Sentinel from huey import RedisHuey sentinel = Sentinel([('10.0.0.1', 26379), ...], socket_timeout=5.0) HUEY = RedisHuey( 'my-app', connection_pool=sentinel.master_for('my-master').connection_pool) ``` -------------------------------- ### Huey Startup Hook Source: https://github.com/coleifer/huey/blob/master/docs/api.md Register a startup hook to run when a worker comes online. Useful for initializing resources like database connections that should not be created at import time. Uncaught exceptions are logged. ```python db_connection = None @huey.on_startup() def setup_db_connection(): global db_connection db_connection = psycopg2.connect(database='my_db') @huey.task() def write_data(rows): cursor = db_connection.cursor() # ... ``` -------------------------------- ### Explicit Task Enqueueing Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Demonstrates creating a Task instance using .s() and enqueuing it explicitly with huey.enqueue(). ```python import huey @huey.task() def add(a, b): return a + b # Create a task representing the execution of add(1, 2). task = add.s(1, 2) # Enqueue the task instance, which returns a Result handle. result = huey.enqueue(task) ``` -------------------------------- ### Task Locking with Decorator Source: https://github.com/coleifer/huey/blob/master/docs/guide.md Prevent concurrent execution of a periodic task using the `lock_task` decorator. This ensures that a task does not start if a previous invocation is still running. ```python @huey.periodic_task(crontab(minute='*/5')) @huey.lock_task('reports-lock') def generate_report(): run_report() ``` -------------------------------- ### Run Huey Consumer (Greenlets) Source: https://github.com/coleifer/huey/blob/master/README.rst Run the Huey consumer with greenlets for IO-bound workloads, specifying 32 workers. ```console $ huey_consumer my_app.huey -k greenlet -w 32 ``` -------------------------------- ### Explicitly managing shared resources Source: https://github.com/coleifer/huey/blob/master/docs/shared_resources.md Example of managing a Peewee database connection as a context manager within a Huey task. ```python import peewee huey = RedisHuey() @huey.task() def check_comment_spam(comment_id): # Open DB connection at start of task, close upon exit. with database: comment = Comment.get(Comment.id == comment_id) if akismet.is_spam(comment.body): comment.is_spam = True comment.save() ``` -------------------------------- ### Task Declarations Source: https://github.com/coleifer/huey/blob/master/docs/mini.md Examples of how to declare tasks using MiniHuey, including regular tasks, scheduled tasks using crontab, and periodic tasks. ```python from huey import crontab from huey.contrib.mini import MiniHuey huey = MiniHuey() @huey.task() def fetch_url(url): return urlopen(url).read() @huey.task(crontab(minute='0', hour='4')) def run_backup(): pass @huey.periodic_task(crontab(hour='10,14', minute='30')) def send_reminders(): pass ``` -------------------------------- ### Enabling Immediate Mode for Tasks Source: https://github.com/coleifer/huey/blob/master/docs/api.md Demonstrates how to enable and disable immediate mode for synchronous task execution within the application, useful for testing. When enabled, Huey defaults to in-memory storage. ```python >>> from demo import add_numbers, huey >>> huey.immediate = True # Tasks executed immediately. >>> res = add_numbers(2, 3) >>> res() 5 ``` -------------------------------- ### Task Creation and Enqueueing Source: https://github.com/coleifer/huey/blob/master/docs/api.md Illustrates how to define a task using the @huey.task() decorator and how to enqueue it for execution, either directly or via a Task instance. ```APIDOC ## Task Creation and Enqueueing ### Description Define a task using the `@huey.task()` decorator and enqueue it for execution. This shows the equivalence between calling a decorated function directly and creating a `Task` instance and enqueuing it. ### Example ```python @huey.task() def add(a, b): return a + b # Direct call (equivalent to enqueueing) ret = add(1, 2) print(ret.get(blocking=True)) # Output: "3" # Equivalent using Task instance and enqueue task_instance = add.s(1, 2) # Create a Task instance. ret = huey.enqueue(task_instance) # Enqueue the task. print(ret.get(blocking=True)) # Output: "3" ``` ``` -------------------------------- ### rate_limit Source: https://github.com/coleifer/huey/blob/master/docs/api.md Simple fixed-window rate-limiting for tasks. When a task fails due to being rate-limited, it will be automatically rescheduled to run at the start of the next window when `retry=True` or the task itself has one or more `retries`. ```APIDOC ## rate_limit(name, limit, per, retry=True) ### Description Simple fixed-window rate-limiting for tasks. When a task fails due to being rate-limited, it will be automatically rescheduled to run at the start of the next window when `retry=True` or the task itself has one or more `retries`. Tasks that fail due to rate-limiting will emit a `SIGNAL_RATE_LIMITED`. ### Parameters #### Path Parameters - **name** (str) - Required - Name to use for the rate-limiter. - **limit** (int) - Required - Maximum invocations in the given window. - **per** (int) - Required - Size of rate-limiting window in seconds. - **retry** (bool) - Optional - Automatically reschedule rate-limited tasks to run at beginning of next rate-limit window. Defaults to True. ### Returns [`RateLimit`](#RateLimit) instance, which can be used as a decorator or context-manager. ``` -------------------------------- ### BlackHoleHuey Initialization Source: https://github.com/coleifer/huey/blob/master/docs/api.md Describes the initialization of `BlackHoleHuey`, a Huey backend that discards all operations, making it suitable for testing scenarios where task execution needs to be verified without side effects. ```APIDOC ## BlackHoleHuey Initialization ### Description Initializes a `BlackHoleHuey` instance, which uses `BlackHoleStorage`. All storage operations are no-ops; tasks are discarded, and the result store is always empty. Ideal for testing task invocation without actual execution or storage. ```