### Install django-cog using pip Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Installs the django-cog library using pip. This is the first step in setting up the library. ```bash pip install django-cog ``` -------------------------------- ### Dockerfile for Celery Application Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md A Dockerfile to build the Celery application image. It uses a Python 3.8 base image, sets up necessary directories for static and media files, installs project dependencies from 'requirements/base.txt', 'requirements/${DJANGO_ENVIRONMENT}.txt', and 'requirements/custom.txt', and upgrades pip. It does not define an entry point as it's handled by Docker Compose. ```dockerfile FROM python:3.8 ENV PYTHONUNBUFFERED 1 ARG DJANGO_ENVIRONMENT # Make the static/media folder. RUN mkdir /var/staticfiles RUN mkdir /var/mediafiles # Make a location for all of our stuff to go into RUN mkdir /app # Set the working directory to this new location WORKDIR /app # Add our Django code ADD . /app/ RUN pip install --upgrade pip # Install requirements for Django RUN pip install -r requirements/base.txt RUN pip install -r requirements/${DJANGO_ENVIRONMENT}.txt RUN pip install -r requirements/custom.txt # No need for an entry point as they are defined in the docker-compose.yml services ``` -------------------------------- ### Run django_cog migrations Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Applies the database migrations specifically for the django_cog app. This must be done after installing the library and adding it to INSTALLED_APPS. ```python python manage.py migrate django_cog ``` -------------------------------- ### Django-Cog Runtime vs. Static Arguments Merging Example Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This example demonstrates the merging behavior of static and runtime arguments in Django-Cog. The `process_data` function receives arguments where runtime arguments override static ones and new arguments are added. ```python from django_cog import cog @cog def process_data(a, b, c, user_id=None, environment=None): # Process data with the provided arguments result = a + b + c print(f"Processing for user {user_id} in {environment} environment") return result ``` ```json # Static arguments in Task Admin: { "a": 1, "b": 2 } ``` ```python # Launching with runtime arguments: pipeline.launch(a=10, c=3, user_id=123, environment='production') # Task will receive: # { # "a": 10, # Runtime argument overrides static # "b": 2, # Static argument preserved # "c": 3, # New runtime argument # "user_id": 123, # Runtime argument # "environment": "production" # Runtime argument # } ``` -------------------------------- ### Sample Docker Compose configuration for Celery and Redis Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md A snippet of a docker-compose.yml file showing how to set up necessary services for Celery workers, Celery-Beat, and Redis. This is typically used in development environments to run background tasks. ```yaml version: '3' ``` -------------------------------- ### Configure PostgreSQL Service in Docker Compose Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Sets up a PostgreSQL database service using the 'postgres:9.6-alpine' image. It configures environment variables for database credentials, restarts the service unless stopped, mounts the persistent data volume, exposes the database port, and connects it to the 'backend' network. ```yaml postgresdb: image: postgres:9.6-alpine environment: - POSTGRES_DB - POSTGRES_USER - POSTGRES_PASSWORD restart: unless-stopped volumes: - volume_postgresdata:/var/lib/postgresql/data ports: - "127.0.0.1:5432:5432" networks: - backend ``` -------------------------------- ### Configure Django Web Service in Docker Compose Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Configures the Django web application service, building from the current directory and using 'djangoapp:latest' image. It maps local project files, persistent static and media volumes, exposes the application port locally, sets up network connections, and defines numerous environment variables for application configuration, including database connection details. ```yaml djangoweb: build: context: . args: - DJANGO_ENVIRONMENT=${DJANGO_ENVIRONMENT:-production} image: djangoapp:latest networks: - backend - celery - frontend volumes: - .:/app/ - volume_django_static:/var/staticfiles - volume_django_media:/var/mediafiles ports: # IMPORTANT: Make sure to use 127.0.0.1 to keep it local. Otherwise, this will be broadcast to the web. - 127.0.0.1:8000:8000 depends_on: - postgresdb - mailhog environment: - POSTGRES_DB - POSTGRES_USER - POSTGRES_PASSWORD - POSTGRES_HOST=postgresdb # Name of the postgresql service. - POSTGRES_PORT - DJANGO_SETTINGS_MODULE - FORCE_SCRIPT_NAME - DJANGO_ENVIRONMENT - SECRET_KEY - DJANGO_ALLOWED_HOSTS links: - "postgresdb" ``` -------------------------------- ### Launching a Django-Cog Pipeline Programmatically with Runtime Arguments Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This snippet illustrates how to launch a Django-Cog pipeline programmatically and pass additional arguments at runtime. These runtime arguments are merged with static task arguments, with runtime arguments taking precedence. ```python from django_cog.models import Pipeline # Get your pipeline pipeline = Pipeline.objects.get(name='my_pipeline') # Launch with additional kwargs pipeline.launch(user_id=123, environment='production', custom_flag=True) ``` -------------------------------- ### Defining Task Arguments in Django-Cog Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This snippet shows how to define a Python function (`add`) with parameters and how to provide these parameters as JSON in the Django admin for a task. This allows for static configuration of task inputs. ```python from django_cog import cog @cog def add(a, b): # add the two numbers together return a + b ``` ```json { "a": 1, "b": 2 } ``` -------------------------------- ### Configure Celery Beat Service in Docker Compose Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Sets up a Celery Beat service, also built from 'Dockerfile.celery'. This service runs the Celery beat scheduler, which is responsible for dispatching periodic tasks. It shares environment variables and volumes with other services and depends on the Celery worker service, connecting to 'backend' and 'celery' networks. ```yaml celerybeat: build: context: . dockerfile: Dockerfile.celery args: - DJANGO_ENVIRONMENT=${DJANGO_ENVIRONMENT:-production} command: celery -A django_cog beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler image: djangoapp_celerybeat:latest volumes: - .:/app/ environment: - POSTGRES_DB - POSTGRES_USER - POSTGRES_PASSWORD - POSTGRES_HOST=postgresdb # Name of the postgresql service. - POSTGRES_PORT - FORCE_SCRIPT_NAME - DJANGO_ENVIRONMENT - DJANGO_SETTINGS_MODULE - SECRET_KEY - DJANGO_ALLOWED_HOSTS depends_on: - celery networks: - backend - celery ``` -------------------------------- ### List discovered cog functions Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Prints a dictionary of all functions that django_cog has discovered and registered as cogs. This is useful for verifying that functions are being picked up correctly. ```python from django_cog import cog print(cog.all) ``` -------------------------------- ### Configure Celery Worker Service in Docker Compose Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Configures a Celery worker service, building from the 'Dockerfile.celery' with specified build arguments. It runs Celery workers in verbose mode, mounts the project directory, and shares environment variables with the Django application, including database credentials and settings. It depends on PostgreSQL and Redis and connects to 'backend', 'celery', and 'frontend' networks. ```yaml celery: build: context: . dockerfile: Dockerfile.celery args: - DJANGO_ENVIRONMENT=${DJANGO_ENVIRONMENT:-production} command: celery -A django_cog worker -l info image: djangoapp_celery:latest volumes: - .:/app/ environment: - POSTGRES_DB - POSTGRES_USER - POSTGRES_PASSWORD - POSTGRES_HOST=postgresdb # Name of the postgresql service. - POSTGRES_PORT - FORCE_SCRIPT_NAME - DJANGO_ENVIRONMENT - DJANGO_SETTINGS_MODULE - SECRET_KEY - DJANGO_ALLOWED_HOSTS depends_on: - postgresdb - redis networks: - backend - celery links: - "postgresdb" ``` -------------------------------- ### Add django_cog to Django INSTALLED_APPS Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Configures the Django application to include django_cog by adding its app config to the INSTALLED_APPS list in settings.py. This is essential for the library to function within the Django project. ```python INSTALLED_APPS = [ ... # Django-Cog: 'django_cog.apps.DjangoCogConfig', # Required for nested inline child views in the Django Admin: 'nested_inline', # Optional (recommended): 'django_celery_beat', ] ``` -------------------------------- ### Configure Redis Service in Docker Compose Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Sets up a Redis service using the 'redis:alpine' image. This service is configured to run on the 'celery' network, typically used for message brokering and caching in distributed task systems. ```yaml redis: image: "redis:alpine" networks: - celery ``` -------------------------------- ### Define Docker Compose Networks Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Defines three distinct networks for service communication: 'frontend' for external access, 'backend' for internal service communication, and 'celery' for task queue related services. These networks allow for controlled and isolated communication between different parts of the application. ```yaml networks: frontend: name: djangocog_frontend backend: name: djangocog_backend celery: name: djangocog_celery ``` -------------------------------- ### Django Settings for Task Weight Sample Size Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This Python code snippet demonstrates how to configure the sample size for calculating task weights in Django settings. The `DJANGO_COG_TASK_WEIGHT_SAMPLE_SIZE` variable controls how many past task runtimes are considered for averaging the task's weight, influencing task prioritization. ```python DJANGO_COG_TASK_WEIGHT_SAMPLE_SIZE = 10 ``` -------------------------------- ### Define Docker Compose Volumes Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Defines persistent storage volumes for PostgreSQL data, Django media files, and Django static files. These volumes ensure data persistence across container restarts and are shared between specific services as needed. ```yaml volumes: volume_postgresdata: # Store the postgres database data. Only linked with postgres. volume_django_media: # Store the Django media files. Volume is shared between djangoweb and nginx. volume_django_static: # Store the Django static files. Volume is shared between djangoweb and nginx. ``` -------------------------------- ### Manually create cog records in Django shell Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Provides a method to manually trigger the creation of Cog records in the Django admin if automatic registration fails. This function should be called within the Django shell. ```python from django_cog.apps import create_cog_records create_cog_records() ``` -------------------------------- ### Register a cog function using the decorator Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md Demonstrates how to register a Python function as a 'cog' using the @cog decorator from the django_cog library. This allows the function to be managed and potentially executed by Celery. ```python from django_cog import cog @cog def my_task(): # do something in a celery worker pass ``` -------------------------------- ### Django Settings to Allow Overlapping Failed Pipeline Runs Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This Python code snippet configures Django settings to allow new pipeline runs even if the previous run failed. Setting `DJANGO_COG_OVERLAP_FAILED` to `True` overrides the default safety feature that prevents repeated execution of potentially problematic pipelines. ```python DJANGO_COG_OVERLAP_FAILED = True ``` -------------------------------- ### Register Custom Cog Error Handler with Task Run Context Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This Python code shows how to register a custom error handler that also receives the task run object. By including the `task_run` keyword argument, the error handler gains access to context about the failed task, enabling more specific error reporting or recovery actions. ```python from django_cog import cog_error_handler @cog_error_handler def myErrorHandler(error, task_run): """ Do something with the error (Excemption type) and know where it came from. """ print("The following error came from the task:", task_run.task.name) print(error) ``` -------------------------------- ### Registering a Python Function as a Cog in Django-Cog Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This snippet demonstrates how to register a Python function as a 'Cog' using the `@cog` decorator. Registered functions can be executed as tasks within Django-Cog pipelines. These functions must be imported in the application's `__init__.py` for auto-discovery. ```python from django_cog import cog @cog def my_task(): # do something in a celery worker pass ``` -------------------------------- ### Configure Celery Worker Queue in Docker Compose Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This snippet shows how to configure a Celery worker in `docker-compose.yml` to listen to a specific queue named 'queue_name'. This allows for dedicated workers for specific sets of tasks, improving organization and resource management. ```yaml services: celery: command: celery -A django_cog worker -l info -Q queue_name ``` -------------------------------- ### Django Settings to Disable Auto Task Weight Calculation Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This Python code snippet shows how to disable the automatic calculation of task weights in Django settings. Setting `DJANGO_COG_AUTO_WEIGHT` to `False` allows for manual weight adjustment through the Django admin, giving direct control over task queuing order. ```python DJANGO_COG_AUTO_WEIGHT = False ``` -------------------------------- ### Register Custom Cog Error Handler Source: https://github.com/david-pettifor-nd/django_cog/blob/master/README.md This Python code demonstrates how to register a custom error handler function for Django Cog tasks. The `@cog_error_handler` decorator registers the `myErrorHandler` function to be called when a task fails, allowing for custom error processing logic. ```python from django_cog import cog_error_handler @cog_error_handler def myErrorHandler(error): """ Do something with the error (Excemption type) """ print(error) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.