Background Tasks

The Distributed Task Execution Layer

A standard web request operates on a strict request-response lifecycle: the user sends a request, your web server processes the logic, and the user waits on a loading screen until the server returns a response.

If your view handles slow, resource-heavy operations—such as sending marketing emails, processing uploaded video assets, generating bulky PDF reports, or syncing data with external APIs—handling them inside the standard view will delay the response, causing a sluggish user experience or network timeout crashes.

Synchronous View:  [User Request] ──► [Wait for Email API (5s)] ──► [Render Success Page]  (Slow UX)

Asynchronous Task: [User Request] ──► [Hand off to Queue]       ──► [Render Success Page]  (Instant UX)
                                              │
                                              ▼
                                      [Celery Worker Runs Task in Background]

To solve this, production web applications use Background Task Queues. This architecture allows your views to instantly hand off heavy workloads to a background worker queue, freeing up your web server to return a response to the user immediately.

1. Core Architecture: Celery & Redis

Celery is an industry-standard, asynchronous task queue framework built for Python. To pass tasks between your Django views and Celery workers, you need a message broker—a temporary storage medium that holds tasks until a worker is free to process them. Redis is commonly used as this message broker.

Step A: Installing Dependencies

pip install celery redis

Step B: Configuring Celery inside Django

To integrate Celery cleanly, initialize its application instance directly inside your project's configuration directory:

Python
# core_gate/celery.py
import os
from celery import Celery

# 1. Set the default Django settings module environment variable
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'core_gate.settings')

# 2. Instantiate the Celery app engine
app = Celery('core_gate')

# 3. Load configurations from your project settings file using a custom namespace
app.config_from_object('django.conf:settings', namespace='CELERY')

# 4. Automatically discover background tasks declared across your apps inside tasks.py files
app.autodiscover_tasks()
Ensure Celery initializes automatically whenever your Django server boots up by updating your project's structural init file:
Python
# core_gate/__init__.py
from .celery import app as celery_app

__all__ = ('celery_app',)

Step C: Configuring Broker Connections in Settings

Add your Redis broker connection endpoints and configuration rules to your settings.py file:

Python
# core_gate/settings.py

# Point Celery to your Redis instance database index node
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/2'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/2'

# Performance Tuning Configs
CELERY_TASK_TIME_LIMIT = 300 # Guardrail: Kill any task running longer than 5 minutes
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'

2. Defining and Dispatching Background Tasks

To create a background task, create a file named tasks.py inside any of your app directories and wrap your processing functions with the @shared_task decorator.

Python
# positions_board/tasks.py (Defining Reusable Background Tasks)
from celery import shared_task
import time
import logging

logger = logging.getLogger(__name__)

@shared_task
def send_bulk_job_alerts_email(recipient_list, prompt_text):
    """
    Simulates sending heavy marketing notification email batches.
    Runs entirely out-of-band, separated from the web server thread.
    """
    logger.info(f"Background Job Initialized: Processing {len(recipient_list)} target alert emails.")
    
    # Simulate a slow network operation
    time.sleep(5) 
    
    logger.info("Background Job Completed Successfully.")
    return f"Dispatched {len(recipient_list)} alerts."

To hand off a task to your background workers, do not execute the function standardly like send_bulk_job_alerts_email(). Instead, use the .delay() method. This method sends the function arguments to your message broker queue instantly, allowing your view code to continue running without waiting for the task to finish.

Python
# positions_board/views.py (Instant Request Hand-Off)
from django.http import JsonResponse
from .tasks import send_bulk_job_alerts_email

def trigger_alerts_gateway(request):
    subscribers = ["[email protected]", "[email protected]", "[email protected]"]
    message = "New developer roles are live on the platform matrix."

    # DISPATCH HOOK: Instantly pushes the task payload out to the Redis broker queue
    # This call takes milliseconds, returning execution flow back to the view instantly
    send_bulk_job_alerts_email.delay(subscribers, message)

    return JsonResponse({
        "status": "queued",
        "message": "Heavy notification processing loop offloaded to background clusters successfully."
    })

3. Periodic and Scheduled Tasks (Cron Jobs)

Often, applications need to run tasks automatically at specific times instead of triggering them from user actions—such as generating systemic data backups every midnight, or clearing stale temporary logs every Sunday morning. Celery handles this using a scheduler component called Celery Beat.

To configure scheduled tasks, use the CELERY_BEAT_SCHEDULE configuration array in your project's settings file:

Python
# core_gate/settings.py
from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    # 1. Interval-Based Schedule: Run a task every 15 minutes
    'sync-platform-metrics-every-quarter': {
        'task': 'positions_board.tasks.aggregate_system_metrics',
        'schedule': 900.0, # Time counter specified in seconds
    },
    
    # 2. Cron-Tab Schedule: Run a cleanup script exactly at 12:00 AM every Sunday night
    'purge-stale-logs-weekly-cron': {
        'task': 'positions_board.tasks.purge_expired_system_logs',
        'schedule': crontab(minute=0, hour=0, day_of_week='sun'),
    },
}

An entry inside your tasks file handles processing matching scheduled tasks:

Python
# positions_board/tasks.py
@shared_task
def purge_expired_system_logs():
    # Executed automatically by the Celery Beat scheduler daemon
    pass

4. Execution Commands for Production

In a production environment, you must run your Celery processes as separate, independent system daemons alongside your main Django web server process.

Command A: Start the Primary Worker Daemon

Instructs Celery to spin up resource threads to listen to your broker queue and process incoming tasks:

celery -A core_gate worker --loglevel=info

Command B: Start the Periodic Scheduler Engine

Instructs Celery to boot up the Beat scheduling loop, which monitors your timetables and dispatches tasks to the workers when schedules hit:

celery -A core_gate beat --loglevel=info

Background Architecture Operational Strategy Matrix