The problem

Signals in django are synchronous, therefore the user doesn’t get a response until all the signals are run. When you have a software where you can chain lots of signals, and every signal can perform some database queries, that can lead to slowness, and slowness lead to poor user experience. Most times, you don’t need to run synchronously, there is when using Celery makes sense.

Django and Celery

Celery is a distributed task queue based on distributed message passing. It can use severals backends such as RabbitMQ and Redis. It allows to run tasks on background speeding up the response to user, thus improving the user experience.

You can install Celery with a simple pip command. Although it isn’t strictly needed I also recommend to install django-celery. django-celery provides several handy tools such as a database scheduler and a database result backend.

$ pip install celery
$ pip install django-celery

Remember to include “djcelery” in your INSTALLED_APPS.

The celery configuration is done in project.celery file unless you want to use the --config flag. My celery.py file looks like this

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('project')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

I have some lines in the settings.py file too.

# CELERY STAFF
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_ACCEPT_CONTENT = ['json', 'pickle']
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

Note that using pickle as a serialiazer has some security concerns, so don’t use it unless you really need it.

Using the autodiscover feature the tasks should live in tasks.py inside an app. A task may looks like this

@shared_task(serializer='json')
def adds(num1=0, num2=0):
    return num1 + num2

adds.delay(2,3)

In order to the task get executed, you need to run a worker. If you have django-celery installed you can run it using the manage.py

python manage.py celery worker -A project -l info

Next steps?

There are a number of things to do:

• Monkey patching signals
• Running celery tasks on other hosts (possibly without all the project in it)
• Benchmark the whole system