Getting a Barebones Celery Application Up and Running

True Story Follows

So I’ve been coding for over a decade and doing it professionally for over 2 years now, and I use celery all the time. But I’ve only actually set up celery one time with a Django application and it took my several hours to get working because of some weird import stuff. Celery is awesome, but it’s also supposed to be easy. So I wanted to create a simple, barebones set up to get it working.

Step 1: Choose a Broker

There are a number of different brokes that can be used for celery in order to send and consume messages with. Because of its ubiquity, general reliability, and re-usability, I’m just going to use Redis as my broker.

Installing Redis

Super easy. If you’re using Heroku, you can choose a free tier Redis plan like Redis To Go. If you’re running things locally, set up takes less than a minute. Just follow the Redis quickstart guide. Now redis can be started with:

redis-server

If you install on Linux, Redis will already run at startup.

Step 2: Install requirements with Python

Here’s my requirements file:

celery==3.1.18
redis==2.10.3

Run through the steps of settings up a virtual environment:

virtualenv .virtualenv
source .virtualenv/bin/activate
pip install -r requirements.txt

Step 3: Create config file

Now create a “celeryconfig.py” file. This can be another filename, but if so, you’ll need to specify that as an argument when you start celery workers. Here’s my config file:

BROKER_URL = 'redis://localhost:6379:0'

CELERY_ACCEPT_CONTENT = ['json', 'pickle']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

CELERY_IMPORTS = [
    'some_file_you_made',
    'some_directory.some_other_file',
]

Define some tasks in Python code

Next, create your taks. Here’s mine:

from celery import Celery
app = Celery('this_can_be_anything', broker='redis://localhost:6379/0')

@app.task(name="some_unique_name")
def arbitrary_task(serializable_input):
    # do anything here
    pass

if __name__ == "__main__":
    arbitrary_task.delay("hello world")

Start Celery Workers

Here’s my command to start my celery app:

celery worker --autoreload --config=celeryconfig --concurrency=2

And that’s it! Obviously there are a ton of settings adjustments that can be made, things to fine tune, and changes to make that are specific to your application, but if you run through the above steps, that is the fastest way to get up and running.

The End