Setting Up Celery for Your Django Project on Ubuntu Server: A Comprehensive Guide

In this tutorial, we'll walk you through the process of configuring Celery, a distributed task queue, with your Django project on an Ubuntu server. Celery enables the asynchronous execution of tasks, enhancing the performance and scalability of your application. We'll break down the configuration of the celeryd and celery.service files step by step, helping even novice developers get started.

Setting Up Celery for Your Django Project on Ubuntu Server

Setting Up Celery

In this section, we will dive into the crucial process of configuring Celery for your Django project. Celery is a powerful tool that enables you to run asynchronous tasks in the background, enhancing the performance and scalability of your web application.

Prerequisites

Before we proceed, make sure you have the following tools and prerequisites in place:

A working Django project.
A Linux server (in this case, we'll focus on configuring Celery on an Ubuntu server).
Python 3.x installed.
A virtual environment is set up for your Django project.

If you haven't set up your Django project or installed Python 3.x, please ensure you do so before continuing.

Creating /etc/default/celeryd

Section 1: celeryd Configuration

Create the Configuration File: To begin, navigate to the /etc/default directory and create a new file named celeryd to hold your Celery configuration.
Defining Node Configuration: Specify the number of nodes for Celery using the CELERYD_NODES variable. For most setups, a single node is sufficient:
```
 CELERYD_NODES="celery"
```
Setting Celery Binary Path: Define the absolute or relative path to the Celery command using the CELERY_BIN variable. For example:
```
 CELERY_BIN="/home/akshaykaushik.eu.org/AI/.venv/bin/celery"
```
Specifying Django App Instance: Set the Django app instance to be used with Celery using the CELERY_APP variable, It is the Django project name:
```
 CELERY_APP="Project"
```
Setting Working Directory: Specify the directory where Celery should operate from using the CELERYD_CHDIR variable(path of directory in which Django manage.py exists):
```
 CELERYD_CHDIR="/home/akshaykaushik.eu.org/AI/"
```

Configuring Logging: Define the logging level and paths for Celery logs using the following variables:

 CELERYD_LOG_LEVEL="INFO"
 CELERYD_LOG_FILE="/home/akshaykaushik.eu.org/AI/log/celery/%n%I.log"
 CELERYD_PID_FILE="/home/akshaykaushik.eu.org/AI/run/celery/%n.pid"

Specifying User and Group: Set the unprivileged user and group for Celery workers using the following variables(user and group of the django folder):
```
 CELERYD_USER="aksha1706"
 CELERYD_GROUP="aksha1706"
```
Directory Creation and Permissions: Enable directory creation for missing PID and log directories and configure them to be owned by the specified user/group:
```
 CELERY_CREATE_DIRS=1
```
Activating Virtual Environment: Specify the virtual environment activation command:
```
 CELERYD_ENV="+/home/akshaykaushik.eu.org/AI/.venv/bin/activate"
```
Setting Permissions: To ensure the Celery worker has the necessary permissions, follow these steps:
- Execute the command chmod +x /etc/init.d/celeryd.

Create the log and PID folders specified in the configuration file and grant them write permissions.

With these permissions in place, your Celery worker will operate smoothly.

# most people will only start one node:
CELERYD_NODES="celery"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/akshaykaushik.eu.org/AI/.venv/bin/celery"

# App instance to use
CELERY_APP="Project"

# Where to chdir at start.
CELERYD_CHDIR="/home/akshaykaushik.eu.org/AI/"

# Set logging level to INFO
CELERYD_LOG_LEVEL="INFO"

# %n will be replaced with the first part of the node name.
CELERYD_LOG_FILE="/home/akshaykaushik.eu.org/AI/log/celery/%n%I.log"
CELERYD_PID_FILE="/home/akshaykaushik.eu.org/AI/run/celery/%n.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="aksha1706"
CELERYD_GROUP="aksha1706"

# If enabled, PID and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

# Activate the virtual environment
CELERYD_ENV="+/home/akshaykaushik.eu.org/AI/.venv/bin/activate"

Creating a systemd Service for Celery

Section 2: celery.service Configuration

Create the Service File: Navigate to the /etc/systemd/system directory and create a new file named celery.service.
Defining the Unit: In the celery.service file, describe the service and set its dependencies:
```
 [Unit]
 Description=Celery Service
 After=network.target
```
Configure the Service: Define the service type, user, group, and reference the celeryd configuration file:
```
 [Service]
 Type=forking
 User=aksha1706
 Group=aksha1706
 EnvironmentFile=/etc/default/celeryd
```

Set Working Directory and ExecStart: Specify the working directory and the command to start the Celery service:

 WorkingDirectory=/home/akshaykaushik.eu.org/AI
 ExecStart=/bin/bash -c "source /home/akshaykaushik.eu.org/AI/.venv/bin/activate && \
   /home/akshaykaushik.eu.org/AI/.venv/bin/celery multi start ${CELERYD_NODES} \
   -A Project --pidfile=${CELERYD_PID_FILE} \
   --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}"

Define ExecStop and ExecReload: Specify the commands to stop and reload the Celery service:

 ExecStop=/bin/bash -c "source /home/akshaykaushik.eu.org/AI/.venv/bin/activate && \
   /home/akshaykaushik.eu.org/AI/.venv/bin/celery multi stopwait ${CELERYD_NODES} \
   --pidfile=${CELERYD_PID_FILE}"
 ExecReload=/bin/bash -c "source /home/akshaykaushik.eu.org/AI/.venv/bin/activate && \
   /home/akshaykaushik.eu.org/AI/.venv/bin/celery multi restart ${CELERYD_NODES} \
   -A Project --pidfile=${CELERYD_PID_FILE} \
   --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}"

Define Installation: Specify the target for installation:

 [Install]
 WantedBy=multi-user.target

 [Unit]
 Description=Celery Service
 After=network.target

 [Service]
 Type=forking
 User=aksha1706
 Group=aksha1706

 EnvironmentFile=/etc/default/celeryd
 WorkingDirectory=/home/akshaykaushik.eu.org/AI
 ExecStart=/bin/bash -c "source /home/akshaykaushik.eu.org/AI/.venv/bin/activate && \
   /home/akshaykaushik.eu.org/AI/.venv/bin/celery multi start ${CELERYD_NODES} \
   -A Project --pidfile=${CELERYD_PID_FILE} \
   --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}"
 ExecStop=/bin/bash -c "source /home/akshaykaushik.eu.org/AI/.venv/bin/activate && \
   /home/akshaykaushik.eu.org/AI/.venv/bin/celery multi stopwait ${CELERYD_NODES} \
   --pidfile=${CELERYD_PID_FILE}"
 ExecReload=/bin/bash -c "source /home/akshaykaushik.eu.org/AI/.venv/bin/activate && \
   /home/akshaykaushik.eu.org/AI/.venv/bin/celery multi restart ${CELERYD_NODES} \
   -A Project --pidfile=${CELERYD_PID_FILE} \
   --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}"

 [Install]
 WantedBy=multi-user.target

Enabling and Starting Celery

To activate and initiate Celery as a systemd service, follow these steps:

Run sudo systemctl daemon-reload to reload systemd manager configuration.
Enable the Celery service: sudo systemctl enable celery.
Start the Celery service: sudo systemctl restart celery.

With these commands, you'll have Celery up and running as a background service.

Running Celery

To use Celery with your Django project, you need to run it. After activating the virtual environment and navigating to your Django project folder, execute the following command:

    celery -A CELERY_APP_NAME worker -l INFO

Make sure to replace CELERY_APP_NAME with your actual project name. This command initiates Celery as a worker, ready to process tasks.

In case you need to make changes or updates, rerun the three commands mentioned above. This ensures that your Celery service remains active and responsive.

Conclusion: Congratulations! You've successfully configured Celery for your Django project on an Ubuntu server. The celeryd and celery.service files are now optimized to handle asynchronous tasks efficiently. By following these detailed steps, even newcomers to development can set up Celery with ease. Don't forget to run the specified commands to finalize the configuration and enjoy enhanced performance and scalability for your Django application. Happy coding!

Setting Up Celery for Your Django Project on Ubuntu Server: A Comprehensive Guide

Comments

More from this blog

Enhance Linux Performance: Step-by-Step Guide to Increase Swap File Space

Common Mistakes When Using React useState Hook (With Code Examples)

Merge Multiple Git Repositories without Breaking File History

Things to Note Before Migrating Servers

Multithreading VS Multiprocessing VS Asyncio (With Code examples)

Setting Up Celery

Prerequisites

Creating /etc/default/celeryd

Creating a systemd Service for Celery

Enabling and Starting Celery

Running Celery

Command Palette

Comments

More from this blog

Setting Up Celery

Prerequisites

Creating /etc/default/celeryd

Creating a systemd Service for Celery

Enabling and Starting Celery

Running Celery