Asynchronous tasks

[downdawn]

Async IO is suitable when we want to perform small BackgroundTasks or to perform tasks using concurrency and parallelism. However, when it comes to performing heavy background computations or complex tasks, we would ideally not like to run them as a part of the same process. As such, to perform these complex tasks as a separate process, we need specialized tools such as Celery.

https://medium.com/cuddle-ai/async-architecture-with-fastapi-celery-and-rabbitmq-c7d029030377

[wu-clan]

Nice offer, but we should remove the APScheduler integration!

I've created a new repository here for APScheduler: fastapi_scheduler

[wu-clan]

I think we should add the celery timed task instance at the same time, is that correct?

[downdawn]

Yes, this is relatively common

[wu-clan]

I've opened a PR #229 and for now, I'm only using redis as a broker and backend, maybe we need to think about whether to use rabbitmq directly as it will make local development more complicated

[downdawn]

Nice, I think it makes sense to use Redis for testing, and for the actual project, RabbitMQ would be a better fit.

[wu-clan]

Yes, but don't rule out the possibility that we can control whether or not to use RabbitMQ with a switch for the user to choose for themselves

[downdawn]

Agree

[wu-clan]

Also may need to add more interfaces and methods to better represent celery

[downdawn]

Chatgpt answer：

Celery has a wide range of common use cases in many applications. Here are some common scenarios:

Email Sending: Sending emails asynchronously to speed up response times in web applications. This includes tasks like registration confirmation emails, password reset notifications, and more.

Scheduled Tasks: Executing periodic tasks, such as generating reports, database backups, cleaning temporary files, or performing system maintenance operations.

Image Processing: Asynchronously processing image operations, such as resizing, cropping, applying filters, or generating thumbnails.

File Handling: Uploading and processing large files, like transcoding audio or video files and converting file formats.

Data Scraping: Regularly fetching data from external sources (e.g., websites, APIs) and storing it in a local database.

Message Queues: Pushing messages to queues to support real-time communication, chat applications, and task collaboration.

Web Crawling and Scraping: Building web crawlers to scrape web content, index data, or monitor changes on specific websites.

Queue Processing: Handling large volumes of queue data, such as log processing, distributed computing, event logging, etc.

User Notifications: Sending real-time notifications and reminders, such as social media activity notifications and new message alerts.

Intensive Calculations: Asynchronously executing tasks that require significant computational resources, like scientific computing, machine learning model training, etc.

Data Transformation and ETL: Converting data from one format to another to support data warehousing, analysis, or report generation.

Resource Management: Dynamically allocating and deallocating resources, such as automatic scaling of cloud server resources.

Monitoring and Log Processing: Asynchronously processing monitoring data, real-time log processing, and generating analysis reports.

Batch Processing Tasks: Handling batch processing tasks for large data sets, such as bulk data import/export.

Distributed System Tasks: Distributing tasks and collaboration in a distributed system to support microservices architecture.

Database Operations: Executing database operations, such as index rebuilding, backup, and data migration.

Producer-Consumer Pattern: Implementing the producer-consumer pattern by separating task producers from task consumers to enhance system scalability and stability.

[downdawn]

For us, the more practical ones are:
Email Sending, Scheduled Tasks, User Notifications, Monitoring and Log Processing

[downdawn]

For decoupling complex businesses, for example, after payment is completed, update order tasks, notification tasks, points tasks, etc. can be initiated asynchronously.