Message Queue
A message queue is a system that stores messages (tasks or events) sent from producers and delivers them to consumers for processing, decoupling the two and enabling asynchronous, reliable communication between system components.
Understanding Message Queue
Message queues solve a fundamental distributed systems problem: how do you reliably pass work between system components when both sides may be unavailable at the same time? Without a queue, if the worker is busy or down when a job arrives, the job is lost. With a queue, the job is stored until a worker is available to process it. The producer-consumer model is simple: a producer (web server, webhook handler, user action) puts a message in the queue. A consumer (background worker) takes messages from the queue and processes them. Multiple producers and consumers can work simultaneously, scaling independently. Popular message queue systems include RabbitMQ (full-featured, supports complex routing), Redis (lightweight, fast, used for simple queues), AWS SQS (managed, serverless), and Apache Kafka (high-throughput streaming). GAIA uses RabbitMQ for complex routing and Redis/ARQ for simpler background jobs. Message queues enable graceful handling of load spikes. If 1000 webhook events arrive simultaneously, they queue immediately and get processed at the rate the consumers can handle — no events are lost, and the system doesn't collapse under load.
How GAIA Uses Message Queue
GAIA uses RabbitMQ for routing events between system components and ARQ (Async Redis Queue) for background job execution. When an email arrives or an automation trigger fires, the event is queued immediately and processed by background workers. This architecture ensures no events are lost during high-load periods and enables reliable retry logic.
Related Concepts
Webhook
A webhook is an HTTP callback mechanism where a system sends an automated HTTP request to a specified URL whenever a defined event occurs, enabling real-time notification and integration between services without polling.
Event-Driven Automation
Event-driven automation is a pattern where workflows are triggered automatically in response to specific events, such as a new email arriving, a calendar event being created, or a message being posted, enabling real-time, reactive processing.
Workflow Automation
Workflow automation is the use of technology to execute repeatable business processes and tasks automatically, reducing manual effort and human error.
Cron Job
A cron job is a scheduled task configured to run automatically at specified time intervals or on specific dates using the cron scheduling syntax, enabling recurring automated processes without manual triggers.
API Integration
API integration is the process of connecting different software applications through their Application Programming Interfaces, enabling them to share data and functionality seamlessly.
