A Message Queue allows application-to-application services to communicate with each other regardless of where the application or data resides.
Queue
A queue is a line of things waiting to be handled in sequential order, starting from the beginning of the line. These things are nothing but a sequence of work objects that need to be executed.
Message
It's the data transported between the sender and receiver. It could be an event or something that tells a system to start processing a task.
What is a Message Queue?
A Message Queue (MQ) is a component of messaging middleware solutions that enable independent applications and services to exchange information.
Message queues store messages—packets of data that applications create for other applications to consume—in the order, they are transmitted until the consuming application can process them.
This enables messages to wait safely until the receiving application is ready, so if there is a problem with the network or receiving the application, the messages in the message queue are not lost.
Need for Message Queues
Sometimes we require asynchronous processing eg. send SMS, email, process orders or pass information between services. This type of background processing is simply not a task that traditional RDBMS is best suited to solve.
With a message queue, we can efficiently support a significantly higher volume of concurrent messages, messages are pushed in real-time instead of periodically polled, messages are automatically cleaned up after being received and we don’t need to worry about any deadlocks or race conditions. In short, message queues present a fundamentally sound strategy for handling high-volume asynchronous processing.
Let's see with an example as to how MQ's can help
Assume you are running marketplace e-commerce. You want to send a set of emails to your customer, seller, logistic partner and your employees for every order processing.
How would you do that in a monolithic application? Your order processing service is dependent on email service, and call sendEmail(..) before processing. Your order processing is waiting till sendEmail gets executed. What about if your traffic spikes, your endEmail(..) has backlog now?
Here come the MQ's. We can use some method say sendEmailsToMq(..) and send every email to MQ's where it is queued and get processed asynchronously with a separate email handling system.
Benefits and Use Cases of Message Queues
Reliable Message Delivery
Ensures that critical messages between applications are not lost and are successfully delivered to the recipient thereby relieving us from implementing additional duplication or loss-prevention logic.
Better Performance
Enable asynchronous communication, which means that the endpoints that are producing and consuming messages interact with the queue, not each other. No component in the system is ever stalled waiting for another, hence optimizing data flow.
Resilience
Asynchronous messaging ensures that application-specific faults don't impact the system which in turn decreases the chances of single point of failure.
Simplified Decoupled Architecture
Message queues remove dependencies between components and significantly simplify the coding of decoupled applications. They provide a simple way to decouple distributed systems, whether you're using monolithic applications, microservices or serverless architectures.
Granular Scalability
When workloads peak, multiple instances of your application can all add requests to the queue without risk of collision. As your queues get longer with these incoming requests, you can distribute the workload across a fleet of consumers. Producers, consumers and the queue itself can all grow and shrink on demand.
Enhanced Security
MQ's are able to identify and authenticate messages, and in some cases, can encrypt messages in rest, transit or end-to-end which can contribute to the overall security of the application.
Widely used Message Queue Tools
Kafka, RabbitMQ, Amazon SQS, Celery, and ActiveMQ are the most popular tools in the category Message Queue. High-throughput is the primary reason developers pick Kafka over its competitors, while It's fast and it works with good metrics/monitoring is the reason why RabbitMQ was chosen.
PS. If you find the article relevant, then please share it with your network on Social Media. 🙏