Redis Vs Rabbitmq as a Data Broker/messaging System in Between Logstash and Elasticsearch

When choosing between Redis and RabbitMQ as a data broker or messaging system between Logstash and Elasticsearch, it’s important to evaluate based on specific use cases and requirements:

1. Redis

• Type: In-memory data structure store (often used as a message broker, cache, or database).
• Message Model: Pub/Sub (publish-subscribe) model with no message persistence by default.
• Throughput: Extremely high throughput due to its in-memory nature, capable of handling millions of messages per second.
• Durability: By default, Redis does not persist messages, but you can configure it with persistence (RDB or AOF) to make it durable.
• Latency: Very low latency (microseconds) since it operates primarily in memory.
• Use Case: Best suited for real-time, high-throughput scenarios where message persistence is not a key requirement or where data loss is tolerable. For instance, it's ideal for caching or short-lived message queues.
• Scalability: Redis can be scaled via clustering, but scaling horizontally can be more complex than RabbitMQ.
• Complexity: Simpler setup compared to RabbitMQ; easy to install, configure, and manage.

2. RabbitMQ

• Type: Dedicated message broker (with support for multiple messaging patterns like Pub/Sub, Message Queues, Topic, Fanout, etc.).
• Message Model: Supports message queues, acknowledgments, persistence, routing keys, and complex delivery guarantees.
• Throughput: Lower throughput compared to Redis, typically handling around tens to hundreds of thousands of messages per second depending on the setup.
• Durability: High durability with message persistence options. Ensures messages are stored on disk and can survive broker failures.
• Latency: Higher than Redis but still performant with millisecond-level latency.
• Use Case: Ideal when message durability, guaranteed delivery, and complex routing (e.g., fanout, topic exchange) are required. Best for long-lived tasks, processing pipelines, and scenarios where data loss is not acceptable.
• Scalability: RabbitMQ can be horizontally scaled via clustering and federation, and it offers flexible routing mechanisms.
• Complexity: More complex setup and configuration compared to Redis, with a steeper learning curve due to the advanced messaging features.

Recommendation

• Redis would be suitable if:
  • You need extremely high throughput with low latency.
  • Durability is not a major concern (or you can afford to lose some messages).
  • You’re looking for simplicity and lightweight architecture.
• RabbitMQ would be better if:
  • You need reliable, durable message delivery with acknowledgments.
  • You have more complex message routing needs or require advanced messaging patterns.
  • You require guaranteed delivery, persistence, and the ability to manage queues more granularly.

For a Logstash to Elasticsearch use case, where message delivery, retries, and potential buffering are essential, RabbitMQ may be a better fit, ensuring that logs don’t get lost and are delivered in the correct order. Redis could be used if you prioritize speed and can afford minor data loss, but it's less common for this specific logging pipeline due to the durability requirements of log systems.

Got an article suggestion? Let us know

Explore more

Logstash Elasticsearch Redis

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.