Redis, Kafka or rabbitmq: which one to choose as the micro service message broker- otonomo

Jiedao jdon 2021-05-04 17:28:48
redis kafka rabbitmq choose micro

When using asynchronous communication for microservices , Message broker is usually used . Agents ensure reliable and stable communication between different microservices , Ensure that messages are managed and monitored within the system , And messages are not lost . You can choose some message agents , They differ in size and data capabilities . This blog post will compare the three most popular ​​ 's agent :RabbitMQ,Kafka and Redis.

But first of all , Let's learn about microservice Communications .


Microservice communication : Synchronous and asynchronous

There are two common ways of communication between microservices : Synchronous and asynchronous . In synchronous communication , The caller waits for a response before sending the next message , And it acts as HTTP Above REST The protocol runs . contrary , In asynchronous communication , Send a message without waiting for a response . This applies to distributed systems , A message broker is usually required to manage messages .

The type of communication you choose should consider different parameters , For example, the structure of microservices , The right infrastructure , Delay , scale , Dependencies and communication purposes . The establishment of asynchronous communication can be more complicated , And you need to add more components to stack , But the benefits of using asynchronous communication for microservices far outweigh the disadvantages .


The advantage of asynchronous communication

First , According to the definition , Asynchronous communication is nonblocking . It also supports better scaling than synchronous operations . Third , In the case of a microservice crash , Asynchronous communication mechanism provides various recovery technologies , Usually better at dealing with crash related errors . in addition , When using a proxy instead of REST When the agreement , The services that receive communications don't actually need to know each other . After the old service has been running for a long time , You can even introduce new services , Better decoupling Services .

Last , When choosing asynchronous operation , You will enhance the ability to create centralized discovery in the future , monitor , Load balancing and even the ability of policy executors . This will give you flexibility in code and system building , Scalability and more functionality .


Choose the right message broker

Asynchronous communication is usually managed by message broker . There are other ways , for example aysncio, But they are more scarce and limited .

When you select an agent to perform an asynchronous operation , The following points should be considered :

  1. Agency Scale – The number of messages sent per second in the system .
  2. Data persistence – The ability to recover messages .
  3. Consumer power – Whether the agent has the ability to manage one-on-one and / Or one to many consumers .


One to many

We checked the latest and best service there , To find the strongest providers in these three categories .



scale : According to the configuration and resources , The speed here is about one second 50K msg.

persistence : Supports persistent and transient messages .

One to one and one to many consumers : Both have .

RabbitMQ On 2007 Released in , Is one of the first common message agents to be created . It's an open source , By implementing advanced message queuing protocol (AMQP) Through point-to-point and pub-sub Method to deliver messages . It's designed to support complex routing logic .

There are hosted services that you can use as SaaS, But it's not part of the main cloud provider stack on the machine .RabbitMQ Support for all major languages , Include Python,Java,.NET,PHP,Ruby,JavaScript,Go,Swift etc. .

In persistent mode , There may be some performance issues .



scale : You can send up to a million messages per second .

persistence : Yes .

one-on-one vs One to many consumers : Only one to many ( It seems strange at first sight , Right ?!).

Kafka from Linkedin On 2011 Created in , Designed to handle high throughput , Low latency processing . As a distributed streaming platform ,Kafka Copied a release - A subscription service . It provides data persistence and stores record streams , Enable it to exchange high quality messages .

Kafka I was in Azure,AWS and Confluent Upper management SaaS. They are all Kafka The creator and main contributor of the project .Kafka Support for all major languages , Include Python,Java,C / C ++,Clojure,.NET,PHP,Ruby,JavaScript,Go,Swift etc. .



scale : You can send up to a million messages per second .

persistence : Basically not , It's a data store in memory .

One to one and one to many consumers : Both have .

Redis It's a little different from other message agents .Redis At the heart of this is a data store in memory , It can be used as a high performance key value store or message broker . Another difference is Redis No persistence , Instead, dump its memory to Disk / DB in . It's also perfect for real-time data processing .

first ,Redis Not one to one and one to many . however , because Redis 5.0 Introduced pub-sub, So the functionality has been enhanced , One to many becomes the real choice .


The message broker for each use case

We introduced RabbitMQ,Kafka and Redis Some of the characteristics of . These three kinds of animals are all their categories , But as mentioned above , They work very differently . This is our suggestion that the correct message broker should be used according to different use cases .


Short lived news :Redis

Redis Of In memory databases are almost suitable for use cases with short messages that do not require persistence . because Redis Provides very fast service and memory capabilities , So it's ideal for short message reservation , In these messages, persistence is not very important , You can tolerate some loss . With 5.0 in Redis Stream Publishing , It has also become a candidate for one to many use cases , Due to limitations and old pub-sub function , Absolutely need to use it .


Large amount of data :Kafka

Kafka It's a high throughput distributed queue , For storing large amounts of data for a long time . For one to many use cases that require persistence ,Kafka It's the ideal choice .


Complex routing :RabbitMQ

RabbitMQ Is an older but mature agent , It has many functions to support complex routing . When the required rate is not high ( More than tens of thousands msg / sec) when , It will even support complex routing communications .


Consider your software stack

Of course , Finally, consider your current software stack . If you are looking for a relatively simple integration process , And don't want to maintain other agents in the stack , Then you may prefer to use proxies that are already supported by the stack .

for example , If you are in RabbitMQ In the system above Celery for Task Queue, Then you'll get a connection with RabbitMQ or Redis Power to use together , Instead of not supporting Kafka And it needs to be rewritten ​​Kafka.

stay Otonomo, We use all of the above through the development and expansion of the platform , And then use it a little bit ! It's important to remember , Each tool has its own advantages and disadvantages , It's about understanding them and working for the job as well as a specific time , The situation is related to the requirement to choose the right tool .

本文为[Jiedao jdon]所创,转载请带上原文链接,感谢

  1. Compare node.js with spring boot- Ryan Gleason
  2. Obvious pitfalls of spring Webflux- Ł ukaszKy ć
  3. Spring founder uncle rod's real thoughts on yaml
  4. 码农飞升记-02-OracleJDK是什么?OracleJDK的版本怎么选择?
  5. What is manong feisheng-02-oracle JDK? How to choose the version of Oracle JDK?
  6. Spring tide surging Xinanjiang
  7. Linux内核软中断
  8. Linux kernel soft interrupt
  9. Linux内核软中断
  10. Linux kernel soft interrupt
  11. Java multithreading Foundation
  12. The construction of Maven private library nexus
  13. I / O stream in Java
  14. JDK 16:Java 16的新功能 - InfoWorld
  15. 在Java中本地进行线程间数据传输的三种方式和源码展示
  16. jdon导致cpu 99%最后tomcat死掉---banq给予回复
  17. 用领域事件模拟AOP注入
  18. JDK 16: new function of Java 16 - InfoWorld
  19. Cartoon: from JVM lock to redis distributed lock
  20. Spring 3.1 终于加入了Cache支持
  21. Prototype与JQuery对比
  22. Three ways of data transmission between threads in Java and source code display
  23. Jdon causes 99% of CPU and Tomcat dies -- banq replies
  24. docker 原理之 user namespace(下)
  25. Simulating AOP injection with domain events
  26. Spring 3.1 finally adds cache support
  27. Comparison between prototype and jquery
  28. User namespace of docker principle (2)
  29. The way to learn java IO stream and XML
  30. Why does a seemingly correct code cause the Dubbo thread pool to be full
  31. 0 基础 Java 自学之路(2021年最新版)
  32. 0 basic Java self study road (latest version in 2021)
  33. c#—基础拾遗(1) 面向对象
  34. C - basic information (1) object oriented
  35. 技术分享|SQL和 NoSQL数据库之间的差异:MySQL(VS)MongoDB
  36. Technology sharing differences between SQL and NoSQL databases: MySQL (VS) mongodb
  37. PHP教程/面向对象-3~构造函数和析构函数
  38. Spring Cloud的Feign客户端入门
  39. 优化Spring Boot应用的Docker打包速度
  40. PHP tutorial / object oriented - 3 ~ constructor and destructor
  41. Introduction to feign client of spring cloud
  42. Optimizing docker packaging speed of spring boot application
  43. 尚硅谷宋红康Java基础教程2019版
  44. 尚硅谷宋红康Java基础教程2019版
  45. Song Hongkang Java foundation course 2019
  46. Song Hongkang Java foundation course 2019
  47. Redis 6 的多线程
  48. Multithreading of redis 6
  49. SpringCloud-微服务架构编码构建
  50. SpringCloud-微服务架构编码构建
  51. Linux作业控制
  52. Coding construction of springcloud microservice architecture
  53. Java中几个常用并发队列比较 | Baeldung
  54. 为什么Java后端在创业企业中并不流行? -reddit
  55. Coding construction of springcloud microservice architecture
  56. Linux job control
  57. Comparison of several common concurrent queues in Java
  58. Why is java backend not popular in start-ups- reddit
  59. docker 资源限制之 cgroup
  60. 大数据环境: hadoop和jdk部署