Capital service, after reading this article, you still don't understand rocketmq, I lose

Yin Jihuan 2020-11-06 01:18:31
capital service reading article don


  1. RocketMQ Introduce
  2. RocketMQ Concept
  3. Why use RocketMQ?
    1. Asynchronous decoupling
    2. Peak shaving and valley filling
    3. Distributed transactions are ultimately consistent
    4. Data dissemination
  4. RocketMQ framework
  5. RocketMQ Message type
    1. General news
    2. Sequential message
    3. Timing message
    4. Transaction message
  6. Best practices
    1. Message retry
    2. Message filtering
    3. Consumption patterns
    4. Consumption idempotent
  7. Local transaction message encapsulation
  8. Reference code

RocketMQ Introduce

Apache RocketMQ Is a Low latency 、 High concurrency 、 High availability 、 Highly reliable distributed message middleware . Message queue RocketMQ It can provide asynchronous decoupling and peak shaving for distributed application systems , At the same time, it also has the massive message accumulation required by Internet applications 、 High throughput 、 Features such as reliable retries .

RocketMQ Concept

  • Topic: Message topic , Used to categorize a class of messages , For example, order subject , That is, all order related messages can be carried by this topic , Producers send messages to this topic .
  • producer : Responsible for production message and send message to Topic Role .
  • consumer : In charge of from Topic Receive and consume messages Role .
  • news : The producer to Topic Sent content , Will be consumed by consumers .
  • Message properties : The producer can customize some business-related properties for the message when sending it , such as Message Key and Tag etc. .
  • Group: A class of producers or consumers , This kind of producer or consumer usually produces or consumes the same kind of news , And the logic of message publishing or subscribing is consistent .

Why use RocketMQ?

Asynchronous decoupling

With the popularity of microservice Architecture , It's very important to sort out the relationship between services . Asynchronous decoupling can reduce the coupling between services , At the same time, it can also improve the throughput of services .

There are many business scenarios that use asynchronous decoupling , Because the business of every industry will be different , With some more general business to illustrate, I believe everyone can understand .

For example, the single business scenario in the e-commerce industry , In terms of the simplest order process , The order process is as follows :

  1. Lock stock
  2. Create order
  3. Users pay
  4. Deducting the inventory
  5. Send SMS notification to users
  6. Add points to users
  7. Inform the merchant to deliver the goods

After we succeed in the following list , Users pay , Payment completion will have a logic called payment callback , You need to do some business logic in callback . Let's first look at the time required for synchronization , Here's the picture :


The order process above starts from 3 To 5 They can be processed by asynchronous processes , For users , After the payment is completed, he does not need to pay attention to the following process . Just take care of it backstage , This simplifies three steps , Improve callback processing time .


Peak shaving and valley filling

Peak shaving and valley filling refers to the impact of large flow , utilize RocketMQ Can withstand the instantaneous large flow , Protect the stability of the system , Enhance user experience .

In the e-commerce industry , The most common traffic shock is the second kill , utilize RocketMQ To achieve a complete second kill business or with a lot of work to do , Beyond the scope of this article , I can talk to you alone later . What I want to tell you is that scenes like this can be used RocketMQ To carry high concurrence , The premise is that the business scenario supports asynchronous processing .


Distributed transactions are ultimately consistent

as everyone knows , Distributed transactions have 2PC,TCC, Finally, consistency, etc . Among them, using message queue to do the final consistency scheme is more commonly used .

In the business scenario of e-commerce , The core business related to transactions must ensure data consistency . By introducing a message queue RocketMQ Version of distributed transactions , It can realize the decoupling between systems , It can also ensure the final data consistency .

Data dissemination

Data distribution refers to the ability to distribute raw data to multiple systems that need to use the data , The requirement of realizing data heterogeneity . The most common is the distribution of data to ES, Redis Provides search for business , Caching and other services .

In addition to manually distributing data through the messaging mechanism , You can also subscribe to Mysql Of binlog To distribute , Distributing this scene , Need to use RocketMQ To ensure data consistency .


RocketMQ framework


Image source: official document of Alibaba cloud

  • Name Server: Is an almost stateless node , It can be deployed in clusters , In the message queue RocketMQ The naming service is provided in the , Update and discover Broker service . It's a registry .
  • Broker: Message relay role , Responsible for storing messages , Forward a message . It is divided into Master Broker and Slave Broker, One Master Broker Can correspond to multiple Slave Broker, But one Slave Broker There's only one Master Broker.Broker After startup, you need to register yourself to Name Server The operation of ; Then every 30s Regularly send to Name Server Report Topic Routing information .
  • producer : And Name Server One of the nodes in the cluster ( Random ) Build long links (Keep-alive), On a regular basis from Name Server Read Topic Routing information , And provide Topic Service Master Broker Build long links , And timing to Master Broker Send a heartbeat .
  • consumer : And Name Server One of the nodes in the cluster ( Random ) Establish long connection , On a regular basis from Name Server Pull Topic Routing information , And provide Topic Service Master Broker、Slave Broker Establish long connection , And timing to Master Broker、Slave Broker Send a heartbeat .Consumer It can be done from Master Broker Subscribe to news , You can also get it from Slave Broker Subscribe to news , Subscription rules are made by Broker Configuration decision .

RocketMQ Message type

RocketMQ Support rich message types , It can meet the business requirements of multiple scenarios . Different messages have different application scenarios , Here are four common types of messages .

General news

Ordinary news means RocketMQ No feature message in . When there is no special business scenario , It's enough to use regular messages . If there's a special scene , You can use special message types , Like sequence , Affairs, etc. .

The synchronous

The synchronous : The sender sends out a message , The result returned by the server will be obtained synchronously .

Send asynchronously

Send asynchronously : The sender sends out a message , No need to wait for the server to return the result , You can then send the next message . The sender can receive the server response through the callback interface , And process the response results .

Send one way

Send one way : The sender is only responsible for sending messages , It doesn't matter when it's sent out , This way of sending is very fast , There is a risk of losing messages .

Sequential message

Sequential message means that the producer releases messages in a certain order ; Consumers subscribe to messages in a given order , That is to say, the news released first will be received by consumers first .

For example, data distribution scenarios , If we subscribe to Mysql Of binlog To heterogeneous data . If there's no order in the message , There will be data confusion .

For example, add a new one id=1 The data of , And then immediately delete . So there are two messages . The normal order of consumption is to add , Then delete , There is no data at this time . If the message is out of order , The deleted ones are consumed first , And then consume the new , The data is still there , It's not deleted , It leads to inconsistencies .

Timing message

Timing message means that the message has the function of timing sending , When the message is sent to the server , Not immediately delivered to consumers . Instead, it will not be delivered to consumers for consumption until the time specified by the message .

Delayed messages are also known as timed messages , Timing messages are sent at a certain point in time , such as 2020-11-11 12:00:00 send out .

The delay message is generally based on the current sending time and how long it will be sent , For example, the current time is 2020-09-10 12:00:00, Delay 10 minute , Then the message will be sent in 2020-09-10 12:10:00 Delivery to consumers .

The timing message can be used in scenarios such as automatic cancellation of an order when the order is overdue .

Transaction message

RocketMQ Provide similar X/Open XA Distributed transaction function of , adopt RocketMQ Transaction message can achieve the final consistency of distributed transaction .

Interaction process :


Image source: official document of Alibaba cloud

  1. The sender first sends a semi transaction message to RocketMQ Server side .
  2. RocketMQ The server receives the message , Then after the message is persisted successfully , Return to the sender Ack Confirm that the message has been sent successfully , In this case, the message is a semi transaction message , Not delivered to consumers .
  3. Receiving a semi transactional message Ack after , The sender starts executing the local transaction logic .
  4. The sender submits the second confirmation to the server according to the local transaction execution result , If the local transaction is executed, the message will be processed Commit, If the execution fails, the message will be processed Rollback, Server received Commit State marks semi transactional messages as deliverable , The consumer will eventually receive the message ; Server received Rollback In the state, the semi transaction message is deleted , The consumer will not receive the message .

  5. If something unexpected happens , step 4 There is no secondary confirmation of the message , After waiting for a fixed time, the server will send a message back check to the message .

  6. After the sender receives the message to check back , You need to check the final result of the local transaction execution of the corresponding message . The sender submits the second confirmation according to the final state of the local transaction , The server still follows the steps 4 Operate on semi transactional messages .

Best practices

Message retry

After the consumer fails to consume ,RocketMQ The server will deliver the message again , Know the news of consumers' successful consumption , Of course, there is a limit to the number of retries , Default 16 Time .

Message retrying ensures that messages are not lost to some extent , Try again to achieve the ultimate goal of consumption . It should be noted that consumers must wait for the success of local business when consuming ACK( Consumer to confirm ), Otherwise, there will be consumption failure , But already ACK, Messages will not be delivered repeatedly .

If we adopt the way of asynchronous consumption , Need to be asynchronous to synchronous , Wait until the asynchronous operation is finished ACK, For details, please refer to an article I wrote earlier

Finally, we need to do the corresponding monitoring , If you try again 4,5 Time or failure , Basically, it's a failure to try again later . It's time for developers to know , If it's time to deal with it, it's time to intervene . Or directly monitor the dead letter queue .

Message filtering

Message topic , It is generally used for the unified classification of a class of messages . For example, order subject , But there are many kinds of messages on orders . For example, create an order , Cancel the order, etc .

Different types of messages have different business processes , We can define the message format in a unified way , Then through a field to distinguish the message type to do different business logic . The bad thing is that all the news is pushed to the consumer side , You can't spend on demand .

stay RocketMQ You can specify tag, adopt tag To differentiate message types . Consumers can base on Tag stay RocketMQ The server completes the message filtering , To ensure that consumers end up spending only on the type of message they care about .

I once met a tag There is no proper way to use it , only one MQ example , use tag To distinguish the environment . All the messages are in one subject , Test environment consumes test environment tag, Online consumption online tag.

The problem with this approach is that the message is not isolated , Online and offline messages are all together . The other is tag It's fixed as a division of the environment , Cannot be used for message type scenarios , As a result, only more than one can be built topic To carry multiple business message types .


Consumption patterns

RocketMQ There are two patterns of consumption , Cluster consumption and broadcast consumption .

Cluster consumption :


Consumers deploy multiple instances, which we call a cluster , Cluster consumption will only be consumed by one instance .

Suitable for most business scenarios , In most scenarios, our message is only allowed to be consumed once , And only one consumer can spend , For example, the payment callback scenario , If a message is consumed by multiple instances at the same time , Then it will appear to modify the order status at the same time , At the same time to reduce the inventory situation .

Radio consumption :


Broadcast consumption will cause each instance in the cluster to consume once .

For example, we use local caching , When the data changes , We need to refresh the local cache of each node , So every node needs to receive a message .

Consumption idempotent

Idempotent problem , Whether in the API Request scenario or message consumption scenario , Will meet . A message can't be consumed many times, which must be guaranteed , Because we can't guarantee that the sender will not send the message many times , There is no guarantee that messages will not be delivered repeatedly .

RocketMQ Of Exactly-Once Delivery semantics , It is used to solve idempotent problems .Exactly-Once The message sent to the message system can only be processed by the consumer and only once , Even if the production side retries the message sending, resulting in repeated delivery of a message , The message is only consumed once on the consumer side .

The best idempotent method is to have a unique business ID , Although every message has MessageId, But it is not recommended to use MessageId To make idempotent judgments , When sending a message , You can set one for each message MessageKey, This MessageKey Can be used to do business unique identification .

How to deal with idempotent , Let's not go into details . You can refer to an article I wrote before, A general implementation scheme of idempotent .


Local transaction message encapsulation

It's about transaction messages ,RocketMQ The transaction message of the system adopts a two-phase commit mode . And combined with the mechanism of message anti checking to ensure the final consistency .

In terms of use , Each business scenario needs to implement a reverse query logic , A little annoyed. .

Here's another way that is often used , It's local transaction news . The scheme of local message table was originally ebay Proposed , Local transaction messages need to create a message table in the database corresponding to the service , When you send a message, you don't actually send it to MQ, Instead, insert a piece of message data into the message table .

The inserted action is the same transaction as the local business logic , If the local transaction is executed successfully , It's the news that makes it to the table , Will be sent to MQ, Local transaction failed , Message data rollback .

Then we need to have a special program to pull the unsent messages from the message list and post them to MQ, If the delivery fails , You can always try again , Until success or human intervention .


The message is written to the message table , And then I'll give it all the time MQ send out , There's no problem with this step . If MQ After receiving the message , The news is still there PageCache In the time ,Broker It's down. , At this time, there will be news loss . Of course, you can also use synchronous brush disk and other ways to avoid loss . If we were to brush the disk asynchronously , Is there any way to keep the message from being lost ?

As we mentioned earlier ,RocketMQ There will be a back look mechanism for transaction messages of , The way message tables are , There also needs to be a mechanism to ensure that messages are consumed , Otherwise, you need to keep retrying to send messages , Until the message is consumed .

There needs to be a field in the message table to identify the current status of the message , such as Not sent , Has been sent , Have consumed . When the message is still unsent, it is sent to MQ, If the transmission is successful , The status is sent . But after a few minutes , The status is still sent , It's time to do some actions .

In this scene , It's possible that consumers can't keep up with the pace of production , The news is piling up , The news has not been consumed . Another possibility is whether the message is missing ?

The corresponding message stack data can be obtained to determine whether the message is stacked , If not, resend the message to MQ, Knowing that the message is being consumed .

The problem is that information is being consumed , How should I know? ?

Like the cloud service I use , There is a corresponding Open API You can directly query the message trace . Open source should also have , No careful study of , It should be similar to the commercial version .

According to the message track, we can know whether the message has been consumed , So that's the end of the process . Message sent to MQ If it fails, it will try again , If the message has not been consumed for a long time , It will also resend , Even if you end up in the dead letter queue , It can also be manually intervened by monitoring the dead letter queue , It must be final consistency .

Compared with the self-contained transaction information , The way of local message table does not need to implement backlookup logic , But add a message table , At the same time, we should also provide various kinds of transmission , Check logic , It's a lot of trouble, too . Especially when there's a lot of news , How to quickly send the message in the message table , A lot of processing is needed , Simple look-up table polling is not suitable for a large number of cases .

Both methods can be used , We can achieve the purpose we want .

Reference code

Code related to local transaction messages can be searched on wechat 「 Ape world 」, Reply key 「kitty」 Get it .

It's not easy to code words , If you can, let's have a triple shot , thank !

About author : Yin Jihuan , Simple technology enthusiasts ,《Spring Cloud Microservices - Full stack technology and case analysis 》, 《Spring Cloud Microservices introduction Actual combat and advanced 》 author , official account Ape world Originator .

I have compiled a complete set of learning materials , Those who are interested can search through wechat 「 Ape world 」, Reply key 「 Learning materials 」 Get what I've sorted out Spring Cloud,Spring Cloud Alibaba,Sharding-JDBC Sub database and sub table , Task scheduling framework XXL-JOB,MongoDB, Reptiles and other related information .

本文为[Yin Jihuan]所创,转载请带上原文链接,感谢

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云