Apache RocketMQ Is a Low latency 、 High concurrency 、 High availability 、 Highly reliable distributed message middleware . Message queue RocketMQ It can provide asynchronous decoupling and peak shaving for distributed application systems , At the same time, it also has the massive message accumulation required by Internet applications 、 High throughput 、 Features such as reliable retries .
With the popularity of microservice Architecture , It's very important to sort out the relationship between services . Asynchronous decoupling can reduce the coupling between services , At the same time, it can also improve the throughput of services .
There are many business scenarios that use asynchronous decoupling , Because the business of every industry will be different , With some more general business to illustrate, I believe everyone can understand .
For example, the single business scenario in the e-commerce industry , In terms of the simplest order process , The order process is as follows ：
After we succeed in the following list , Users pay , Payment completion will have a logic called payment callback , You need to do some business logic in callback . Let's first look at the time required for synchronization , Here's the picture ：
The order process above starts from 3 To 5 They can be processed by asynchronous processes , For users , After the payment is completed, he does not need to pay attention to the following process . Just take care of it backstage , This simplifies three steps , Improve callback processing time .
Peak shaving and valley filling refers to the impact of large flow , utilize RocketMQ Can withstand the instantaneous large flow , Protect the stability of the system , Enhance user experience .
In the e-commerce industry , The most common traffic shock is the second kill , utilize RocketMQ To achieve a complete second kill business or with a lot of work to do , Beyond the scope of this article , I can talk to you alone later . What I want to tell you is that scenes like this can be used RocketMQ To carry high concurrence , The premise is that the business scenario supports asynchronous processing .
as everyone knows , Distributed transactions have 2PC,TCC, Finally, consistency, etc . Among them, using message queue to do the final consistency scheme is more commonly used .
In the business scenario of e-commerce , The core business related to transactions must ensure data consistency . By introducing a message queue RocketMQ Version of distributed transactions , It can realize the decoupling between systems , It can also ensure the final data consistency .
Data distribution refers to the ability to distribute raw data to multiple systems that need to use the data , The requirement of realizing data heterogeneity . The most common is the distribution of data to ES, Redis Provides search for business , Caching and other services .
In addition to manually distributing data through the messaging mechanism , You can also subscribe to Mysql Of binlog To distribute , Distributing this scene , Need to use RocketMQ To ensure data consistency .
Image source: official document of Alibaba cloud
RocketMQ Support rich message types , It can meet the business requirements of multiple scenarios . Different messages have different application scenarios , Here are four common types of messages .
Ordinary news means RocketMQ No feature message in . When there is no special business scenario , It's enough to use regular messages . If there's a special scene , You can use special message types , Like sequence , Affairs, etc. .
The synchronous ： The sender sends out a message , The result returned by the server will be obtained synchronously .
Send asynchronously ： The sender sends out a message , No need to wait for the server to return the result , You can then send the next message . The sender can receive the server response through the callback interface , And process the response results .
Send one way ： The sender is only responsible for sending messages , It doesn't matter when it's sent out , This way of sending is very fast , There is a risk of losing messages .
Sequential message means that the producer releases messages in a certain order ; Consumers subscribe to messages in a given order , That is to say, the news released first will be received by consumers first .
For example, data distribution scenarios , If we subscribe to Mysql Of binlog To heterogeneous data . If there's no order in the message , There will be data confusion .
For example, add a new one id=1 The data of , And then immediately delete . So there are two messages . The normal order of consumption is to add , Then delete , There is no data at this time . If the message is out of order , The deleted ones are consumed first , And then consume the new , The data is still there , It's not deleted , It leads to inconsistencies .
Timing message means that the message has the function of timing sending , When the message is sent to the server , Not immediately delivered to consumers . Instead, it will not be delivered to consumers for consumption until the time specified by the message .
Delayed messages are also known as timed messages , Timing messages are sent at a certain point in time , such as 2020-11-11 12:00:00 send out .
The delay message is generally based on the current sending time and how long it will be sent , For example, the current time is 2020-09-10 12:00:00, Delay 10 minute , Then the message will be sent in 2020-09-10 12:10:00 Delivery to consumers .
The timing message can be used in scenarios such as automatic cancellation of an order when the order is overdue .
RocketMQ Provide similar X/Open XA Distributed transaction function of , adopt RocketMQ Transaction message can achieve the final consistency of distributed transaction .
Interaction process ：
Image source: official document of Alibaba cloud
The sender submits the second confirmation to the server according to the local transaction execution result , If the local transaction is executed, the message will be processed Commit, If the execution fails, the message will be processed Rollback, Server received Commit State marks semi transactional messages as deliverable , The consumer will eventually receive the message ; Server received Rollback In the state, the semi transaction message is deleted , The consumer will not receive the message .
If something unexpected happens , step 4 There is no secondary confirmation of the message , After waiting for a fixed time, the server will send a message back check to the message .
After the sender receives the message to check back , You need to check the final result of the local transaction execution of the corresponding message . The sender submits the second confirmation according to the final state of the local transaction , The server still follows the steps 4 Operate on semi transactional messages .
After the consumer fails to consume ,RocketMQ The server will deliver the message again , Know the news of consumers' successful consumption , Of course, there is a limit to the number of retries , Default 16 Time .
Message retrying ensures that messages are not lost to some extent , Try again to achieve the ultimate goal of consumption . It should be noted that consumers must wait for the success of local business when consuming ACK( Consumer to confirm ), Otherwise, there will be consumption failure , But already ACK, Messages will not be delivered repeatedly .
If we adopt the way of asynchronous consumption , Need to be asynchronous to synchronous , Wait until the asynchronous operation is finished ACK, For details, please refer to an article I wrote earlier https://mp.weixin.qq.com/s/Bbh1GDpmkLhZhw5f0POJ2A.
Finally, we need to do the corresponding monitoring , If you try again 4,5 Time or failure , Basically, it's a failure to try again later . It's time for developers to know , If it's time to deal with it, it's time to intervene . Or directly monitor the dead letter queue .
Message topic , It is generally used for the unified classification of a class of messages . For example, order subject , But there are many kinds of messages on orders . For example, create an order , Cancel the order, etc .
Different types of messages have different business processes , We can define the message format in a unified way , Then through a field to distinguish the message type to do different business logic . The bad thing is that all the news is pushed to the consumer side , You can't spend on demand .
stay RocketMQ You can specify tag, adopt tag To differentiate message types . Consumers can base on Tag stay RocketMQ The server completes the message filtering , To ensure that consumers end up spending only on the type of message they care about .
I once met a tag There is no proper way to use it , only one MQ example , use tag To distinguish the environment . All the messages are in one subject , Test environment consumes test environment tag, Online consumption online tag.
The problem with this approach is that the message is not isolated , Online and offline messages are all together . The other is tag It's fixed as a division of the environment , Cannot be used for message type scenarios , As a result, only more than one can be built topic To carry multiple business message types .
RocketMQ There are two patterns of consumption , Cluster consumption and broadcast consumption .
Cluster consumption ：
Consumers deploy multiple instances, which we call a cluster , Cluster consumption will only be consumed by one instance .
Suitable for most business scenarios , In most scenarios, our message is only allowed to be consumed once , And only one consumer can spend , For example, the payment callback scenario , If a message is consumed by multiple instances at the same time , Then it will appear to modify the order status at the same time , At the same time to reduce the inventory situation .
Radio consumption ：
Broadcast consumption will cause each instance in the cluster to consume once .
For example, we use local caching , When the data changes , We need to refresh the local cache of each node , So every node needs to receive a message .
Idempotent problem , Whether in the API Request scenario or message consumption scenario , Will meet . A message can't be consumed many times, which must be guaranteed , Because we can't guarantee that the sender will not send the message many times , There is no guarantee that messages will not be delivered repeatedly .
RocketMQ Of Exactly-Once Delivery semantics , It is used to solve idempotent problems .Exactly-Once The message sent to the message system can only be processed by the consumer and only once , Even if the production side retries the message sending, resulting in repeated delivery of a message , The message is only consumed once on the consumer side .
The best idempotent method is to have a unique business ID , Although every message has MessageId, But it is not recommended to use MessageId To make idempotent judgments , When sending a message , You can set one for each message MessageKey, This MessageKey Can be used to do business unique identification .
How to deal with idempotent , Let's not go into details . You can refer to an article I wrote before https://mp.weixin.qq.com/s/9fhqnbeXPz7-7x0Eadd8DA, A general implementation scheme of idempotent .
It's about transaction messages ,RocketMQ The transaction message of the system adopts a two-phase commit mode . And combined with the mechanism of message anti checking to ensure the final consistency .
Here's another way that is often used , It's local transaction news . The scheme of local message table was originally ebay Proposed , Local transaction messages need to create a message table in the database corresponding to the service , When you send a message, you don't actually send it to MQ, Instead, insert a piece of message data into the message table .
The inserted action is the same transaction as the local business logic , If the local transaction is executed successfully , It's the news that makes it to the table , Will be sent to MQ, Local transaction failed , Message data rollback .
Then we need to have a special program to pull the unsent messages from the message list and post them to MQ, If the delivery fails , You can always try again , Until success or human intervention .
The message is written to the message table , And then I'll give it all the time MQ send out , There's no problem with this step . If MQ After receiving the message , The news is still there PageCache In the time ,Broker It's down. , At this time, there will be news loss . Of course, you can also use synchronous brush disk and other ways to avoid loss . If we were to brush the disk asynchronously , Is there any way to keep the message from being lost ？
As we mentioned earlier ,RocketMQ There will be a back look mechanism for transaction messages of , The way message tables are , There also needs to be a mechanism to ensure that messages are consumed , Otherwise, you need to keep retrying to send messages , Until the message is consumed .
There needs to be a field in the message table to identify the current status of the message , such as Not sent , Has been sent , Have consumed . When the message is still unsent, it is sent to MQ, If the transmission is successful , The status is sent . But after a few minutes , The status is still sent , It's time to do some actions .
In this scene , It's possible that consumers can't keep up with the pace of production , The news is piling up , The news has not been consumed . Another possibility is whether the message is missing ？
The corresponding message stack data can be obtained to determine whether the message is stacked , If not, resend the message to MQ, Knowing that the message is being consumed .
The problem is that information is being consumed , How should I know? ？
Like the cloud service I use , There is a corresponding Open API You can directly query the message trace . Open source should also have , No careful study of , It should be similar to the commercial version .
According to the message track, we can know whether the message has been consumed , So that's the end of the process . Message sent to MQ If it fails, it will try again , If the message has not been consumed for a long time , It will also resend , Even if you end up in the dead letter queue , It can also be manually intervened by monitoring the dead letter queue , It must be final consistency .
Compared with the self-contained transaction information , The way of local message table does not need to implement backlookup logic , But add a message table , At the same time, we should also provide various kinds of transmission , Check logic , It's a lot of trouble, too . Especially when there's a lot of news , How to quickly send the message in the message table , A lot of processing is needed , Simple look-up table polling is not suitable for a large number of cases .
Both methods can be used , We can achieve the purpose we want .
Code related to local transaction messages can be searched on wechat 「 Ape world 」, Reply key 「kitty」 Get it .
It's not easy to code words , If you can, let's have a triple shot , thank ！
About author ： Yin Jihuan , Simple technology enthusiasts ,《Spring Cloud Microservices - Full stack technology and case analysis 》, 《Spring Cloud Microservices introduction Actual combat and advanced 》 author , official account Ape world Originator .
I have compiled a complete set of learning materials , Those who are interested can search through wechat 「 Ape world 」, Reply key 「 Learning materials 」 Get what I've sorted out Spring Cloud,Spring Cloud Alibaba,Sharding-JDBC Sub database and sub table , Task scheduling framework XXL-JOB,MongoDB, Reptiles and other related information .