How does Alibaba rocketmq make double 11 peak 0 fail?

Alibaba cloud developer 2021-04-16 20:12:50
alibaba rocketmq make double peak

brief introduction :2020 The peak of trading on November 11 reached 58.3 W pen / second , Message middleware RocketMQ For years to come 0 The fault silky smooth perfectly supports all kinds of business smoothly promoted by the whole group .

 The first figure .png

author | Yu'an
source | Alibaba cloud official account

2020 The peak of trading on November 11 reached 58.3 W pen / second , Message middleware RocketMQ For years to come 0 The fault silky smooth perfectly supports all kinds of business smoothly promoted by the whole group .2020 Promotion of the 11th National Congress of the Communist Party of China , Message middleware RocketMQ The following changes have taken place :

  • Yunyuan biochemical practice : Complete the cloud original biochemical transformation at the operation and maintenance level , Realization Kubernetes turn .
  • performance optimization : Message filtering optimizes transaction cluster performance 30%.
  • New consumption model : Provide a new consumption model for delay sensitive business , Reduce due to release 、 The consumption delay caused by restart and other scenarios .

Yunyuan biochemical practice

1. background

Kubernetes As an important link in the practice of Yunyuan biochemical technology stack , Its ecology has been gradually established and enriched . at present , Serving within the group RocketMQ Clusters have a huge scale and a variety of historical factors , Therefore, there are quite a few pain points in operation and maintenance , We hope to try to find the corresponding solution through the cloud native technology stack , At the same time, it can reduce the cost and improve the efficiency , To achieve unattended automatic operation and maintenance .

Message oriented middleware is as early as 2016 year , Through the middleware deployment platform provided by the internal team, containerization and automatic release are realized , Overall operation and maintenance ratio 2016 There was a lot of improvement a year ago , But as a stateful service , There are still many problems in operation and maintenance .

The middleware deployment platform helps us complete the resource application , Container creation 、 initialization 、 Image installation and a series of basic work , But because middleware products have their own different deployment logic , So in the release of applications , Each application has its own customization . The development of middleware deployment platform does not fully understand the group RocketMQ What's the deployment process for .

So in 2016 In the year , The deployment platform requires us to implement the application release code of message middleware by ourselves . Although the deployment platform greatly improves our operation and maintenance efficiency , And even one click Publishing , But there are also many problems with such a scheme . It is obvious that , When our release logic changes , You also need to modify the code corresponding to the deployment platform , We need to deploy platform upgrades to support us , In a recent popular saying , It's not cloud native .

Also replace in the failure machine 、 Cluster shrinking and other operations , There are some manual jobs , Such as shear flow , Confirmation of stacked data, etc . We have tried to integrate more message middleware's own operation and maintenance logic in the deployment platform , But write your own business code in the projects of other teams , It's also an unfriendly implementation , So we hope to pass Kubernetes To realize the message middleware's own operator . We also hope to reduce the cost of our machines and reduce the complexity of primary and standby operation and maintenance by using the multi replica capability of cloud disks after cloud .

After a period of follow-up and discussion , Finally, the internal team undertakes the task of building cloud native application operation and maintenance platform again , And rely on the experience of middleware deployment platform , With the help of cloud native technology stack , Realize the breakthrough of automatic operation and maintenance of stateful applications .

2. Realization


The overall implementation scheme is shown in the figure above , Through custom CRD Abstract the business model of message middleware , The original business release and deployment logic in middleware deployment platform is sunk into message middleware's own operator in , Hosted internally Kubernetes On the platform . The platform is responsible for the production of all the containers 、 Initialization and baseline deployment of all online environments within the group , Screen out IaaS All the details of the layer .

Operator Took on all the new clusters 、 Capacity expansion 、 Shrinkage capacity 、 The whole logic of migration , Including each pod Corresponding brokerName Automatic generation 、 The configuration file , Various switches configured according to different functions of the cluster , Synchronous replication of metadata and so on . At the same time, some previous manual operations , For example, the flow observation when cutting flow , The stack data observation before offline is all integrated into operator in . When we need to modify all kinds of operation and maintenance logic again , And no longer rely on a generic implementation , Modify your own operator that will do .

Finally, the actual deployment on the line removes all the replica Standby machine . stay Kubernetes The idea of , The state of each instance in a cluster is consistent , No dependency , However, if the original master-slave deployment scheme of message middleware is followed , There is a strict correspondence between master and slave , And in the online and offline release process, there are strict sequence requirements , This deployment mode is in Kubernetes Is not advocated under the system of . If we still adopt the above old architecture , It will lead to the complexity and uncontrollability of instance control , At the same time, we also hope to follow more Kubernetes The operation and maintenance concept of .

After clouding ECS Using a high-speed cloud disk , The underlying layer will make multiple backups of the data , So the availability of data is guaranteed . And the performance of high-speed cloud disk is fully satisfied MQ Synchronous brush set , therefore , At this point, you can change the previous asynchronous disk brushing to synchronous , Ensure that the message is not lost when writing . Cloud native mode , All instance environments are consistent , Relying on container technology and Kubernetes Technology , Any instance can be hung up ( Including the hang up caused by downtime ), It's self-healing , Fast recovery .

After the availability of data and service , The architecture of the whole cloud can be simplified , Only broker The concept of , There's no master-slave .

3. Promote verification


Above, Kubernetes After the launch, the double 11 will promote the sending of the day RT Statistics , It can be seen that the transmission during the promotion period RT More stable , Overall as expected , Yunyuan biochemical practice has completed a key milestone .

performance optimization

1. background

RocketMQ It's been seven years in a row 0 Failure support group's double 11 Promotion . since RocketMQ Since the birth of , In order to be able to fully carry all kinds of key services including core links such as platform transaction messages in group business , Reuse the original upper layer protocol logic , It makes all kinds of business parties switch to RocketMQ On , And at the same time fully enjoy the more stable and powerful RocketMQ Various features of message middleware .

At present , The number of business parties applying to subscribe to the core trading messages of Taiwan has been continuously increasing , And as the complexity of various businesses increases , The configuration of message subscription on the business side has also become more complicated , As a result, the calculation logic of transaction cluster filtering becomes more complex . Some of these business parties follow the old protocol logic (Header Filter ), Some use RocketMQ Peculiar SQL Filter .

2. The main cost is

Now within the group RocketMQ Most of the machine costs are related to transaction message clusters , During the double eleven zero peak period , The peak value of trading cluster is directly proportional to the peak value of transaction , The addition of complex subscriptions added each year brings extra CPU Filter computing logic , Trading clusters are the places where the cost of machines in China has increased most .

3. The optimization process

For historical reasons , Most business parties still use Header Filter , The internal implementation is actually aviator expression . Carefully observe the business side filtering expression of transaction message cluster , You can see that most of them are designated as similar to MessageType == xxxx Such conditions . Look over aviator It can be found that such conditions will eventually be called Java String comparison of String.compareTo().

Because a lot of business news includes MessageType, There are at least a few thousand on record , As business processes become more complex ,MessageType The growth rate is more diverse . As the trading peak increases , The peak value of trading news increased proportionally , Overlay this part with more complex filtering , A growing future , The cost of trading clusters is likely to increase exponentially with the peak value of transactions , So determined to optimize this part .

The original filtering process is as follows , Each transaction message needs to match different group The subscription relation expression of , If it fits the expression , Then select the corresponding group The delivery machine . As shown in the figure below :


The idea of optimizing this process needs some inspiration , Here with the help of the idea of database index : The original process can treat all subscriber's filter expressions as database records , Each message filtering is equivalent to a database query with specific conditions , Put all matching queries ( news ) The record of ( Filter expression ) Pick it up as a result . To speed up query results , You can choose MessageType Index as an index field , Each query changes to match first MessageType Main index , Then, the records matching the primary index are subjected to other conditions ( The following figure sellerId and testA ) matching , The optimization process is shown in the figure below :


After the above optimization process is determined , There are two technical points to focus on :

  • How to extract... From each expression MessageType Field ?
  • How to MessageType Fields are indexed ?

For technical points 1 , Need to aim at aviator The compilation process of hook , thorough aviator After the source , You can find aviator The compiler is typical of Recursive descent, At the same time, we need to consider the short-circuit problem of the parent expression after extraction .

In the process of compiling, it aims at messageType==XXX After this type is extracted , Put the original message==XXX Turn into true/false Two cases , And then for true、false The short circuit of the expression can get the optimized extraction of the expression . for example :

 expression :
messageType=='200-trade-paid-done' && buyerId==123456
Extract into two subexpressions :
subexpression 1(messageType==200-trade-paid-done):buyerId==123456
subexpression 2(messageType!=200-trade-paid-done):false
  • Specific to the aviator In the realization of , Expression compilation will put each token Construct a List , Similar to the figure below ( For the sake of understanding , The green box is token , The other boxes represent the combination of specific conditions of the expression ):


Extracted messageType , There are two situations :

  • Situation 1 :messageType == '200-trade-paid-done', Then put the previous token The position of the merger into true, Then the expression is short circuited , Finally, it is optimized to buyerId==123456 , As follows :


  • Situation two :messageType != '200-trade-paid-done', Then put the previous token The position of the merger into false , The expression is short circuited , Finally, it is optimized to false , As follows :


So it's done messageType Extraction of . There may be a question here , Why consider the above situation 2 ,messageType != '200-trade-paid-done', This is because when multiple conditions have to be considered , such as :

(messageType=='200-trade-paid-done' && buyerId==123456) || (messageType=='200-trade-success' && buyerId==3333)

We have to take into account the situation that is not equal to . Empathy , If you consider the nesting of multiple expressions , Short circuit calculation needs to be carried out step by step . But the whole logic is similar , No more details here .

After finishing the technical point 1, We continue to focus on Technology 2, Considering efficient filtration , Use it directly HashMap Structure can be indexed , Namely the messageType The value of HashMap Of key , Take the extracted subexpression as HashMap Of value , In this way, each filter passes directly through hash Calculation can filter out most of the inappropriate expressions , Greatly improve the filtration efficiency .

3. The optimization effect

This optimization mainly reduces CPU Calculation logic , According to the performance comparison before and after optimization , We find that the higher the complexity of subscriber subscription expression in different transaction clusters , The better the optimization effect , This is in line with our expectations , The biggest of them CPU Optimization has 32% The promotion of , Greatly reduced this year RocketMQ The deployment machine cost of .

New consumption model —— POP consumption

1. background

RocketMQ Of PULL Consumption is abnormal for machines hang It's not very friendly . If you encounter a client machine hang live , But in a half dead state , And broker When the heart of your heart doesn't break , client rebalance The consumption queue will still be allocated to hang On the machine , also hang When the machine consumption speed is very slow or even unable to consume , This will lead to the accumulation of consumption . In addition, there are similar services Broker When it was released , It will also be due to the client many times rebalance Lead to consumption delay and other unavoidable problems . As shown in the figure below :


When Pull Client 2 happen hang Machine time , It's assigned three Broker Upper Q2 There are serious red accumulations . For this , We've added a new consumption model —— POP consumption , This kind of stability problem can be solved . As shown in the figure below :


POP Consumption , Three clients don't need rebalance To allocate the consumption queue , In its place , They all use POP Request all broker Get information to spend .broker The internal will allocate the messages of its three queues to the requested ones according to a certain algorithm POP Client. Even if Pop Client 2 appear hang, But messages from internal queues also make Pop Client1 and Pop Client2 Consumption . So you hang The machine avoids the accumulation of consumption .

2. Realization

POP Consumption and the original PULL Consumption comparison , The biggest thing is that it weakens the concept of queues ,PULL Consumption needs the client to pass rebalance hold broker The queue is assigned , So as to consume and allocate to their own exclusive queue , new POP Consumption , The client machine will go directly to each broker For request consumption , broker The message is assigned back to the waiting machine . After that, the client will return the corresponding Ack Result notification broker,broker Mark the message consumption result again , If the timeout does not respond or the consumption fails , We'll try again .


POP The architecture of consumption is shown in the figure above .Broker For every time POP Request , There are three operations :

  • Lock the corresponding queue , And then from store Layer gets the message of the queue ;
  • And then write CK news , Indicates that the acquired message is to be POP consumption ;
  • Finally submit the current locus , And release the lock .

CK The message actually records POP The timing message of the specific location of the message , When the client time-out does not respond ,CK The news will be re broker consumption , And then put CK The message is written to the retrying queue at the location of the message . If broker Receiving the consumption result of the client Ack , Delete corresponding CK news , Then, according to the specific results, we can judge whether we need to try again .

From the overall process, we can see that ,POP Consumption doesn't need reblance , You can avoid rebalance The consumption delay brought about by it , At the same time, the client can consume broker All the queues of , So you can avoid machines hang And that leads to the problem of accumulation .


Link to the original text :

Copyright notice : The content of this article is contributed by alicloud real name registered users , The copyright belongs to the original author , Alicloud developer community does not own its copyright , It also does not bear the corresponding legal liability . Please check the specific rules 《 Alicloud developer community user service agreement 》 and 《 Alibaba cloud developer community intellectual property protection guidelines 》. If you find any suspected plagiarism in this community , Fill in the infringement complaint form to report , Once verified , The community will immediately delete the suspected infringement content .
本文为[Alibaba cloud developer]所创,转载请带上原文链接,感谢

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云