A message is only to be ISR All in Follower From the Leader Copy the past to be considered submitted . This prevents some data from being written in Leader, I haven't got any time to be Follower Replication goes down , Data is lost . And for Producer for , It can choose whether to wait for messages commit, This can be done by request.required.acks To set up . This mechanism ensures that as long as ISR One or more of them follower, One of them is commit You won't lose your message .
What is? ISR？ Reference here ：【 Big data beeps 20210123】 Don't ask , Asking is Kafka Highly reliable
There is a very important problem when Leader It's down. , How to be in Follower To elect a new Leader, because Follower Maybe a lot behind or directly crash 了 , So make sure you choose the latest Follower As new Leader. One basic principle is this , If Leader No more , new Leader Must have the original Leader commit All the news of . This requires a compromise , If Leader In a message by commit Wait for more Follower confirm , So after it goes down, there are more Follower Can be new Leader, But it can also cause a drop in throughput .
A very common election Leader The way is “ The minority is subordinate to the majority “,Kafka Not in this way . In this mode , If we had 2f+1 Copies , So in commit You have to make sure you have f+1 individual replica Copy the message , At the same time, in order to ensure that the new Leader, The number of failed copies cannot exceed f individual . There's a big advantage to this approach , The delay of the system depends on the fastest machines , In other words, for example, the number of copies is 3, So the delay depends on the fastest one Follower Not the slowest one .“ The minority is subordinate to the majority ” There are also some disadvantages in the way of , In order to ensure Leader The normal conduct of the election , It can tolerate failure Follower The number is relatively small , If you want to tolerate 1 individual Follower Hang up , Well, at least 3 More than copies , If you want to tolerate 2 individual Follower Hang up , There must be 5 More than copies . in other words , In order to ensure high fault tolerance in the production environment , There has to be a lot of copies , And a large number of copies will lead to a sharp decline in performance in the case of large amounts of data . This algorithm is more used in Zookeeper This shared cluster configuration is rarely used in systems that require large amounts of data .HDFS Of HA Functionality is also based on “ The minority is subordinate to the majority ” The way , But its data storage is not in this way .
actually ,Leader There are a lot of election algorithms , such as Zookeeper Of Zab、Raft as well as Viewstamped Replication. and Kafka What is used Leader The election algorithm is more like Microsoft's PacificA Algorithm .
Kafka stay Zookeeper Where is each Partition Dynamic maintenance of a ISR, This ISR All in replica They're all following Leader, Only ISR In order to be elected to Leader The possibility of （unclean.leader.election.enable=false）. In this mode , about f+1 Copies , One Kafka Topic Can guarantee not to lose already commit Toleration on the premise of news f Failed copies , In most usage scenarios , This model is very beneficial . in fact , In order to tolerate f Failed copies ,“ The minority is subordinate to the majority ” And ISR stay commit The number of copies to wait before is the same , however ISR The total number of copies required is almost “ The minority is subordinate to the majority ” Half of the way .
Mentioned above , stay ISR At least one of them Follower when ,Kafka You can make sure that you have commit Data of is not lost , But if one of them Partition All of the replica It's all gone , There is no guarantee that the data will not be lost . In this case, there are two possible solutions ：
This requires a simple choice between usability and consistency . Be sure to wait if ISR Medium replica“ live ” To come over , The unavailability time may be relatively long . And if the ISR All of the replica No way “ live ” Over here. , Or the data is missing , This Partition Will never be available . Select first “ live ” Over here replica As Leader, And this replica No ISR Medium replica, So even if it doesn't guarantee that it already includes everything that has been commit The news of , It will also become Leader And as a Consumer Data source . By default ,Kafka Use the second strategy , namely unclean.leader.election.enable=true, You can also set this parameter to false To enable the first strategy .
unclean.leader.election.enable This parameter is for leader The election of 、 The availability of the system and the reliability of the data have a crucial impact . Let's analyze several typical scenarios .
If the picture above shows , Suppose that one Partition The number of copies in is 3,replica-0, replica-1, replica-2 Store separately Broker0, Broker1 and Broker2 in .AR=(0,1,2),ISR=(0,1). Set up request.required.acks=-1, min.insync.replicas=2,unclean.leader.election.enable=false. There will be Broker0 Also known as Broker0 At first Broker0 by Leader,Broker1 by Follower.
This article is from WeChat official account. - Big data is fun （havefun_bigdata）
The source and reprint of the original text are detailed in the text , If there is any infringement , Please contact the email@example.com Delete .
Original publication time ： 2021-01-24
Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .