Click on the above In blue , choice “ Set to star ”
reply ” resources “ Get more surprises
Kafka As a commercial message middleware , The importance of message reliability can be imagined . This paper starts from Producter Go to Broker Send a message 、Topic Partition copy and Leader Election several angles introduce the reliability of data .
Topic Partition copy
Producer Go to Broker Send a message
If we are going to Kafka The corresponding subject sends a message , We need to pass Producer complete . We talked about Kafka The theme corresponds to multiple partitions , There are multiple replicas under each partition ; In order to let users set data reliability , Kafka stay Producer It provides message confirmation mechanism . That is to say, we can decide to send a message to several copies of the corresponding partition through configuration . You can define Producer Through acks Parameter assignment （ stay 0.8.2.X The previous version was through request.required.acks Parameter setting ）.
This parameter supports the following three values ：
acks = 0： It means that if producers can send messages over the network , The message is considered to have been successfully written Kafka. In this case, it's still possible to make mistakes , For example, the sent object cannot be serialized or the network card fails , But if the partition is offline or the whole cluster is unavailable for a long time , Then you don't get any mistakes . stay acks=0 The running speed in mode is very fast （ That's why many benchmarks are based on this pattern ）, You get amazing throughput and bandwidth utilization , But if you choose this model , There's bound to be some loss of information .
acks = 1： Meaning if Leader After receiving the message and writing it to the partition data file （ It doesn't have to be synchronized to disk ） Will return a confirmation or error response . In this mode , If normal Leader The election , The producer will receive a LeaderNotAvailableException abnormal , If the producer can handle this error properly , It will try again to send the message , Eventually the message will arrive safely in the new Leader Where? . However, it is still possible to lose data in this mode , For example, the message has been successfully written Leader, But before the message is copied to follower Before the copy Leader There's a breakdown .
acks = all（ This and request.required.acks = -1 Same meaning ）： signify Leader Before returning an acknowledgement or error response , Will wait for all synchronized copies to receive a silent message . If and min.insync.replicas Parameters together , You can decide at least how many copies can receive the message before you return the confirmation , The producer will try again until the message is successfully submitted . But it's also the slowest way , Because the producer needs to wait for all copies to receive the current message before continuing to send other messages .
According to the actual application scenario , We set up different acks, In order to ensure the reliability of data .
in addition ,Producer Sending messages can also choose to synchronize （ Default , adopt producer.type=sync To configure ） Or asynchronous （producer.type=async） Pattern . If set to asynchronous , Although it will greatly improve the performance of message sending , But this increases the risk of data loss . If you need to make sure the message is reliable , Must be producer.type Set to sync.
Leader The election
Introducing Leader Before the election , Let's take a look at ISR（in-sync replicas） list . For each division leader Will maintain a ISR list ,ISR It's in the list follower Replica Borker Number , Only to keep up with Leader Of follower Copy can be added to ISR Inside , This is through replica.lag.time.max.ms Parameter configuration . Only ISR Only the members in have been chosen as leader The possibility of .
This article is from WeChat official account. - Big data is fun （havefun_bigdata）
The source and reprint of the original text are detailed in the text , If there is any infringement , Please contact the email@example.com Delete .
Original publication time ： 2021-01-20
Participation of this paper Tencent cloud media sharing plan , You are welcome to join us , share .