[big data beeps 20210124] someone asked me about Kafka leader election? I'm not in a panic

Wang Zhiwu 2021-01-24 12:00:22
big data beeps kafka leader


A message is only to be ISR All in Follower From the Leader Copy the past to be considered submitted . This prevents some data from being written in Leader, I haven't got any time to be Follower Replication goes down , Data is lost . And for Producer for , It can choose whether to wait for messages commit, This can be done by request.required.acks To set up . This mechanism ensures that as long as ISR One or more of them follower, One of them is commit You won't lose your message .

What is? ISR? Reference here :【 Big data beeps 20210123】 Don't ask , Asking is Kafka Highly reliable

There is a very important problem when Leader It's down. , How to be in Follower To elect a new Leader, because Follower Maybe a lot behind or directly crash 了 , So make sure you choose the latest Follower As new Leader. One basic principle is this , If Leader No more , new Leader Must have the original Leader commit All the news of . This requires a compromise , If Leader In a message by commit Wait for more Follower confirm , So after it goes down, there are more Follower Can be new Leader, But it can also cause a drop in throughput .

A very common election Leader The way is “ The minority is subordinate to the majority “,Kafka Not in this way . In this mode , If we had 2f+1 Copies , So in commit You have to make sure you have f+1 individual replica Copy the message , At the same time, in order to ensure that the new Leader, The number of failed copies cannot exceed f individual . There's a big advantage to this approach , The delay of the system depends on the fastest machines , In other words, for example, the number of copies is 3, So the delay depends on the fastest one Follower Not the slowest one .“ The minority is subordinate to the majority ” There are also some disadvantages in the way of , In order to ensure Leader The normal conduct of the election , It can tolerate failure Follower The number is relatively small , If you want to tolerate 1 individual Follower Hang up , Well, at least 3 More than copies , If you want to tolerate 2 individual Follower Hang up , There must be 5 More than copies . in other words , In order to ensure high fault tolerance in the production environment , There has to be a lot of copies , And a large number of copies will lead to a sharp decline in performance in the case of large amounts of data . This algorithm is more used in Zookeeper This shared cluster configuration is rarely used in systems that require large amounts of data .HDFS Of HA Functionality is also based on “ The minority is subordinate to the majority ” The way , But its data storage is not in this way .

actually ,Leader There are a lot of election algorithms , such as Zookeeper Of Zab、Raft as well as Viewstamped Replication. and Kafka What is used Leader The election algorithm is more like Microsoft's PacificA Algorithm .

Kafka stay Zookeeper Where is each Partition Dynamic maintenance of a ISR, This ISR All in replica They're all following Leader, Only ISR In order to be elected to Leader The possibility of (unclean.leader.election.enable=false). In this mode , about f+1 Copies , One Kafka Topic Can guarantee not to lose already commit Toleration on the premise of news f Failed copies , In most usage scenarios , This model is very beneficial . in fact , In order to tolerate f Failed copies ,“ The minority is subordinate to the majority ” And ISR stay commit The number of copies to wait before is the same , however ISR The total number of copies required is almost “ The minority is subordinate to the majority ” Half of the way .

Mentioned above , stay ISR At least one of them Follower when ,Kafka You can make sure that you have commit Data of is not lost , But if one of them Partition All of the replica It's all gone , There is no guarantee that the data will not be lost . In this case, there are two possible solutions :

  • wait for ISR Any one of replica“ live ” To come over , And choose it as Leader
  • Select first “ live ” Over here replica( It doesn't have to be ISR in ) As Leader

This requires a simple choice between usability and consistency . Be sure to wait if ISR Medium replica“ live ” To come over , The unavailability time may be relatively long . And if the ISR All of the replica No way “ live ” Over here. , Or the data is missing , This Partition Will never be available . Select first “ live ” Over here replica As Leader, And this replica No ISR Medium replica, So even if it doesn't guarantee that it already includes everything that has been commit The news of , It will also become Leader And as a Consumer Data source . By default ,Kafka Use the second strategy , namely unclean.leader.election.enable=true, You can also set this parameter to false To enable the first strategy .

unclean.leader.election.enable This parameter is for leader The election of 、 The availability of the system and the reliability of the data have a crucial impact . Let's analyze several typical scenarios .

file

If the picture above shows , Suppose that one Partition The number of copies in is 3,replica-0, replica-1, replica-2 Store separately Broker0, Broker1 and Broker2 in .AR=(0,1,2),ISR=(0,1).
Set up request.required.acks=-1, min.insync.replicas=2,unclean.leader.election.enable=false. There will be Broker0 Also known as Broker0 At first Broker0 by Leader,Broker1 by Follower.

  • When ISR Medium replica-0 appear crash situations ,Broker1 Vote for the new Leader[ISR=(1)], Because of receiving min.insync.replicas=2 influence ,write Can't serve , however read Can continue normal service . In this case, the recovery plan :
  1. Try to recover ( restart )replica-0, If you can get up , The system is OK ;2. If replica-0 Cannot resume , Need to put min.insync.replicas Set to 1, recovery write function .
  • When ISR Medium replica-0 appear crash, Then replica-1 There have been crash, here [ISR=(1),leader=-1], Can't provide external services , In this case, the recovery plan :
  1. Try to recover replica-0 and replica-1, If you can get up , Then the system returns to normal ;
  1. If replica-0 get up , and replica-1 Can't get up , At this time, we still can't choose Leader, Because when setting unclean.leader.election.enable=false when ,leader Only from ISR The middle election , When ISR After all the copies in are invalid , need ISR The last invalid copy can be restored before the election Leader, namely replica-0 Fail first ,replica-1 Post failure , need replica-1 You can't vote until you recover Leader. Conservative proposal setting unclean.leader.election.enable=true, But there will be data loss , This will restore read service . The same needs to be done min.insync.replicas Set to 1, recovery write function ;
  2. replica-1 recovery ,replica-0 Cannot resume , This situation has been encountered above ,read Service available , Need to put min.insync.replicas Set to 1, recovery write function ;
  3. replica-0 and replica-1 Can't recover , This situation can be referred to the situation 2.
  • When ISR Medium replica-0,replica-1 Simultaneous downtime , here [ISR=(0,1)], Can't provide external services , In this case, the recovery plan : Try to recover replica-0 and replica-1, When any one of these copies returns to normal , It can be provided to the outside world read service . until 2 Copies back to normal ,write Function can be restored , Or will min.insync.replicas Set to 1.

Welcome to your attention ,《 The way of big data becoming God 》 Series articles

Welcome to your attention ,《 The way of big data becoming God 》 Series articles

Welcome to your attention ,《 The way of big data becoming God 》 Series articles

版权声明
本文为[Wang Zhiwu]所创,转载请带上原文链接,感谢
https://javamana.com/2021/01/20210124115952337r.html

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云