Convenient method to build zookeeper server, easy to use and collect~

osc_ facwbzof 2021-01-23 12:57:23
convenient method build zookeeper server

What is? ZooKeeper

ZooKeeper yes Apache A top-level project of , For distributed applications to provide efficient 、 Highly available distributed coordination services , Provides such things as data publishing / subscribe 、 Load balancing 、 Naming service 、 Distributed coordination / Distributed infrastructure services such as notification and distributed locks . because ZooKeeper Easy to use 、 Excellent performance and good stability , It is widely used for example Hadoop、HBase、Kafka and Dubbo In large distributed systems . Put together a copy of Java Interview book full version PDF

Zookeeper There are three modes of operation : standalone mode 、 Pseudo cluster mode and cluster mode .

  • standalone mode : This pattern is generally suitable for developing test environments , On the one hand, we don't have so many machine resources , In addition, the usual development and debugging do not need excellent stability .

  • Cluster pattern : One ZooKeeper Clusters are usually made up of a set of machines , commonly 3 More than one platform can form a usable ZooKeeper Clustered . form ZooKeeper Each machine in the cluster maintains the current server state in memory , And every machine keeps communicating with each other .

  • Pseudo cluster mode : This is a special clustering model , That is, all servers in the cluster are deployed on one machine . When you have a better machine on hand , If deployed as a stand-alone mode , It's a waste of resources , In this case ,ZooKeeper Allows you to start multiple ports on a machine by starting different ports ZooKeeper Service instance , In this way, we can provide external services with the characteristics of cluster .

ZooKeeper Knowledge about

Zookeeper The role of :

  • The leader (leader): Responsible for the initiation and resolution of the vote , Update system status .

  • follower (follower): Used to receive client requests and return results to clients , Vote in the election process .

  • The observer (observer): Can accept client connections , Forward the write request to leader, however observer The process of not voting , Just to expand the system , Increase the speed of reading .

 Convenient construction Zookeeper Server method , To use , Collection ~

Zookeeper Data model of

  • Hierarchical directory structure , Naming meets general file system specifications , Be similar to Linux.

  • Each node is in Zookeeper called Znode, And it has a unique path identifier .

  • node Znode Can contain data and child nodes , however EPHEMERAL A node of type cannot have children .

  • Znode The data in can have multiple versions , For example, there are multiple data versions in a certain path , Then you need to bring the version to query the data in this path .

  • The client application can set the monitor on the node .

  • The node does not support partial read / write , It's a one-time, complete read-write .
     Convenient construction Zookeeper Server method , To use , Collection ~

ZooKeeper Node characteristics of

ZooKeeper Nodes are lifecycle , It depends on the type of node . stay ZooKeeper in , Nodes can be divided into persistent nodes according to their duration (PERSISTENT)、 Temporary node (EPHEMERAL), It can be divided into sequential nodes according to whether they are ordered or not (SEQUENTIAL)、 And unordered nodes ( Default is out of order ).

Once a persistent node is created , Unless you take the initiative to remove , Otherwise it will always be stored in Zookeeper in ( It will not disappear because the session of the client who created the node fails ).

Zookeeper Application scenarios of

ZooKeeper It is a highly available distributed data management and system coordination framework . Based on the Paxos Implementation of algorithm , The framework ensures the strong consistency of data in distributed environment , It is based on such characteristics , bring ZooKeeper Solve a lot of distributed problems .

It is worth noting that ,ZooKeeper It's not built for these scenarios , Later, many developers based on the characteristics of their framework , Take advantage of the range of API Interface ( Or primitives ), Find out the typical use of .

Data publishing and subscription ( Configuration center )

Publish and subscribe model , The so-called configuration center , As the name implies, publishers publish data to ZooKeeper Node , For subscribers to dynamically access data , Realize centralized management and dynamic update of configuration information . For example, global configuration information , The service address list of service-oriented service framework is very suitable for use .

Some configuration information used in the application is put into ZooKeeper Centralized management . This kind of scene is usually like this : When the application is started, it will take the initiative to obtain a configuration , At the same time, register a Watcher. thus , Every time the configuration is updated later , Will notify the subscription client in real time , Always get the latest configuration information .

Distributed search services , The meta information of the index and the node status of the server cluster machine are stored in ZooKeeper Some specified nodes of , For each client subscription .

Distributed log collection system

The core work of this system is to collect logs distributed on different machines . Collectors usually allocate collection units by application , So you need to ZooKeeper Create an application name as path The node of P, And put all the machines in this application IP, Register to the node as a child node P On . In this way, when the machine changes , It can notify the collector to adjust the task allocation in real time .

Some information in the system needs to be acquired dynamically , And there will be questions to manually modify this information . It's usually exposing the interface , for example JMX Interface , To get some runtime information . introduce ZooKeeper after , You don't have to implement a plan by yourself , Just store the information in the designated ZooKeeper On the node .

Be careful : In the application scenario mentioned above , There is a default premise —— Small amount of data , But the data update may be faster .

Load balancing

Load balancing here refers to soft load balancing . In a distributed environment , To ensure high availability , Usually, the provider of the same application or service will deploy multiple copies , Reach peer-to-peer service . And consumers need to choose one of these peer servers to execute the relevant business logic , One of the more typical is the producer of message middleware , Consumer load balancing .

Naming service (Naming Service)

Naming service is also a common scenario in distributed system . In distributed systems , By using the naming service , The client application can get the address of the resource or service according to the specified name , Information such as providers . The named entity can usually be a machine in a cluster , Service address provided , Remote objects and so on —— We can call them all names (Name). One of the more common is the service address list in some distributed service frameworks . By calling ZooKeeper Provided to create nodes API, It's easy to create a globally unique path, This path Can be used as a name .

Alibaba Group's open source distributed service framework Dubbo Use in ZooKeeper As its naming service , Maintain the global service address list . stay Dubbo In the implementation of :

  • When the service provider starts , towards ZooKeeper Specified node on /dubbo/${serviceName}/providers Write your own URL Address , This operation completes the service release .

  • When the service consumer starts , subscribe /dubbo/serviceName/providers Provider under Directory URL Address , And to /dubbo/{serviceName}/providers Provider under Directory URL Address , And to /dubbo/{serviceName} /consumers Write your own URL Address .

Be careful : All to ZooKeeper All registered addresses on are temporary nodes , This ensures that service providers and consumers can automatically sense changes in resources .

in addition ,Dubbo And monitoring for service granularity . The way is to subscribe to /dubbo/${serviceName} Information about all providers and consumers in the directory .

Distributed notification / Coordinate

ZooKeeper Middle endemic Watcher Registration and asynchronous notification mechanism , It can well realize the notification and coordination between different systems in the distributed environment , Realize real-time processing of data changes . It is usually used in different systems ZooKeeper The last one Znode To register , monitor Znode The change of ( Include Znode The content of itself and the ), One of the systems Update 了 Znode, Then another system can receive notifications , And deal with it accordingly .

Another heartbeat detection mechanism : The detection system is not directly related to the detected system , But through ZooKeeper The last node is associated with , Greatly reduce system coupling .

Another system scheduling mode : A system has two parts: console and push system , The responsibility of the console is to control the push system to carry out the corresponding push work . Some operations that managers do in the console , It's actually a modification ZooKeeper The state of some nodes on , and ZooKeeper Just notify them of these changes and register Watcher The client of , Push system . therefore , Make the corresponding push task .

Another mode of work report : Some are similar to task distribution systems . After the subtask starts , To ZooKeeper To register a temporary node , And regularly report their progress ( Write the progress back to this temporary node ). In this way, the task manager can know the progress of the task in real time .

Distributed lock

Distributed locks mainly benefit from ZooKeeper For us to ensure a strong consistency of data . There are two types of lock services : One is to keep exclusive , The other is control timing .

Keep exclusive , It's all the clients trying to get this lock , In the end, only one can get the lock . The usual way is to ZooKeeper On the one Znode Think of it as a lock , adopt create znode The way to achieve . All clients are going to create /distribute_lock node , Finally, the client successfully created owns the lock .

Control timing , To get all the client views , In the end, they will be arranged to execute , It's just a global sequence . It's basically similar to the above , Just here /distribute_lock Already exist in advance , The client creates a temporary ordered node under it ( This can be controlled by the properties of the node :CreateMode.EPHEMERAL_SEQUENTIAL To specify the ).ZooKeeper Parent node (/distribute_lock) Maintain a share of sequence, Ensure the timing of child node creation , Thus, the global timing of each client is formed .

  1. Because the names of the child nodes of the same node cannot be the same , So just create it under a node Znode, Successful creation indicates successful locking . Registered listeners listen to this Znode, Just delete this Znode Inform other clients to lock .

  2. Create a temporary order node : Create a node under a node , A request creates a node , Because it's sequential , So the one with the smallest sequence number gets the lock , When the lock is released , Inform the next sequence number to get the lock .

Distributed queues

Queue aspect , In short, there are two kinds of : One is the regular first in, first out queue , The other is to wait for the queue members to get together before executing in order . For the first kind of queue, the basic principle is consistent with the scenario of controlling timing in the above-mentioned distributed lock service , I won't repeat it here .

The second kind of queue is actually FIFO An enhancement is made on the basis of the queue . Usually in /queue This Znode Create a new one in advance /queue/num node , And the assignment is n( Or directly to /queue assignment n) Indicates the size of the queue . Then every time a queue member joins , Judge whether the queue size has been reached , Decide whether you can start to execute .

A typical scenario for this usage is : In distributed environment , A big task Task A, It needs to be done in many subtasks ( Or the conditions are ready ) It can only be carried out under the circumstances . This is the time , If one of the subtasks is completed ( be ready ), Then go /taskList Create your own temporary timing node (CreateMode.EPHEMERAL_SEQUENTIAL). When /taskList Find that the number of child nodes below satisfies the specified number , You can proceed to the next step and process in sequence .

Use dokcer-compose Set up the cluster

Above we introduced about ZooKeeper There are so many scenarios , Next, learn how to build ZooKeeper Cluster and then carry out actual combat. The above application scenario .

The directory structure of the file is as follows :

├── docker-compose.yml

To write docker-compose.yml file

docker-compose.yml The contents of the document are as follows :

version: '3.4'
image: zookeeper
restart: always
hostname: zoo1
- 2181:2181
ZOO_SERVERS: server.1=;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
image: zookeeper
restart: always
hostname: zoo2
- 2182:2181
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=;2181 server.3=zoo3:2888:3888;2181
image: zookeeper
restart: always
hostname: zoo3
- 2183:2181
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=;2181

In this configuration file ,Docker It's running 3 individual Zookeeper Mirror image , adopt ports Fields will be local 2181, 2182, 2183 The corresponding port bound to the container 2181 On port .

ZOO_MY_ID and ZOO_SERVERS It's construction Zookeeper Two environment variables needed by cluster .ZOO_MY_ID Identify the service id, by 1-255 Integer between , Must be unique in the cluster .ZOO_SERVERS Is the list of hosts in the cluster .

stay docker-compose.yml Execute in the directory docker-compose up, You can see the startup log .

Connect ZooKeeper

After the cluster is started, we can connect to ZooKeeper It carries on the node related operation .

  1. First you need to download ZooKeeper.

  2. Unzip it .

  3. Enter its conf/ Catalog , take zoo_sample .cfg Change to zoo.cfg.

Profile description

# The number of milliseconds of each tick
# tickTime:CS The number of communication heartbeat 
# Zookeeper Time interval between servers or between clients and servers to maintain heartbeat , That is, every one of them tickTime Time will send a heartbeat .tickTime In Milliseconds .

# The number of ticks that the initial
# synchronization phase can take
# initLimit:LF Initial communication time limit 
# In the cluster follower The server (F) And leader The server (L) The maximum number of heartbeats that can be tolerated during the initial connection between (tickTime The number of ).

# The number of ticks that can pass between
# sending a request and getting an acknowledgement
# syncLimit:LF Synchronous communication time limit 
# In the cluster follower The server and leader The maximum number of heartbeats that can be tolerated between requests and responses between servers (tickTime The number of ).

# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# dataDir: Data file directory 
# Zookeeper The directory where the data is stored , By default ,Zookeeper Save the log file that writes the data in this directory .

# dataLogDir: Log file directory 
# Zookeeper The directory where the log files are stored .

# the port at which the clients will connect
# clientPort: Client connection port 
# Client connection Zookeeper The port of the server ,Zookeeper Will listen to this port , Accept client access requests .

# the maximum number of client connections.
# increase this if you need to handle more clients
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
# The number of snapshots to retain in dataDir
# Purge task interval in hours
# Set to "0" to disable auto purge feature

# Server name and address : Cluster information ( Server number , Server address ,LF Communication port , Election port )
# This configuration item is written in a special format , The following rules :

# server.N=YYY:A:B

# among N Indicates the server number ,YYY Represents the... Of the server IP Address ,A by LF Communication port , Represents the server and the leader The port of information exchanged .B For the election port , To vote for a new leader When the servers communicate with each other ( When leader Hang up , The rest of the servers will communicate with each other , Choose a new leader). Generally speaking , For each server in the cluster A Ports are the same , For each server B The same is true for ports . But when it's a pseudo cluster ,IP The addresses are the same , Only when A Port and B Ports are different .

It can be left unchanged zoo.cfg, Use default configuration . Next, after decompressing bin/ Execute the command in the directory ./ -server You can connect .

Welcome to ZooKeeper!
2020-06-01 15:03:52,512 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server localhost/\. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2020-06-01 15:03:52,576 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/, initiating session
2020-06-01 15:03:52,599 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/, sessionid = 0x100001140080000, negotiated timeout = 30000
WatchedEvent state:SyncConnected type:None path:null
[zk: 0]

Next, you can use the command to view the nodes :

  • Use ls Command to view the current ZooKeeper What's included in . command :ls /

    [zk: 10] ls /

  • Created a new znode node zk And the string associated with it . command :create /zk myData

    [zk: 11] create /zk myData

Created /zk [zk: 12] ls / [zk, zookeeper] [zk: 13]

  • obtain znode node zk. command :get /zk

    [zk: 13] get /zk myData cZxid = 0x400000008 ctime = Mon Jun 01 15:07:50 CST 2020 mZxid = 0x400000008 mtime = Mon Jun 01 15:07:50 CST 2020 pZxid = 0x400000008 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 6 numChildren = 0

  • Delete znode node zk. command :delete /zk Put together a copy of Java Interview book full version PDF

    [zk: 14] delete /zk [zk: 15] ls / [zookeeper]

本文为[osc_ facwbzof]所创,转载请带上原文链接,感谢

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云