Preface

storm and kafka Cluster installation is not necessarily related , I put these two together , Because they're all made up of zookeeper managed , It all depends on JDK Environment , In order not to write the configuration again , So I put these two together . If only one is needed , Just choose the reading you choose .

The dependence of the two is as follows :

  • Storm colony :JDK1.8 , Zookeeper3.4,Storm1.1.1;
  • Kafa colony : JDK1.8 ,Zookeeper3.4 ,Kafka2.12;

explain : Storm1.0 and Kafka2.0 Yes JDK The requirement is 1.7 above ,Zookeeper3.0 above .

Download address :

Zookeeper:https://zookeeper.apache.org/releases.html#download

Storm: http://storm.apache.org/downloads.html

Kafka: http://kafka.apache.org/downloads

JDK install

Every machine has to be equipped with JDK!!!

explain : commonly CentOS Bring it with you openjdk, But what we're using here is oracle Of JDK. So write and unload openjdk, And then install it in oracle Download good JDK. If it has been unloaded , You can skip this step .

First type java -version

Check to see if it is installed JDK, If installed , But if the version doesn't fit , Just uninstall

Input

rpm -qa | grep java

Check the information



Then input :

rpm -e --nodeps “ You want to uninstall JDK Information about ”

Such as : rpm -e --nodeps java-1.7.0-openjdk-1.7.0.99-2.6.5.1.el6.x86_64

After confirming that it's gone , Unzip the downloaded JDK

tar -xvf jdk-8u144-linux-x64.tar.gz

Move to opt/java In the folder , New if not , Then rename the folder to jdk1.8.

mv jdk1.8.0_144 /opt/java
mv jdk1.8.0_144 jdk1.8

Then edit profile file , Add the following configuration

Input :

vim /etc/profile

add to :

export JAVA_HOME=/opt/java/jdk1.8
export JRE_HOME=/opt/java/jdk1.8/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export PATH=.:${JAVA_HOME}/bin:$PATH

After adding successfully , Input

source /etc/profile
java -version

See if the configuration is successful

Zookeeper Environmental installation

1, Document preparation

Will download the Zookeeper The configuration file of

stay linux On the input :

 tar -xvf zookeeper-3.4.10.tar.gz

Then move to /opt/zookeeper Inside , New if not , Then rename the folder to zookeeper3.4

Input

mv zookeeper-3.4.10 /opt/zookeeper
mv zookeeper-3.4.10 zookeeper3.4

2, Environment configuration

edit /etc/profile file

Input :

export ZK_HOME=/opt/zookeeper/zookeeper3.4
export PATH=.:${JAVA_HOME}/bin:${ZK_HOME}/bin:$PATH

Input :

source /etc/profile

Make configuration effective

3, Modify the configuration file

3.3.1 Create files and directories

Create these directories on the servers in the cluster

mkdir /opt/zookeeper/data
mkdir /opt/zookeeper/dataLog

And in /opt/zookeeper/data Create under directory myid file

Input :

touch myid

Once created , change myid file .

I'm here for convenience , take master、slave1、slave2 Of myid The content of the document is changed to 1,2,3

3.3.2 newly build zoo.cfg

Switch to /opt/zookeeper/zookeeper3.4/conf Under the table of contents

without zoo.cfg This document , Just copy zoo_sample.cfg And rename it to file zoo.cfg.

Modify this new zoo.cfg file

dataDir=/opt/zookeeper/data
dataLogDir=/opt/zookeeper/dataLog
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888

explain :client port, seeing the name of a thing one thinks of its function , Client connection zookeeper Port of service . This is a TCP port.dataLogDir It's the sequence log that you put in (WAL). and dataDir It's a memory data structure snapshot, Easy to recover quickly . To maximize performance , It is generally recommended that dataDir and dataLogDir Split it into different disks , In this way, we can make full use of the disk sequential write feature .dataDir and dataLogDir To create yourself needs , The catalogue can be made by itself , Corresponding .server.1 In this 1 Need and master On this machine dataDir In the directory myid The values in the file correspond to .server.2 In this 2 Need and slave1 On this machine dataDir In the directory myid The values in the file correspond to .server.3 In this 3 Need and slave2 On this machine dataDir In the directory myid The values in the file correspond to . Of course , You can use any number you like , Just correspond .2888 and 3888 You can also use the port number of , Because on different machines , It doesn't matter if you use it the same way .

1.tickTime:CS The number of communication heartbeat

Zookeeper Time interval between servers or between clients and servers to maintain heartbeat , That is, every one of them tickTime Time will send a heartbeat .tickTime In Milliseconds .

tickTime=2000

2.initLimit:LF Initial communication time limit

In the cluster follower The server (F) And leader The server (L) The maximum number of heartbeats that can be tolerated during the initial connection between (tickTime The number of ).

initLimit=10

3.syncLimit:LF Synchronous communication time limit

In the cluster follower The server and leader The maximum number of heartbeats that can be tolerated between requests and responses between servers (tickTime The number of ).

syncLimit=5

Still will zookeeper Transfer it to other machines , Remember to change /opt/zookeeper/data Under the myid, This is not consistent .

Input :

scp -r /opt/zookeeper root@slave1:/opt
scp -r /opt/zookeeper root@slave2:/opt

4, start-up zookeeper

because zookeeper It's an electoral system , Its master-slave relationship is not like hadoop That's what it's supposed to be , See the official documentation for details .

Successfully configured zookeeper after , Start on each machine zookeeper.

Switch to zookeeper Under the table of contents

cd /opt/zookeeper/zookeeper3.4/bin

Input :

zkServer.sh start

After a successful launch

View status input :

 zkServer.sh status

You can look at the machines zookeeper Of leader and follower

Storm Environmental installation

1, Document preparation

Will download the storm The configuration file of

stay linux On the input :

 tar -xvf apache-storm-1.1.1.tar.gz

Then move to /opt/storm Inside , New if not , Then rename the folder to storm1.1

Input

mv apache-storm-1.1.1 /opt/storm
mv apache-storm-1.1.1 storm1.1

2, Environment configuration

edit /etc/profile file

add to :

export STORM_HOME=/opt/storm/storm1.1
export PATH=.:${JAVA_HOME}/bin:${ZK_HOME}/bin:${STORM_HOME}/bin:$PATH

Input storm version View version information

3, Modify the configuration file

edit storm/conf Of storm.yarm.

Edit as follows :

Input :

vim storm.yarm

storm.zookeeper.servers:
- "master"
- "slave1"
- "slave2" storm.local.dir: "/root/storm" nimbus.seeds: ["master"] supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703



explain :

  1. storm.zookeeper.servers Is designated zookeeper Service address of .

    because storm The information stored in zookeeper On , So configure zookeeper Service address of . If zookeeper If it's a stand-alone machine, just specify one !
  2. storm.local.dir Represents the storage directory .

    Nimbus and Supervisor The daemons need to store a directory on the local disk to store a small amount of state ( such as jar,confs wait ). It can be created on every machine , And give them permission .

    3.nimbus.seeds Represents a candidate host .

    worker Need to know which machine is the host candidate (zookeeper Cluster is an electoral system ), So you can download topology jars and confs.

    4.supervisor.slots.ports Express worker port .

    For each of these supervisor machine , We can use this to configure how much worker On this machine . every last worker Use a separate port To receive the news , This port also defines which ports are open to use . If you define it here 5 Ports , It means this supervisor You can run at most 5 individual worker. If you define 3 Ports , It means it can run at most 3 individual worker. By default ( That is, it is configured in defaults.yaml in ), There will be four workers Running on the 6700, 6701, 6702, and 6703 port .

    supervisor It doesn't start these four immediately at startup worker. It's when you get the assignment , Will start , Specific start a few worker Also according to our Topology In this supervisor How many do you need worker To make sure . If specified Topology Only one worker perform , that supervisor Just start one worker, It doesn't start all .

notes : There should be no spaces in front of these configurations !!!, Otherwise, it will report a mistake . The host name is used here ( I did the mapping ), You can also use IP. The actual one is subject to one's own .

have access to scp Order or ftp The software will storm Copy to other machines

After successful configuration , Then you can start Storm 了 , But make sure JDK、Zookeeper Has been installed correctly , also Zookeeper Started successfully .

4, start-up Storm

Switch to storm/bin Under the table of contents

At the main node (master) Start input :

storm nimbus >/dev/null 2>&1 &

visit web Interface (master) Input :

storm ui

From the node (slave1,slave2) Input :

storm supervisor >/dev/null 2>&1 &

Enter... In the browser interface : 8080 port

The interface is successfully opened , Indicates that the environment configuration is successful :

kafka Environment installation of

1, Document preparation

Will download the Kafka The configuration file of

stay linux On the input :

 tar -xvf kafka_2.12-1.0.0.tgz

Then move to /opt/kafka Inside , New if not , Then rename the folder to kafka2.12

Input

mv kafka_2.12-1.0.0 /opt/kafka
mv kafka_2.12-1.0.0 kafka2.12

2, Environment configuration

edit /etc/profile file

Input :

export KAFKA_HOME=/opt/kafka/kafka2.12
export PATH=.:${JAVA_HOME}/bin:${KAFKA_HOME}/bin:${ZK_HOME}/bin:$PATH

Input :

source /etc/profile

Make configuration effective

3, Modify the configuration file

notes : What I want to say is , If it's a single machine ,kafka The configuration file of can not be modified , Direct to bin Just start in the directory . But we have clusters here , So just change it a little bit .

Switch to kafka/config Under the table of contents

edit server.properties file

What needs to be changed is Zookeeper The address of :

find Zookeeper Configuration of , Appoint Zookeeper Cluster address , The settings are as follows

zookeeper.connect=master:2181,slave1:2181,slave2:2181
zookeeper.connection.timeout.ms=6000

Other options are

1 ,num.partitions Represents the specified partition , The default is 1

2,log.dirs kafka Log path of , This can be changed according to personal needs

3, broker.id: Non-negative integer , For unique identification broker, Each one is different

...

notes : There are other configurations , You can view the official documentation , If there is no special requirement , Just use the default .

After configuration , Remember to use scp The command is transferred to other clusters , Remember to change server.properties file !

4, start-up kafka

Cluster each cluster needs to operate !

Switch to kafka/bin Under the table of contents

Input :

kafka-server-start.sh

Then input jps Name to see if it started successfully :

After a successful launch , You can do a simple test

First create topic

Input :

kafka-topics.sh --zookeeper master:2181 --create --topic t_test --partitions 5 --replication-factor 2

explain : Here's a new one called t_test Of topic, And specify that 5 Zones , Each partition specifies 2 Number of copies . If you don't specify a partition , The default partition is configured by the configuration file .

And then the production data

Input :

kafka-console-producer.sh --broker-list master:9092 --topic t_test

Can be used to Ctrl+D sign out

And then we open another one xshell window

Consumption

Input :

kafka-console-consumer.sh --zookeeper master:2181 --topic t_test --from-beginning



Can be used to Ctrl+C sign out

You can see that the data has been consumed normally .

5,kafka Some common commands of

1. Startup and shutdown kafka

bin/kafka-server-start.sh config/server.properties >>/dev/null 2>&1 &
bin/kafka-server-stop.sh

2. see kafka Message queues and concrete queues in the cluster

View all of the cluster topic

 kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181 --list

View one topic Information about

 kafka-topics.sh --zookeeper master:2181 --describe --topic t_test

3. establish Topic

kafka-topics.sh --zookeeper master:2181 --create --topic t_test --partitions 5 --replication-factor 2

4. Production data and consumption data

kafka-console-producer.sh --broker-list master:9092 --topic t_test

Ctrl+D sign out

kafka-console-consumer.sh --zookeeper master:2181 --topic t_test --from-beginning

Ctrl+C sign out

5.kafka The delete command of

kafka-topics.sh --delete --zookeeper master:2181 --topic t_test

6, Add partition

kafka-topics.sh --alter --topict_test --zookeeper master:2181 --partitions 10

Other

Storm Refer to the official documents for environment construction :

http://storm.apache.org/releases/1.1.1/Setting-up-a-Storm-cluster.html

Kafka Refer to the official documents for environment construction :

http://kafka.apache.org/quickstart

Here we are , In this paper, the end , Thank you for reading !

kafka and strom Cluster environment installation of more related articles

  1. kafka and storm Cluster environment installation

    Preface storm and kafka Cluster installation is not necessarily related , I put these two together , Because they're all made up of zookeeper managed , It all depends on JDK Environment , In order not to write the configuration again , So I put these two together . If only one ...

  2. elasticsearch7.5.0+kibana-7.5.0+cerebro-0.8.5 Installation and configuration of cluster production environment elasticsearch-migration Tools for data migration between new and old clusters

    One . Server preparation There are two at present 128G Memory server , So prepare to start two for each es example , Add another virtual machine , There are five nodes , Guarantee down The data of two nodes of a server is not affected . Two . System initialization See my last kafka System initialization :ht ...

  3. kafka learning - Cluster configuration and installation

    1. Software version kafka2.10_0.9.0.0 zookeeper_3.4.6 2. Cluster nodes Altogether 3 Taiwan machine . 192.168.14.100 slave-01 192.168.14.105 slav ...

  4. Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( The 21st )NIFI1.7.1 install

    One .nifi Basic configuration 1. Modify the host name of each node , modify /etc/hosts The contents of the document . 192.168.0.120 master 192.168.0.121 slave1 192.168.0.122 sla ...

  5. Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( Twelve )VMW Install four CentOS, And realize that the machine can interact with them , The internal implementation of the virtual machine can access the Internet .

    Centos7 Something unusual happened :Failed to start LSB: Bring up/down networking. according to <Kafka:ZK+Kafka+Spark Streaming Cluster environment ...

  6. Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( Ten ) install hadoop2.9.0 build HA

    How to build configuration centos Virtual machine, please refer to <Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( One )VMW Install four CentOS, And realize that the machine can interact with them , The internal implementation of the virtual machine can access the Internet .& ...

  7. Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( Nine ) install kafka_2.11-1.1.0

    How to build configuration centos Virtual machine, please refer to <Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( One )VMW Install four CentOS, And realize that the machine can interact with them , The internal implementation of the virtual machine can access the Internet .& ...

  8. Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( 8、 ... and ) install zookeeper-3.4.12

    How to build configuration centos Virtual machine, please refer to <Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( One )VMW Install four CentOS, And realize that the machine can interact with them , The internal implementation of the virtual machine can access the Internet .& ...

  9. Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( 3、 ... and ) install spark2.2.1

    How to build configuration centos Virtual machine, please refer to <Kafka:ZK+Kafka+Spark Streaming Cluster environment construction ( One )VMW Install four CentOS, And realize that the machine can interact with them , The internal implementation of the virtual machine can access the Internet .& ...

Random recommendation

  1. LeetCode-3LongestSubstringWithoutRepeatingCharacters(C#)

    # subject 3. Longest Substring Without Repeating Characters Given a string, find the length of the longes ...

  2. UITextField The use of,

    turn iOS in UITextField The use of, (1) // initialization textfield And set the position and size   UITextField *text = [[UITextField alloc]initWithFr ...

  3. ~/.bashrc Common use of alias Set up

    centos6.5 In the system ,alias It's defined in /etc/bashrc, They are written separately in /etc/profile.d/*.sh in , You can add my.sh, alias attrib='chmod'alias c ...

  4. HDU-1234(string character string )

    Description The first person to the computer room every day should open the door , The last person to leave is to close the door . There are a lot of messy machine room signs To . Check out record , Please find out who opened and closed the door according to the record .   Input The first line of the test input gives the total number of records ...

  5. memcached session share

    http://www.baidu.com/s?wd=memcached%20session%E5%85%B1%E4%BA%AB&rsv_spt=1&issp=1&f=8& ...

  6. HTML5 Quick reference table

    HTML5 Quick reference table label describe edition attribute <!--...--> Definition notes 4 / 5 none <!DOCTYPE> Define document types 4 / 5 none <a> set ...

  7. CoreCLR Source code exploration ( 7、 ... and ) JIT How it works ( Introduction )

    quite a lot C# All beginners have this question , .Net How the program code is loaded and executed by the machine ? The simplest answer is , C# Through the compiler (CodeDom, Roslyn) Translate it into IL Code , then CLR(.Net Framew ...

  8. OPEN A PO ORDER OR SO ORDER

    OPEN PO ORDER fnd_function.Execute(Function_Name => 'PO_POXPOEPO', Open_Flag => 'Y', Session_F ...

  9. MySQL Update fields in one table with fields in another

    from https://blog.csdn.net/anxpp/article/details/73173274 update table1 t1 left join table2 t2 on t1.ke ...

  10. Infiniband Network performance test

    1. Bandwidth test   stay server End execution [ibtests]# ib_send_bw -a -c UD -d mlx4_0 -i 1 ---------------------------------- ...