stay Docker Running in the container Elasticsearch Kibana and Cerebro

Machine information

10.160.13.139
10.160.9.162
10.160.11.171

1. install docker and docker compose

install docker

# install docker Community Edition 
yum -y install docker-ce docker-ce-cli containerd.io
# start-up Docker
systemctl start docker
# see docker edition
docker version
# Boot up Docker
systemctl enable docker

install docker-compose

sudo curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-`uname -s`-`uname -m` \
> /usr/local/bin/docker-compose
# Add execute permission
sudo chmod +x /usr/local/bin/docker-compose
# verification
docker-compose --version
docker-compose version 1.27.4, build 40524192

2. Download mirroring

docker pull elasticsearch:7.10.1
docker pull kibana:7.10.1

3. take kibana Upload to private harbor

[@master software]# docker images |grep kibana
kibana 7.10.1 3e014820ee3f 3 weeks ago 992MB
[@master software]# docker tag 3e014820ee3f www.test.com/elk/kibana:7.10.1
[@master software]# docker login www.test.com
Username: song
Password:
Login Succeeded
[@master software]# docker push www.test.com/elk/kibana:7.10.1
The push refers to repository [www.test.com/elk/kibana]
00cfe116c0b3: Pushed
20889a7593bc: Pushed
d1409dea493d: Pushed
3448a6e61306: Pushed
f3a4630cb1b0: Pushed
0e3f51adbdfe: Pushed
7bb6eb357f15: Pushed
0263a48f9a3a: Pushed
5acb536ef82f: Pushed
58d51bd5abeb: Pushed
e116279745ca: Pushed
cc33c1bea858: Pushed
5ec5e9c8cf6f: Pushed
291f6e44771a: Pushed
7.10.1: digest: sha256:1731793b7f3e453c65ebaf92ec0b55f4029310ba8abae9e04753a4680dd8210b size: 3247
[@master software]#

4. take elasticsearch Upload to private harbor

[@master software]# docker images |grep elastic
elasticsearch 7.10.1 558380375f1a 3 weeks ago 774MB
[@master software]# docker tag 558380375f1a www.test.com/elk/elasticsearch:7.10.1
[@master software]# docker push www.test.com/elk/elasticsearch:7.10.1
The push refers to repository [www.test.com/elk/elasticsearch]
d6208dd58075: Pushed
9117213e7470: Pushed
b662ef6f4fc1: Pushed
3f481e912123: Pushed
b456348ecd84: Pushed
e2834bada95f: Pushed
8f84abff69b8: Pushed
911490c8ab00: Pushed
291f6e44771a: Mounted from elk/kibana
7.10.1: digest: sha256:e9a1fe65f68b2d2b9583287d1190f67f23af08582eac4d2a8dc342e4219c7306 size: 2202
[@master software]#

5 . Operating system optimization

The operating system and JVM Tuning is mainly for installation elasticsearch Machine . For the operating system , Several kernel parameters need to be adjusted

vim /etc/sysctl.conf
# fs.file-max It mainly configures the maximum number of open file descriptors of the system , It is suggested to amend it to 655360 Or higher
fs.file-max=655360
# vm.max_map_count influence Java Number of threads , Used to limit what a process can have VMA( Virtual memory area ) Size
# The system default is 65530, It is suggested to amend it to 262144 Or higher .
vm.max_map_count = 262144
sysctl -p
sysctl -a

in addition , You also need to adjust the process's maximum open file descriptor (nofile)、 Maximum number of user processes (nproc) And maximum lock memory address space (memlock),

vim /etc/security/limits.conf
* soft nproc 20480
* hard nproc 20480
* soft nofile 65536
* hard nofile 65536
* soft memlock unlimited
* hard memlock unlimited
# remarks : * representative Linux All user names ( such as es), preservation , sign out , Login again to take effect
# see
ulimit -a

Last , It needs to be revised

# /etc/security/limits.d/90-nproc.conf(centos6.x System )
# /etc/security/limits.d/20-nproc.conf(centos7.x System ) vim /etc/security/limits.d/20-nproc.conf
* soft nproc 4096
# It is amended as follows :
* soft nproc 20480

Or delete /etc/security/limits.d/20-nproc.conf File is also OK .

6. JVM tuning

JVM Tuning is mainly for elasticsearch Of JVM Optimize memory resources ,elasticsearch The memory resource configuration file for is jvm.options,7.1 The default setting for download is 1GB

from ES 6 Start , Only support 64 Bit JVM( To configure config/jvm.options)

Configuration suggestions

  • Will memory Xms and Xmx Set it to the same , avoid heap resize It's a pause
  • Xmx Don't set more than physical memory 50%, On a single node , The maximum memory is recommended not to exceed 32G Memory
  • Production environment ,JVM You have to use Server Pattern
  • close JVM Swapping
vim /usr/local/elasticsearch/config/jvm.options
-Xms30g
-Xmx30g

7. install elasticsearch

You can create a new one first elasticsearch Folder . And then in this folder , Create a new one docker-compose.yml file .

function docker-compose, Build a more efficient development environment locally , More intuitive understanding of Elasticsearch Distributed features

Integrate Cerebro, Easy to view cluster status

mkdir -p /search/odin/elasticsearch/{data01,data02,data03}
mkdir -p /Users/song/study/elasticsearch/plugins/ik
# ES_JAVA_OPTS="-Xms64m -Xmx512m" Memory limit , Minimum 64m, Maximum 512m, It can be adjusted according to the need , In addition, other configurations need not be modified
cat /search/odin/elasticsearch/docker-compose.yml
version: '2.2'
services:
cerebro:
image: lmenezes/cerebro:0.8.3
container_name: cerebro
ports:
- "9000:9000"
command:
- -Dhosts.0.host=http://elasticsearch:9200
networks:
- elastic
kibana:
image: kibana:7.10.1
container_name: kibana7
environment:
- I18N_LOCALE=zh-CN
- XPACK_GRAPH_ENABLED=true
- TIMELION_ENABLED=true
- XPACK_MONITORING_COLLECTION_ENABLED="true"
ports:
- "5601:5601"
networks:
- elastic
elasticsearch:
image: elasticsearch:7.10.1
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
ulimits:
nproc: 65535
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
- /Users/song/study/elasticsearch/plugins/ik:/usr/share/elasticsearch/plugins/ik
ports:
- 9200:9200
- 9300:9300
networks:
- elastic
elasticsearch2:
image: elasticsearch:7.10.1
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
ulimits:
nproc: 65535
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
- /Users/song/study/elasticsearch/plugins/ik:/usr/share/elasticsearch/plugins/ik
ports:
- 9201:9201
- 9301:9301
networks:
- elastic
elasticsearch3:
image: elasticsearch:7.10.1
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
ulimits:
nproc: 65535
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
- /Users/song/study/elasticsearch/plugins/ik:/usr/share/elasticsearch/plugins/ik
ports:
- 9202:9202
- 9302:9302
networks:
- elastic
elastichd:
image: containerize/elastichd:latest
container_name: elasticsearch-hd
networks:
- elastic
ports:
- 9800:9800
depends_on:
- "elasticsearch"
links:
- "elasticsearch:demo" volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local networks:
elastic:
driver: bridge

function docker-compose, install Elasticsearch

start-up

screen
docker-compose up

Access test

curl -X GET "localhost:9200/_cat/nodes?v&pretty"

Stop container

docker-compose down

Stop the container and remove the data

docker-compose down -v

Delete individual containers

$docker rm Container name / Containers ID
-f, –force=false;
-l, –link=false Remove the specified link and not the underlying container;
-v, –volumes=false Remove the volumes associated to the container

Delete all containers

$docker rm `docker ps -aq`

stop it 、 start-up 、 Kill 、 Restart a container

$ docker stop Container name / Containers ID
$ docker start Container name / Containers ID
$ docker kill Container name / Containers ID
$ docker restart Container name / Containers ID

Displays container resource usage statistics

docker stats Containers ID

Use docker Conduct ElasticSearch + Kibana More articles on cluster building

  1. Docker Series of swarm Cluster building

    Study Docker For a long time , Share today Docker Of swarm The process of building a cluster is very simple The first step is Every machine has to be equipped with docker I use centos7 operating system , Use 3 Too many virtual machines ,3 Too many virtual machines must be networked ...

  2. elasticsearch Distributed cluster building

    elasticsearch Environment building and single node building can refer to my last article :http://www.cnblogs.com/xuwenjin/p/8745624.html This article takes Elaticsearch 6.2.2 ...

  3. Elasticsearch Hot and cold cluster building

    ES edition :6.2.4 Cluster environment :7 Taiwan machine , One for each master node . among 3 I'm going to deploy 2 individual hot node , in addition 4 I'm going to deploy 2 individual warm node . common 21 Nodes . 1. Hanging plate According to the actual situation , On a machine 2 individual data section ...

  4. ElasticSearch introduction —— Cluster building

    One . Environment introduction and installation preparation 1. Environmental statement 2 Virtual machines ,OS by ubuntu13.04,ip Respectively xxx.xxx.xxx.140 and xxx.xxx.xxx.145. 2. Installation preparation ElasticSearch( Jane ...

  5. 【ELK】【docker】【elasticsearch】1. Use Docker and Elasticsearch+ kibana 5.6.9 Build full text search engine applications colony , install ik Word segmentation is

    Series articles :[ It is suggested to start from chapter two ] [ELK][docker][elasticsearch]1. Use Docker and Elasticsearch+ kibana 5.6.9 Build full text search engine applications colony , Ann ...

  6. Elasticsearch Cluster construction and use Java The client stores and queries data

    Two pieces of this blog , The first part is how to build a Elastic colony , The second half is based on Java Write and aggregate statistics on data . One .Elastic Cluster building 1. Environmental preparation . The cluster environment is based on VMware virtual machine .CentOS ...

  7. build ELK Log analysis platform ( On )—— ELK Introduce and build Elasticsearch Distributed cluster

    Notes : build ELK Log analysis platform ( On )-- ELK Introduce and build Elasticsearch Distributed cluster note date :2018-03-02 27.1 ELK Introduce 27.2 ELK Preparation for installation 27.3 install e ...

  8. Elasticsearch(ELK) Cluster building

    One . Preface Elasticsearch It is an open source and highly extensible distributed full-text retrieval engine , It can store almost in real time . Retrieving data : It's very extensible , It can be extended to hundreds of servers , Handle PB Level of data .Elasticsearch Also used ...

  9. elasticsearch Cluster building example

    elasticsearch Cluster building example We're going to start searching again next month , It hasn't moved for several months. Fortunately, it hasn't fallen down .  In the evening, I built a simple search cluster on my virtual machine , Share it . Operating system environment : Red Hat 4.8.2-16 el ...

  10. Docker Next ETCD Cluster building

    Before setting up the cluster, first prepare to install two CentOS 7 The host , And install it on it Docker. Master 10.100.97.46 Node 10.100.97.64 ETCD There are three ways to build a cluster , Namely Sta ...

Random recommendation

  1. java Memory model -volatile

    volatile Characteristics of When we declare the shared variable as volatile after , Reading this variable / Writing will be special . understand volatile A good way to create features is : Put right volatile Single reading of variables / Write , Think of it as using the same ...

  2. 30 Learn to use in minutes grunt Package the front end code

    http://www.cnblogs.com/yexiaochai/p/3603389.html

  3. In depth understanding of java virtual machine (4)--- Class loading mechanism

    Class loading process includes : load class To the memory , data verification , Transformation and analysis , initialization , Use using And uninstall unloading The process . Except for the parsing phase , The order of the other processes is fixed . Parsing can be done after initialization , The purpose is to support dynamic ...

  4. TOMCAT Report errors :HTTP Status 404 -

    structure struts2 The engineer ,tomcat Report errors : HTTP Status 404 - type Status report message description The requested resour ...

  5. Understanding postgresql.conf : log*

    After loooong pause, adding next (well, second) post to the “series“. This time, I'd like to describ ...

  6. chinaOS

    Promote the operating system in terms of public opinion and policies , This is a good thing for domestic operating systems , Bad thing ? In a way , It's a good thing Using the power of Administration , It's really a good thing to take the lead in popularizing domestic operating systems in state organs , The first is to ensure national information security , Secondly, it can ...

  7. AFNetworking Internal details

    AFNetworking Is a suitable for IOS and Mac OSX Network library of two platforms , He is in Foundation URL Loading System   On the basis of a package , And provide a wealth of API Interface ...

  8. How to be in vuejs Extract the common code from the

    When we are using vue When building medium and large projects , We usually encounter some frequently used methods and properties , For example, build an employee management system , Requested url Need a common prefix , Or at some point view It's going to take time , This time is formatted by some method ...

  9. 【Peaks Enhanced Edition BZOJ 3551】 Are you in a hole ?

    The optimization time is 20s About the topic finally in the rice cake two days of entanglement . Goof . It's hard to AC 了 . But unfortunately , The rice cake number one code is actually more concise , But so far I can't find BUG, I put it down there , Maybe one day another big one from afar ...

  10. java send out http Of get、post request 【 Memo 】

    class package com.dsideal.kq.Controller; import java.io.BufferedReader; import java.io.IOException; impo ...