Docker - build log monitoring system

docker build log monitoring

Centralized log collection tools are commonly used in projects

  • Logstash

Logstash Is an open source data collection engine , With real-time pipeline function .Logstash It can dynamically unify data from different data sources , And standardize the data to your chosen destination .

  • advantage

    Logstash The main thing is its flexibility , Mainly because it has a lot of plug-ins , Detailed documentation and straightforward configuration format make it applicable in a variety of scenarios . We can basically find a lot of resources on the Internet , It can handle almost any problem .

  • shortcoming

    Logstash The fatal problem is its performance and resource consumption ( The default heap size is 1GB). Although its performance has greatly improved in recent years , It's much slower than its substitutes . Here you are Logstash And rsyslog Performance comparison and Logstash And filebeat Performance comparison of . It can be a problem in the case of large amounts of data .

  • Filebeat

As Beats A member of the family ,Filebeat Is a lightweight log transfer tool , Its existence is making up for Logstash The shortcomings of :Filebeat As a lightweight log transmission tool, it can push logs to the center Logstash.

  • advantage

    Filebeat It's just a binary without any dependencies . It takes up very little resources , Even though it's very young , Formal because it's simple , So there's little that can go wrong , So its reliability is very high . It also provides us with a lot of adjustable points , for example : How it searches for new files , And when the file hasn't changed for a while , When to choose to close the file handle .

  • shortcoming

    Filebeat The scope of application is very limited , So in some scenarios we have problems . for example , If you use Logstash As a downstream pipeline , We also have performance problems . Because of that ,Filebeat It's expanding . At the beginning of the , It can only send logs to Logstash and Elasticsearch, Now it can send logs to Kafka and Redis, stay 5.x In the version , It also has the ability to filter .

  • Fluentd (Docker Log driven support )

Fluentd The purpose of creation is to use it as much as possible JSON Output as log , Therefore, the transmission tool and its downstream transmission line do not need to guess the types of fields in the substring . such , It provides libraries for almost all languages , It also means that , We can plug it into our custom program .

  • advantage

    And most of them Logstash The plug-in is the same ,Fluentd The plug-in uses Ruby Language development is very easy to write and maintain . So it's a lot , Almost all source and target stores have plug-ins ( The maturity of each plug-in is also different ). It also means that we can use Fluentd To connect everything .

  • shortcoming

    Because in most scenarios , We will pass Fluentd Get structured data , It's not very flexible . But we can still use regular expressions , To parse unstructured data . Even though , Performance is good in most scenarios , But it's not *** Of , and syslog-ng equally , Its buffer only exists with the output , Single threaded core and Ruby GIL The plug-in implemented means that its performance is limited under large nodes , however , Its resource consumption is acceptable in most scenarios . For small or embedded devices , You may need to see Fluent Bit, It and Fluentd The relationship with Filebeat and Logstash The relationship between them is similar to .

Use Docker-Compose build EFK Collection center

  1. establish docker-compose.yml

Create a new one efk Catalog , Then go to the directory :

    image: httpd
      - "80:80"
      - fluentd
        tag: httpd.access
    build: ./fluentd
      - ./fluentd/conf:/fluentd/etc
      - "elasticsearch"
      - "24224:24224"
      - "24224:24224/udp"
      - "discovery.type=single-node"
      - "9200"
      - "9200:9200"
      - "elasticsearch"
      - "5601:5601"
  1. establish fluentd Image and configuration config And plug-ins

newly build fluentd/Dockerfile

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem""install""fluent-plugin-elasticsearch""--no-document""--version""4.3.3"]
USER fluent

newly build fluentd/conf/fluent.conf

  @type forward
  port 24224
<match *.**>
  @type copy
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
    @type stdout
  1. Start the service

docker-compose up
  1. The request for many times httpd Service generation log

$ curl localhost:80
  1. Verify log collection

Open browser access http://localhost:5601

Initialize creation fluentd-* Indexes

 Create index

Create index

Now you can see that Httpd The generated logs have been collected



Use fluentd Collect key points

  1. How to specify fluentd drive

  • modify daemon.json( overall situation )

  • Single container

    #  Start adding  
    --fluentd-address=localhost:24224  --log-driver=fluentd
    # Be careful : Be careful , If at this time fluentd Service to hang   The service doesn't start up , When the service starts   add 


Welcome to the official account ! The official account replied : The group of , Scan code to join our communication group !  Scan code to pay attention to official account for more learning materials

本文为[Junior General of national defense]所创,转载请带上原文链接,感谢

  1. Hand in hand teaching you to read and debug large open source project zookeeper
  2. 基于SpringBoot 在线答题系统 含小程序!
  3. 10个优秀开源JavaScript模板引擎
  4. JavaScript基础知识及写法
  5. Based on springboot online answering system, including small procedures!
  6. 重磅!谷歌Fuchsia操作系统将支持运行Linux应用程序
  7. (四十) springcloud分布式商城之跟我学习SpringCloud-Gateway整合Eureka路由转发
  8. Spring IOC 特性有哪些,不会读不懂源码!
  9. 手摸手教你阅读和调试大型开源项目 ZooKeeper
  10. 10 excellent open source JavaScript template engines
  11. Basic knowledge and writing method of JavaScript
  12. win10安装mysql5.7
  13. Heavy weight! Google's Fuchsia operating system will support running Linux applications
  14. (40) Spring cloud distributed mall learn from me spring cloud gateway integrates Eureka routing and forwarding
  15. What are the spring IOC features? I can't understand the source code!
  16. Hand in hand teaching you to read and debug large open source project zookeeper
  17. Install mysql5.7 in win10
  18. SCIP:构造数据抽象--数据结构中队列与树的解释
  19. Spring Cloud+Nacos实现服务注册中心(Hoxton版本)
  20. rabbitmq-server的安装与升级
  21. SCIP: constructing data abstraction -- Explanation of queue and tree in data structure
  22. Spring cloud + Nacos to implement service registry (Hoxton version)
  23. Installation and upgrade of rabbitmq server
  24. Lei Jun: Xiaomi's spring conference has been largely streamlined, which will take at least four hours. Some products will be directly released on Weibo
  25. IntelliJ IDEA 2021最新激活码(亲测有效,可激活至 2089 年)
  26. java版本spring cloud+spring boot+mybatis 分布式商城 微服务商城 多租户商城 电子商务 直播带货商城 社交电商
  27. win10安装Redis5.0
  28. (十五)springboot电子商务商城之SpringCloud-使用Eureka集群搭建实现高可用服务注册中心
  29. (十四)springboot电子商务商城之SpringCloud-Eureka自我保护模式和InstanceID的配置
  30. 花生壳内网穿透(Linux版)
  31. Docker部署elasticsearch(单机)
  32. (十三)springboot电子商务商城之SpringCloud-使用Eureka集群搭建实现高可用服务注册中心
  33. (十二)springboot电子商务商城之Eureka注册中心开启密码认证
  34. 七、Spring Boot 集成 Thymeleaf 模板引擎
  35. mysql 命令行秒复制数据库
  36. Windows安装Mysql(msi 图形安装)
  37. The latest activation code of IntelliJ idea 2021
  38. Java应用全链路启动速度提升至15s,阿里云SAE能力再升级
  39. Linux基础命令
  40. Who moved your red envelope? Risk control report of 2021 spring festival activities
  41. Java version spring cloud + spring boot + mybatis distributed mall micro Service Mall multi tenant mall e-commerce live delivery mall social E-commerce
  42. Java 任意音频转MP3
  43. Docker 的 DNS
  44. Docker-搭建日志监控系统
  45. ssm+mysql+maven+shiro进销存系统wms
  46. Installing redis5.0 on win10
  47. (15) Springcloud of springboot E-commerce mall - using Eureka cluster to build and implement high availability service registry
  48. (14) Springcloud Eureka self protection mode and instanceid configuration of springboot E-commerce mall
  49. Peanut shell intranet penetration (Linux version)
  50. Deploying elastic search with docker (stand alone)
  51. (13) Springcloud of springboot E-commerce mall - using Eureka cluster to build and implement high availability service registry
  52. (12) Eureka registry of springboot E-commerce mall opens password authentication
  53. 爱上 Java 的10 大理由!
  54. 7、 Spring boot integrates thymeleaf template engine
  55. 【DB宝41】监控利器PMM的使用--监控MySQL、PG、MongoDB、ProxySQL等
  56. 【DB宝42】MySQL高可用架构MHA+ProxySQL实现读写分离和负载均衡
  57. MySQL command line second replication database
  58. Windows installation of MySQL (MSI graphic installation)
  59. The full link startup speed of Java applications has been increased to 15s, and the SAE capability of alicloud has been upgraded again
  60. Linux basic command