Linux system learning series -- Linux system log management (2)

Programmer grey 2020-11-09 17:45:03
linux learning series linux log


Logs are important system files , All important events in the system are recorded and saved . But log files also need regular maintenance , Because log files are growing , If you don't do log maintenance at all , And let it increase at will , It won't take long , Our hard disk will be full of .

The main job of log maintenance is to delete the old log files , To make room for new log files . If the work is done manually by the Administrator , It's actually very cumbersome , And it's easy to forget . that Linux Whether the system can automatically complete the rotation of the log ?

logrotate It's used for log rotation ( Also called log dump ) Of , That is to move and rename the old log file , At the same time, create a new empty log file to record the new log , Old log files are deleted when they are out of range .
image.png

need Linux Information and Linux Beginner Subjects +qun832218493 obtain

Naming rules for log files

The main function of log rotation is to move and rename old log files , Also create a new empty log file , Old log files are deleted when they are out of range . that , After the old log file was renamed , How to name it ? Mainly depends on /etc/logrotate.conf In the configuration file “dateext” Parameters .

If there is... In the configuration file “dateext” Parameters , Then the log will use the date as the suffix of the log file , Such as “secure-20130605”. In this way, the log file names will not overlap , There is no need to rename the log file , Only the specified number of logs need to be saved , Delete redundant log files .

If there is no “dateext” Parameters , Then the log file needs to be renamed . When the first log rotation , Current “secure” The log will automatically be renamed “secure.1”, Then build “secure” journal , To save new logs ; When the log rotation is performed for the second time ,“secure.1” It will automatically change its name to “secure.2”, Current “secure” The log will automatically be renamed “secure.1”, And then it's going to be new “secure” journal , To save new logs ; And so on .

logrotate The configuration file

Let's take a look at logrotate Configuration file for /etc/logrotate.conf Default content of .

[root@localhost ~]# vi /etc/logrotate.conf

see "man logrotate" for details

rotate log files weekly

weekly

Rotate the log files once a week

keep 4 weeks worth of backlogs rotate 4

preservation 4 Log files , in other words , If it does 5 Next log rotation , The first backup log will be deleted

create new (empty) log files after rotating old ones create

In log rotation , Automatically create new log files

use date as a suffix of the rotated file dateext

Use date as suffix of log rotation file

uncomment this if you want your log files compressed #compress

Is the log file compressed . If uncomment , The log will be compressed at the same time as the dump

The above log configuration is the default configuration , If the log needs to be rotated, no independent parameters are set , Then the above parameters will be followed

If the rotation log is configured with independent parameters , Then independent parameters have a higher priority

RPM packages drop log rotation information into this directory include /etc/logrotate.d

contain /etc/logrotate.d/ All the sub configuration files in the directory . in other words , It will read all the sub configuration files in this directory , Log rotation

no packages own wtmp and btmp -- we'11 rotate them here

The following two rotations have their own independent parameters , If it conflicts with the default parameters , Then the independent parameters take effect

/var/log/wtmp {

The following parameters are only valid for this directory

monthly

Rotate the log files once a month

create 0664 root utmp

New log files created , Permissions are 0664, The owner is root, The group is utmp Group

minsize 1M

The minimum rotation size of the log file is 1MB. That is to say, the log must exceed 1MB It will take turns , Otherwise, even if the time reaches one month , And we don't have to take turns

rotate 1

Keep only one log backup . That is to say, only keep wtmp and wtmp.1 Yuezhi )

/var/log/btmp {

The following parameters are only for /var/log/btmp take effect

missingok

If the log does not exist , Ignore the warning message in the log

monthly
create 0600 root utmp
rotate 1
}

system-specific logs may be also be configured here.

  • In this configuration file , It is divided into three parts : The first part is the default settings , If there is no special configuration for the log file to be dumped , The default parameters are followed ;
  • The second part is reading /etc/logrotate.d/ The sub configuration file of log rotation in directory , in other words , stay /etc/logrotate.d/ All sub configuration files in the directory that conform to the syntax rules will also be log rotated ;
  • The third part is about wtmp and btmp Set the rotation of log files , If this setting conflicts with the default parameter , The current settings take effect ( Such as wtmp The current parameter set for the rotation time is monthly , The rotation time of the default parameter is weekly , On the other hand wtmp This log file says , The rotation time is monthly , The current setting parameters take effect ).

logrotate The main parameters of the configuration file are shown in the table 1 Shown .

Among these parameters, the more difficult to understand should be prerotate/endscript and postrotate/endscript, We make use of “man logrotate” To explain these two parameters . for example :

"/var/log/httpd/access.log" /var/log/httpd/error.log {

The log rotation is /var/log/httpd/ in RPM Package installed by default apache Correct access log and error log

    rotate 5
    # Rotation 5 Time
    mail www@my.org
    # Send the message to the designated mailbox
    size 100k
    # The log is larger than 100KB Only when the log rotation , No longer rotate according to time
    sharedscripts
    # The following script is executed only once
    postrotate
    # After the log rotation , Execute the following script
    /usr/bin/killall -HUP httpd
    # restart apache service
endscript

Script end

}

prerotate and postrotate It is mainly used to execute the specified script while the log rotation is in progress , It is generally used to restart the service after log rotation . Let's emphasize here , If your log is written rsyslog Of the configuration file for the service , Then add the new log to logrotate after , Make sure you reboot rsyslog service , Otherwise you will find out , Although the new log set up , But the data is still written into the old log . That's because although logrotate Know that the log rotation , however rsyslog The service didn't know .

Empathy , If you use the source package to install apache、Nginx Etc , You need to restart apache or Nginx service , At the same time, you have to restart rsyslog service , Otherwise, the log will not rotate normally .

however , A typical application here is to add specific logs chattr Of a attribute . If the system file is added with a attribute , Then this file can only add data , You can't delete or modify existing data ,root Users are no exception .

therefore , We will add... To important log files a attribute , In this way, the log file can be protected from malicious modification . however , Once joined a attribute , So in the log rotation , This log file cannot be renamed , Of course, there is no log rotation . We can use prerotate and postrotate Parameter to modify the log file chattr Of a attribute .

Add your own log to the log rotation

If some logs do not add log rotation by default ( For example, the log of the service installed by the source code package , Or add your own log ), Then these logs will not rotate by default , This certainly does not meet our requirements for log management . If you need to add these logs to the log rotation , Then how to operate ?

  • There are two ways : The first way is directly to /etc/logrotate.conf Rotation policy written to the log in the configuration file , To add the log to the rotation ;
  • The second way is to /etc/logrotate.d/ The rotation file of the newly created log in the directory , Write the correct rotation policy in the rotation file , Because the files in this directory will be included in the main configuration file , So you can also add logs to rotation .

We recommend the second method , Because there are a lot of logs that need to rotate in the system , If all are written directly into /etc/logrotate.conf The configuration file , Then the manageability of this file will be very poor , Not conducive to the maintenance of this file .

It's complicated to say , Let's take an example . Remember that we made it ourselves /var/log/alert.log Journal ? This log is not the default log of the system , It's through /etc/rsyslog.conf The log generated by the configuration file itself , So by default, the log will not be rotated . If we need to add this log to the log rotation strategy , Then how to achieve it ? We use the second method , That is to say /etc/logrotate.d/ Create a rotation file for this log in the directory .

The specific steps are as follows :

[root@localhost ~]# chattr +a /var/log/alert.log # First give the log file chattr Of a attribute , Keep the log safe
[root@localhost ~]# vi /etc/logrotate.d/alter

establish alter Rotation documents , hold /var/log/alert.log Join the rotation

/var/log/alert.log {
    weekly
    # Rotate once a week
    rotate 6
    # Retain 6 One rotation says ambition
    sharedscripts
    # The following command is executed only once
    prerotate
    # Execute before log rotation
        /usr/bin/chattr -a /var/log/alert.log
        # Cancel before log rotation a attribute , So that the logs can rotate
    endscript
    # The script ends with
    sharedscripts
    postrotate
    # Execute after log rotation
        /usr/bin/chattr +a /var/log/alert.log
        # After the log rotation , Rejoin a attribute
    endscript
    sharedscripts
    postrotate
    /bin/kill -HUP $(/bin/cat /var/run/syslogd.pid 2>/dev/null) fi>/dev/null
    endscript
    # restart rsyslog service , Make sure that the log rotation works properly
}

So we generate our own logs /var/log/alert.log It's time for log rotation , Of course, these configuration information can also be written directly /etc/logrotate.conf Of this configuration file .

  1. Linux logrotate Command usage is explained in detail : Log dump ( Rotation )

The reason why log rotation can backup logs at a specified time , Because it depends on the system's timed tasks . If you remember /etc/cron.daily/ Catalog , You will find that there are logrotate Of documents , Take a look at this file , The order is as follows :

[root@localhost ~]# vi /etc/cron.daily/logrotate

!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1

The most important thing is to implement logrotate command

EXITVALUE=$?
if [ $EXITVALUE!= 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

in other words , The system runs every day /etc/cron.daily/logrotate file , Run... In this file “/usr/sbin/logrotate/etc/logrotate.conf>/dev/null 2>&1” command .logrotate Orders will be based on /etc/logrotate.conf Configuration of configuration file , To determine whether the logs in the configuration file meet the conditions of log rotation ( such as , Log backup time has been full for a week ), If meet , The journal will rotate . So , Log rotation or by crond Service initiated .

logrotate What is the format of the command ? Let's learn about .

[root@localhost ~]# logrotate [ Options ] Profile name

  • Options : If there are no options for this command , The log rotation will be performed according to the conditions in the configuration file
  • -v: Show log rotation process . Joined the -v Options , The log rotation process will be displayed
  • -f: Force log rotation . Whether the conditions of log rotation are met or not , Force rotation of all logs in the configuration file

We execute logrotate command , And take a look at the execution process .

[root@localhost ~]# logrotate -v /etc/logrotate.conf

Check the log rotation process

… Omit part of the output …
rotating pattern:/var/log/alert.log weekly (6 rotations)

That's what we've joined in the rotation alert.log journal

empty log files are rotated, old logs are removed
considering log /var/log/alert.log
log does not need rotating

It's not a week , So there's no log rotation

… Omit part of the output …

We found that ,/var/log/alert.log Added log rotation , Has been logrotate Identify and call , It's just that time doesn't meet the criteria for rotation , So there was no rotation . Then we force a log rotation , See what happens .

[root@localhost ~]# logrotate -vf /etc/logrotate.conf

Force log rotation , Whether or not the rotation conditions are met

… Omit part of the output …
rotating pattern:/var/log/alert.log forced from command line (6 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/alert.log
log needs rotating

The journal needs to rotate

rotating log /var/log/alert.log,log->rotateCount is 6
dateext suffix '-20130607'

Extract date parameters

glob pattern '-0-90-90-90-9'
glob finding old rotated logs failed
running prerotate script
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/alert.log to /var/log/alert.log-20130607

The old log was renamed

creating new /var/log/alert.log mode = 0600 uid = 0 gid = 0

Create a new log file , Also specify permissions 、 Owners and groups

running postrotate script
… Omit part of the output …

We found that ,alert.log The log has completed the log rotation . Take a look at the newly generated logs and the old logs , as follows :

[root@localhost ~]# ll /var/log/alert.log*
-rw-------.1 root root 0 6 month 7 10:07 /var/log/alert.log
-rw-------.1 root root 237 6 month 7 09:58 /var/log/alert.log-20130607

The old log files have been rotated

[root@localhost ~]# lsattr /var/log/alert.log
-----a-------e- /var/log/alert.log

New log files are added automatically chattr Of a attribute

logrotate The command is using “-f” After the option , It doesn't matter if the log meets the rotation conditions , And mandatory rotation of all logs .

  1. Linux Log analysis tool (logwatch) Installation and use

Logs are very important system files , The important work of the administrator every day is to analyze and view the log of the server , Determine the health status of the server . But log management is a very boring job , If you need an administrator to view all logs on the server manually , It was a very painful job . Some administrators are lazy , Omit log detection , But doing so can easily lead to server problems .

So do we have alternatives ? Yes , That's the log analysis tool . These log analysis tools look at the logs in detail , Analyze these logs at the same time , And send the results of the analysis to root user . such , We just check the email of log analysis tool every day , You can know the basic situation of the server , Instead of checking the logs one by one . In this way, the system administrator can be free from the heavy daily work , To deal with more important work .

stay CentOS It comes with a log analysis tool , Namely logwatch. However, this tool is not installed by default ( Because we chose “Basic Server”), So you need to install it by hand . The installation command is as follows :

[root@localhost Packages]# yum -y install logwatch

After installation , It needs to be generated manually logwatch Configuration file for . The default profile is /etc/logwatch/conf/logwatch.conf, But this configuration file is empty , You need to copy the template configuration file . The order is as follows :

[root@localhost ~]# cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/logwatch.conf

Copy profile

Most of the contents of this configuration file are comments , Let's get rid of the comments , The contents of this configuration file are as follows :

[root@localhost ~]# vi /etc/logwatch/conf/logwatch.conf

View the configuration file

LogDir = /var/log

logwatch Can analyze and count /var/log/ Log in

TmpDir = /var/cache/logwatch

Appoint logwatch The temporary directory of

MailTo = root

Log analysis results , to root Users send email

MailFrom = Logwatch

The sender of the mail is Logwatch, Show when receiving mail

Print =

Whether or not to print . If you choose “yes”, Then log analysis will be printed to standard output , And don't send email . We don't print here ,# But to root Users send email

Save = /tmp/logwatch

If you turn this on , Log analysis doesn't send email , It's stored in /tmp/logwatch In file

If you turn this on , Log analysis doesn't send email , It's stored in /tmp/logwatch In file

Range = yesterday

Analyze which day's log . Can identify “All”“Today”“Yesterday”, Used for analysis “ All logs ”“ Today's Journal ”“ Yesterday's diary ”

Detail = Low

The level of detail in the log . Can identify “Low”“Med”“High”. It can also be expressed in numbers , The scope is 0~10,“0” It means the least detailed ,“10” For the most detailed

Service = All

Analyze and monitor all logs

Service = "-zz-network"

But not monitoring “-zz-network” Logs of services .“- service name ” Indicates that the log of this service is not analyzed and monitored

Service = "-zz-sys"
Service = "-eximstats"

This configuration file doesn't need to be modified ( I put Range Item changed to All, Otherwise, there are too few logs that can be analyzed in the experiment later ), It will default to daily execution . Why does it run every day ? Smart readers have thought of , It must be crond The function of service . you 're right ,logwatch Once installed , Will be in /etc/cron.daily/ Create in directory “0logwatch” file , Used to execute at regular intervals every day logwatch command , Analyze and monitor relevant logs .

If you want this log analysis to be performed immediately , Then just execute logrotate Command is enough . The order is as follows :

[root@localhost ~]# logwatch

Do it now logwatch Log analysis tool

[root01ocalhost ~]# mail

Check email

Heirloom Mail version 12.4 7/29/08. Type ? for help, "/var/spool/mail/root": 5 messages 1 new 2 unread
1 logwatch@localhost.1 Fri Jun 7 11:17 42/1482 "Logwatch for localhost.localdomain (Linux)"
U 2 logwatch@localhost.1 Fri Jun 7 11:19 42/1481 "Logwatch for localhost.localdomain (Linux)"
3 logwatch@localhost.1 Fri Jun 7 11:23 1234/70928 "Logwatch for localhost.localdomain (Linux)"
4 logwatch@localhost.1 Fri Jun 7 11:24 190/5070 "Logwatch for localhost.localdomain (Linux)"
5 logwatch@localhost.1 Fri Jun 7 11:55 41/1471 "Logwatch for localhost.localdomain (Linux)"

N 6 logwatch@localhost.1 Fri Jun 7 11:57 189/5059 "Logwatch for localhost.localdomain (Linux)"

The first 6 An email is a newly generated blog analysis email ,"N" The representative didn't check

& 6
Message 6:
From root@localhost.localdomain Fri Jun 7 11:57:35 2013 Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
To: root@localhost.localdomain
From: logwatch@localhost.localdomain
Subject: Logwatch for localhost.localdomain (Linux)
Content-Type: text/plain; charset="iso-8859-1"
Date: Fri, 7 Jun 2013 11:57:33 +0800 (CST)
Status: R

Logwatch 7.3.6 (05/19/07)

Processing Initiated: Fri Jun 7 11:57:33 2013
Date Range Processed: all
Detail Level of Output: 0
Type of Output: unformatted
Logfiles for Host: localhost.localdomain

The above is the time and date of the analysis

... Omit part of the output ...
--------- Connections (secure-log) Begin-----------

analysis secure.log Contents of the log . Count which users and groups have been created , And the wrong login information New Users:

    bb (501)
    def (503)
    hjk (504)
    zhangsan (505)
    dovecot (97)
    dovenull (498)
    aa (500)

New Groups:
    bb (501)
    def (503)
    hjk (504)
    zhangsan (505)
    dovecot (97)
    dovenull (498)
    aa (500)

Failed logins:
    User root:
    (null): 3 Time(s)

Root logins on tty's: 7 Time(s).

Unmatched Entries
groupadd: group added to /etc/group: name=dovecot, GID=97: 1 Time(s)
groupadd: group added to /etc/group: name=dovenul1, GID=498: 1 Time(s)
groupadd: group added to /etc/gshadow: name=dovecot: 1 Time(s)groupadd: group added to /etc/gshadow: name=dovenull: 1 Time(s)
--------Connections (secure-log)End-------
-------------SSHD Begin-------------------

analysis SSHD Log . You can know what IP The address is connected to the server

SSHD Killed: 7 Time(s)
SSHD Started: 24 Time(s)
Users logging in through sshd:
192.168.0.104: 10 times
192.168.0.108: 8 times
192.168.0.101: 6 times
192.168.0.126: 4 times
192.168.0.100: 3 times
192.168.0.105: 3 times
192.168.0.106: 2 times
192.168.0.102: 1 time
192.168.0.103: 1 time
SFTP subsystem requests: 3. Time(s)
Unmatched Entries
Exiting on signal 15 : 6 time(s)
----------------SSHD End-----------

--------------- yum Begin ---------

Statistics yum Installed software . You can know what software we have installed

Packages Installed:
    perl-YAML-Syck-1.07-4.el6.i686
    perl-Date-Manip-6.24-1.el6.noarch
    logwatch-7.3.6-49.el6.noarch
-----------yum End-------------

--------Disk Space Begin-------

Statistics of disk space

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 20G 1.9G 17G 11% /
/dev/sda1 194M 26M 158M 15% /boot
/dev/sr0 3.5G 3.5G 0 100% /mnt/cdrom
---------Disk Space End-----------------

Logwatch End

With this log analysis tool , Log management will be much easier . Of course , stay Linux Can support many log analysis tools , We only introduce here CentOS Self contained logwatch, You can choose the corresponding log analysis tool according to your own habits .

版权声明
本文为[Programmer grey]所创,转载请带上原文链接,感谢

  1. 【计算机网络 12(1),尚学堂马士兵Java视频教程
  2. 【程序猿历程,史上最全的Java面试题集锦在这里
  3. 【程序猿历程(1),Javaweb视频教程百度云
  4. Notes on MySQL 45 lectures (1-7)
  5. [computer network 12 (1), Shang Xuetang Ma soldier java video tutorial
  6. The most complete collection of Java interview questions in history is here
  7. [process of program ape (1), JavaWeb video tutorial, baidu cloud
  8. Notes on MySQL 45 lectures (1-7)
  9. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  10. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  11. 精进 Spring Boot 03:Spring Boot 的配置文件和配置管理,以及用三种方式读取配置文件
  12. Refined spring boot 03: spring boot configuration files and configuration management, and reading configuration files in three ways
  13. 【递归,Java传智播客笔记
  14. [recursion, Java intelligence podcast notes
  15. [adhere to painting for 386 days] the beginning of spring of 24 solar terms
  16. K8S系列第八篇(Service、EndPoints以及高可用kubeadm部署)
  17. K8s Series Part 8 (service, endpoints and high availability kubeadm deployment)
  18. 【重识 HTML (3),350道Java面试真题分享
  19. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  20. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  21. [re recognize HTML (3) and share 350 real Java interview questions
  22. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  23. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  24. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  25. RPC 1: how to develop RPC framework from scratch
  26. 造轮子系列之RPC 1:如何从零开始开发RPC框架
  27. RPC 1: how to develop RPC framework from scratch
  28. 一次性捋清楚吧,对乱糟糟的,Spring事务扩展机制
  29. 一文彻底弄懂如何选择抽象类还是接口,连续四年百度Java岗必问面试题
  30. Redis常用命令
  31. 一双拖鞋引发的血案,狂神说Java系列笔记
  32. 一、mysql基础安装
  33. 一位程序员的独白:尽管我一生坎坷,Java框架面试基础
  34. Clear it all at once. For the messy, spring transaction extension mechanism
  35. A thorough understanding of how to choose abstract classes or interfaces, baidu Java post must ask interview questions for four consecutive years
  36. Redis common commands
  37. A pair of slippers triggered the murder, crazy God said java series notes
  38. 1、 MySQL basic installation
  39. Monologue of a programmer: despite my ups and downs in my life, Java framework is the foundation of interview
  40. 【大厂面试】三面三问Spring循环依赖,请一定要把这篇看完(建议收藏)
  41. 一线互联网企业中,springboot入门项目
  42. 一篇文带你入门SSM框架Spring开发,帮你快速拿Offer
  43. 【面试资料】Java全集、微服务、大数据、数据结构与算法、机器学习知识最全总结,283页pdf
  44. 【leetcode刷题】24.数组中重复的数字——Java版
  45. 【leetcode刷题】23.对称二叉树——Java版
  46. 【leetcode刷题】22.二叉树的中序遍历——Java版
  47. 【leetcode刷题】21.三数之和——Java版
  48. 【leetcode刷题】20.最长回文子串——Java版
  49. 【leetcode刷题】19.回文链表——Java版
  50. 【leetcode刷题】18.反转链表——Java版
  51. 【leetcode刷题】17.相交链表——Java&python版
  52. 【leetcode刷题】16.环形链表——Java版
  53. 【leetcode刷题】15.汉明距离——Java版
  54. 【leetcode刷题】14.找到所有数组中消失的数字——Java版
  55. 【leetcode刷题】13.比特位计数——Java版
  56. oracle控制用户权限命令
  57. 三年Java开发,继阿里,鲁班二期Java架构师
  58. Oracle必须要启动的服务
  59. 万字长文!深入剖析HashMap,Java基础笔试题大全带答案
  60. 一问Kafka就心慌?我却凭着这份,图灵学院vip课程百度云