Linux 日志管理

文章目录Linux 日志管理rsyslog 日志配置rsyslog 服务配置配置文件位置日志记录规则配置文件内容查看日志内容自定义日志记录规则最佳实践补充systemd-journald 日志systemd-journald 日志持久化保存systemd-journald 日志分析最佳实践故障1配置文件丢失故障2配置文件参数错误故障3配置文件参数错误Linux 日志管理操作系统内核和程序记录了发生的事件日志这些日志用于审核系统并解决问题。日志以文本方式保存在/var/log目录中。可以使用普通文本实用程序如less和tail检查这些日志。Linux 内置了基于Syslog协议的标准日志记录系统。许多程序使用此系统记录事件并将其组织到日志文件中。CentOS 7 中systemd-journald和rsyslog服务负责处理syslog消息。systemd-journald 服务是操作系统事件记录体系结构的核心收集系统各方面事件消息包括内核、引导过程早期阶段的输出、守护程序启动和运行时的输出、syslog事件然后将它们重组为标准格式并写入结构化的索引系统日志中。rsyslog 服务读取systemd-journald日志然后记录到日志文件或根据自己的配置将日志保存到不同的文件中以及转发给其他程序。rsyslog 日志配置rsyslog 服务配置配置文件位置主配置 /etc/rsyslog.conf。主配置文件中以下配置作用是引入从配置目录中配置文件。bash # Include all config files in /etc/rsyslog.d/ include(file/etc/rsyslog.d/*.conf modeoptional)从配置/etc/rsyslog.d/*.conf。日志记录规则每一条日志消息都可以通过消息类型facility和priority分类参考rsyslog.conf(5)。日志记录规则格式:facility连接符号priority 处理方式例如cron.infofacility设备类型facility说明authpam产生的日志authprivsshftp等登录信息的验证信息cron周期性任务计划相关kern内核Ipr打印mail邮件news新闻组user用户程序产生的相关信息local 0~7自定义的日志设备本地使用priority优先级优先级优先级名称严重性0emerg系统不可用1alert必须立即采取措施2crit临界情况3err非验证错误状况4warning警告情況5notice正常但重要的事件6info信息性事件7debug调式级别消息8none代表什么都不存储连接符号连接符号作用.表示大于等于xxx级别的信息.表示等于xXx级别的信息.!表示在xXx之外的等级的信息处理方式记录到文件发送到终端转发给其他服务器配置文件内容/etc/rsyslog.conf 中部分内容如下### RULES ##### Log all kernel messages to the console.#kern.* /dev/console*.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron# Everybody gets emergency messages*.emerg :omusrmsg:*# Save news errors of level crit and higher in a special file.uucp,news.crit /var/log/spooler# Save boot messages also to boot.loglocal7.* /var/log/boot.log日志存储位置/var/log/messages大多数系统日志消息记录在此处不包括与身份验证、电子邮件处理和调度作业执行相关的消息以及纯碎与调试相关的消息。/var/log/secure与安全性和身份验证事件相关的syslog消息。/var/log/maillog与邮件服务器相关的syslog消息。/var/log/cron与调度作业执行相关的syslog消息。/var/log/boot.log与系统启动相关的非syslog控制台消息。查看日志内容[rootserver ~ 09:17:57]# tail -f /var/log/messagesJul2409:17:57 server systemd: Stopped OpenSSH server daemon. Jul2409:17:57 server systemd: Starting OpenSSH server daemon... Jul2409:17:57 server sshd: /etc/ssh/sshd_config: line140: Bad configuration option: hello Jul2409:17:57 server sshd: /etc/ssh/sshd_config: terminating,1bad configuration options Jul2409:17:57 server systemd: sshd.service: main process exited,codeexited,status255/n/......日志内容说明Jul 24 09:17:57代表日志产生时间。centos7产生日志的主机名。systemd-logind产生日志的进程。最后一个区域是日志内容例如“Removed session 15.”。[rootserver ~]# tail -f /var/log/secure......Nov1010:27:26 centos7 sshd[2755]: Accepted passwordforlaoma from10.1.8.1 port5530ssh2 Nov1010:27:27 centos7 sshd[2755]: pam_unix(sshd:session): session openedforuser laoma by(uid0)Nov1010:27:31 centos7 sshd[2755]: pam_unix(sshd:session): session closedforuser laoma自定义日志记录规则[rootserver ~]# vim /etc/rsyslog.d/laoma.conflocal5.* /var/log/laoma.log[rootserver ~]# systemctl restart rsyslog.service[rootserver ~]# ls /var/log/laoma.logls: 无法访问/var/log/laoma.log: 没有那个文件或目录# 手动生成一个消息[laomacentos7 ~]$ logger-plocal5.infotest my log[rootserver ~]# cat /var/log/laoma.logNov1010:32:26 centos7 laoma:testmy log最佳实践1.故障1sshd_config文件丢失模拟[rootserver ~ 09:03:12]# mv /etc/ssh/sshd_config .[rootserver ~ 09:04:05]# systemctl restart sshdJobforsshd.service failed because the control process exited with error code. Seesystemctl status sshd.serviceandjournalctl -xefordetails.监控日志[rootserver ~ 09:17:57]# tail -f /var/log/messages提示一下错误根据提示恢复文件。[rootserver ~ 09:04:13]# cp sshd_config /etc/ssh[rootserver ~ 09:06:18]# systemctl restart sshd2.故障2sshd_config配置错误模拟故echo hello world /etc/ssh/sshd_config[rootserver ~ 09:06:30]# echo hello world /etc/ssh/sshd_config监控日志[rootserver ~ 09:17:57]# tail -f /var/log/messagesJul2409:17:57 server systemd: Stopped OpenSSH server daemon. Jul2409:17:57 server systemd: Starting OpenSSH server daemon...#提示错误Jul2409:17:57 server sshd: /etc/ssh/sshd_config: line140: Bad configuration option: hello Jul2409:17:57 server sshd: /etc/ssh/sshd_config: terminating,1bad configuration options Jul2409:17:57 server systemd: sshd.service: main process exited,codeexited,status255/n/a Jul2409:17:57 server systemd: Failed to start OpenSSH server daemon. Jul2409:17:57 server systemd: Unit sshd.service entered failed state. Jul2409:17:57 server systemd: sshd.service failed.根据提示修复配置文件错误。3.故障3/etc/ssh/sshd_config 里的将PasswordAuthentication改成Passw0rdAuthentication[rootserver ~ 09:26:54]# vim /etc/ssh/sshd_config[rootserver ~ 09:27:15]# systemctl restart sshdJobforsshd.service failed because the control process exited with error code. Seesystemctl status sshd.serviceandjournalctl -xefordetails.[rootserver ~ 09:29:51]# tail -f /var/log/messages 提示一下错误补充虽然系统提供了日志服务但并不会记录所有内容。系统中的应用程序是否使用 rsyslog 服务记录日志取决于应用程序设计。例如httpd 服务使用自己的日志记录。sshd 服务使用 rsyslog 服务记录登录和退出日志。systemd-journald 日志systemd-journald 日志持久化保存systemd-journald 配置文件/etc/systemd/journald.conf详情参考 journald.conf(5)Storage 参数更改参数值后需要重启systemd-journald服务。persistent将日志存储在/var/log/journal目录中这可在系统重启后持久保留。如果/var/log/journal目录不存在systemd-journald服务会创建它。volatile将日志存储在易失性/run/log/journal目录中。因为/run文件系统是临时的仅存在于运行时内存中存储在其中的数据包括系统日志不会在系统启后持久保留。auto如果/var/log/journal目录存在那么rsyslog会使用持久存储否则使用易失性存储。如果未设置Storage参数此为默认操作。[rootserver ~10:33:00]# vim /etc/systemd/journald.conf......[Journal]# StorageautoStoragepersistent......[rootserver ~10:39:35]# systemctl restart systemd-journaldsystemd-journald 日志分析[rootserver ~10:39:40]# ls /var/log/journal/a54662f266c147a995aaacaf13321dfa# systemd-journald日志是以二进制方式存储不能使用常规文本工具查看# 查看所有日志条目默认使用less查看文档[rootserver ~10:39:59]# journalctl-- Logs begin at Fri2026-07-24 09:01:37 CST, end at Fri2026-07-2410:40:14 CST. -- Jul2409:01:37 centos7.wanho.cloud systemd-journal[95]: Runtime journal is using8.0M(max a Jul2409:01:37 centos7.wanho.cloud kernel: Initializing cgroup subsys cpuset Jul2409:01:37 centos7.wanho.cloud kernel: Initializing cgroup subsys cpu Jul2409:01:37 centos7.wanho.cloud kernel: Initializing cgroup subsys cpuacct Jul2409:01:37 centos7.wanho.cloud kernel: Linux version3.10.0-1160.71.1.el7.x86_64(mockbu Jul2409:01:37 centos7.wanho.cloud kernel: Command line:BOOT_IMAGE/vmlinuz-3.10.0-1160.71.......# 动态查看所有日志条目[rootserver ~]# journalctl -f# 查看本次系统启动到现在所有日志条目[rootserver ~]# journalctl -b 0# 1 第1次启动的日志# 2 第2次启动的日志# N 第n次启动的日志# 查看最近5条日志[rootserver ~]# journalctl -n 5# 查看error级别日志[rootserver ~]# journalctl -p err[rootserver ~]# journalctl -p err | cat# 根据时间查看日志[rootserver ~]# journalctl --since today[rootserver ~]# journalctl --since 2019-02-10 20:30:00 --until 2019-02-13 12:00:00[rootserver ~]# journalctl --since -1 hour# 详细查看所有日志[rootserver ~]# journalctl -o verbose# 查看特定unit日志[rootserver ~]# journalctl -u sshd.service最佳实践故障1配置文件丢失[rootserver ~10:45:24]# mv /etc/ssh/sshd_config .mv: overwrite ‘./sshd_config’?yes[rootserver ~10:45:33]# systemctl restart sshdJobforsshd.service failed because the control process exited with error code. Seesystemctl status sshd.serviceandjournalctl -xefordetails.处理过程通过日志发现/etc/ssh/sshd_config: No such file or directory文件丢失。# 重启服务时动态监控日志[rootserver ~]# journalctl -fJul2410:46:26 server.myz.cloud systemd[1]: Starting OpenSSH server daemon... Jul2410:46:26 server.myz.cloud sshd[6404]: /etc/ssh/sshd_config: No suchfileor directory Jul2410:46:26 server.myz.cloud systemd[1]: sshd.service: main process exited,codeexited,status1/FAILURE Jul2410:46:26 server.myz.cloud systemd[1]: Failed to start OpenSSH server daemon.......# 移动回来并重启服务[rootserver ~]# mv sshd_config /etc/ssh/sshd_config[rootserver ~]# systemctl restart sshd故障2配置文件参数错误[rootserver ~10:48:27]# echo PermitRootLogin hahaha /etc/ssh/sshd_config[rootserver ~10:48:39]# systemctl restart sshdJobforsshd.service failed because the control process exited with error code. Seesystemctl status sshd.serviceandjournalctl -xefordetails.处理过程通过日志发现/etc/ssh/sshd_config line 141: unsupported option hahaha.# 重启服务时动态监控日志[rootserver ~]# journalctl -fJul2410:49:30 server.myz.cloud systemd[1]: Stopped OpenSSH server daemon. Jul2410:49:30 server.myz.cloud systemd[1]: Starting OpenSSH server daemon... Jul2410:49:30 server.myz.cloud sshd[6548]: /etc/ssh/sshd_config line142: unsupported optionhahaha.Jul2410:49:30 server.myz.cloud systemd[1]: sshd.service: main process exited,codeexited,status255/n/a Jul2410:49:30 server.myz.cloud systemd[1]: Failed to start OpenSSH server daemon. Jul2410:49:30 server.myz.cloud systemd[1]: Unit sshd.service entered failed state.# 清理对应无效记录并重启服务[rootserver ~10:49:42]# sed -i /hahaha/d /etc/ssh/sshd_config[rootserver ~10:50:43]# systemctl restart sshd故障3配置文件参数错误[rootserver ~]# yum install -y httpd[rootserver ~]# sed -i s/Listen 80/Listen 80000/g /etc/httpd/conf/httpd.conf[rootserver ~11:03:07]# systemctl restart httpdJobforhttpd.service failed because the control process exited with error code. Seesystemctl status httpd.serviceandjournalctl -xefordetails.处理过程通过日志发现AH00526: Syntax error on line 42 of /etc/httpd/conf/httpd.conf# 重启服务时动态监控日志[rootserver ~]# journalctl -fJul2411:04:53 server.myz.cloud systemd[1]: Starting The Apache HTTP Server... Jul2411:04:53 server.myz.cloud httpd[7168]: AH00526: Syntax error on line42of /etc/httpd/conf/httpd.conf: Jul2411:04:53 server.myz.cloud httpd[7168]: Invalid address or port Jul2411:04:53 server.myz.cloud systemd[1]: httpd.service: main process exited,codeexited,status1/FAILURE Jul2411:04:53 server.myz.cloud systemd[1]: Failed to start The Apache HTTP Server. Jul2411:04:53 server.myz.cloud systemd[1]: Unit httpd.service entered failed state. Jul2411:04:53 server.myz.cloud systemd[1]: httpd.service failed.# 修改回来并重启服务[rootserver ~]# sed -i s/Listen 80000/Listen 80/g /etc/httpd/conf/httpd.conf[rootserver ~]# systemctl restart httpd/FAILURE Jul2411:04:53 server.myz.cloud systemd[1]: Failed to start The Apache HTTP Server. Jul2411:04:53 server.myz.cloud systemd[1]: Unit httpd.service entered failed state. Jul2411:04:53 server.myz.cloud systemd[1]: httpd.service failed.# 修改回来并重启服务[rootserver ~]# sed -i s/Listen 80000/Listen 80/g /etc/httpd/conf/httpd.conf[rootserver ~]# systemctl restart httpd