遇到一台CentOS 7.x的服务器异常,通过dmesg看到有大量刷屏如下信息刷屏:(可能长期持续甚至会导致宕机)。
1 2 3 4 |
[ 445.509296] systemd-journald[13720]: Failed to write entry (15 items, 350 bytes), ignoring: Cannot assign requested address [ 451.494537] systemd-journald[13720]: Failed to write entry (19 items, 505 bytes), ignoring: Cannot assign requested address [ 451.506036] systemd-journald[13720]: Failed to write entry (19 items, 505 bytes), ignoring: Cannot assign requested address [ 451.517485] systemd-journald[13720]: Failed to write entry (19 items, 545 bytes), ignoring: Cannot assign requested address |
单独重启了systemd-journald服务也没有用,网上找了一会有了方案,最后解决了。
从这个错误来看就是,journald写日志错误。
通过 journalctl --verify 命令找到损坏的journal文件,然后删除或者mv移动它,再重启systemd-journald服务即可。
1 2 3 4 5 |
$sudo journalctl --verify 13bf258: Invalid object File corruption detected at /var/log/journal/21fbd9cf5ebc4d1ea8e83b6f1be17eaf/system.journal:13bf258 (of 25165824 bytes, 82%). FAIL: /var/log/journal/21fbd9cf5ebc4d1ea8e83b6f1be17eaf/system.journal (Cannot assign requested address) PASS: /run/log/journal/111bd9cf5ebc4d1ea8e83b6f1b132433/system.journal |
重启systemd-journald服务。
1 |
sudo systemctl restart systemd-journald.service |
解决后,通过dmesg查看信息,有如下信息,然后也没有再继续打印“Failed to write entry”的错误了。
1 2 3 4 |
[ 451.550833] systemd-journald[13720]: Failed to write entry (19 items, 534 bytes), ignoring: Cannot assign requested address [ 452.086954] systemd-journald[13720]: Failed to write entry (21 items, 664 bytes), ignoring: Cannot assign requested address [ 452.098319] systemd-journald[13720]: /var/log/journal/21fbd9cf5ebc4d1ea8e83b6f1be17eaf/system.journal: Journal file has been deleted, rotating. [ 456.087325] systemd-journald[13720]: Received SIGTERM from PID 1 (systemd). |
参考文档:
https://www.claudiokuenzler.com/blog/627/systemd-journald-stopped-logging-centos-7-failed-to-write-entry#.Wn2G74KYPOQ
https://bugzilla.redhat.com/show_bug.cgi?id=1292447