Using Linux Logs

Using Linux Logs


Log files on Linux can provide a lot of useful information on what's happening on your system. The below commands can help you sort through the data and pinpoint problems.


Linux systems maintain quite a collection of log files, many of which you are probably rarely tempted to view. Some of these log files are quite valuable, though, and options for exploring them might be more interesting and varied than you imagine. Let's look at some system logs and get a handle on some of the ways in which log data might be easier to probe. Log file rotation


First, there's the issue of log rotation. Some Linux log files are “rotated.” In other words, the system stores more than one "generation" of these files, mostly to keep them from using too much disk space. The older logs are then compressed but left available for a while. Eventually, the oldest in a series of rotated log files will be automatically deleted in the log rotation process, but you’ll still have access to a number of the older logs so that you can examine log entries that were added in the last few days or weeks when and if you need to look a little further back into some issue you're tracking.


To get a feel for what types of system information are being saved, simply cd over to the /var/log directory and list its contents.


/var/log# ls
alternatives.log btmp.1 kern.log.2.gz syslog.3.gz
alternatives.log.1 cups kern.log.3.gz syslog.4.gz
alternatives.log.2.gz dist-upgrade kern.log.4.gz syslog.5.gz
alternatives.log.3.gz dpkg.log lastlog syslog.6.gz
alternatives.log.4.gz dpkg.log.1 mail.err syslog.7.gz
alternatives.log.5.gz dpkg.log.2.gz mail.err.1 sysstat
apport.log dpkg.log.3.gz mail.err.2.gz tallylog
apport.log.1 dpkg.log.4.gz mail.err.3.gz ufw.log
apt dpkg.log.5.gz mail.err.4.gz ufw.log.1
atop faillog mail.log ufw.log.2.gz
auth.log fontconfig.log mail.log.1 ufw.log.3.gz
auth.log.1 gdm3 mail.log.2.gz ufw.log.4.gz
auth.log.2.gz gpu-manager.log mail.log.3.gz unattended-upgrades
auth.log.3.gz hp mail.log.4.gz wtmp
auth.log.4.gz installer speech-dispatcher wtmp.1
boot.log journal syslog
bootstrap.log kern.log syslog.1
btmp kern.log.1 syslog.2.gz


This is fairly large collection of logs and log directories — 69 files and directories in /var/log in this case, but 180 files when you include the files inside those directories.


$ cd /var/log
$ ls | wc -l
69
$ find . -type f -print | wc -l
180


When you examine your log files, you will see pretty clearly which are generations of the same basic log. For example, one of the primary log files — the syslog file — is broken into nine separate files. These represent what is basically a week's worth of historical data along with the current file. Most of the older files are zipped to preserve space.

$ ls -l syslog*
-rw-r----- 1 syslog adm 588728 Oct 15 20:42 syslog
-rw-r----- 1 syslog adm 511814 Oct 15 00:09 syslog.1
-rw-r----- 1 syslog adm 31205 Oct 14 00:06 syslog.2.gz
-rw-r----- 1 syslog adm 34797 Oct 13 00:06 syslog.3.gz
-rw-r----- 1 syslog adm 61107 Oct 12 00:08 syslog.4.gz
-rw-r----- 1 syslog adm 31682 Oct 11 00:06 syslog.5.gz
-rw-r----- 1 syslog adm 32004 Oct 10 00:07 syslog.6.gz
-rw-r----- 1 syslog adm 32309 Oct 9 00:05 syslog.7.gz


The syslog files contain messages from many different system services — cron, sendmail and the kernel itself are just examples. You'll also see evidence of user sessions and cron (scheduled tasks).
Most Linux systems no longer use the old messages and dmesg files that served as landing places for the bulk of our system messages for many years. Instead, a large variety of files and some special commands have become available to help present the log information that is likely to be most relevant to what you are looking for.
Depending on the file in question, you might simply use more or tail commands, or you might use a file-specific command like this use of the who command to pull user login data from the wtmp log.
$ who wtmp
shs pts/1 2018-10-05 08:42 (192.168.0.10)
shs pts/1 2018-10-08 09:41 (192.168.0.10)
shs pts/1 2018-10-11 14:00 (192.168.0.10)
shs :0 2018-10-14 19:11 (:0)
shs pts/0 2018-10-14 19:16 (192.168.0.25)
shs pts/0 2018-10-15 07:39 (192.168.0.25)
shs :0 2018-10-15 19:58 (:0)
dory pts/0 2018-10-15 20:01 (192.168.0.11)
shs pts/0 2018-10-15 20:42 (192.168.0.6)
shs pts/0 2018-10-16 07:18 (192.168.0.6)
nemo pts/1 2018-10-16 07:46 (192.168.0.14)


Similarly, you might see nothing when you run a tail faillog command, but a command like this shows you that it's simply full of zeroes:
# od -bc faillog
0000000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
\0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0076600


You may also see very little when you try to tail lastlog only to discover that you need to use the lastlog command to view that log's data.
So, here is a listing of log files in /var/log and some descriptions of what they contain and how to view their contents.

alternatives.log -- "run with" suggestions from update-alternatives
apport.log -- information on intercepted crashes
auth.log -- user logins and authentication mechanisms used
boot.log -- boot time messages
btmp -- failed login attempts
dpkg.log -- information on when packages were installed or removed
lastlog -- recent logins (use the lastlog command to view
faillog -- information on failed login
attempts -- all zeroes if none have transpired (use faillog command to view)
kern.log -- kernel log messages
mail.err -- information on errors detected by the mail server
mail.log -- information from mail server syslog -- system services log
ufw -- firewall log
wtmp -- login records

journalctl

In addition to the log files maintained in /var/log, there is also the systemd journal. While not a simple "log file" in the usual sense of a single log file, this journal represents an important collection of information on user and kernel activity. The information is retrieved from a variety of sources on the system.
To view the information that has been collected, you would use the journalctl command.
How much information you will see depends on whether you are a member of the adm group or not. Non-adm users will see relatively little information, but members of the adm group will have access to a massive amount of data — as shown in this example, which is merely showing us how many lines of information are available for this adm group member to review:

$ journalctl | wc -l
666501

That's more than 666,000 lines of text! To pare this down to a hopefully more digestible display, you're probably going to want to use arguments that tailor what you will see displayed. Some of the options available with journalctl include:
--utc (change the time format to UTC) -b (only show records added since the last boot) -b -1 (only show records added since the previous to last boot) --since and --until (only show records added within the specify timeframe, e.g., --since "2018-10-15" -until "2018-10-11 06:00"

Here's an example:

$ journalctl --since "2018-10-16 13:28"
-- Logs begin at Mon 2018-05-14 15:16:11 EDT, end at Tue 2018-10-16 13:28:57 EDT. -
Oct 16 13:28:25 vswLaptop kernel: [UFW BLOCK] IN=enp0s25 OUT= MAC=01:00:5e:00:00:01:02:
Oct 16 13:28:25 vswLaptop kernel: [UFW BLOCK] IN=enp0s25 OUT= MAC=01:00:5e:00:00:fb:00:
Oct 16 13:28:57 vswLaptop su[7784]: pam_unix(su:session): session closed for user root
Oct 16 13:28:57 vswLaptop sudo[7783]: pam_unix(sudo:session): session closed for user root lines 1-5/5 (END)

You can also examine log entries just for some particular service. This is probably one of the more useful things that the journalctl command can do for you:


$ journalctl -u networking.service
-- Logs begin at Mon 2018-05-14 15:16:11 EDT, end at Tue 2018-10-16 08:06:31 EDT
May 14 15:16:12 vsw-System76-Galaga systemd[1]: Starting Raise network interfaces.
May 14 15:16:12 vsw-System76-Galaga systemd[1]: Started Raise network interfaces.
May 14 15:49:18 vswLaptop systemd[1]: Stopping Raise network interfaces...
May 14 15:49:18 vswLaptop systemd[1]: Stopped Raise network interfaces. -- Reboot -
May 14 15:49:50 vswLaptop systemd[1]: Starting Raise network interfaces...
May 14 15:49:51 vswLaptop systemd[1]: Started Raise network interfaces. -- Reboot -


Notice how the system reboots are displayed in this output.


To get a list of services, try a command such as this:


$ service --status-all | column
[ + ] acpid [ + ] network-manager
[ - ] alsa-utils [ - ] networking
[ - ] anacron [ - ] plymouth
[ + ] apparmor [ - ] plymouth-log
[ + ] apport [ - ] pppd-dns
[ + ] atd [ + ] procps
[ + ] atop [ - ] quota
[ + ] atopacct [ - ] quotarpc
[ + ] avahi-daemon [ - ] rsync
[ - ] bluetooth [ + ] rsyslog
[ - ] console-setup.sh [ - ] saned
[ + ] cron [ + ] sendmail
[ + ] cups [ + ] speech-dispatcher
[ + ] cups-browsed [ - ] spice-vdagent
[ + ] dbus [ + ] ssh
[ - ] dns-clean [ + ] sysstat
[ + ] gdm3 [ - ] thermald
[ + ] grub-common [ + ] udev
[ - ] hwclock.sh [ + ] ufw
[ + ] irqbalance [ + ] unattended-upgrades
[ + ] kerneloops [ - ] uuidd
[ - ] keyboard-setup.sh [ + ] whoopsie
[ + ] kmod [ - ] x11-common

In the display above:
+ = active
- = inactive
? = no status option available


Here's a useful command for getting a quick report on disk space usage:
$ journalctl --disk-usage
Archived and active journals take up 824.1M in the file system.
If you want to focus on a particular process, you can do that by providing a PID (truncated) as in the example below.


$ journalctl _PID=787
-- Logs begin at Mon 2018-05-14 15:16:11 EDT, end at Tue 2018-10-16 08:25:17 EDT
Aug 03 18:02:46 vswLaptop apport[787]: * Starting automatic crash report genera
Aug 03 18:02:46 vswLaptop apport[787]: ...done.
-- Reboot -
Sep 16 13:26:34 vswLaptop atopacctd[787]: Version: 2.3.0 - 2017/03/25 09:59:59
Sep 16 13:26:34 vswLaptop atopacctd[787]: accounting to /run/pacct_source
-- Reboot -
Oct 03 18:08:41 vswLaptop apport[787]: * Starting automatic crash report genera
Oct 03 18:08:41 vswLaptop apport[787]: ...done.
-- Reboot -
Oct 15 14:07:11 vswLaptop snapd[787]: AppArmor status: apparmor is enabled but s
Oct 15 14:07:12 vswLaptop snapd[787]: AppArmor status: apparmor is enabled but s
Oct 15 14:07:12 vswLaptop snapd[787]: daemon.go:344: started snapd/2.35.2 (serie
Oct 15 14:07:12 vswLaptop snapd[787]: autorefresh.go:376: Cannot prepare auto-re

NOTE: The systemd journal's configuration file is /etc/systemd/journald.conf.