System Logs & Monitoring
System logs are records of system activities (including dates, processes, and messages). Log files provide the essential paper trail needed for system troubleshooting, security auditing, and automated analysis.
Logging Services
A logging service is a daemon that tracks logs or errors. Log messages are stored in separate files in the /var/log directory. The main log file is often /var/log/messages or /var/log/syslog.
Automating Log Analysis
During maintenance sessions, instead of manually parsing large files, you can automate log analysis by writing Perl or Bash scripts. For example, write a script to parse a mail log and notify you of rejected messages, and execute it via a crontab entry.
Automatic Rotation
A system of regular rotation of logs to maintain a minimum log file size using the logrotate utility.
- When executed, logrotate adds a
.1to the end of the current log file. - Previously rotated files become
.2,.3, etc. - Can be scheduled daily, weekly, or monthly.
- Why? Saves disk space! Older log files are pushed out when a limit is reached, preventing the hard drive from running out of free space and crashing the computer.
The Central Network Log Server
A server used to implement centralized logging services across a network. It receives syslog messages from Linux/Windows servers and network devices (routers, switches, firewalls).
- Performs log analysis, data mining, and generates alerts/reports.
- Ensures logs are safe and available even if the originating computer or disk crashes.
MRTG
Multi Router Traffic Grapher (MRTG) is free software licensed under GPL. It is used to monitor and measure the traffic load on network links, representing the data in graphical form.
The syslogd Utility
Tracks remote and local system logs. Settings are configured in /etc/syslog.conf. Syntax: syslogd [options] (Install via sudo apt install inetutils-syslogd)
| Option | Used To |
|---|---|
| -d | Turn on debug mode. |
| -f {file} | Specify a new config file instead of syslog.conf. |
| -m {int} | Time interval between timestamp marks in the log. |
| -r | Enable receiving messages from a network. |
logger
The logger command is the command interface to the system log module. It has options that allow you to customize the content that needs to be logged directly from the terminal or shell scripts.
The rsyslog Utility
A more modern alternative to syslogd. Tracks, forwards, and stores messages via the syslog protocol. Configured in /etc/rsyslog.conf. Syntax: rsyslog [options]
| Option | Used To |
|---|---|
| -d | Turn on debug mode. |
| -f {file} | Specify a new config file instead of default. |
| -N {level} | Check configuration files for validity (level 1+ controls verbosity). |
The journalctl Utility
A component of systemd that manages and views log files created by the systemd-journald service. Configured in /etc/systemd/journald.conf. Syntax: journalctl [options]
| Option | Used To |
|---|---|
| -n {lines} | Specify number of lines to display. |
| -o {format} | Format output (short, verbose, export). |
| -f | Follow logs (like tail -f). |
| -p {priority} | Filter by priority (alert, err, warning, info, etc.). |
| -b | Show log messages from the current boot only. |
/var/log/journal/
By default, the Systemd Journal only stores logs in memory, cleared on reboot. To persist logs, simply create the /var/log/journal/ directory. Systemd will automatically store logs there if the directory exists.
Common Log Files
By default, syslogd stores most messages in /var/log/messages. The location is controlled by /etc/syslog.conf. Some specific logs include:
| /var/log/syslog | General system info. |
| /var/log/maillog | Mail messages. |
| /var/log/samba | Samba (SMB) messages. |
| /var/log/mrtg | MRTG traffic graphs. |
| /var/log/httpd | Apache web server messages. |
The lastlog Command
Utilizes data from /var/log/lastlog to display the latest login details of all users.
- Displays login name, date, time, and terminal.
- Highly useful for administrators to view user accounts that have never been used, identifying potential security risks or orphaned accounts.
The grep Command
Searches a file for a string and prints matching lines. Syntax: grep [options] {keyword} {file}
| -h | Print matching lines without file names. |
| -w | Restrict search to whole words only. |
| -c | Display a count of matching lines instead of lines. |
| -i | Ignore case. |
| -l | List file names containing matches. |
| -n | Precede each line with its line number. |
| -s | Suppress error messages. |
| -e | Specify one or more search patterns. |
fgrep and egrep
fgrep searches for multiple text patterns but does not use regular expressions. egrep uses an extended set of regular expression elements.
The tail Command
Retrieves data from a file, displaying the last 10 lines by default. Syntax: tail [options] {file}
| --retry | Force tail to open a file that cannot be opened. |
| -n {num} | Print specified number of lines from the end. |
| -c {bytes} | Print specified number of bytes from the end. |
| -f | "Follow" — update the output automatically if changes are made to the file. |