If you're like me, it probably seems obvious that command logging is an important part of system administration. Any change I make, whether it's on a router or server, is logged somewhere. Although folks give plenty of reasons why they don't use tools like sudo for super-user functions (inconvenience and potential embarrassment seem to be the leading causes), none of those reasons outweigh the benefits of command logging.
- Ability to quickly undo changes
- Crash debugging
- Reactive education of junior administrators
<!--more-->
Easily Undo Changes
I recently stumbled across a white paper advocating version control for software developers. The very first point the author makes is that version control gives developers the ability to undo any changes they've made. This is exactly why command logging is so vital to system administration.
As a system administrator, when you receive a report of a problem on a server or network, your first thought should be "What has changed?". If your logs don't include a timestamped list of commands run by your fellow administrators, you'll have to begin the debugging process with only the information included in the problem report.
Debug Crashes
When a system crashes, there will hopefully be some evidence in the logs explaining what caused the crash. Often, whatever caused the crash, also kept the system from writing to its logging facility. If you go back and look through the command log, you might find that one of the administrators made a change shortly before the crash. For example, when trying to increase the number of file descriptors on a Linux server, it's possible to accidentally decrease the available file descriptors with a typo. Without that command log, a mistake like that might take a long time to debug.
Educate Junior Administrators
One aspect of a senior-level administration role is to mentor junior administrators and guide their development. In addition, administrators that are new to the team might not fully understand the standards and procedures the team employs. With this in mind, senior-level team members should regularly review the command logs and make sure that any discrepancies or learning experiences are pointed out. This practice should not be approached as negative feedback, but instead used to build a positive environment for young administrators.
