Unix Log Analysis Tricks

2017-12-20 20:36:00

I frequently have to analyze logs on Linux. Typically this is the best way to get to the root cause of a problem you are having whether you are a developer, a sysadmin, or even a regular user. In the beginning, reviewing logs can be a daunting task, but there are a few tricks that make the process much more painless. I'll share a few of the things 've picked up with you here. These are all some common Unix tools. Information is readily available on them and the man pages can tell you all about them (just type man COMMAND in the terminal), but I'll simplify it down a bit and show how I use them day to day.

A note on piping and log locations

One thing we will be using a lot is piping. This is when we place the | between two commands to redirect output and input. The syntax is command 1 | command 2. The effect of this is to redirect the standard output of command 1 to the standard input of command 2. This allows us to perform consecutive processing operations on log text to generate our final readable output.

Log files are generally located in the /var/log directory unless otherwise specified in a configuration file. Take a look in this directory and see what is available to you. As a desktop user you should at least have an Xorg log and a syslog with lots of juicy information for you to play with.

cat

If you want to see the contents of a file without opening it in an editor, you can use the cat command. The syntax is cat [filename]. Multiple file names can be specified and they will be displayed one after another.

Now this is useful, but for a large log file this doesn't cut it. Enter less.

less

If we want to scroll through a large file, we can pipe the output of cat to less with cat [filename] | less. Piping with the | character will redirect the standard output of the cat command into the standard input of the less command. Alternatively, we can just call less on the file directly e.g. less [filename]. Less will open a navigation screen. We can scroll up and down in the file using the arrow keys. You can directly go to the beginning and end of the file by entering g or G.

Less also allows you to search through the file by typing / followed by the text you want to search for. You can use n and N to jump to the next and previous instances of the search pattern.

To exit less type q.

grep

Grep allows us to search through a file for a pattern. The syntax is grep [pattern] [filename]. This will output all lines of the file that contain the given pattern specified. An alternative syntax to achieve the same thing would be cat [filename] | grep [pattern].

One useful option for grep is -v. It means filter out the given pattern. For example, if I want to show my NGINX access log without showing traffic from 127.0.0.1 I would use grep -v 127.0.0.1 /var/log/nginx/access.log.

Another useful option is -i. It means ignore case. For example, grep -i iNdEx /var/log/nginx/access.log will show me all lines that match index, INDEX, indeX, etc - case is not taken into account. An example of this might be if I want to find my graphical device descriptor from my Xorg log.


tim@localhost:/var/log$ grep -i ">device" Xorg.0.log
[ 27.439] (**) | |-->Device "nvidia"


We can also pipe the output of grep into less. If we take our example of filtering out 127.0.0.1, we can use grep -v 127.0.0.1 /var/log/nginx/access.log | less to navigate through our entire access log without 127.0.0.1 entries.

Another nifty thing we can use is regular expressions in our search. We do this using the -E option. An example would be if I was to show all 5XX range errors from my nginx access log I would use grep -E '5[0-9]{2}' /var/log/nginx/access.log. In this example [0-9] will match all numerical digits and {2} indicates that we want to match two digits in a row.

tail

Tail will show the end of the a text file. The syntax is tail [filename]. By default it shows the last 10 lines. This can be changed with -n. For example, tail -n 30 /var/log/syslog will show the last 30 lines of the system log.

One option I use all the time with tail is -f. This stands for follow. This will output the end of the log as it updates in real time. This is especially useful for debugging. Let's say that I have a web application and want to observe 5XX range errors in real time while I interact with my application. I might use the following: tail -f /var/log/nginx/access.log | grep -E '5[0-9]{2}'.

head

Head is like tail by for the beginning of the file. I use it much less than I use tail as it doesn't make much sense to follow the head, but it is still quite useful.

In Conclusion

These are a few of the tricks I use to look through logs. There is no limit to how many pipes you can use and a lot of the power to this strategy lies in piping the output of these commands into each other. Both grep and searching within less allow for the usage of regular expressions to add variability to your searches.

Often when people are new to troubleshooting and debugging using logs, they can be a bit overwhelming. Hopefully this helps to make them a bit more manageable.