Network Monitoring

131 CHAPTER 6 Network Monitoring Knowing When It Goes Wrong Without Watching It A s an administrator, it is your responsibility to know when things are about to go wrong. You can, of course, go sit by your server all day and figure out if everything is going all right, but you probably have better things to do. Nagios offers services to monitor the network for you. In this chapter you’ll learn how to install and use Nagios. Starting with Nagios Nagios is a network- wide monitoring tool. In this chapter you’ll learn how to set it up on your servers. Once it is set up, you can watch the status of servers in your network via a web browser. Don’t want to watch a web browser all time? That’s fine, because you can configure Nagios to send relevant security alerts to some specified users on the network if something goes wrong. Nagios allows you to monitor local server events, such as running out of disk space, as network events. Before you install Nagios, make sure that you have a web server configured (you can read more about configuring Apache Web Server in Chapter 11 of my book Beginning Ubuntu Server Administration, from Apress) and running. Nagios uses a web interface to show its information, so you can’t do without that. Once you have confirmed it is up and running, install the j]ceko packages: ]lp)capejop]hhj]ceko.j]ceko)lhqcejoj]ceko)ei]cao This command installs about 40 MB of data on your server. Once that is done, you have to complete the installation by setting up authentication. Nagios uses the file CHAPTER 6 N NETWORK MONITORING 132 +ap_+j]ceko.+dpl]oos`*qoano , but this file is not created automatically. The following command creates it for you, puts a user with the name j]ceko]`iej in it, and prompts for a password: dpl]oos`)_+ap_+j]ceko.+dpl]oos`*qoanoj]ceko]`iej There are two configuration files related to user authentication. First, +ap_+ j]ceko.+]l]_da.*_kjb contains all settings that allow Nagios to communicate with Apache. Listing 6-1 shows its contents. Listing 6-1. /etc/nagios2/apache2.conf Sets Up Communication Between Nagios and Apache nkkp<iah6+ap_+j]ceko._]p]l]_da.*_kjb O_nelp=he]o+_ce)^ej+j]ceko.+qon+he^+_ce)^ej+j]ceko. O_nelp=he]o+j]ceko.+_ce)^ej+qon+he^+_ce)^ej+j]ceko. =he]o+j]ceko.+opuhaodaapo+ap_+j]ceko.+opuhaodaapo =he]o+j]ceko.+qon+od]na+j]ceko.+dp`k_o 8@ena_pknuI]p_d$+qon+od]na+j]ceko.+dp`k_ox+qon+he^+_ce)^ej+j]ceko.%: KlpekjoBkhhksOuiHejgo @ena_pknuEj`atej`at*dpih =hhksKranne`a=qpd?kjbec Kn`an=hhks(@aju =hhksBnki=hh =qpdJ]iaJ]ceko=__aoo =qpdPula>]oe_ =qpdQoanBeha+ap_+j]ceko.+dpl]oos`*qoano namqenar]he`)qoan 8+@ena_pknuI]p_d: As you can see, the ]l]_da.*_kjb file contains the authentication settings and some basic paths that Nagios has to use. The other relevant configuration file is +ap_+j]ceko.+ _ce*_bc , which contains the name of the admin user that is used for different purposes, as well as other settings that are related to the CGI scripts that Nagios uses. The interesting part of this script is that you can change admin names in it. By default, j]ceko]`iej is the only user who has administrative permissions to perform different tasks. If, for instance, you want to use another user account for hosts and services- related commands, change it in the _ce*_bc file. Listing 6-2 shows its contents. CHAPTER 6 N NETWORK MONITORING 133 N Note For better readability, I have removed all comment lines. Consult the configuration file on disk to see the comment lines as well. Listing 6-2. cgi.cfg Contains the Authorizations of the admin User j]ceko[_da_g[_kii]j`9+qon+he^+j]ceko+lhqcejo+_da_g[j]cekoX +r]n+_]_da+j]ceko.+op]pqo*`]p1#+qon+o^ej+j]ceko.# qoa[]qpdajpe_]pekj9- ]qpdkneva`[bkn[ouopai[ejbkni]pekj9j]ceko]`iej ]qpdkneva`[bkn[_kjbecqn]pekj[ejbkni]pekj9j]ceko]`iej ]qpdkneva`[bkn[ouopai[_kii]j`o9j]ceko]`iej ]qpdkneva`[bkn[]hh[oanre_ao9j]ceko]`iej ]qpdkneva`[bkn[]hh[dkopo9j]ceko]`iej ]qpdkneva`[bkn[]hh[oanre_a[_kii]j`o9j]ceko]`iej ]qpdkneva`[bkn[]hh[dkop[_kii]j`o9j]ceko]`iej `ab]qhp[op]pqoi]l[h]ukqp91 `ab]qhp[op]pqosnh[h]ukqp90 lejc[oujp]t9+^ej+lejc)j)Q)_1 DKOP=@@NAOO nabnaod[n]pa95, At this point, you have a very basic Nagios server up and running. Before you start to configure it, you need to find out if it works properly. From a workstation, start your browser and connect to the following URL: dppl6++ukqn[j]ceko[oanran+j]ceko. This should give you a login prompt at which you can enter the name and password of the admin user you have just created. After entering these, you should see the Nagios web interface, as shown in Figure 6-1. Don’t bother clicking around in it, because you haven’t set up anything yet. Therefore, you won’t see much for the moment. Read the fol- lowing sections to find out how to configure Nagios. CHAPTER 6 N NETWORK MONITORING 134 Figure 6-1. After installing Nagios, connect to it to see if it works. N Note The Nagios web interface gives access to some documentation that is installed on your server as well. You can use this documentation, but be aware that the paths on Ubuntu Server are different from the pathnames referred to in the documentation. CHAPTER 6 N NETWORK MONITORING 135 Configuring Nagios Nagios uses lots of configuration files. The most difficult part of managing Nagios is to find the right configuration file for a specific purpose. To make it even more difficult, Nagios distinguishes between core configuration files and plug- in configuration files, add- on files that can be used as an extension to the default functionality of Nagios. Location of the Configuration Files When you first start working with Nagios, it looks like configuration files are located just about everywhere! To help you pinpoint the locations of these files, the following list identifies the most common directories in which Nagios stores information: s +ap_+j]ceko. : This is the master configuration directory. It contains the most important configuration files, among which you will find the j]ceko*_bc config- uration file. s +qon+he^+j]ceko+lhqcejo : As mentioned, Nagios works with plug- ins. Every plug- in allows you to monitor an additional service. For example, Nagios by itself doesn’t know how to monitor Oracle. If, however, the Oracle plug- in has been installed in this directory (which is the case after a default installation), the plug- in can man- age Oracle. s +ap_+j]ceko.+_kjb*` : This directory contains some of the most important Nag- ios configuration files. If the file you are looking for is not in here, also check +ap_+j]ceko)lhqcejo+_kjb ig. s +ap_+j]ceko)lhqcejo+_kjbec : This directory contains the configuration files for the plug- ins that are installed on your server. s +r]n+he^+j]ceko. : Nagios writes its output to this directory. When Nagios has been up and running for some time, you’ll find *kqp files in this directory. These files contain the information that is used by the Nagios web interface. s +r]n+hkc+j]ceko. : This is the directory where Nagios writes its log files. Use it if anything goes wrong with your Nagios environment. Before diving deep into the different configuration files, you should also be aware of the +ap_+j]ceko.+_kii]j`o*_bc file. To do its work, Nagios uses its own command set. The _kii]j`o*_bc file defines the most important commands. Listing 6-3 gives a partial example. CHAPTER 6 N NETWORK MONITORING 136 Listing 6-3. /etc/nagios2/commands.cfg Defines the Most Common Nagios Commands nkkp<iah6+ap_+j]ceko._]p_kii]j`o*_bc #lnk_aoo)dkop)lanb`]p]#_kii]j``abejepekj `abeja_kii]j`w _kii]j`[j]ialnk_aoo)dkop)lanb`]p] _kii]j`[heja+qon+^ej+lnejpb!^ H=OPDKOP?DA?G Xp DKOPJ=IA ± Xp DKOPOP=PA Xp DKOP=PPAILP Xp DKOPOP=PAPULA Xp ± DKOPATA?QPEKJPEIA Xp DKOPKQPLQP Xp DKOPLANB@=P= Xj:: ± +r]n+he^+j]ceko.+dkop)lanb`]p]*kqp y #lnk_aoo)oanre_a)lanb`]p]#_kii]j``abejepekj `abeja_kii]j`w _kii]j`[j]ialnk_aoo)oanre_a)lanb`]p] _kii]j`[heja+qon+^ej+lnejpb!^ H=OPOANRE?A?DA?G Xp DKOPJ=IA ± Xp OANRE?A@AO? Xp OANRE?AOP=PA Xp OANRE?A=PPAILP Xp ± OANRE?AOP=PAPULA Xp OANRE?AATA?QPEKJPEIA Xp OANRE?AH=PAJ?U ± Xp OANRE?AKQPLQP Xp OANRE?ALANB@=P= Xj::+r]n+he^+j]ceko.+oanre_a)lanb`]p]*kqp y Nagios commands are well structured. If you feel you are missing any functionality in the default Nagios command set, you can create your own Nagios commands as well. The _kii]j`o*_bc file contains some hints on how to do that. The Master Configuration File: nagios.cfg The master configuration file that Nagios uses is +ap_+j]ceko.+j]ceko*_bc . This file determines where Nagios should read and write specific information. By using _bc[beha statements, it also tells Nagios what additional configuration files to read. For example, these statements can refer to configuration files for specific modules that you want to use. By default, all of these configuration files are disabled, which means that Nagios basically monitors nothing. Of course, it makes sense to enable them, but only after you have modified the configuration file according to your needs. Listing 6-4 shows the part of j]ceko*_bc that indicates what configuration files to use. Be aware, though, that these are only example files, and in some cases refer to files that don’t even exist at the location that is indicated. CHAPTER 6 N NETWORK MONITORING 137 Listing 6-4. From nagios.cfg, Additional Configuration Files Are Included ?kii]j``abejepekjo _bc[beha9+ap_+j]ceko.+_kii]j`o*_bc Pdaoakpdanat]ilhao]nap]gajbnkiqlopna]i#oo]ilha_kjbecqn]pekj behao* Ukq_]jolhepkpdanpulaokbk^fa_p`abejepekjo]_nkoooaran]h _kjbecbehaoebukqseod$]o`kjadana%(kngaalpdai]hhej] oejcha_kjbecbeha* _bc[beha9+ap_+j]ceko.+_kjp]_pcnkqlo*_bc _bc[beha9+ap_+j]ceko.+_kjp]_po*_bc _bc[beha9+ap_+j]ceko.+`alaj`aj_eao*_bc _bc[beha9+ap_+j]ceko.+ao_]h]pekjo*_bc _bc[beha9+ap_+j]ceko.+dkopcnkqlo*_bc _bc[beha9+ap_+j]ceko.+dkopo*_bc _bc[beha9+ap_+j]ceko.+oanre_ao*_bc _bc[beha9+ap_+j]ceko.+peialanek`o*_bc Atpaj`a`dkop+oanre_aejbk`abejepekjo]najksopkna`]hkjcsepd kpdank^fa_p`abejepekjo6 _bc[beha9+ap_+j]ceko.+dkopatpejbk*_bc _bc[beha9+ap_+j]ceko.+oanre_aatpejbk*_bc Ukq_]j]hokpahhJ]cekopklnk_aoo]hh_kjbecbehao$sepd]*_bc atpajoekj%ej]l]npe_qh]n`ena_pknu^uqoejcpda_bc[`en `ena_pera]oodksj^ahks6 _bc[`en9+ap_+j]ceko.+oanrano _bc[`en9+ap_+j]ceko.+lnejpano _bc[`en9+ap_+j]ceko.+osep_dao _bc[`en9+ap_+j]ceko.+nkqpano As a Nagios administrator, it is also useful if you know about the other important lines in the j]ceko*_bc file. The following list provides an overview of the most important definitions it contains: CHAPTER 6 N NETWORK MONITORING 138 s hkc[beha9+r]n+hkc+j]ceko.+j]ceko*hkc : This parameter tells Nagios where to log its information. s _bc[`en9+ap_+j]ceko.+_kjb*` : This line tells Nagios to include all configuration files in the specified directory. s _bc[beha9+ap_+j]ceko.+_kii]j`o*_bc : This line tells Nagios to load the configura- tion file _kii]j`o*_bc as well. Likewise, other _bc[beha lines are used to refer to additional configuration files that Nagios should include. s op]pqo[beha9+r]n+_]_da+j]ceko.+op]pqo*`]p : This file contains current status infor- mation about all hosts and services that are monitored. The CGI scripts from the Nagios web server interpret this file and display its contents in a graphical way. s _da_g[atpanj]h[_kii]j`9, : This default line makes sure that no external commands can be executed. If you want to manage Nagios using a web server (which should always be the case), you need to enable this option by giving it the value 1. s hkc[nkp]pekj[iapdk`9` : This line specifies in what way the Nagios log file should be rotated. By default, this will happen daily. Valid values for this parameter follow: s j : Don’t rotate the log s d : Rotate hourly s ` : Rotate daily s s : Rotate weekly s i : Rotate monthly s hkc[]n_dera[l]pd9+r]n+hkc+j]ceko.+]n_derao : If log rotation is enabled, this param- eter describes where the archive of log files should be written to. Creating Essential Nagios Configuration Files Nagios needs some minimal configuration files, and they should reside in one of the directories defined in the j]ceko*_bc file using the _bc[`en directive. The default location to put them would be +ap_+j]ceko.+_kjb*` . Make sure that you create at least the follow- ing configuration files: s _kjp]_po*_bc : This file defines which people should get a message in case of trouble. s _kjp]_pcnkqlo*_bc : All contacts specified in _kjp]_po*_bc should be a member of at least one contact group. Use this file to define the contact group. CHAPTER 6 N NETWORK MONITORING 139 s pailh]pao*_bc : This file defines templates that can be used by other configuration files. s dkopo*_bc : Use this file to define the hosts that Nagios will monitor. s dkopcnkqlo*_bc : In large networks, it is useful to subdivide hosts into host groups, such as servers, switches, routers, and so on. s oanre_ao*_bc : The file defines specific services that you want to monitor for each host. s peialanek`o*_bc : This file defines time periods used in all configuration files. Now it is time to start the real work, which unfortunately involves a lot of typing. In the rest of this chapter, we will work on a small example network in which four Linux servers are used. Three of these are on the internal network, and one of them is on the Internet. Nagios can monitor other operating systems as well, but let’s try to set up Linux- based host monitoring first. The following servers are monitored: s -5.*-24*-*55 : DHCP, NFS, web, Nagios, SSH s -5.*-24*-*-,, : Samba, SSH s -5.*-24*-*-,- : Web, FTP, SSH s 4,*25*5/*.-2 : Web, SSH Creating a Contacts File Start with the creation of the _kjp]_po*_bc file. As specified in +ap_+j]ceko.+j]ceko*_bc , this file should reside in +ap_+j]ceko. , so make sure to create it there. Listing 6-5 gives an example of what this file may look like. Listing 6-5. Example contacts.cfg File _kjp]_p`abejepekjbknhej`] `abeja_kjp]_pw _kjp]_p[j]iahej`] ]he]ohej`]pdkioaj oanre_a[jkpebe_]pekj[lanek`skngdkqno dkop[jkpebe_]pekj[lanek`skngdkqno oanre_a[jkpebe_]pekj[klpekjo_(n dkop[jkpebe_]pekj[klpekjo`(n CHAPTER 6 N NETWORK MONITORING 140 oanre_a[jkpebe_]pekj[_kii]j`ojkpebu)^u)ai]eh dkop[jkpebe_]pekj[_kii]j`odkop)jkpebu)^u[ai]eh ai]ehhej`]<hk_]hdkop y The interesting part of this configuration file is that there are quite a few cross- references. That is, the _kjp]_po*_bc file depends on what you do in other configuration files. For instance, the lines oanre_a[jkpebe_]pekj[lanek` and dkop[jkpebe_]pekj[lanek` are periods that you will define later in the peialanek`o*_bc file. In the example _kjp]_po*_bc file in Listing 6-6, you also see that some oanre_a[jkpebe_]pekj[klpekjo and dkop[jkpebe_]pekj[klpekjo parameters are used. The following oanre_a[jkpebe_]pekj[klpekjo parameters can be used: s j : Do not notify at all s s : Notify on WARNING states s q : Notify on UNKNOWN states s _ : Notify on CRITICAL states s n : Notify when the service recovers and returns to OK state Likewise, the following dkop[jkpebe_]pekj[klpekjo parameters can be used: s j : Do not notify at all s ` : Notify on DOWN host states s q : Notify if host is unreachable s n : Notify when host recovers Defining a Contacts Group After defining the contacts file, you may want to create a contact group as well. This makes it easier in large implementations to address all contacts at once. Listing 6-6 shows what a contact group may look like. [...]... option, which gives you a graphical representation of the host in the Nagios monitoring network Figure 6-4 The individual host view allows you to see exactly what is happening on a host CHAPTER 6 NETWORK MONITORING The last options from the Monitoring section that I want to cover here are Service Problems, Host Problems, and Network Outages Each of these gives you an overview of current problems, sorted... for the plug-ins you want to use before you configure the file Listing 6-9 Nagios Plug-ins Often Have Lots of Options to Define What You Want Them to Do CHAPTER 6 NETWORK MONITORING 145 146 C HAPTER 6 NET W OR K MONITOR ING CHAPTER 6 NETWORK MONITORING After you have familiarized yourself with the possibilities the plug-ins have to offer, you can start creating the services file Listing 6-10 shows a... the report displayed in the browser window Summary In this chapter you have learned how to set up a Nagios server for network monitoring You have learned how Nagios helps you to monitor critical parameters, including both parameters on the individual host that you are monitoring and network parameters You have also learned how to enable NRPE, which allows you to monitor parameters on a remote host,...CHAPTER 6 NETWORK MONITORING Listing 6-6 Example of a Contact Group Defining Hosts and Host Groups After defining whom to contact if things go wrong, you have to define hosts and, if so required, hostnames The hosts... three parameters, separated by exclamation marks If you look at the help output of lp, you see the result shown in Listing 6-11 Listing 6-11 Partial Output of check_disk Usage Information CHAPTER 6 NETWORK MONITORING As you can see, this command has three options that are required: , , and or generates a warning if less than the specified percentage of disk space is free, generates a critical event... required Listing 6-12 gives an example of its contents Listing 6-12 Using timeperiods_nagios2.cfg to Define During What Times of Day Nagios Has to Act 149 150 C HAPTER 6 NET W OR K MONITOR ING CHAPTER 6 NETWORK MONITORING Restarting Nagios with Your Configuration Your configuration files should now be in place, which means it is time to restart Nagios To do this, use the command If you have applied changes... click Go to continue In the File Download dialog box that asks you whether you want to save the archive to disk, click Save Store it somewhere on disk; the directory is a reasonable choice CHAPTER 6 NETWORK MONITORING 3 Assuming that you have just downloaded the package to the directo go to that directory, and then use the command tory, use to extract it This creates a subdirectory with the name n> 4... output the version of the NRPE process on the remote server If that worked, next add the command to the file This file should contain a section that defines that looks like the following: CHAPTER 6 NETWORK MONITORING Now that you have defined the command, it’s time to add a section to the file In this section you will use the generic command, with the particular command you want to execute on the remote... interface In this section, you’ll get a quick tour of the web interface The first window that you should check out is the Tactical Monitoring Overview, shown in Figure 6-2 Especially when managing many nodes with Nagios, this view gives the fastest insight into what is happening on your network The right part of the window shows you a performance summary, and the bottom part of the window shows you which hosts... HAPTER 6 NET W OR K MONITOR ING Figure 6-2 The Tactical Monitor Overview window shows you in one view what critical events have happened on your network recently If you observe that something may be wrong, you’ll want to get more details about what is happening The Monitoring section of Nagios offers different ways of displaying this information In fact, all options look at the same information, but filter . to monitor the network for you. In this chapter you’ll learn how to install and use Nagios. Starting with Nagios Nagios is a network- wide monitoring tool 131 CHAPTER 6 Network Monitoring Knowing When It Goes Wrong Without Watching It A s an administrator,