The mount options check

The mount options check in Check_MK was developed first and foremost to detect broken ext3 filesystems by monitoring if those were in "read-only" mode.

The uses:

  • Ext3/Ext4 filesystems switch to read-only if the "remount-ro" mount option is set, which is default on some Linux distros. If the check alerts you, you can react by trying to stop the applications on the server, check for consistency  errors, etc. and fix the issue.
  • It also handles other mount options though and can be very helpful, for example if you use a readonly-root filesystem like me you can be alerted if you forgot to mount -o remount,ro / after maintenance.
  • If you make use of extended attributes for modern filesystems like Ceph is also imporant to monitor that user_xattr is enabled on the backing stores.
  • using checks = [] you can create an enforcing rule that overrides inventory and just requires mount options to be "this, that, and those"

 

The check can do all of that and it does it mostly automatically.

 

The downsides:

  • The check cannot monitor any options that you imprinted on the FS using tune2fs - it only sees those that are explicitly listed in /proc/mounts.
  • you might get false alerts if you do changes to the system, like upgrading to a newer kernel version that enables inode64 on XFS
  • the check does not fix those on re-inventory (even with a full one)
  • It has a bug that prints "OK" but returns "WARNING"

 

 

 

 

So, how do you deal with such issues?

 

Locate the "autochecks" file for the server - this file has all the info that was gathered on the system at inventory time.

You could just delete it, I usually update it to have the right content:

Before adjusting mount options:

$ grep mount $OMD_ROOT/var/check_mk/autochecks/vfile.mk 
  ('vfile', 'mounts', '/', ['data=ordered', 'relatime', 'rw']),
  ('vfile', 'mounts', '/boot', ['data=ordered', 'relatime', 'rw']),
  ('vfile', 'mounts', '/home', ['nodev', 'noexec', 'noquota', 'nosuid', 'relatime', 'rw']),
  ('vfile', 'mounts', '/home/wwwroot', ['nodev', 'noexec', 'noquota', 'nosuid', 'relatime', 'rw']),

 

 

After adjusting mount options:

$ grep mount $OMD_ROOT/var/check_mk/autochecks/vfile.mk 
  ('vfile', 'mounts', '/', ['data=ordered', 'relatime', 'rw']),
  ('vfile', 'mounts', '/boot', ['data=ordered', 'relatime', 'rw']),
  ('vfile', 'mounts', '/home', ['nodev', 'noexec', 'noquota', 'nosuid', 'inode64', 'relatime', 'rw']),
  ('vfile', 'mounts', '/home/wwwroot', ['nodev', 'noexec', 'noquota', 'nosuid', 'inode64', 'relatime', 'rw']),

The order doesn't matter, this is just a python "list" and each mount options is verified separately.

 

Do I have to hand-edit all that? Are you kidding me?

No, not really (smile)

If you use vi you can bulk edit the lines in question.

use :set nu to enable line numbers

And then give a range of lines to edit using sed. In the sed line, do something that replaces one existing mount option with the existing one and our new one:

 

Make it show:

After that you just need to rewrite your config and restart Nagios using

cmk -R 

if someone runs cmk -I or similar at a situation like that, hit them on the back of their head (smile)

Once the restart is done, wait for the next update after checking the host in question and your "recent events" should show everything as OK now:

 

 

Online documentation

PLEASE also keep in mind that all the checks have man pages and those are also visible online!

The man page for the "mounts" check is at http://mathias-kettner.de/checkmk_check_mounts.html