Rotating logs w/ Monitor Witness Script

Last month I released the Hive witness tool I created to keep an eye on my witnesses and make sure they are still in sync even if they are not the primary witness. You can read about and download it here.

One of the requirements to use my tool is to have logging enabled for your witness node. I highly recommend you do this reguardless as it is the only way to really troubleshoot problems that may arise. These log files can get big quickly at around 20 Mb or more growth daily.

Fortunately, there is a native tool for handling log files on Linux that makes this really easy. Logrotate will automatically rotate log files by breaking up your logs by whatever criteria and compressing older log files.

When logging from Hived (typically through screen) you have one log file that just keeps growing until you run out of disk space. Even if you already know how to use logrotate, you might run into an issue with hived where it no longer writes to the file name specified and this will trip up Montior Witness causing false alerts.

This happens as log rotate will move and rename log files automatically but some processes will continue to use the same file handle and if you were writing to witness.log, you will end up writing to witness.log.1 causing Monitor Witness to fail because it cannot detect the last block processed.

You can just cut/paste my logrotate config that handles this situation without the false alerts. The big change to a standard logrotate config is I use copytruncate rather than create. This will introduce a very brief split second gap in the logging while it makes the cut over, in most cases you will never notice it, at worse case one line might be missing.

To configure this, all you need to do is create the following file: /etc/logrotate.d/witness as root, and enter in the following:

 /home/ubuntu/witness.log { 
    su ubuntu ubuntu         
    daily
    copytruncate
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
}

You will need to change the first line to be the file path to your log file and the second line to be your username and group you want for the permissions (so it isn't owned by root). This configuration will rotate your log file daily, compress them, and keep 30 daily logs. It of course will use copytruncate instead of create which is important to avoid false alarms.

There is no service to restart and it will take place immediately upon the next time it would usually do a log rotate. If you check back in a few days you should see something along these lines.

If you haven't checked out Monitor Witness, I highly recommend running it on your witness node(s), especially if you have at least one backup node. Without it, you will never know if your backup nodes are in sync prior to switching over to them and finding out it fails.

Considering a witness node can take days to resync, knowing when a node is out of sync before it is used to sign blocks is a huge help.

If you are having trouble with any of this, just leave a comment below or hit me up on Discord.