LOG CLEANERS -- Make your Apache and Sendmail logs more readable

LogCleaners -- Make your Apache and Sendmail logs more useful

These are so simple that they are almost not worth listing. I figure someone might be able to make use of them but, thruthfully, a real programmer could do this in their sleep.

I got tired of trying to view log files from Apache and having them wrap on my screen. Unless I was using 4-point font on my 21" monitor, it was a pain to quickly scan the logs for interesting stuff. It was also littered with stuff that I didn't care about. A PERFECT application for Perl.

All this code does is parse the log files and clean them up a bit. It outputs this cleaned version to another file. I run these hourly via a cron job. You can have this output file go to your webserver directory so you can see your logs from anywhere, without the need for an SSH client.

NOTE: There is a significant security risk with placing your log files in a publicly accessible web page. I suggest you either:

Don't do it. Have it dump the output log to a privileged directory.
Put it on an obscure URL that nobody can stumble across. Also, make a .htaccess file containing "Options -Indexes" for that directory.
Password protect the page.

For the Watching Eye

I've added a version that works for watching logs "live" also. This is nice for admins who like to keep a terminal window open while they are working. Tailing /var/log/secure would be a good one also.

The live versions follow right after the post-process versions and are almost devoid of documentation.

What Do They Do?

Not a whole heck of a lot.

Ideally I guess the substitutions should be loaded into a hash but for such a small task, who cares.

The Apache Log Cleaner:

Makes some simple substitutions for things like IP addresses. I got tired of trying to remember my dynamic IP address or my friend's work IP.
Exlcudes lines which contain "Don't care" items.
Truncates lines once they get to the User Agent. Certainly some people have reason to care about this. I seldom do.
Truncates at a fixed length. Gets rid of those annoying wraps on my screen.

The Sendmail Log Cleaner:

Makes some simple substitutions for things like IP addresses. I got tired of trying to remember my dynamic IP address or my friend's work IP. This usually shows up for IMAP logins and the like.
Exlcudes lines which contain "Don't care" items.
Makes generic substitutions for maillog entries of sendmail, IMAP, POP, SpamAssassin.
Eliminates some of the detail hash that Sendmail logs. Probably changing a "log level" would reduce this but its nice to have if you ever need it -- I just don't need to see it every day.
Truncates at a fixed length. Gets rid of those annoying wraps on my screen. (Note: I have mine set fairly wide because SpamAssassin 3.0 now reports hits in the maillog. You can adjust as required.)

The Apache Log Cleaner Code

#! /usr/bin/perl -w
#
# logcleaner.pl
#
# See http://www.pettingers.org/code/logcleaner.html for notes and details
#
use strict;
$| = 1;
# Enter things in the exclusion string below that you do not want
# to see.  Note that if these things are contained anywhere in
# the log line, the ENTIRE line is deleted from the output.
#
our $excl = "stinky\.cgi|images\/|html_report|WebDAV"; 
#
our $login1 = "/var/log/httpd/access_log";
my  $logonesize = '300'; # Number of lines from log one to clean
our $login2 = "/var/log/httpd/access_log-sitetwo_log";
my  $logtwosize = '100'; # Number of lines from log two to clean
our $login3 = "/var/log/httpd/access_log-sitethree_log";
my  $logthreesize = '200'; # Number of lines from log three to clean

our $logoutput = "/var/log/httpd/weblogout.txt";

our $maxwidth = 156; # Set for your screen size/resolution/font

##########################################################


   print "Processing your log...\n\n";
   open (USEROUT, ">$logoutput")|| die ("Could not open file. \n $!");
   print USEROUT "\n\n================  First Site =================\n\n";
   open (USERIN, $login1)|| die ("Could not open file. \n $!");
   print USEROUT Cleaner($logonesize);
   close (USERIN);
#
#### Delete the following eight lines if you only have one log ####
   print USEROUT "\n\n================= Second Site =================\n\n";
   open (USERIN, $login2)|| die ("Could not open file. \n $!");
   print USEROUT Cleaner($logtwosize);
   close (USERIN);

   print USEROUT "\n\n================= Third Site =================\n\n";
   open (USERIN, $login3)|| die ("Could not open file. \n $!");
   print USEROUT Cleaner($logthreesize);
   close (USERIN);

####
   close (USEROUT);
exit;

##################################################################

sub Cleaner
{
my ($text, $pos) = '';
my $count = $_[0];
my @build = ();
my @final = ();

$text = <USERIN>;

while ($text) {
   if ($text !~ m/$excl/i){

# Enter things to substitute in the form of s/string-to-replace/replacement-string/

        $text =~ s/18\.72\.0\.3 /      WORK      /;
        $text =~ s/24\.20\.100\.240 /    BUBBA\@HOME  /;
        $text =~ s/4\.3\.170\.200 /    MICHAEL     /;
#
        unless ($text =~ m/google|msn\.com|yahoo|bot/) {
             chop($text);
             $text = substr($text, 0, $maxwidth);
             $text = substr($text, 0, (index($text,"\"Mozi"))) . "\n";
        }
        push (@build, $text); # Build of a reduced copy of the log
   }
   $text = <USERIN>;
}

$pos = @build; # Count the number of lines 
if ($pos > $count) {
        # work from the end and build up the final line count
        while ($count > 0) {
                $pos --;
                $count --;
                unshift (@final, $build[$pos]);
        } # End while
} # End if
else {@final = @build;}

return @final;

} # End sub Cleaner

#################################################################

The LIVE Version for Apache Logs

#!/usr/bin/perl -w
##############################################################################
#
# See http://www.pettingers.org/code/logcleaner.html for notes and details
#
use strict;
use Socket;
my($LOG) = '/var/log/httpd/access_log';

# enter the strings for lines you do NOT want to see
my($REASONS) = '(stinky\.cgi|images)';
#
our $maxwidth = 156; # Set for your screen size/resolution/font

print "\nInitializing...\n";
taillog();

sub taillog {
   my($offset, $line);
   $offset = (-s $LOG); # Don't start at begining, go to end

   while (1==1) {
       sleep(1);
       $| = 1;
       if ((-s $LOG) < $offset) {
           print "Log shrunk, resetting..\n";
           $offset = 0;
       }
       open(TAIL, $LOG) || print STDERR "Error opening $LOG: $!\n";

        if (seek(TAIL, $offset, 0)) {
           # found offset, log not rotated
       } else {
           # log reset, follow
           $offset=0;
           seek(TAIL, $offset, 0);
       }
       while ($line = <TAIL>) {
           chop($line);
           if (($REASONS) && ($line !~ m/$REASONS/)) {


             $line =~ s/18\.72\.0\.3 /      WORK      /;
             $line =~ s/24\.20\.100\.240 /    BUBBA\@HOME  /;
             $line =~ s/4\.3\.170\.200 /    MICHAEL     /;
             $line = substr($line, 0, $maxwidth);
             $line = substr($line, 0, (index($line,"\"Mozi"))) . "\n";



             print $line;



               next;
           }
       }
       $offset=tell(TAIL);
       close(TAIL);
   }
}

Sendmail Log Cleaner Code

#! /usr/bin/perl -wT
#
# mailcleaner.pl
#
#
# See http://www.pettingers.org/code/logcleaner.html for notes and details
#
use strict;
$| = 1;
# Enter things in the exclusion string below that you do not want
# to see.  Note that if these things are contained anywhere in
# the log line, the ENTIRE line is deleted from the output.
#
our $excl = "STARTTLS=|Command stream end of file";
#
our $maillogin = "/var/log/maillog"; # Log to analyze
our $maillogout = "/var/www/html/yoursite/sendmaillogs.txt"; # Output file
our $maxwidth = 250; # Set for your screen size/resolution/font

##########################################################


   print "Processing your log...\n\n";
   open (USERIN, $maillogin);
   open (USEROUT, ">$maillogout");
   Cleaner();
   close (USERIN);
   close (USEROUT);
exit;

##################################################################

sub Cleaner
{
my $text = '';
$text = <USERIN>;

while ($text) {
   if ($text !~ m/$excl/i){
   
# Enter things to substitute in the form of s/string-to-replace/replacement-string/

        $text =~ s/18\.72\.0\.3 /      WORK      /;
        $text =~ s/24\.20\.100\.240 /    BUBBA\@HOME  /;
        $text =~ s/4\.3\.170\.200 /    MICHAEL     /;
#
        $text =~ s/ \w+ sendmail\[[0-9]+\]: \w+/:-sendmail/;
        $text =~ s/ \w+ spamd\[[0-9]+\]/:   -SA-  /;
        $text =~ s/ \w+ imapd\[[0-9]+\]/:   imap  /;
        $text =~ s/ \w+ ipop3d\[[0-9]+\]/:   pop3  /;
        $text =~ s/ \(Message received: .+\@/ \(Message received: XX\@/;
        $text =~ s/ delay=.+,|xdelay=.+,|class=[0-9],|ctladdr=.+,|pri=.+,|nrcpts=.+,|proto=.+,|daemon=.+//g;
        $text = substr($text, 0, $maxwidth);
        if ($text !~ m/\n$/) {
                $text .= "\n";
        }
        print USEROUT $text;
        }
   $text = <USERIN>;
   print "\.";
}
print "\n";
return;
}

#################################################################

The LIVE Version for Sendmail Logs

#!/usr/bin/perl -w
##############################################################################
#
# See http://www.pettingers.org/code/logcleaner.html for notes and details
#
use strict;
use Socket;
my($LOG) = '/var/log/maillog';


my($REASONS) = '(STARTTLS=|Command stream end of file)';
#
our $maxwidth = 156; # Set for your screen size/resolution/font

print "\nInitializing...\n";
taillog();

sub taillog {
   my($offset, $line, $reason);
   $offset = (-s $LOG); # Don't start at begining, go to end

   while (1==1) {
       sleep(1);
       $| = 1;
       if ((-s $LOG) < $offset) {
           print "Log shrunk, resetting..\n";
           $offset = 0;
       }
       open(TAIL, $LOG) || print STDERR "Error opening $LOG: $!\n";

        if (seek(TAIL, $offset, 0)) {
           # found offset, log not rotated
       } else {
           # log reset, follow
           $offset=0;
           seek(TAIL, $offset, 0);
       }
       while ($line = <TAIL>) {
           chop($line);
           if (($REASONS) && ($line !~ m/$REASONS/)) {
               $reason = $1;

             $line =~ s/ \w+ sendmail\[[0-9]+\]: \w+/:-sendmail/;
             $line =~ s/ \w+ spamd\[[0-9]+\]/:   -SA-  /;
             $line =~ s/ \w+ imapd\[[0-9]+\]/:   imap  /;
             $line =~ s/ \(Message received: .+\@/ \(Message received: XX\@/;
             $line =~ s/\,mid=\<.+\>//;
             $line =~ s/ delay=.+,|xdelay=.+,|class=[0-9],|ctladdr=.+,|pri=.+,|nrcpts=.+,|proto=.+,|daemon=.+//g;
             $line = substr($line, 0, $maxwidth);
                if ($line !~ m/\n$/) {
                     $line .= "\n";
                }
             print $line;

               next;
           }
       }
       $offset=tell(TAIL);
       close(TAIL);
   }
}

Prerequisites

Root access. Required to get to the logs unless your hosting company allows you to access these for your site.