I happened to take these at a time when one particular person was acting well outside their normal calling pattern.

For starters.. the whole platform is written in perl using a forking model with a Postgres database backend.  So far all the development has been with CDRs coming from Broadsoft but records from ACME Packet are next to be incorporated.  The CDR files are tailed by another program I’ve already released here and it generates radius records on the wire back to a pair of redundant Radiator radius servers.  This is an excellent commercial radius server that is also written in perl and is extensible through the use of various code hooks.  In those hooks I do about dozen different things to the incoming records.. enriching them.. normalizing.. filtering.. and finally sorting and storing as either ‘interesting’ call traffic or not. For performance.. the only time radiator touches the database is to store the records that are going to be kept.

Radiator is an independent process..  but it is controlled through a daemonized network listener that runs on every node of this system.  The listener provides both an interactive network command interface as well as a remote API of sorts.  Nodes can use this telnet like protocol to talk to other nodes and exchange more complex data using Memcached when needed.  Code changes can be verified and deployed to the entire cluster with a single command through this interface and all remote processes can be queried for status and started and stopped gracefully with minimal interruption.

Each node can run one or more collector processes that are tied back to a specific source of call data… like the AS cluster on the west coast for example… so that calls from the same set of customers are always handled within the same process.  Interesting calls are analyzed seconds after they come in and are scored through a somewhat complex process (secret sauce) that uses dynamic baselining for both ends of the call. When this scored traffic is further analyzed… fraudulent activity becomes easy to spot.

That’s the theory anyway.  There is still plenty to do before this is a complete system that’s ready to play well with others.. but the basics are in place and have been scoring calls for several weeks now with only a few minor tweaks as I find corner cases in the traffic patterns.  I’ve also got the start of the web interface.. not nearly complete.. but good enough to be useful.

So some of what they showed us were examples of emails from other companies that were covering news items or special deals from their advertisers etc.  All were overproduced email through… that is.. they looked like web pages instead of email. Lots of graphics.. large photos.. etc.  I and several others pointed out that this is not the sort of stuff we want to get or have to read on our smart phones and how often we are working with limited bandwidth.

Most of these example emails were 80% ‘advertising’ or branding or whatever and 20% message content.  To me.. for an email that is supposed to be communicating the content.. that’s broken. It looks great.. and if you have the screen and bandwidth to see it properly I’m sure it would leave a good impression… but it’s not appropriate for email.  Also, you are emailing people you already have a relationship with. You don’t need half the screen taken up by some logo banner to remind them who you are or what your call letters are.  You also don’t need to repeat the same information 3 times in a message.  Keeping mobile devices in mind.. you should make use of the From address and Subject lines to let someone know without even opening it if it’s breaking news.. some advertising deal or an upcoming special guest on a show that day.

My similar gripe is with DirecTV.  I switched back to them a few months ago after TimeWarner demonstrated that the combination of Tivo/CableCard/Tuning Adapter just wasn’t going to work reliably. One of things I like about DVRs is the 30 second skip.. I use it constantly.  One thing I’ve noticed though is that when you encounter an advertisement FOR DirectTV.. ON DirecTV it won’t let you skip past it.  You can still fast forward.. but not skip.  It’s hard to imagine what the reasoning is for this… seeing as I’m obviously already a customer. What marketing genius thought it was a good idea to FORCE me to watch an advert for a service I’m already paying for??  This is broken behavior and all it does is annoy me every single time it happens. It’s also a reminder that as good as their stuff is.. it’s a long way from Tivo.

• Get it on the Project Page

I was fortunate enough to take the SANS security course on Wireless Ethical Hacking, Penetration Testing, and Defenses a few years ago. While I totally recommend the SANS courses.. they are really top notch in the world of tech training.. one of the things I learned as a result of that course is that very few people/organizations take security seriously.   Security should be thought of as existing on a continuum along with ease of use.  That is.. something could be totally secure and totally unusable or very easy to use and totally insecure.  SCADA systems have been operating at that end of the scale for decades now and I doubt very seriously that’s going to change anytime soon.  If the customers who buy these systems cared at all about security they would demand the systems actually be more secure.  That doesn’t happen though.. and I blame human nature.

Incidentally… you may think your world isn’t personally touched by these systems but you would be wrong.  In fact.. in some areas you may already have a vulnerable SCADA component bolted right on your own home.  Heard of the SmartGrid?  The very same researcher who taught my wireless hacking class has found some serious issues with the power meters used in smart grid systems.  Imagine a worm that could infect a network of power company smart meters.. giving control over the power they regulate to some 3rd party.  At that point it would be trivial to crash the regional electrical grid on demand.. and we know from what happened accidentally in the north east a few years ago that can take days to recover from. Sleep tight!

The script is generalized so as to be easy to modify and there isn’t much here that’s specific to postgres.. so it could easily be adapted for use with other systems like Oracle. You will need to put in the DDL for the child tables you will create but otherwise it’s pretty straight forward.  Please let me know if you find this useful as I couldn’t find anything else out there like it.

Visit the project page for details and the download.

Update: Several important updates to the code and my examples since I first published this.  Be sure to grab the latest version which is starting to behave reasonably now.

I’d been using iSync back before I got an iPhone..  and had never thought to disable it.  It looks like what was happening is that mail and other apps (ical, address book) had been building a huge database of stuff that needed to be synced to my old phone.  Once I went in and reset the sync history and disabled iSync.. everything calmed down and now a running instance of Mail.app with 3 imap accounts and an Exchange account is using about 60 MB of ram.. and it’s not steadily climbing as it had been before.  This may have also been the source of the problems I’d been having with Entourage.

I’m big on understanding the fundamentals of linux. If someone comes to me with a resume showing 10+ years of experience building and managing production unix/linux systems there are certain things I’d expect them to know.. and to a certain depth. If they obviously don’t.. then I have to question the validity of what’s on the resume. So what I’ll usually do is pick a few key areas and start off with some general (easy) questions and then drill down a bit to discover the level of understanding on that particular topic. As an example.. I’ll share one of my favorites and lay it out the way I might do it during an interview.

Q: If I wanted to know who was logged in, how busy a system was and how long it had been up what command would tell me all that?

(Assuming they get that I’m looking for the ‘w’ command and mention the system load)

Q: Why are there three numbers for the system load?

(Assuming they know about the 3 time periods)

Q: What is the system load.. what do those averages actually represent?

(This usually starts to trip up the junior people but assuming they know it’s the run queue length)

Q: How does a multi-cpu system affect your interpretation of system load?

(Assuming they say something about dividing load by CPU count)

Q: Describe the relationship between the load average measurement and the percentage busy you might get from ‘top’.

This is usually about as far as I’ll take something like this.. but it can lead to a discussion about things like different kernel schedulers and how they can be tweaked etc. If a person can answer these and have that sort of discussion it tells me they have the right depth of understanding a senior person should have… at least on this area. Someone who has been an admin (but not what I’d classify as an engineer) should be able to answer at least the first 2 questions.

I used the google TTS engine to record certain fixed statements that wold be common across calls.. converting the audio to the proper format for asterisk (SLN16) with sox.  For the specific alert text I’m using a simplified version of festival from the command line called ‘flite’.   The asterisk part is done entirely in a perl agi script and allows the called person to repeat the alert or acknowledge it within nagios. If they don’t answer or don’t ack the alert nagios will initiate another call in a few minutes.. and I’m able to use service escalations to notify different people if it goes too long without a response.

This project has more moving pieces than I usually like to use but it was interesting in just how easy it was to get working.  It’s gotten me thinking about doing a more full featured voice fronted to nagios that I could release.

See the code on page 2

Google TTS perl script:

Source code

#!/usr/bin/perl
 
# This takes a short string of text an spits out an mp3 audio
# file of it being spoken. This can then be converted for use
# in asterisk in the native sln16 format. 
 
use LWP;
 
$speak = $ARGV[0];
if (length($speak) ==0 || length($speak) > 100) { die ('String is too long'); }
 
my @headers1 = (
'Host' => 'translate.google.com',
'User-Agent' => 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.5) Gecko/20111114 Ubuntu/9.10 (karmic) Firefox/8.0.1',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language' => 'en-us,en;q=0.5',
'Accept-Encoding' => 'gzip,deflate',
'Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Keep-Alive' => '300',
'Connection' => 'keep-alive',
);
 
my $browser = LWP::UserAgent->new;
 
$speak =~ s/ /+/g;
 
$sec = time();
$resp = $browser->get("http://translate.google.com/translate_tts?q=$speak",@headers1);
 
open(FILE,">$sec.mp3");
print FILE $resp->content;
close(FILE);
print "Wrote $sec.mp3\n";
 
# To convert for asterisk use
# sox <file.mp3> -t raw outfile.sln16

Nagios listener…

Source code

#!/usr/bin/perl -w
 
package NagiosListener;
 
use strict;
use Data::Dumper;
use Net::Server::PreFork;
our @ISA = qw(Net::Server::PreFork);
use Nagios::Cmd;
 
my $NAGSOCKET = '/var/lib/nagios3/rw/nagios.cmd';
my $PIDfile = "/var/run/naglisten.pid";
my $TIME = time();
 
if (not -w $NAGSOCKET) { die ("Unable to open $NAGSOCKET for writing\n"); }
 
my $nagios = Nagios::Cmd->new( $NAGSOCKET );
 
NagiosListener->run(port=>5777, user=>'nagios', group=>'www-data', background=>1);
 
open (PID, '>', $PIDfile) or
   die("Unable to open PID file for writing\n");
print PID $$ +1 ."\n";
close PID;
 
$SIG{'TERM'} = sub
{
    unlink $PIDfile or die "Unable to remove PID file: $PIDfile\n";
};
 
$SIG{'HUP'} = sub
{
    unlink $PIDfile or die "Unable to remove PID file: $PIDfile\n";
    exec($0, '') or die "Couldn't restart: $!\n";
};
 
sub process_request {
    my $self = shift;
    eval
    {
        local $SIG{'ALRM'} = sub { die "Timed Out!\n" };
        my $timeout = 300; # give the user 300 seconds to type some lines
 
        my $previous_alarm = alarm($timeout);
        while (<STDIN>)
        {
            s/\r?\n$//;
            my $col = trim($_);
 
            print STDERR "Command: $col\n";
 
            if ($col eq 'sysload')
            {
                my $load = trim(`uptime |tr -s ' ' |cut -d ' ' -f 11 |cut -d , -f 1`);
                print "$load\r\n";
                next;
            }
 
            # Check for a properly formatted response
            if ($col =~ /^[\w\.]+;[\w\s_]+;\d{1}?;.+/)
            {
                my ($hostname, $service, $status, $desc) = split /;/, $col;
 
                $nagios->service_check($hostname, $service, $status, $desc);
 
                print "OK\r\n";
                next;
            }
 
            print "ERROR: Invalid Command\r\n";
 
        }
 
        alarm($previous_alarm);
 
    };
 
    if ($@ =~ /timed out/i)
    {
           print STDOUT "Timed Out.\r\n";
           return;
    }
}
 
sub trim
{
    my $string = shift;
    if (not $string) { return $string }
    $string =~ s/^\s+//;
    $string =~ s/\s+$//;
    return $string;
}
 
1;

Asterisk AGI script…

Source code

#!/usr/bin/perl
 
# extensions.conf lines
#[nagios-out]
#exten => s,1,AGI(nagiosalert.pl,${Params})
 
use strict;
use warnings;
use Nagios::Cmd;
use Asterisk::AGI;
 
# Don't buffer output
$|=1;
 
#my %input = $AGI->ReadParse();
 
my ($hostname, $service, $username, $message) = split /\|/, $ARGV[0];
$message =~ s/\+/,/g;
 
#$message =~ s/'(.+)'/$1/;
 
my $NAGSOCKET = '/var/lib/nagios3/rw/nagios.cmd';
my $TIME = time();
my ($digit, $t, $repeat);
 
my $AGI = new Asterisk::AGI;
 
if (not -w $NAGSOCKET) { $AGI->verbose ("Unable to open $NAGSOCKET for writing. User: $>  Group $)\n", 1); }
 
my $nagios = Nagios::Cmd->new( $NAGSOCKET );
 
# Prepare the tts audio file
$t = `flite -t "$message" -voice rms -o /tmp/$username.wav`;
$t = `sox /tmp/$username.wav -t raw /tmp/$username.sln16`;
$t = `chown asterisk.asterisk /tmp/$username.sln16`;
$t = `cp /tmp/$username.sln16 /var/lib/asterisk/sounds/en/`;
 
$AGI->answer();
$AGI->exec('WAIT', '1');
$AGI->exec('playback', 'followingisanalert');
 
# Playback the tts error message
ERRMSG: processDigit($AGI->get_data("$username", '200', '1'));
 
$repeat = 2;
while ($repeat > 0)
{
    processDigit($AGI->get_data('1torepeat7toclear', '5000', '1'));
    $repeat--;
}
 
if (not $digit) { $AGI->exec('playback', 'vm-goodbye'); exit 0; }
 
sub processDigit
{
    my ($digit) = @_;
 
    #$AGI->verbose("Digit:$digit");
 
    if (not $digit) { return; }
 
    if ($digit == 1) { goto ERRMSG; }
 
    if ($digit == 7)
    {
        $nagios->service_check($hostname, $service, 1, 'Alert Cleared Via Phone');
 
        $AGI->exec('playback', 'alert_ack');
        exit 0;
    }
}

Nagios Notification Script

Source code

#!/usr/bin/perl
 
#use warnings;
use strict;
use Getopt::Std;
use Switch;
 
my %options;
my ($t, $dial, $text_raw);
 
getopts("H:m:s:p:", \%options);
 
if (not $options{H} or not $options{m} or not $options{s} or not $options{p} )
{
    print "\nUsage: $0 -H 'host_name' -s 'service_name' -p 'phone number' -m 'tts message'\n\n";
    exit;
}
 
# Depending on the kind of alert.. you may want to customize the message a litle.
switch ($options{s})
{
    case /international_forwarding_alert/i		{ $text_raw = "Possible fraud on, $options{H} $options{m}"; }
 
    else			{ $text_raw = "No match. Something is wrong on host $options{H}. The message is, $options{m}"; }
 
}
 
my $text = speechify($text_raw);
 
# Can't pass commas through asterisk parameter list
$text =~ s/,/+/g;
 
#print "$text\n";
#exit;
 
# Generate the call file
$dial  = "Channel: SIP/SJC/$options{p}\n";
$dial .= "Callerid: Nagios <7045551212>\n"; # Originating number..
$dial .= "MaxRetries: 5\n";
$dial .= "RetryTime: 60\n";
$dial .= "WaitTime: 60\n";
$dial .= "Context: nagios-out\n";
$dial .= "Extension: s\n";
$dial .= "Priority: 1\n";
$dial .= "Set: Params=$options{H}|$options{s}|$text\n";
 
open my $FP, '>', "/var/spool/asterisk/tmp/dial.call" or die ("Unable to write temp call file");
 
print $FP $dial;
close $FP;
 
$t = `chown asterisk.asterisk /var/spool/asterisk/tmp/dial.call`;
$t = `mv /var/spool/asterisk/tmp/dial.call /var/spool/asterisk/outgoing/`;
 
sub speechify
{
    my ($r) = @_;
 
    # Transform a into b to sound right when using TTS
    my %s = (
          'as1.iad01' => 'a s 1 dot i a d 0 1, ',
          'as2.iad01' => 'a s 2 dot i a d 0 1, ',
          'as1.sjc01' => 'a s 1 dot s j c 0 1, ',
          'as2.sjc01' => 'a s 2 dot s j c 0 1, ',
          'as1.ams01' => 'a s 1 dot a m s 0 1, ',
          'as2.ams02' => 'a s 2 dot a m s 0 1, ',
          'as1.smb01' => 'a s 1 dot s m b 0 1, ',
          'as2.smb01' => 'a s 2 dot s m b 0 1, ',
          'groupid'   => 'group i d, ',
          'userid'    => 'user i d, ',
          'calledby'  => 'called by, ',
         );
 
    my @split = split / /, $r;
    my @out = ();
    my $lastbite;
 
    BITE: foreach my $bite (@split)
    {
 
        # Check the hash for a known word replacement
        foreach my $key ( keys %s )
        {
            if ($bite =~ /$key/i)
            {
                $lastbite = $bite;
                $bite =~ s/$bite/$s{$key}/i;
                push @out, $bite;
                next BITE;
            }
        }
 
        # The thing/word folling any of these needs to be spelled out
        if ($lastbite =~ /groupid/i or $lastbite =~ /userid/i)
        {
            $lastbite = $bite;
            $bite =~ s/(\w)(?=\w)/$1 /g;
            push @out, "$bite, ";
            next BITE;
        }
 
        push @out, $bite;
        $lastbite = $bite;
    }
 
    return "@out";
}