Tag Archives: HowTo

Zabbix: how to monitor Radius (and other services) with external check items and netcat (nc)

You can monitor Radius (and other services too, such as DNS and other) with Zabbix external check feature and netcat (nc).

How do external check items work?

“External check” items are monitored by Zabbix using external scripts, running on the server; you can create your own scripts and put them in the ExternalScripts directory, as defined by the zabbix_server.conf file:

# Location of external scripts
ExternalScripts=/etc/zabbix/externalscripts

You can setup an external check item using the following syntax for the key parameter:

Type: External check
Key: script[parameters]

For example, you can configure Zabbix to run the script checkradius.sh and to pass it the host IP address:

Description: Radius – Authentication
Type: External check
Key: checkradius.sh[{IPADDRESS}]
Type of information: Numeric (unsigned)

Zabbix will execute checkradius.sh HOSTNAME|HOSTIPADDRESS IPADDRESS, where HOSTNAME|HOSTIPADDRESS is the host name or IP address (it depends on the “Connect to” host parameter), and IPADDRESS is the value of the macro {IPADDRESS} used as parameter.

How to use netcat with external check items

With netcat (nc) you can send a host UDP or TCP data and get a response from it. For example, you can send a Radius authentication packet and wait for a response from the server.

Of course Netcat knows nothing about Radius or other protocols, it simply sends and receives data, so you have to forge an Access-Request packet and to parse an Access-Accept response.

Radius - Access-Request packetTo build the Radius Access-Request packet I simply sniffed a real packet using Wireshark and then I exported it to my Zabbix server (you can see it in the picture).

Once you have sent the Access-Request packet, you should receive an Access-Accept response from your server, so you can parse the response to see if it is the one you expected. You can do this using od to convert netcat output in hex and then grep the Radius Access-Accept code (0x02).

External check item and script configuration

My script uses only one argument, the one Zabbix always passes to external scripts, so I did’nt configure it to pass other parameters:

Description: Radius – Authentication
Type: External check
Key: checkradius.sh[]
Type of information: Numeric (unsigned)

The script gets the Radius packet to send to the server from the $1.rad file, where $1 is the host name or IP address; for example, for the Radius server at 10.0.0.1 I will put the Access-Request packet in the 10.0.0.1.rad file.

EDIT 2011-10-13: I think I forgot the “#! /bin/bash” line!

UPDATE 2011-12-01: I added the timelimit command to the script (you can install it with apt-get install timelimit). This command runs another command (nc in my script) and kills it after a specified time lapse. This is useful to handle endless netcat timeout. Remember to raise the default Timeout in the Zabbix configuration file in order to match your command timeout (file /etc/zabbix/zabbix_server.conf, parameter Timeout=10).

#! /bin/bash

cat /etc/zabbix/externalscripts/$1.rad | 
        timelimit -q -t 5 -T 5 nc -u -w 1 $1 1812 | od -t x1 | 
        grep "0000000 02" > /dev/null
if [ $? == 0 ]; then
        echo 1
        exit 1
else
        echo 0
        exit 0
fi

References

Radius RFC: http://www.ietf.org/rfc/rfc2865.txt

Zabbix: http://www.zabbix.com/

Netcat: http://netcat.sourceforge.net/

Wireshark: http://www.wireshark.org/

OpenACS configuration and scripting

As first, congrats to Audrys, aka au3, OpenACS author; I played a little with OpenACS and it seems to be a really good TR-069 framework, even if in beta version.

In the last days I have found some bugs, immediately fixed by the author in the svn version; at this time (svn revision 181 and release 0.03) I suggest you to download the latest svn version and compile it (on Debian you need subversion package):

svn co http://openacs.svn.sourceforge.net/svnroot/openacs openacs
ant -f b.xml
cp dist/acs.ear /opt/jboss/server/default/deploy/

If you already have it running, be aware you need to drop db tables and lets the svn version to create them.

OpenACS exposes a javascript engine you can use to implement your configuration server logic. Of course, you need to know TR-069 (CWMP) and related Technical Reports; you can find these documents here: Broadband Forum Technical Reports

OpenACS runs the “Default” script for each Inform request it receives: in this script you can do your business and call other script.
On each script you can access objects exposed by OpenACS: you can find some information on the OpenACS wiki page on SourceForge.

You can find a Default script example at the end of this post.

Actually I’m testing OpenACS with AVM Fritz!Box Fon 7170 and they really speak the same language! Firmware upgrade and configuration changes work good, I have not yet tested parameter attributes and notification changes.

If you are interested in this matter, stay tuned! 😉

My Default script:

var i;
var sData;

sData = 'n==========================================================';
sData += 'nDeviceId:';
sData += 'n   Manufacturer: ' + cpe.Inform.DeviceId.Manufacturer;
sData += 'n   OUI: ' + cpe.Inform.DeviceId.OUI;
sData += 'n   ProductClass: ' + cpe.Inform.DeviceId.ProductClass;
sData += 'n   SerialNumber: ' + cpe.Inform.DeviceId.SerialNumber;
sData += 'nMisc:';
sData += 'n   MaxEnvelopes: ' + cpe.Inform.MaxEnvelopes;
sData += 'n   RetryCount: ' + cpe.Inform.RetryCount;
sData += 'n   CurrentTime: ' + cpe.Inform.CurrentTime;

sData += 'nEvents:' ;

for( i=0; i<=cpe.Inform.Event.length-1; i++ )
   sData += 'n   ' + cpe.Inform.Event[i].EventCode+' ['+cpe.Inform.Event[i].CommandKey + ']';

sData += 'nParams:';

for( i=0; i<=cpe.Inform.ParameterList.length-1; i++ )
   sData += 'n   ' + cpe.Inform.ParameterList[i].Name+'='+cpe.Inform.ParameterList[i].Value;

sData += 'n';
sData += 'n==========================================================';

logger( sData );

// ------------------------------------------------------------------------------
for( i=0; i<=cpe.Inform.ParameterList.length-1; i++ )
{
       switch ( cpe.Inform.ParameterList[i].Name )
       {
               case 'InternetGatewayDevice.ManagementServer.ParameterKey':
                       cpedb.ManagementServer_ParameterKey = cpe.Inform.ParameterList[i].Value;
                       break;

               case 'InternetGatewayDevice.DeviceInfo.SpecVersion':
                       cpedb.DeviceInfo_SpecVersion = cpe.Inform.ParameterList[i].Value;
                       break;

               case 'InternetGatewayDevice.DeviceInfo.HardwareVersion':
                       cpedb.DeviceInfo_HardwareVersion = cpe.Inform.ParameterList[i].Value;
                       break;

               case 'InternetGatewayDevice.DeviceInfo.SoftwareVersion':
                       cpedb.DeviceInfo_SoftwareVersion = cpe.Inform.ParameterList[i].Value;
                       break;

               case 'InternetGatewayDevice.DeviceInfo.ProvisioningCode':
                       cpedb.DeviceInfo_ProvisioningCode = cpe.Inform.ParameterList[i].Value;
                       break;

               case 'InternetGatewayDevice.ManagementServer.ConnectionRequestURL':
                       cpedb.ManagementServer_ConnectionRequestURL = cpe.Inform.ParameterList[i].Value;
                       break;

               default:
                       if( cpe.Inform.ParameterList[i].Name.indexOf( 'InternetGatewayDevice.WANDevice.' ) >= 0 )
                       {
                               cpedb.DefaultWANConnection = cpe.Inform.ParameterList[i].Name.substr( 0, cpe.Inform.ParameterList[i].Name.indexOf( '.ExternalIPAddress' ) );
                               cpedb.DefaultWANConnection_ExternalIPAddress = cpe.Inform.ParameterList[i].Value;
                       }
                       break;
       }
}
cpedb.Save()

var sEvent;
var sCommandKey;
for( i=0; i<=cpe.Inform.Event.length-1; i++ )
{
       sEvent = cpe.Inform.Event[i].EventCode;
       sCommandKey = cpe.Inform.Event[i].CommandKey;

       switch ( sEvent )
       {
               case '0 BOOTSTRAP':
                       break;

               case '1 BOOT':
                       break;

               case '2 PERIODIC':
                       break;

               case '3 SCHEDULED':
                       break;

               case '4 VALUE CHANGE':
                       break;

               case '5 KICKED':
                       break;

               case '6 CONNECTION REQUEST':
                       break;

               case '7 TRANSFER COMPLETE':
                       break;

               case '8 DIAGNOSTICS COMPLETE':
                       break;
       }
}

DoSomething();

Installing TR-069 OpenACS on a fresh Debian setup

As the title says, OpenACS is a TR-069 based automatic configuration server, implementing CPE configuration protocol CWMP.
It’s an opensource project you can find on Source Forge (http://sourceforge.net/projects/openacs/), actually in Beta status.

I put here a brief HowTo to have it running on a fresh Debian setup. Of course, the result of this installation if for testing purpose only, and not for production environment.

EDIT: I tested this on both Debian 4.0 (Etch) and Debian 5.0 (Lenny) and it’s working fine.

I still didn’t test its features, actually I just had it running.
As soon as I get TR-069 capable CPEs and a bit of time to test them I will add more content in the blog!

Install JDK 1.5

Make sure to have “contrib” in your apt source list; if you don’t have, add and update aptitude.

nano /etc/apt/sources.list

deb http://YOUR_MIRROR/debian/ etch main contrib
deb-src http://YOUR_MIRROR/debian/ etch main contrib

Install some utilities to build JDK Debian package:

apt-get install java-package fakeroot

As non-root user, get Sun JDK 5.0 Update 17 from http://java.sun.com/j2se/1.5.0/download.jsp (non-RPM file):

wget URL

Build the Debian package

fakeroot make-jpkg jdk-1_5_0_17-linux-i586.bin

Install the .deb package (as root)

dpkg -i sun-j2sdk1.5_1.5.0+update17_i386.deb

Install JBoss

Get JBoss Application Server 4.2.2 zip file from http://www.jboss.org/download/:

wget URL

Unzip it:

unzip jboss-4.2.2.GA.zip -d /opt/
cd /opt
mv jboss-4.2.2.GA/ jboss

Run the server to test it:

cd /opt/jboss/bin
./run.sh -b 0.0.0.0

If the server fails starting, check it’s using the right Java VM; you can edit the bin/run.conf file and set JAVA_HOME=”/usr/lib/j2sdk1.5-sun”

Test the server: browse the homepage at http://YOUR_IP_ADDRESS:8080/

Hit CTRL+C to stop the server.

Install MySQL

apt-get install mysql-server-5.0

Install MySql Connector/J

Get Connector/J from http://dev.mysql.com/downloads/connector/j/5.1.html

wget URL

Extract and put mysql-connector-java-5.1.7-bin.jar into jboss/server/default/lib/

tar -xzvf mysql-connector-java-5.1.7.tar.gz
cd mysql-connector-java-5.1.7
mv mysql-connector-java-5.1.7-bin.jar /opt/jboss/server/default/lib/

Compile and deploy OpenACS

Install Apache Ant:

apt-get install ant

Get openacs-src file (openacs-src-0.03.zip) from SourceForge:

wget URL
unzip openacs-src-0.03.zip
cd openacs

Edit build.properties and set the right path to jboss (jboss=/opt/jboss/server/default)

nano build.properties

Edit the web.xml file and set the right path to the firmware directory (org.openacs.fwbase context-param):

nano acs-war/web/WEB-INF/web.xml

[… cut …]
    <context-param>
        <description>Path for firmware images</description>
        <param-name>org.openacs.fwbase</param-name>
        <param-value>/firmware/</param-value>
    </context-param>
[… cut …]
Run ant to build OpenACS

ant

Copy dist/acs.ear to jboss/server/default/deploy:

cp dist/acs.ear /opt/jboss/server/default/deploy/

Create and edit jboss/server/default/deploy/openacs-ds.xml, configuring data source

nano /opt/jboss/server/default/deploy/openacs-ds.xml

openacs-ds.xml:

<?xml version="1.0" encoding="UTF-8"?>
<datasources>
    <local-tx-datasource>
        <jndi-name>ACS</jndi-name>
        <connection-url>jdbc:mysql://localhost/ACS</connection-url>
        <driver-class>com.mysql.jdbc.Driver</driver-class>
        <user-name>openacs</user-name>
        <password>openacs</password>
        <min-pool-size>5</min-pool-size>
        <max-pool-size>20</max-pool-size>
        <idle-timeout-minutes>5</idle-timeout-minutes>
    </local-tx-datasource>
</datasources>

Create openacs-service.xml in jboss/server/default/deploy/jms

nano /opt/jboss/server/default/deploy/jms/openacs-service.xml

<?xml version="1.0" encoding="UTF-8"?>
<server>
    <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=acsQueue">
        <depends optional-attribute-name="DestinationManager">jboss.mq:service=DestinationManager</depends>
    </mbean>
</server>

Create ACS database and openacs user on MySQL, as in openacs-ds.xml:

mysql
CREATE DATABASE ACS;
GRANT ALL ON ACS.* TO openacs IDENTIFIED BY 'openacs';

Create ACS tables:

echo "CREATE TABLE HardwareModelBean (
  id int(11) NOT NULL auto_increment,
  oui varchar(250) default NULL,
  hclass varchar(250) default NULL,
  DisplayName varchar(250) default NULL,
  manufacturer varchar(250) default NULL,
  PRIMARY KEY  (id)
);

CREATE TABLE HostsBean (
  id int(11) NOT NULL auto_increment,
  serialno varchar(250) default NULL,
  url varchar(250) default NULL,
  configname varchar(250) default NULL,
  currentsoftware varchar(250) default NULL,
  sfwupdtime timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  sfwupdres varchar(250) default NULL,
  cfgupdres varchar(250) default NULL,
  lastcontact timestamp NOT NULL default '0000-00-00 00:00:00',
  cfgupdtime timestamp NOT NULL default '0000-00-00 00:00:00',
  hardware varchar(250) default NULL,
  cfgversion varchar(250) default NULL,
  props longblob,
  hwid int(11) default NULL,
  username varchar(250) default NULL,
  password varchar(250) default NULL,
  authtype int(11) default NULL,
  customerid varchar(250) default NULL,
  conrequser varchar(250) default NULL,
  conreqpass varchar(250) default NULL,
  PRIMARY KEY  (id)
);

CREATE TABLE ConfigurationBean (
  name varchar(250) NOT NULL,
  hardware varchar(250) default NULL,
  config longblob,
  filename varchar(250) default NULL,
  version varchar(250) default NULL,
  PRIMARY KEY  (name)
);

CREATE TABLE ScriptBean (
  name varchar(250) NOT NULL,
  script longblob,
  description varchar(250) default NULL,
  PRIMARY KEY  (name)
);

CREATE TABLE SoftwareBean (
  hardware varchar(250) NOT NULL,
  version varchar(250) NOT NULL,
  minversion varchar(250) default NULL,
  url varchar(250) default NULL,
  size bigint(20) NOT NULL,
  filename varchar(250) default NULL,
  PRIMARY KEY  (hardware,version)
);" | mysql ACS;

Create the firmware directory:

mkdir /firmware

Run the server:

cd /opt/jboss/bin
./run.sh -b 0.0.0.0

Browse the OpenACS web interface at http://YOUR_IP_ADDRESS:8080/openacs/index.jsf

Some useful links:
OpenACS Wiki: http://openacs.wiki.sourceforge.net/
Getting JDK 1.5 and Tomcat 5.5 up and running in Debian Linux: http://nileshk.com/node/36
JBoss on Debian quickstart: http://lorenzod8n.wordpress.com/2008/03/02/jboss-on-debian-quickstart/

Zabbix, SNMP traps mapped to the right host

In my opinion Zabbix is a really good NMS, but it’s lacking on SNMP traps handling.

Traps can be received using Net-SNMP suite and snmptrap.sh, a script released within the misc directory of Zabbix.

This is how it works: snmptrapd starts listening on port UDP 162, receives traps and sends them to a handler, which runs the script and pass trap information to its standard input. Finally, the script runs zabbix_sender to send information to the Zabbix server. On Zabbix, you can setup a fake Host with an Item of type “ZABBIX Trapper”: in the original script, both host and item’s key are referenced as snmptraps.

It works! The problem is that, whatever the sender is, trap data is always binded to one host: snmptraps.

zabbix_sender, used to send traps information to the server, can’t translate IP address to hostname:

# ./zabbix_sender -h

ZABBIX send v1.6.2 (16 January 2009)

usage: zabbix_sender [-Vhv] {[-zpsI] -ko | [-zpI] -i } [-c ]
Options:

[cut]

-s --host <Hostname> Specify host name.
Host IP address and DNS name will not work.

[cut]

So, we need to translate the sender’s IP address to its Zabbix hostname, in order to runs zabbix_sender with the right -s option value.

I make a very simple script to build an { IP / Zabbix hostname } file, using mysql client:

DST="/home/zabbix/zabbix-1.6.2/misc/snmptrap/zabbixhosts"
mysql --batch --silent -e "SELECT CONCAT( '[', IP, ']', Host )
FROM zabbix.hosts WHERE IP <> '' AND IP <> '0.0.0.0'" > $DST

Running this script every 30 minutes I have a file containing pairs of IP/HostName such these:

[192.168.0.1]MYWEBSERVER
[192.168.0.2]MYMAILSERVER

With few changes to the original script I can send traps to the right Zabbix host, grabbing the hostname from the mySql dump (zabbixhosts):

# CONFIGURATION

ZABBIX_SERVER="127.0.0.1";
ZABBIX_PORT="10051";

# path to zabbix_sender
ZABBIX_SENDER="/home/zabbix/zabbix-1.6.2/src/zabbix_sender/zabbix_sender";

# path to zabbixhosts, containing IP/hostname pairs
ZABBIX_HOSTSFILE="/home/zabbix/zabbix-1.6.2/misc/snmptrap/zabbixhosts"

# item key used to grab snmp data
KEY="snmptrap";

# used if the script can't find the hostname
DEFAULTHOST="Default_Trapper";

# END OF CONFIGURATION

read hostname
read ip
read uptime
read oid
read var1
read var2
read var3

oid=`echo $oid|cut -f2 -d' '`

# get hostname from the mySql dump
ZABBIX_HOST=`grep "[$hostname]" $ZABBIX_HOSTSFILE`

if [ $? -eq 0 ]; then
        hostname=`echo "$ZABBIX_HOST" | cut -f2 -d]`
else
        hostname="$DEFAULTHOST"
fi

str="$oid $var1 $var2 $var3"

result=`$ZABBIX_SENDER -z $ZABBIX_SERVER -p $ZABBIX_PORT
-s $hostname -k $KEY -o "$str"`

With this script you need an Item with type=”ZABBIX Trapper” and key=”snmptrap” for each host you want trap handling. You can also create a template and attach it to hosts you want to monitor.
You can create triggers based on snmp trap content as you want.

I use crontab to update the IP-to-hostname file two times each hour.

Installing Zabbix agent on Debian Etch

Base on Nick Anderson’s post Zabbix 1.4.4 from source on Debian Etch this is a simple guide line to compile and install Zabbix agent on a Debian (Etch) machine.

Create Zabbix user and group:

groupadd zabbix
useradd -c 'Zabbix' -d /home/zabbix -g zabbix -s /bin/bash zabbix
mkdir /home/zabbix
chown zabbix:zabbix /home/zabbix

Download Zabbix source (version 1.6.2 here):

su - zabbix
wget http://dfn.dl.sourceforge.net/sourceforge/zabbix/zabbix-1.6.2.tar.gz
tar zxvf zabbix-1.6.2.tar.gz
cd zabbix-1.6.2

Compile it:

./configure --prefix=/usr --enable-agent
make
exit
cd /home/zabbix/zabbix-1.6.2
make install

Update /etc/services:

echo "
zabbix_agent 10050/tcp # Zabbix ports
zabbix_trap 10051/tcp" >> /etc/services

Copy the config files:

mkdir -p /etc/zabbix
chown -R zabbix:zabbix /etc/zabbix
cp misc/conf/zabbix_agent* /etc/zabbix

Edit the config file (set the server IP address):

nano /etc/zabbix/zabbix_agentd.conf

Copy the init.d script and edit it, setting the right DAEMON path to /usr/sbin/${NAME}:

cp /home/zabbix/zabbix-1.6.2/misc/init.d/debian/zabbix-agent /etc/init.d/
nano /etc/init.d/zabbix-agent

Update the startup links and let the agent to sartup automatically:

update-rc.d zabbix-agent defaults 90

Start the agent!

/etc/init.d/zabbix-agent start

If you have a firewall running, remember to update its policies to allow traffic from and to the Zabbix server!