Tag Archives: Provider

DNS-amplification attack reflection on backhaul circuit

As many of us already know, DNS amplification attacks are a big plague for who fights every day for the sake of Internet security and service availability.

Infected hosts are instructed by botnet controllers to send DNS queries to recursive open resolvers, asking them for big zones with spoofed UDP packets containing the victim’s IP address in the source field, so that a small request would generate a big traffic toward the victim.

Small efforts are needed in order to mitigate those attacks – a proper DNS resolvers configuration to avoid open recursion, IP source validation (such as Cisco uRPF) to block source IP spoofing at the access network layer – but they may not be sufficient to immunize a network against annoying issues.

An unpleasant side effect

Even on secure networks an unpleasant side effect may occur: attack reflection against infected hosts, with the consequent backhaul circuit saturation and users’ downstream degradation.

Take, for example, the following not uncommon scenario:

DNS Amplification Reflection - Scenario

An ISP, running a properly configured DNS resolver, connects many users with a shared backhaul link between its core network and a local metro area; one or more users have infected devices responding to a botnet C&C server who aims to launch a DDoS against a given target.

A well implemented network access layer would stop spoofed packets whose source IP can not be reached through the same link on which they came from. At the same time a properly configured DNS resolver would not let recursive queries to go on by untrusted sources. The problem raises when proper DNS queries came in from trusted users and go to the ISP DNS resolver.

A not-really-failed attack attempt

Failed Attack

In the above diagram, at step 1, the botnet controller instructs the infected host to start a DNS amplification attack against the victim’s IP address 1.2.3.4. In the step 2 the malicious software tries to send a spoofed packet containing the victim’s address in the source field but something goes wrong: the operating system doesn’t let the malware to forge such a packet and rewrites it using its LAN address, or the router/firewall/CPE changes it with the WAN IP address (NAT). Anyway, at step 3, a proper DNS query comes out the user’s network and heads to the ISP DNS resolver, which in turn sends back a response with the huge DNS zone (step 4).

It’s easy to understand how this behaviour could lead to ISP internal issues regarding the backhaul link saturation and the users experience’s deterioration.

Consequences

A small upstream user’s query (65 bytes for an ANY query on isc.org) produces a big downstream response (~ 4 KB for isc.org zone), with a ~ 60x multiplicative factor. Every infected host may send many and many queries over a long period, even more than 1 query per second for many days, and many compromised hosts may be triggered at the same time by the same botnet controller.

Backhaul links may be rent from incumbent local carriers and may be characterized by an overbooking ratio calculated over the expected usage by customers who share them; high speed links which connect DNS resolvers to the core may overwhelm them when filled by UDP response packets and lead to traffic stagnation because of traffic policing operated by the carrier.

Customers also may report a bad user experience: it’s true, their links are operating at 100% of their capacity, but Facebook is slow and the VoIP is unusable.

A very big headache, even for an ISP with a properly configured network.

Symptoms

The first symptom that can be observed is an abnormal peak in resolvers bandwidth usage:

DNS resolver bandwidth usage during an attack attempt - response traffic in green

DNS resolver bandwidth usage during an attack attempt – response traffic in green

During an attack attempt the network usage (servers’ upstream) may raise up to hundreds of times higher than average.

NetFlow also may help us to identify this kind of traffic; big response UDP datagrams may be fragmented over the network and they would be shown as port-0 UDP packets in the output of nfdump or similar tools, with an high Bpp (bytes-per-packet) ratio:

Proto Src IP Addr:Port  Dst IP Addr:Port   Packets    Bytes  pps     bps   Bpp Flows
UDP    RESOLVER_1:0   ->  A.B.1.155:0        78966  106.6 M   48  519300  1350    79
UDP    RESOLVER_1:0   ->   G.H.4.73:0        35798   48.3 M   25  274100  1350    38
UDP    RESOLVER_1:53  ->  I.J.5.101:14068     7430    9.3 M    4   46712  1249   187
A 65-bytes request generated a 4157 bytes response in 3 segments - calculated at IP level

A 65-bytes request generated a 4157 bytes response in 3 segments – calculated at IP level

Mitigation

Unfortunately, as far as I know, there are still no specific implementations aimed to mitigate those kind of attack.

BIND9 has a generic rate-limit option which prevents a requestor to be told the same answer more than a specific number of times within a one-second interval, but there is no way to apply it only to a subset of responses (like the ones used in DDoS attack, such as ANY to isc.org or ripe.net). DNS RRL (Response Rate Limiting) is focused on authoritative servers, not on recursive ones.

A suitable way would be the use of the iptables recent module on recursive resolvers, but other aspects have to be considered, such as servers load and performances degradation.
A first deep-packet inspection of the incoming DNS requests would filter those DNS queries whose type has been set to ANY, then the recent module would lookup the source IP address on a local list and drop the packet if it violates the predetermined policy. For example, a policy may allow one or two queries with type = ANY every 5 seconds, so that “regular” usage would be allowed while malware initiated traffic would be dropped within few seconds.

Number of different IP addresses on the recent module's queue - peak during an attack attemp

Number of different IP addresses on the recent module’s queue – peak during an attack attemp

References

“Alert (TA13-088A) DNS Amplification Attacks”, US-Cert: http://www.us-cert.gov/ncas/alerts/TA13-088A

“DNS Response Rate Limiting (DNS RRL)”, Paul Vixie, ISC – Vernon Schryver, Rhyolite: http://ss.vix.su/~vixie/isc-tn-2012-1.txt

Graphing near realtime PPPoE/PPPoA link speed using SNMP Traffic Grapher (STG to its friends)

Sometimes it happens to me that, for troubleshooting reasons, I need to graph PPPoE or PPPoA connections speed from the NAS/BRAS side. These links are terminated on Cisco routers, where other hundreds of CPEs are connected; connections are from dialin users and I can’t have static graphs, mostly because I don’t need endusers monitoring on a fulltime basis and it would only be a huge waste of resources.

In this case a little program helps me: STG, SNMPTrafficGrapher.

STG - SNMPTrafficGrapher

STG – SNMPTrafficGrapher

It’s a small Windows utility that uses SNMP to get counters data and put them on a graph, like MRTG does. It’s easy and fast to deploy (run it, set SNMP OID and it’s ready), does not use many resources and can give you graphs updated every second.

Configuration

From the View / Settings menu you just have to set the device’s IP address and SNMP community, and then to select OID and polling frequency.

As said, users have dialin connections which go up and down and there is no way to predict their SNMP interface’s index; to obtain the right OID we can use the show snmp mib ifmib ifindex command.

Initially we get the actual Virtual-Access interface for the user we need to monitor:

Router#sh users | include MyUserName
  Vi1.195      MyUserName	   PPPoATM      -        10.11.12.13

Then we get it’s SNMP index:

Router#show snmp mib ifmib ifindex Virtual-Access 1.195
Interface = Virtual-Access1.195, Ifindex = 257

And finally we can use it to configure STG:

STG setup

Geen OID = 1.3.6.1.2.1.2.2.1.10.257 (ifInOctets.257)
Blue OID = 1.3.6.1.2.1.2.2.1.16.257 (ifOutOctets.257)

Where 257 is the dynamic SNMP ifIndex of our user’s Virtual-Access interface.

References

STG homepage: http://leonidvm.chat.ru/

GNS-3 Lab: 4-byte Autonomous System Number – interactions between 32 and 16 bit speaking BGP routers

Such as IPv4 addresses, 2-byte pool of Autonomous System (AS) Numbers is going to be exhausted soon. By the time I’m writing, it seems Internet will run out of 16-bit AS Number on 26-Sep-2012. To prevent this situation, IANA extended the AS Number field to 32 bits and, on 2007, RIRs started assigning them… or, at least, to offer them!
Of course, in order to let 4-byte ASN to work properly, a little adjustement was needed in the BGP protocol. Enter RFC 4893, BGP Support for Four-octet AS Number Space.

4-byte ASN - Lab topologyIn this post I don’t want to cover the mechanisms of this protocol extension, if you want to understand them I please you to follow some links in the References section; I prefer to show some interactions between OLD 2-byte ASN and NEW 4-byte ASN BGP speakers. I built a little GNS3/Dynamips lab using a couple of 7200s with IOS 12.2(33)SRE as NEW speakers, and a couple of 3640s as OLD peers.

In the topology every router announces any subnet to the others; BGP peering sessions follow the physical topology. Green ASs support 4-byte AS, while gray do not.
For the sake of readibility I’ll use asdot notation in this post, with the exception of some configuration blocks where I’ll use asplain just to show both notation usage.

I also uploaded two packet captures on PacketLife.net: they show BGP UPDATES with and without the NEW_AS_PATH attributes. They are 4-byte_AS_numbers_Full_Support.cap and 4-byte_AS_numbers_Mixed_Scenario.cap.

Partial 4-byte ASN support: A-B routers

Router A is on AS 10.1 / 655361 (asdot / asplain notation):

A#sh run | sec bgp
router bgp 655361
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 neighbor 172.16.3.2 remote-as 2
 no auto-summary
B#sh run | sec bgp
router bgp 2
 no synchronization
 bgp log-neighbor-changes
 network 20.0.0.0
 neighbor 172.16.3.1 remote-as 23456
 no auto-summary

As you can see, on router B neighbor statement we can’t use the real ASN to peer with A, so we have to use the AS_TRANS (23456). Let’s wait until our sessions come up, then show BGP neighbors and tables:

A#sh ip bgp neighbors 172.16.3.2
BGP neighbor is 172.16.3.2,  remote AS 2, external link
  BGP version 4, remote router ID 20.0.0.1
  BGP state = Established, up for 00:00:27
  Last read 00:00:27, last write 00:00:27, hold time is 180, keepalive interval is 60 seconds
  Neighbor sessions:
    1 active, is not multisession capable
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised
    Address family IPv4 Unicast: advertised and received
    Multisession Capability: advertised
...

Please note the Four-octets ASN Capability: advertised line: A advertised this capability but did not receive it back from B.

A#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         0.0.0.0                  0         32768 i
*> 20.0.0.0         172.16.3.2               0             0 2 i

As expected, A receives the B‘s route but…

B#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.3.1               0             0 23456 i
*> 20.0.0.0         0.0.0.0                  0         32768 i

B only sees A‘s subnet as from AS_TRANS (23456).

This is because B does not support 4-byte ASN, and router A knows this (capabilities exchange during session setup), so it just sends AS_PATH attributes containing the 16-bit AS_TRANS in place of the real 32-bit AS number.

You can find a similar UPDATE on the first capture I sent on PacketLife.net (4-byte_AS_numbers_Mixed_Scenario.cap), packet number 2.

Full 4-byte ASN support: A-D routers

Let’s setup the BGP session between routers A and D; they both support 4-byte ASN.

A#sh run | sec bgp
router bgp 655361
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 neighbor 172.16.1.2 remote-as 2621441
 neighbor 172.16.3.2 remote-as 2
 no auto-summary
D#sh run | sec bgp
router bgp 2621441
 no synchronization
 bgp log-neighbor-changes
 network 40.0.0.0
 neighbor 172.16.1.1 remote-as 655361
 no auto-summary

When sessions come up…

A#sh ip bgp neighbors 172.16.1.2
BGP neighbor is 172.16.1.2,  remote AS 40.1, external link
  BGP version 4, remote router ID 40.0.0.1
  BGP state = Established, up for 00:04:04
  Last read 00:00:46, last write 00:00:53, hold time is 180, keepalive interval is 60 seconds
  Neighbor sessions:
    1 active, is multisession capable
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised and received
    Address family IPv4 Unicast: advertised and received
    Multisession Capability: advertised and received
...

Here A both sent and received the 4-byte ASN capability to/by B, because it supports this feature.

A#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         0.0.0.0                  0         32768 i
*> 20.0.0.0         172.16.3.2               0             0 2 i
*> 40.0.0.0         172.16.1.2               0             0 2621441 i
D#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.1.1               0             0 655361 i
*> 20.0.0.0         172.16.1.1                             0 655361 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i

Both A and D have subnets with the full 32-bit AS number.

You can see the capture on the file 4-byte_AS_numbers_Full_Support.cap on PacketLife.net.

As I already said, let’s switch to asdot notation now, just to improve readibility:

D#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
D(config)#router bgp 40.1
D(config-router)#bgp asnotation dot
D(config-router)#do show ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.1.1               0             0 10.1 i
*> 20.0.0.0         172.16.1.1                             0 10.1 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i
A#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
A(config)#router bgp 10.1
A(config-router)#bgp asnotation dot

Ok, 10.1 is better than 655361, and 40.1 than 2621441!! :)

Mixed scenario: D-B routers

Let’s have more fun now and bring D-B session up!

D(config-router)#neighbor 172.16.2.1 remote-as 2
B(config-router)#neighbor 172.16.2.2 remote-as 23456

As we already saw, B doesn’t support 4-byte ASN, so we have to use AS_TRANS to peer with D.

D#show ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*  10.0.0.0         172.16.2.1                             0 2 10.1 i
*>                  172.16.1.1               0             0 10.1 i
*> 20.0.0.0         172.16.2.1               0             0 2 i
*                   172.16.1.1                             0 10.1 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i

The interesting thing we can see now on D is the presence of A‘s subnet (10.0.0.0) advertised by B: even if B does not support 4-octect ASN, D receives it with the real 32-bit path. When A advertises the subnet to B it makes the AS_PATH attribute up using AS_TRANS, but it also adds the transitive attribute NEW_AS_PATH, containing the full 32-bit AS numbers; when B advertises the subnet to D it adds this attribute and so D can rebuild the real 32-bit path.

As before, you can find a similar UPDATE on the capture I sent on PacketLife.net; the file is 4-byte_AS_numbers_Mixed_Scenario.cap, packet number 3.

Another interesting aspect of 4-byte AS lack of support can be seen in the following table:

B#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*  10.0.0.0         172.16.2.2                             0 23456 23456 i
*>                  172.16.3.1               0             0 23456 i
*> 20.0.0.0         0.0.0.0                  0         32768 i
*> 40.0.0.0         172.16.2.2               0             0 23456 i
*                   172.16.3.1                             0 23456 23456 i

Here B has two entries for the 10.0.0.0 subnet: they both seem to be originated from the same AS, but that’s not true: the first comes from D (AS 40.1), while the second (the selected one) comes directly from A.

Similar behaviour is for the 40.0.0.0 subnet. Of course, this is not a really big problem, but may lead to false assumptions and compromise traffic patterns analysis.

16-bit stub router

Just to complete all the possible scenarios, here is C configuration:

B#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
B(config)#router bgp 2
B(config-router)#neighbor 172.16.4.2 remote-as 3
C#sh run | sec bgp
router bgp 3
 no synchronization
 bgp log-neighbor-changes
 network 30.0.0.0
 neighbor 172.16.4.1 remote-as 2
 no auto-summary
C#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.4.1                             0 2 23456 i
*> 20.0.0.0         172.16.4.1               0             0 2 i
*> 30.0.0.0         0.0.0.0                  0         32768 i
*> 40.0.0.0         172.16.4.1                             0 2 23456 i

It has no idea of what 4-byte ASN are, so it sees both 10.0.0.0 and 40.0.0.0 subnets as originated from the same AS.

Thanks

I have to say thanks to Marco Rizzi for his kind support on helping me to build this lab! ;-) And, of course, I suggest you to visit his blog: Marco Rizzi Blog – networking with passion!!!

Download the lab

To download the lab and configuration files click here.

References

Report on consumption of AS Numbers: http://www.potaroo.net/tools/asns/

BGP Support for Four-octet AS Number Space: RFC 4893

APRICOT 2007: 4-Byte AS Numbers (PDF)

Cisco.com: Migration Guide for Explaining 4-Byte Autonomous System

Cisco.com: Cisco IOS BGP 4-Byte ASN Support

Cisco.com: Explaining 4-Byte Autonomous System (AS) ASPLAIN and ASDOT Notation for Cisco IOS

NetFlow: installation and configuration of NFDUMP and NfSen on Debian

After the brief overview about the installation of flow-tools and FlowViewer, in this post I’d like to share my experience about the setup of a basic solution based on another pair of tools: NFDUMP and NfSen. As always on my posts, the starting point is a fresh Debian 5.0 setup.

Components used in the solution

As it was for the previous solution, two main components are involved, one to acquire NetFlow data, the other to analyze them in an easy way.

NFDUMP is a suite of tools composed by many programs; nfcapd is one of them and it is the daemon which listens for incoming NetFlow data. It is launched by NfSen.

NfSen is the web based front end we use to analyze NFDUMP NetFlow data. It is splitted up in two parts: a Perl program which runs in the background and launches nfcapd, and some PHP web pages.

The biggest difference between a NFDUMP/NfSen based solution and a flow-tools/FlowViewer solution is that the former does support NetFlow v9, while the latter may result in a simpler and easier tool to work with. Anyway, there is not one perfect solution for all the needs!

In this post I’ll use NFDUMP 1.6.1 and NfSen 1.3.2, the latest stable releases I can find on SourceForge while I’m writing this blog entry; you can find their web pages in the References section of this post. I’ll build them from source code.

NFDUMP

Let’s add some tools we’ll need to build the package:

apt-get install gcc flex librrd-dev make

We are ready to start! The steps are very simple: download source code and extract the tar:

cd /usr/local/src/
wget http://sourceforge.net/projects/nfdump/files/stable/nfdump-1.6.1/nfdump-1.6.1.tar.gz/download
gzip -dc nfdump-1.6.1.tar.gz | tar -xf -
cd nfdump-1.6.1

In order to use NFDUMP with NfSen we need to enable nfprofile in the configure step; then we have to make && make install it:

./configure --enable-nfprofile 
make
make install

Now it is ready to be ran.

If we want to use it without NfSen we can stop here and create a simple init.d script to launch NFDUMP at startup; it will acquire NetFlow data and we can use command line tools (such as nfdump) to analyze them. You can find a basic init.d script at the end of this post (“Annex A: stand-alone NFDUMP”).

But that’s not our goal, we want NfSen!

NfSen

As said, NfSen has a PHP front-end so it needs a web server with PHP support in order to be used. It also uses RRD to store data and paint nice graphs and some Perl modules to produce alerts: let’s satisfy its prerequisites:

apt-get install apache2 libapache2-mod-php5 php5-common libmailtools-perl rrdtool librrds-perl

Now it’s time to download and configure the program:

cd /usr/local/src/
wget http://sourceforge.net/projects/nfsen/files/stable/nfsen-1.3.2/nfsen-1.3.2.tar.gz/download
gzip -dc nfsen-1.3.2.tar.gz | tar -xf -
cd nfsen-1.3.2

Once extracted, we have to copy the config file into /etc and modify it:

cp etc/nfsen-dist.conf /etc/nfsen.conf
nano /etc/nfsen.conf

As you can see from the support web page it presents a lot of parameters, but many of them may be left unchanged; in this example I chose to leave as many parameters as possible at their default settings, so we have all the files in a single directory: /data/nfsen. Here the parameters I changed:

$USER    = "www-data";
$WWWUSER  = "www-data";
$WWWGROUP = "www-data";

%sources = (
    'MYROUTER'    => { 'port'    => '9995', 'col' => '#0000ff', 'type' => 'netflow' },
);

$MAIL_FROM   = 'MYEMAIL@MYDOMAIN.COM';
$SMTP_SERVER = 'MY.SMTPSERVER.COM';

For a better setup I suggest you to read the configuration guide on the project’s web site.

Let’s make the destination directory and complete the installation by running the install.pl script:

mkdir -p /data/nfsen
./install.pl /etc/nfsen.conf

We are ready to run the nfsen program and let it executes nfcapd daemon:

cd /data/nfsen/bin
./nfsen start

We can also schedule it to be ran at startup:

ln -s /data/nfsen/bin/nfsen /etc/init.d/nfsen
update-rc.d nfsen defaults 20

It’s ready: point your browser at http://YOUR_IP_ADDRESS/nfsen/nfsen.php and enjoy! Of course, this is just a very basic setup, both NFDUMP and NfSen have many options you can set and use to have better performances and to improve stability and scalability. If you want to go deep into their configuration I suggest you to use the links you can find at the end of this post.

P.S.: Don’t forget to configure your routers to export NetFlow data! ;-)

Annex A: stand-alone NFDUMP

As said, NFDUMP may be used in a stand-alone way, without the help of NfSen.

We just have to build a destination directory for NetFlow data…

mkdir -p /var/flows/MYROUTER
mkdir -p /var/flows/MYSECONDROUTER

and add an init.d script like the following one:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          nfcapd
# Required-Start:    $network
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: netflow capture daemon
# Description:       nfcapd is the netflow capture daemon of the nfdump tools.
### END INIT INFO

# Author: Erik Wenzel <erik@debian.org>
# Edited by Pierky for the blog post "NetFlow: installation and configuration of 
# NFDUMP and NfSen on Debian" on http://blog.pierky.com

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin
DESC="netflow capture daemon"
NAME=nfcapd
DAEMON=/usr/local/bin/$NAME
DATA_BASE_DIR="/var/flows"
PIDFILE=/var/run/$NAME.pid
DAEMON_ARGS="-D -w -S 1 -P $PIDFILE -n MYROUTER,192.168.0.1,$DATA_BASE_DIR/MYROUTER"
SCRIPTNAME=/etc/init.d/nfdump

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null 
                || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- 
                $DAEMON_ARGS 
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        #
        # Disabled second call, because is kills nfsen controlled nfcapd
        #start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        #[ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}

case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ q"$VERBOSE" != qno ] && log_end_msg 0 ;;
                2)   [ p"$VERBOSE" != pno ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "r$VERBOSE" != rno ] && log_end_msg 0 ;;
                2) [ "s$VERBOSE" != sno ] && log_end_msg 1 ;;
        esac
        ;;
  #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
  restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0|1) log_end_msg 0 ;;
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac
exit 0

I edited this script starting from the one included in the Debian’s package.

Please note the DATA_BASE_DIR and the DAEMON_ARGS variables. The first contains the base destination directory for our NetFlow data. The second represents arguments we want to use when launching nfcapd; you can add as many -n options as you want, one for each NetFlow source you want to collect data from. Remember: man nfcapd is your friend! ;)

Finally:

chmod a+x /etc/init.d/nfdump
update-rc.d nfdump defaults 20
/etc/init.d/nfdump start

References

Cisco.com: NetFlow version 9

NFDUMP: http://nfdump.sourceforge.net/

NfSen: http://nfsen.sourceforge.net/

NetFlow: how to install and configure flow-tools and FlowViewer on a fresh Debian setup

NetFlow is a very useful tool/protocol to monitor network traffic’s patterns. Many tools have been developed to collect and analyze NetFlow data, here I chose flow-tools and FlowViewer packages, and I would like to show how to get them work on a fresh Debian 5.0 (Lenny) setup.

Components used in the solution

There are two main packages involved in the solution: flow-tools, used to acquire and collect NetFlow data generated by our routers, and FlowViewer, used to process, view and paint them on nice graphs.

The flow-tools package is built up by a lot of components, many of them are transparently used by FlowViewer; here I’ll focus on the flow-capture program, the one which acquires and collect the data.

The FlowViewer package is splitted up on 3 CGIs: FlowViewer, FlowGrapher and FlowTracker. They can be used through a web-server and they let us analyze data collected by flow-capture. The package contains 2 programs also, FlowTracker_Collector and FlowTracker_Grapher, which run periodically and build MRTG-like graphs, storing data in RRD databases.

Acquiring data

First off, we need to acquire NetFlow data generated by our routers; flow-tools is the package we need:

apt-get install flow-tools

Let’s edit the main configuration file, flow-capture.conf, where we tell flow-capture what we want to acquire and how we want to store it:

cd /etc/flow-tools
nano flow-capture.conf

Its configuration file is pretty simple; it’s built up by many lines containing the command line arguments of flow-capture: each line is used to run an instance of the program.

You can execute man flow-capture to view all the arguments it accepts; here I used the following example:

# MYROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYROUTER 0.0.0.0/192.168.0.1/3001

# MYSECONDROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYSECONDROUTER 0.0.0.0/192.168.0.2/3002

So, I acquire data from MYROUTER, which sends NetFlow version 5 data from 192.168.0.1; this flow-capture instance will be listening on port 3001 of every local IP address (0.0.0.0) and it will store data on the /var/flows/MYROUTER directory, with a nesting level of type 3, that is directories like /var/flows/MYROUTER/YYYY/YYYY-MM/YYYY-MM-DD/. It will keep files up to a maximum of 5 GB.

This may be a sample configuration for MYROUTER (Cisco), where 192.168.0.9 is the IP address of our NetFlow box:

ip flow-export source FastEthernet0/0.1
ip flow-export version 5
ip flow-export destination 192.168.0.9 3001

interface FastEthernet0/0
 description LAN facing
 no ip address

interface FastEthernet0/0.1
 encapsulation dot1Q 1 native
 ip address 192.168.0.1 255.255.255.0

interface FastEthernet0/1
 description WAN facing
 ip address 10.0.0.1 255.0.0.0
 ip route-cache flow

So on for MYSECONDROUTER…

We just have to build the destination directories and then run the program:

mkdir -p /var/flows/MYROUTER
mkdir -p /var/flows/MYSECONDROUTER

/etc/init.d/flow-capture start

The tool is now working:

ls -l -R /var/flows/MYROUTER/
/var/flows/MYROUTER/:
totale 4
drwxr-xr-x 3 root root 4096  5 mar 10:11 2010

/var/flows/MYROUTER/2010:
totale 4
drwxr-xr-x 3 root root 4096  5 mar 10:11 2010-03

/var/flows/MYROUTER/2010/2010-03:
totale 4
drwxr-xr-x 2 root root 4096  5 mar 10:11 2010-03-05

/var/flows/MYROUTER/2010/2010-03/2010-03-05:
totale 4
-rw-r--r-- 1 root root 92  5 mar 10:11 ft-v05.2010-03-05.101125+0100

FlowViewer

Once we are collecting data using flow-tools we want to analyze them, so we need to get FlowViewer up and running!

Requirements

As we can see from the FlowViewer web site we have to satisfy some requirements in order to run it:

- a web server with CGI support;

- Perl 5.0 or later;

- FlowTools;

- GD and GD:Graph;

- RRDTool.

Let’s start installing them:

apt-get install apache2

apt-get install libgd-graph-perl

apt-get install rrdtool

As we’ll see later in the FlowViewer configuration, it uses also another utility to resolve IP addresses in host names: dig. We have to install the dnsutils Debian package in order to have it:

apt-get install dnsutils

Installation and configuration

Download and untar the package:

cd /usr/local/src
wget http://ensight.eos.nasa.gov/FlowViewer/FlowViewer_3.3.1.tar
tar -xf FlowViewer_3.3.1.tar

Now that all requiremets are met and FlowViewer is on the disk, let’s start configuring it!

In order to get FlowViewer up and running we have to edit its configuration file and build some directories it needs. The user guide provided by the author is very complete, you can find there any information you need. The web site’s FAQ section is very useful too. Here I’ll just provide a basic configuration and layout.

In this sample configuration I use the Apache’s default web site as starting point, so I have:

- / (the root) on /var/www/

- /cgi-bin/ on /usr/lib/cgi-bin/

Let’s move FlowViewer into the cgi-bin directory…

mv FlowViewer_3.3.1 /usr/lib/cgi-bin/

… and edit the configuration file:

cd /usr/lib/cgi-bin/FlowViewer_3.3.1
nano FlowViewer_Configuration.pm

There are a bit of parameters to change, but remember: the user guide is your friend.

Here is the diff of my file against the original:

diff -y --suppress-common-lines -W 250 FlowViewer_Configuration.pm FlowViewer_Configuration.pm.ORIG
$FlowViewer_server       = "192.168.0.9";                                                             |   $FlowViewer_server       = "www.yourcompany.com";     # (IP address or hostname)
$FlowViewer_service      = "http";                                                                    |   $FlowViewer_service      = "https";           # (http, or https)
$reports_directory       = "/var/www/FlowViewer";                                                     |   $reports_directory       = "/htp/htdocs/FlowViewer_3.3.1";
$reports_short           = "/FlowViewer";                                                             |   $reports_short           = "/FlowViewer_3.3.1";
$graphs_directory        = "/var/www/FlowGrapher";                                                    |   $graphs_directory        = "/htp/htdocs/FlowGrapher_3.3.1";
$graphs_short            = "/FlowGrapher";                                                            |   $graphs_short            = "/FlowGrapher_3.3.1";
$tracker_directory       = "/var/www/FlowTracker";                                                    |   $tracker_directory       = "/htp/htdocs/FlowTracker_3.3.1";
$tracker_short           = "/FlowTracker";                                                            |   $tracker_short           = "/FlowTracker_3.3.1";
$cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                       |   $cgi_bin_directory       = "/htp/cgi-bin/FlowViewer_3.3.1";
$work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";                          |   $work_directory          = "/htp/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory          = "/var/www/FlowViewer_Saves";                                               |   $save_directory          = "/htp/htdocs/FlowViewer_Saves";
$names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                       |   $names_directory         = "/htp/cgi-bin/FlowViewer_3.3.1";
$filter_directory        = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Filters"; |   $filter_directory        = "/htp/cgi-bin/FlowTracker_Files/FlowTracker_Filters";
$rrdtool_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRDtool"; |   $rrdtool_directory       = "/htp/cgi-bin/FlowTracker_Files/FlowTracker_RRDtool";
$flow_data_directory     = "/var/flows";                                                              |   $flow_data_directory     = "/htp/flows";
$exporter_directory      = "/var/flows/all_routers";                                                  |   $exporter_directory      = "/htp/flows/all_routers";
$rrdtool_bin_directory   = "/usr/bin";                                                                |   $rrdtool_bin_directory   = "/usr/local/rrdtool-1.2.12/bin";
$trackings_title         = "FlowViewer Saves";                                                        |   $trackings_title         = "Your Company Name";
$user_hyperlink          = "/FlowViewer_Saves";                                                       |   $user_hyperlink          = "http://www.yourcompany.com/";
@devices                 = ("MYROUTER","MYSECONDROUTER");                                             |   @devices                 = ("router_1","router_2","router_3");
$log_directory      = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                            |   $log_directory      = "/htp/cgi-bin/FlowViewer_3.3.1";

And here is every “directory” or “short” parameter:

cat FlowViewer_Configuration.pm | grep "directory|short"
$reports_directory       = "/var/www/FlowViewer";
$reports_short           = "/FlowViewer";
$graphs_directory        = "/var/www/FlowGrapher";
$graphs_short            = "/FlowGrapher";
$tracker_directory       = "/var/www/FlowTracker";
$tracker_short           = "/FlowTracker";
$cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$cgi_bin_short           = "/cgi-bin/FlowViewer_3.3.1";
$work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$work_short              = "/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory          = "/var/www/FlowViewer_Saves";
$save_short              = "/FlowViewer_Saves";
$names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$filter_directory        = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Filters";
$rrdtool_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRDtool";
$flow_data_directory     = "/var/flows";
$exporter_directory      = "/htp/flows/all_routers";
$flow_bin_directory      = "/usr/bin";
$rrdtool_bin_directory   = "/usr/bin";
$log_directory      = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$log_collector_short= "Y";
$log_grapher_short  = "Y";

Apart from the directories, please note the $FlowViewer_server and $FlowViewer_service parameters, and the @devices array, containing the comma-separated list of routers we already configured in flow-captures.

Now, we have to build the directories used by FlowViewer and, of course, we have to set the needed permissions on them.

Here they are:

mkdir -p /var/www/FlowViewer
chmod -R a=rwx /var/www/FlowViewer
...
do for every directory which needs to be created and set the rights permissions

It just remains to copy some images into the proper directories…

cp Generic_Logo.jpg /var/www/FlowViewer/
cp FlowViewer_Save.png /var/www/FlowViewer/
cp FlowViewer.png /var/www/FlowViewer_Saves/

… and we are ready to use our NetFlow solution!! Simply point your browser at http://your_server_IP_address/cgi-bin/FlowViewer_3.3.1/FlowViewer.cgi

Is that all? No, it isn’t! :)

FlowTracker Grapher and Collector

FlowViewer is not just a CGIs collection, it also includes two programs intended to be ran continously on background: they are FlowTracker_Grapher and FlowTracker_Collector.

Once you defined some trackings in FlowTracker they constantly grab data from flow-capture files, store them in RRD databases and build graphs, like MRTG does with SNMP data. They also send you a warning when traffic goes over the thresholds you defined.

To start them up automatically I edited the flowcap script included within FlowViewer package (/usr/lib/cgi-bin/FlowViewer_3.3.1/flowcap); here it is:

#!/bin/sh
#
# FlowTracker:      Starts all processes concerning FlowTracker
#
# description:  This script starts up the the FlowTracker tools (Collector and Grapher)
#
# processname:  There is not a single process associated with these
#               actions, rather there are multiple processes. This
#               script takes care of all of them.
#
# can be restarted by using the following command:
#
# sudo /etc/init.d/FlowTracker restart

RETVAL=0

start() {
    echo -n $"Starting FlowTracker processes: "

    echo -n "FlowTracker_Collector "

    cd /usr/lib/cgi-bin/FlowViewer_3.3.1
    ./FlowTracker_Collector &> /dev/null &
    RETVAL=$?

    echo -n "FlowTracker_Grapher "

    cd /usr/lib/cgi-bin/FlowViewer_3.3.1
    ./FlowTracker_Grapher &> /dev/null &
    RETVAL=$?

    echo ""
}
stop() {
    echo -n $"Stopping FlowTracker processes: "
    RETVAL=0

    for p in `pidof perl`
    do
        ps $p | grep FlowTracker_Collector > /dev/null
        if [ $? -eq 0 ]; then
            echo -n "FlowTracker_Collector "
            kill $p
        fi

        ps $p | grep FlowTracker_Grapher > /dev/null
        if [ $? -eq 0 ]; then
            echo -n "FlowTracker_Grapher "
            kill $p
        fi
    done

    echo ""
}

# See how we were called.

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    *)
        echo $"Usage: $0 {start|stop|restart}"
        ;;
esac
exit $RETVAL

Then I moved it in the /etc/init.d directory as FlowTracker, made it executable and scheduled it to be executed at startup:

mv flowcap /etc/init.d/FlowTracker
chmod a+x /etc/init.d/FlowTracker
update-rc.d FlowTracker defaults 30

Finally:

/etc/init.d/FlowTracker

Housekeeping

Just to clean some files now and then, add an entry in your crontab file pointing to a cleanup script, like the following one:

cd /usr/lib/cgi-bin/FlowViewer_3.3.1/
/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowViewer_CleanFiles &> /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup.log

Give it execution permissions:

chmod a+x /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Then put it in your crontab:

# m h dom mon dow user  command
5   0    *    *    *     /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Everything is done! Enjoy exploring your network traffic and stay tuned for more NetFlow posts! ;)

References

Cisco.com: Cisco IOS NetFlow

Wikipedia: Netflow

Flow-tools: http://www.splintered.net/sw/flow-tools/

FlowViewer: FlowViewer Web Site and F.A.Q.

Switch.ch: List of NetFlow related software