NetFlow is a very useful tool/protocol to monitor network traffic’s patterns. Many tools have been developed to collect and analyze NetFlow data, here I chose flow-tools and FlowViewer packages, and I would like to show how to get them work on a fresh Debian 5.0 (Lenny) setup.

Components used in the solution

There are two main packages involved in the solution: flow-tools, used to acquire and collect NetFlow data generated by our routers, and FlowViewer, used to process, view and paint them on nice graphs.

The flow-tools package is built up by a lot of components, many of them are transparently used by FlowViewer; here I’ll focus on the flow-capture program, the one which acquires and collect the data.

The FlowViewer package is splitted up on 3 CGIs: FlowViewer, FlowGrapher and FlowTracker. They can be used through a web-server and they let us analyze data collected by flow-capture. The package contains 2 programs also, FlowTracker_Collector and FlowTracker_Grapher, which run periodically and build MRTG-like graphs, storing data in RRD databases.

Acquiring data

First off, we need to acquire NetFlow data generated by our routers; flow-tools is the package we need:

apt-get install flow-tools

Let’s edit the main configuration file, flow-capture.conf, where we tell flow-capture what we want to acquire and how we want to store it:

cd /etc/flow-tools
nano flow-capture.conf

Its configuration file is pretty simple; it’s built up by many lines containing the command line arguments of flow-capture: each line is used to run an instance of the program.

You can execute man flow-capture to view all the arguments it accepts; here I used the following example:

# MYROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYROUTER 0.0.0.0/192.168.0.1/3001

# MYSECONDROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYSECONDROUTER 0.0.0.0/192.168.0.2/3002

So, I acquire data from MYROUTER, which sends NetFlow version 5 data from 192.168.0.1; this flow-capture instance will be listening on port 3001 of every local IP address (0.0.0.0) and it will store data on the /var/flows/MYROUTER directory, with a nesting level of type 3, that is directories like /var/flows/MYROUTER/YYYY/YYYY-MM/YYYY-MM-DD/. It will keep files up to a maximum of 5 GB.

This may be a sample configuration for MYROUTER (Cisco), where 192.168.0.9 is the IP address of our NetFlow box:

ip flow-export source FastEthernet0/0.1
ip flow-export version 5
ip flow-export destination 192.168.0.9 3001

interface FastEthernet0/0
 description LAN facing
 no ip address

interface FastEthernet0/0.1
 encapsulation dot1Q 1 native
 ip address 192.168.0.1 255.255.255.0

interface FastEthernet0/1
 description WAN facing
 ip address 10.0.0.1 255.0.0.0
 ip route-cache flow

So on for MYSECONDROUTER…

We just have to build the destination directories and then run the program:

mkdir -p /var/flows/MYROUTER
mkdir -p /var/flows/MYSECONDROUTER

/etc/init.d/flow-capture start

The tool is now working:

ls -l -R /var/flows/MYROUTER/
/var/flows/MYROUTER/:
totale 4
drwxr-xr-x 3 root root 4096  5 mar 10:11 2010

/var/flows/MYROUTER/2010:
totale 4
drwxr-xr-x 3 root root 4096  5 mar 10:11 2010-03

/var/flows/MYROUTER/2010/2010-03:
totale 4
drwxr-xr-x 2 root root 4096  5 mar 10:11 2010-03-05

/var/flows/MYROUTER/2010/2010-03/2010-03-05:
totale 4
-rw-r--r-- 1 root root 92  5 mar 10:11 ft-v05.2010-03-05.101125+0100

FlowViewer

Once we are collecting data using flow-tools we want to analyze them, so we need to get FlowViewer up and running!

Requirements

As we can see from the FlowViewer web site we have to satisfy some requirements in order to run it:

– a web server with CGI support;

– Perl 5.0 or later;

– FlowTools;

– GD and GD:Graph;

– RRDTool.

Let’s start installing them:

apt-get install apache2

apt-get install libgd-graph-perl

apt-get install rrdtool

As we’ll see later in the FlowViewer configuration, it uses also another utility to resolve IP addresses in host names: dig. We have to install the dnsutils Debian package in order to have it:

apt-get install dnsutils

Installation and configuration

Download and untar the package:

cd /usr/local/src
wget http://ensight.eos.nasa.gov/FlowViewer/FlowViewer_3.3.1.tar
tar -xf FlowViewer_3.3.1.tar

Now that all requiremets are met and FlowViewer is on the disk, let’s start configuring it!

In order to get FlowViewer up and running we have to edit its configuration file and build some directories it needs. The user guide provided by the author is very complete, you can find there any information you need. The web site’s FAQ section is very useful too. Here I’ll just provide a basic configuration and layout.

In this sample configuration I use the Apache’s default web site as starting point, so I have:

– / (the root) on /var/www/

– /cgi-bin/ on /usr/lib/cgi-bin/

Let’s move FlowViewer into the cgi-bin directory…

mv FlowViewer_3.3.1 /usr/lib/cgi-bin/

… and edit the configuration file:

cd /usr/lib/cgi-bin/FlowViewer_3.3.1
nano FlowViewer_Configuration.pm

There are a bit of parameters to change, but remember: the user guide is your friend.

Here is the diff of my file against the original:

diff -y --suppress-common-lines -W 250 FlowViewer_Configuration.pm FlowViewer_Configuration.pm.ORIG
$FlowViewer_server       = "192.168.0.9";                                                             |   $FlowViewer_server       = "www.yourcompany.com";     # (IP address or hostname)
$FlowViewer_service      = "http";                                                                    |   $FlowViewer_service      = "https";           # (http, or https)
$reports_directory       = "/var/www/FlowViewer";                                                     |   $reports_directory       = "/htp/htdocs/FlowViewer_3.3.1";
$reports_short           = "/FlowViewer";                                                             |   $reports_short           = "/FlowViewer_3.3.1";
$graphs_directory        = "/var/www/FlowGrapher";                                                    |   $graphs_directory        = "/htp/htdocs/FlowGrapher_3.3.1";
$graphs_short            = "/FlowGrapher";                                                            |   $graphs_short            = "/FlowGrapher_3.3.1";
$tracker_directory       = "/var/www/FlowTracker";                                                    |   $tracker_directory       = "/htp/htdocs/FlowTracker_3.3.1";
$tracker_short           = "/FlowTracker";                                                            |   $tracker_short           = "/FlowTracker_3.3.1";
$cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                       |   $cgi_bin_directory       = "/htp/cgi-bin/FlowViewer_3.3.1";
$work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";                          |   $work_directory          = "/htp/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory          = "/var/www/FlowViewer_Saves";                                               |   $save_directory          = "/htp/htdocs/FlowViewer_Saves";
$names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                       |   $names_directory         = "/htp/cgi-bin/FlowViewer_3.3.1";
$filter_directory        = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Filters"; |   $filter_directory        = "/htp/cgi-bin/FlowTracker_Files/FlowTracker_Filters";
$rrdtool_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRDtool"; |   $rrdtool_directory       = "/htp/cgi-bin/FlowTracker_Files/FlowTracker_RRDtool";
$flow_data_directory     = "/var/flows";                                                              |   $flow_data_directory     = "/htp/flows";
$exporter_directory      = "/var/flows/all_routers";                                                  |   $exporter_directory      = "/htp/flows/all_routers";
$rrdtool_bin_directory   = "/usr/bin";                                                                |   $rrdtool_bin_directory   = "/usr/local/rrdtool-1.2.12/bin";
$trackings_title         = "FlowViewer Saves";                                                        |   $trackings_title         = "Your Company Name";
$user_hyperlink          = "/FlowViewer_Saves";                                                       |   $user_hyperlink          = "http://www.yourcompany.com/";
@devices                 = ("MYROUTER","MYSECONDROUTER");                                             |   @devices                 = ("router_1","router_2","router_3");
$log_directory      = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                            |   $log_directory      = "/htp/cgi-bin/FlowViewer_3.3.1";

And here is every “directory” or “short” parameter:

cat FlowViewer_Configuration.pm | grep "directory|short"
$reports_directory       = "/var/www/FlowViewer";
$reports_short           = "/FlowViewer";
$graphs_directory        = "/var/www/FlowGrapher";
$graphs_short            = "/FlowGrapher";
$tracker_directory       = "/var/www/FlowTracker";
$tracker_short           = "/FlowTracker";
$cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$cgi_bin_short           = "/cgi-bin/FlowViewer_3.3.1";
$work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$work_short              = "/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory          = "/var/www/FlowViewer_Saves";
$save_short              = "/FlowViewer_Saves";
$names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$filter_directory        = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Filters";
$rrdtool_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRDtool";
$flow_data_directory     = "/var/flows";
$exporter_directory      = "/htp/flows/all_routers";
$flow_bin_directory      = "/usr/bin";
$rrdtool_bin_directory   = "/usr/bin";
$log_directory      = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$log_collector_short= "Y";
$log_grapher_short  = "Y";

Apart from the directories, please note the $FlowViewer_server and $FlowViewer_service parameters, and the @devices array, containing the comma-separated list of routers we already configured in flow-captures.

Now, we have to build the directories used by FlowViewer and, of course, we have to set the needed permissions on them.

Here they are:

mkdir -p /var/www/FlowViewer
chmod -R YOUR_PERMISSIONS /var/www/FlowViewer
...
do for every directory which needs to be created and set the rights permissions

It just remains to copy some images into the proper directories…

cp Generic_Logo.jpg /var/www/FlowViewer/
cp FlowViewer_Save.png /var/www/FlowViewer/
cp FlowViewer.png /var/www/FlowViewer_Saves/

… and we are ready to use our NetFlow solution!! Simply point your browser at http://your_server_IP_address/cgi-bin/FlowViewer_3.3.1/FlowViewer.cgi

Is that all? No, it isn’t! 🙂

FlowTracker Grapher and Collector

FlowViewer is not just a CGIs collection, it also includes two programs intended to be ran continously on background: they are FlowTracker_Grapher and FlowTracker_Collector.

Once you defined some trackings in FlowTracker they constantly grab data from flow-capture files, store them in RRD databases and build graphs, like MRTG does with SNMP data. They also send you a warning when traffic goes over the thresholds you defined.

To start them up automatically I edited the flowcap script included within FlowViewer package (/usr/lib/cgi-bin/FlowViewer_3.3.1/flowcap); here it is:

#!/bin/sh
#
# FlowTracker:      Starts all processes concerning FlowTracker
#
# description:  This script starts up the the FlowTracker tools (Collector and Grapher)
#
# processname:  There is not a single process associated with these
#               actions, rather there are multiple processes. This
#               script takes care of all of them.
#
# can be restarted by using the following command:
#
# sudo /etc/init.d/FlowTracker restart

RETVAL=0

start() {
    echo -n $"Starting FlowTracker processes: "

    echo -n "FlowTracker_Collector "

    cd /usr/lib/cgi-bin/FlowViewer_3.3.1
    ./FlowTracker_Collector &> /dev/null &
    RETVAL=$?

    echo -n "FlowTracker_Grapher "

    cd /usr/lib/cgi-bin/FlowViewer_3.3.1
    ./FlowTracker_Grapher &> /dev/null &
    RETVAL=$?

    echo ""
}
stop() {
    echo -n $"Stopping FlowTracker processes: "
    RETVAL=0

    for p in `pidof perl`
    do
        ps $p | grep FlowTracker_Collector > /dev/null
        if [ $? -eq 0 ]; then
            echo -n "FlowTracker_Collector "
            kill $p
        fi

        ps $p | grep FlowTracker_Grapher > /dev/null
        if [ $? -eq 0 ]; then
            echo -n "FlowTracker_Grapher "
            kill $p
        fi
    done

    echo ""
}

# See how we were called.

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    *)
        echo $"Usage: $0 {start|stop|restart}"
        ;;
esac
exit $RETVAL

Then I moved it in the /etc/init.d directory as FlowTracker, made it executable and scheduled it to be executed at startup:

mv flowcap /etc/init.d/FlowTracker
chmod a+x /etc/init.d/FlowTracker
update-rc.d FlowTracker defaults 30

Finally:

/etc/init.d/FlowTracker

Housekeeping

Just to clean some files now and then, add an entry in your crontab file pointing to a cleanup script, like the following one:

cd /usr/lib/cgi-bin/FlowViewer_3.3.1/
/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowViewer_CleanFiles &> /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup.log

Give it execution permissions:

chmod a+x /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Then put it in your crontab:

# m h dom mon dow user  command
5   0    *    *    *     /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Everything is done! Enjoy exploring your network traffic and stay tuned for more NetFlow posts! 😉

References

Cisco.com: Cisco IOS NetFlow

Wikipedia: Netflow

Flow-tools: http://www.splintered.net/sw/flow-tools/

FlowViewer: FlowViewer Web Site and F.A.Q.

Switch.ch: List of NetFlow related software

The following two tabs change content below.

• Bio
• Latest Posts

Pier Carlo Chiodi

Italian, born in 1980, I started working in the IT/telecommunications industry in the late '90s; I'm now a system and network engineer with a deep knowledge of the global Internet and its core architectures, and a strong focus on network automation.

Latest posts by Pier Carlo Chiodi (see all)

• Good MANRS for IXPs route servers made easier - 11 December 2020
• Route server feature-rich and automatic configuration - 13 February 2017
• Large BGP Communities playground - 15 September 2016