NetFlow: how to install and configure flow-tools and FlowViewer on a fresh Debian setup

NetFlow is a very useful tool/protocol to monitor network traffic’s patterns. Many tools have been developed to collect and analyze NetFlow data, here I chose flow-tools and FlowViewer packages, and I would like to show how to get them work on a fresh Debian 5.0 (Lenny) setup.

Components used in the solution

There are two main packages involved in the solution: flow-tools, used to acquire and collect NetFlow data generated by our routers, and FlowViewer, used to process, view and paint them on nice graphs.

The flow-tools package is built up by a lot of components, many of them are transparently used by FlowViewer; here I’ll focus on the flow-capture program, the one which acquires and collect the data.

The FlowViewer package is splitted up on 3 CGIs: FlowViewer, FlowGrapher and FlowTracker. They can be used through a web-server and they let us analyze data collected by flow-capture. The package contains 2 programs also, FlowTracker_Collector and FlowTracker_Grapher, which run periodically and build MRTG-like graphs, storing data in RRD databases.

Acquiring data

First off, we need to acquire NetFlow data generated by our routers; flow-tools is the package we need:

apt-get install flow-tools

Let’s edit the main configuration file, flow-capture.conf, where we tell flow-capture what we want to acquire and how we want to store it:

cd /etc/flow-tools
nano flow-capture.conf

Its configuration file is pretty simple; it’s built up by many lines containing the command line arguments of flow-capture: each line is used to run an instance of the program.

You can execute man flow-capture to view all the arguments it accepts; here I used the following example:

# MYROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYROUTER 0.0.0.0/192.168.0.1/3001

# MYSECONDROUTER
-V 5 -E 5G -N 3 -w /var/flows/MYSECONDROUTER 0.0.0.0/192.168.0.2/3002

So, I acquire data from MYROUTER, which sends NetFlow version 5 data from 192.168.0.1; this flow-capture instance will be listening on port 3001 of every local IP address (0.0.0.0) and it will store data on the /var/flows/MYROUTER directory, with a nesting level of type 3, that is directories like /var/flows/MYROUTER/YYYY/YYYY-MM/YYYY-MM-DD/. It will keep files up to a maximum of 5 GB.

This may be a sample configuration for MYROUTER (Cisco), where 192.168.0.9 is the IP address of our NetFlow box:

ip flow-export source FastEthernet0/0.1
ip flow-export version 5
ip flow-export destination 192.168.0.9 3001

interface FastEthernet0/0
 description LAN facing
 no ip address

interface FastEthernet0/0.1
 encapsulation dot1Q 1 native
 ip address 192.168.0.1 255.255.255.0

interface FastEthernet0/1
 description WAN facing
 ip address 10.0.0.1 255.0.0.0
 ip route-cache flow

So on for MYSECONDROUTER…

We just have to build the destination directories and then run the program:

mkdir -p /var/flows/MYROUTER
mkdir -p /var/flows/MYSECONDROUTER

/etc/init.d/flow-capture start

The tool is now working:

ls -l -R /var/flows/MYROUTER/
/var/flows/MYROUTER/:
totale 4
drwxr-xr-x 3 root root 4096  5 mar 10:11 2010

/var/flows/MYROUTER/2010:
totale 4
drwxr-xr-x 3 root root 4096  5 mar 10:11 2010-03

/var/flows/MYROUTER/2010/2010-03:
totale 4
drwxr-xr-x 2 root root 4096  5 mar 10:11 2010-03-05

/var/flows/MYROUTER/2010/2010-03/2010-03-05:
totale 4
-rw-r--r-- 1 root root 92  5 mar 10:11 ft-v05.2010-03-05.101125+0100

FlowViewer

Once we are collecting data using flow-tools we want to analyze them, so we need to get FlowViewer up and running!

Requirements

As we can see from the FlowViewer web site we have to satisfy some requirements in order to run it:

– a web server with CGI support;

– Perl 5.0 or later;

– FlowTools;

– GD and GD:Graph;

– RRDTool.

Let’s start installing them:

apt-get install apache2

apt-get install libgd-graph-perl

apt-get install rrdtool

As we’ll see later in the FlowViewer configuration, it uses also another utility to resolve IP addresses in host names: dig. We have to install the dnsutils Debian package in order to have it:

apt-get install dnsutils

Installation and configuration

Download and untar the package:

cd /usr/local/src
wget http://ensight.eos.nasa.gov/FlowViewer/FlowViewer_3.3.1.tar
tar -xf FlowViewer_3.3.1.tar

Now that all requiremets are met and FlowViewer is on the disk, let’s start configuring it!

In order to get FlowViewer up and running we have to edit its configuration file and build some directories it needs. The user guide provided by the author is very complete, you can find there any information you need. The web site’s FAQ section is very useful too. Here I’ll just provide a basic configuration and layout.

In this sample configuration I use the Apache’s default web site as starting point, so I have:

– / (the root) on /var/www/

– /cgi-bin/ on /usr/lib/cgi-bin/

Let’s move FlowViewer into the cgi-bin directory…

mv FlowViewer_3.3.1 /usr/lib/cgi-bin/

… and edit the configuration file:

cd /usr/lib/cgi-bin/FlowViewer_3.3.1
nano FlowViewer_Configuration.pm

There are a bit of parameters to change, but remember: the user guide is your friend.

Here is the diff of my file against the original:

diff -y --suppress-common-lines -W 250 FlowViewer_Configuration.pm FlowViewer_Configuration.pm.ORIG
$FlowViewer_server       = "192.168.0.9";                                                             |   $FlowViewer_server       = "www.yourcompany.com";     # (IP address or hostname)
$FlowViewer_service      = "http";                                                                    |   $FlowViewer_service      = "https";           # (http, or https)
$reports_directory       = "/var/www/FlowViewer";                                                     |   $reports_directory       = "/htp/htdocs/FlowViewer_3.3.1";
$reports_short           = "/FlowViewer";                                                             |   $reports_short           = "/FlowViewer_3.3.1";
$graphs_directory        = "/var/www/FlowGrapher";                                                    |   $graphs_directory        = "/htp/htdocs/FlowGrapher_3.3.1";
$graphs_short            = "/FlowGrapher";                                                            |   $graphs_short            = "/FlowGrapher_3.3.1";
$tracker_directory       = "/var/www/FlowTracker";                                                    |   $tracker_directory       = "/htp/htdocs/FlowTracker_3.3.1";
$tracker_short           = "/FlowTracker";                                                            |   $tracker_short           = "/FlowTracker_3.3.1";
$cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                       |   $cgi_bin_directory       = "/htp/cgi-bin/FlowViewer_3.3.1";
$work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";                          |   $work_directory          = "/htp/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory          = "/var/www/FlowViewer_Saves";                                               |   $save_directory          = "/htp/htdocs/FlowViewer_Saves";
$names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                       |   $names_directory         = "/htp/cgi-bin/FlowViewer_3.3.1";
$filter_directory        = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Filters"; |   $filter_directory        = "/htp/cgi-bin/FlowTracker_Files/FlowTracker_Filters";
$rrdtool_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRDtool"; |   $rrdtool_directory       = "/htp/cgi-bin/FlowTracker_Files/FlowTracker_RRDtool";
$flow_data_directory     = "/var/flows";                                                              |   $flow_data_directory     = "/htp/flows";
$exporter_directory      = "/var/flows/all_routers";                                                  |   $exporter_directory      = "/htp/flows/all_routers";
$rrdtool_bin_directory   = "/usr/bin";                                                                |   $rrdtool_bin_directory   = "/usr/local/rrdtool-1.2.12/bin";
$trackings_title         = "FlowViewer Saves";                                                        |   $trackings_title         = "Your Company Name";
$user_hyperlink          = "/FlowViewer_Saves";                                                       |   $user_hyperlink          = "http://www.yourcompany.com/";
@devices                 = ("MYROUTER","MYSECONDROUTER");                                             |   @devices                 = ("router_1","router_2","router_3");
$log_directory      = "/usr/lib/cgi-bin/FlowViewer_3.3.1";                                            |   $log_directory      = "/htp/cgi-bin/FlowViewer_3.3.1";

And here is every “directory” or “short” parameter:

cat FlowViewer_Configuration.pm | grep "directory|short"
$reports_directory       = "/var/www/FlowViewer";
$reports_short           = "/FlowViewer";
$graphs_directory        = "/var/www/FlowGrapher";
$graphs_short            = "/FlowGrapher";
$tracker_directory       = "/var/www/FlowTracker";
$tracker_short           = "/FlowTracker";
$cgi_bin_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$cgi_bin_short           = "/cgi-bin/FlowViewer_3.3.1";
$work_directory          = "/usr/lib/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$work_short              = "/cgi-bin/FlowViewer_3.3.1/Flow_Working";
$save_directory          = "/var/www/FlowViewer_Saves";
$save_short              = "/FlowViewer_Saves";
$names_directory         = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$filter_directory        = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_Filters";
$rrdtool_directory       = "/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowTracker_Files/FlowTracker_RRDtool";
$flow_data_directory     = "/var/flows";
$exporter_directory      = "/htp/flows/all_routers";
$flow_bin_directory      = "/usr/bin";
$rrdtool_bin_directory   = "/usr/bin";
$log_directory      = "/usr/lib/cgi-bin/FlowViewer_3.3.1";
$log_collector_short= "Y";
$log_grapher_short  = "Y";

Apart from the directories, please note the $FlowViewer_server and $FlowViewer_service parameters, and the @devices array, containing the comma-separated list of routers we already configured in flow-captures.

Now, we have to build the directories used by FlowViewer and, of course, we have to set the needed permissions on them.

Here they are:

mkdir -p /var/www/FlowViewer
chmod -R YOUR_PERMISSIONS /var/www/FlowViewer
...
do for every directory which needs to be created and set the rights permissions

It just remains to copy some images into the proper directories…

cp Generic_Logo.jpg /var/www/FlowViewer/
cp FlowViewer_Save.png /var/www/FlowViewer/
cp FlowViewer.png /var/www/FlowViewer_Saves/

… and we are ready to use our NetFlow solution!! Simply point your browser at http://your_server_IP_address/cgi-bin/FlowViewer_3.3.1/FlowViewer.cgi

Is that all? No, it isn’t! 🙂

FlowTracker Grapher and Collector

FlowViewer is not just a CGIs collection, it also includes two programs intended to be ran continously on background: they are FlowTracker_Grapher and FlowTracker_Collector.

Once you defined some trackings in FlowTracker they constantly grab data from flow-capture files, store them in RRD databases and build graphs, like MRTG does with SNMP data. They also send you a warning when traffic goes over the thresholds you defined.

To start them up automatically I edited the flowcap script included within FlowViewer package (/usr/lib/cgi-bin/FlowViewer_3.3.1/flowcap); here it is:

#!/bin/sh
#
# FlowTracker:      Starts all processes concerning FlowTracker
#
# description:  This script starts up the the FlowTracker tools (Collector and Grapher)
#
# processname:  There is not a single process associated with these
#               actions, rather there are multiple processes. This
#               script takes care of all of them.
#
# can be restarted by using the following command:
#
# sudo /etc/init.d/FlowTracker restart

RETVAL=0

start() {
    echo -n $"Starting FlowTracker processes: "

    echo -n "FlowTracker_Collector "

    cd /usr/lib/cgi-bin/FlowViewer_3.3.1
    ./FlowTracker_Collector &> /dev/null &
    RETVAL=$?

    echo -n "FlowTracker_Grapher "

    cd /usr/lib/cgi-bin/FlowViewer_3.3.1
    ./FlowTracker_Grapher &> /dev/null &
    RETVAL=$?

    echo ""
}
stop() {
    echo -n $"Stopping FlowTracker processes: "
    RETVAL=0

    for p in `pidof perl`
    do
        ps $p | grep FlowTracker_Collector > /dev/null
        if [ $? -eq 0 ]; then
            echo -n "FlowTracker_Collector "
            kill $p
        fi

        ps $p | grep FlowTracker_Grapher > /dev/null
        if [ $? -eq 0 ]; then
            echo -n "FlowTracker_Grapher "
            kill $p
        fi
    done

    echo ""
}

# See how we were called.

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    *)
        echo $"Usage: $0 {start|stop|restart}"
        ;;
esac
exit $RETVAL

Then I moved it in the /etc/init.d directory as FlowTracker, made it executable and scheduled it to be executed at startup:

mv flowcap /etc/init.d/FlowTracker
chmod a+x /etc/init.d/FlowTracker
update-rc.d FlowTracker defaults 30

Finally:

/etc/init.d/FlowTracker

Housekeeping

Just to clean some files now and then, add an entry in your crontab file pointing to a cleanup script, like the following one:

cd /usr/lib/cgi-bin/FlowViewer_3.3.1/
/usr/lib/cgi-bin/FlowViewer_3.3.1/FlowViewer_CleanFiles &> /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup.log

Give it execution permissions:

chmod a+x /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Then put it in your crontab:

# m h dom mon dow user  command
5   0    *    *    *     /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup

Everything is done! Enjoy exploring your network traffic and stay tuned for more NetFlow posts! 😉

References

Cisco.com: Cisco IOS NetFlow

Wikipedia: Netflow

Flow-tools: http://www.splintered.net/sw/flow-tools/

FlowViewer: FlowViewer Web Site and F.A.Q.

Switch.ch: List of NetFlow related software

The following two tabs change content below.
Italian, born in 1980, I started working in the IT area in the late '90s; I'm now a system and network administrator with a deep knowledge of the global Internet and its core architectures.

24 Comments

  1. ua0ljj says:

    Using FlowViewer_3.3.1

    I doing:

    cat /usr/lib/cgi-bin/FlowViewer_3.3.1/cleanup
    Нет такого файла или каталога

    Where it is script?

    • pierky says:

      Hi,

      the Housekeeping paragraph was a bit confusing, I changed it.

      The cleanup script is not included in FlowViewer, I made it and there I show how I did.
      So, if you want, you have to edit that file and schedule it on your crontab.

      Bye
      Pierky

  2. megezo says:

    Excellent tutorial

    I have an important question I think, about how flow-capture manages disk space.

    What happens when the total size of netflow files reaches the configured limit of 5GB ?
    Will flow-capture stop recording ?

    How do you face this problem ?

    • pierky says:

      Hi Megezo,

      the man page says:

      -E expire_size Retain the maximum number of files so that the total storage is less than expire_size. The letters b,K,M,G can be used as multipliers, ie 16 Megabytes is 16M. Default to 0 (do not expire).

      I never used it in a production environment, so I have not a personal experience, but I guess it clears old flows and keeps capturing new data.

      Pierky

      • Megezo says:

        Hello Pierky,

        Indeed it appears that flow-capture clears old files, although the total size of netflow files exceeds a little bit the “Expiration” parameter.

        Here’s my test:

        1) Set the Expiration parameter to some value:

        vim /etc/flow-tools/flow-capture.conf

        # Pierky’s blog configuration
        -V 5 -E 17500K -N 3 -w /var/flows/MYROUTER 0.0.0.0/192.168.43.1/5502

        2) Display subdirectories total size:

        netflow-collector-2:~# date
        dimanche 28 novembre 2010, 11:08:34 (UTC+0100)
        netflow-collector-2:~#
        netflow-collector-2:~# du -h /var/flows/MYROUTER/2010/2010-11
        840K /var/flows/MYROUTER/2010/2010-11/2010-11-28
        4,9M /var/flows/MYROUTER/2010/2010-11/2010-11-26
        4,3M /var/flows/MYROUTER/2010/2010-11/2010-11-25
        6,7M /var/flows/MYROUTER/2010/2010-11/2010-11-27
        17M /var/flows/MYROUTER/2010/2010-11

        And the same command, a day later. Note that the oldest directory (2010-11-25) has shrinked, and the total size exceeds slightly the Expiration limit:

        netflow-collector-2:~# du -h /var/flows/MYROUTER/2010/2010-11
        6,7M /var/flows/MYROUTER/2010/2010-11/2010-11-28
        3,2M /var/flows/MYROUTER/2010/2010-11/2010-11-26
        4,0K /var/flows/MYROUTER/2010/2010-11/2010-11-25
        6,7M /var/flows/MYROUTER/2010/2010-11/2010-11-27
        1,8M /var/flows/MYROUTER/2010/2010-11/2010-11-29
        19M /var/flows/MYROUTER/2010/2010-11

    • pierky says:

      Hi Megezo,

      many thanks for sharing your experience on the blog!

      Pierky

  3. Megezo says:

    Well, thank you for this excellent blog 🙂

    Megezo

  4. dano says:

    can you setup NetFlowViewer on CentOS-5.5 ? thank before

    • pierky says:

      Hi,

      I never installed it on CentOS but I think you can do it, using flow-tools and FlowViewer source code.

      Bye

      • dano says:

        I already try to do it but when i install flow-tools after i start the service i saw the log file unlink (/var/run/flow-captuer.pid.8818) Permission denied.
        I try to fix this problem but can not do you any idea ?
        Thank for your reply.

  5. Hook says:

    Be careful when changing directory permissions. Your code will include /usr/bin (since it’s in the config file) which you don’t want to modify.


    cat FlowViewer_Configuration.pm | grep directory | awk -F " '{print $2}'

    Thank you for documenting the FlowViewer configuration.

  6. Lay says:

    Hi Pierky,

    I am installing FlowViewer from http://ensight.eos.nasa.gov/FlowViewer/ on Ubuntu 64-bit edition and I can now access FlowViewer via my browser http://10.6.192.97/cgi-bin/FlowViewer_3.4/FlowViewer.cgi but it doesn’t show any outputs.

    Could you please help me advise the possible cause of why FlowViewer can’t seem to read flow data while Flow Capture seems to be receiving flows from the router?

    setup@ubuntu:~$ ls -l -R /var/flows/MYROUTER/
    /var/flows/MYROUTER/:
    total 4
    drwxrwxrwx 3 root root 4096 2011-08-24 20:17 2011

    /var/flows/MYROUTER/2011:
    total 4
    drwxrwxrwx 3 root root 4096 2011-08-24 20:17 2011-08

    /var/flows/MYROUTER/2011/2011-08:
    total 4
    drwxrwxrwx 2 root root 4096 2011-08-24 23:14 2011-08-24

    /var/flows/MYROUTER/2011/2011-08/2011-08-24:
    total 18700
    -rwxrwxrwx 1 root root 1566957 2011-08-24 20:30 ft-v05.2011-08-24.201715-0700
    -rwxrwxrwx 1 root root 1904374 2011-08-24 20:45 ft-v05.2011-08-24.203000-0700
    -rwxrwxrwx 1 root root 1810963 2011-08-24 21:00 ft-v05.2011-08-24.204500-0700
    -rw-r–r– 1 root root 1649026 2011-08-24 21:15 ft-v05.2011-08-24.210000-0700
    -rw-r–r– 1 root root 1493551 2011-08-24 21:30 ft-v05.2011-08-24.211500-0700
    -rw-r–r– 1 root root 1517829 2011-08-24 21:45 ft-v05.2011-08-24.213000-0700
    -rw-r–r– 1 root root 1530559 2011-08-24 22:00 ft-v05.2011-08-24.214500-0700
    -rw-r–r– 1 root root 1557358 2011-08-24 22:15 ft-v05.2011-08-24.220000-0700
    -rw-r–r– 1 root root 1588123 2011-08-24 22:30 ft-v05.2011-08-24.221500-0700
    -rw-r–r– 1 root root 1580850 2011-08-24 22:45 ft-v05.2011-08-24.223000-0700
    -rw-r–r– 1 root root 1641061 2011-08-24 23:00 ft-v05.2011-08-24.224500-0700
    -rw-r–r– 1 root root 92738 2011-08-24 23:00 ft-v05.2011-08-24.230000-0700
    -rw-r–r– 1 root root 1162143 2011-08-24 23:14 ft-v05.2011-08-24.230303-0700
    -rw-r–r– 1 root root 16468 2011-08-24 23:14 tmp-v05.2011-08-24.231405-0700
    setup@ubuntu:~$

    Thank you very much in advance for your kind assistance.

    Regards,
    Lay

  7. Yogesh Sathe says:

    Hi,

    I find this tool useful to collect data from brach routers. (all routers are Cisco routers)
    But for some routers, I am getting the data in /var/flows/ in branch folders directory when seen through ls- l – R command but actually it dont show data in flowviewer tool even after selecting appropriate options I am bigineer in linux and started with Ubuntu desktop 10.04 lucid.

    For other routers I am able to see the netflow version 5 data.

    Please advice.

  8. Vaquzanf says:

    Do you know the address? http://uorysisory.de.tl bd company nymphets I know some working girls would like doing something like this instead of fucking. Personally, I’d rather get fucked and move on to the next.

  9. Luca Maranzano says:

    Please Please Please read CAREFULLY this command:

    for d in `cat FlowViewer_Configuration.pm | grep directory | awk -F ” ‘{print $2}’`; do chmod -R a=rwx $d/; done

    that file in my configuration contains directories like these:
    /etc
    /usr/bin

    so you are going to do a command like this:
    chmod -R a=rwx /etc
    chmod -R a=rwx /usr/bin
    ……

    YOU ARE GOING TO BREAK AND MESS UP YOUR ENTIRE SYSTEM!

    Besides, why keep using an insanely insecure permission model like this ??

    Good Luck

  10. Jignesh Shah says:

    I have installed FlowvIewer 4.0 and want to capture v9 packet from cisco asa 5500 device. Currently I have configured V 5 and flow capture is started but I cant see any data in FLowViewer, FlowTracker and FlowGrapher. Can please let me know the configuration of Flow-capture to capture v9 packets.

  11. […] the brief overview about the installation of flow-tools and FlowViewer, in this post I’d like to share my experience about the setup of a basic solution based on […]

  12. Jay says:

    Thanks Luca for your warning about the permissions. I just wish I had read all the comments BEFORE I did it. It did indeed mess up my entire server but thankfully it was a VPS with snapshot backups so I was able to revert back to a working state.

    But I do agree, 777 on anything is a BAD IDEA.

  13. Netadmin says:

    When I generate the repor, ntothing to see .

    It say sh: 1: /usr/local/flow-tools/bin/flow-stat: not found.

    How can I solve ?

  14. Joseph Kim says:

    mille grazie ! Era molto utile

  15. bumblebee says:

    hi pierky,

    thanks for your post. i was able to install on ubuntu but i do not see output in the files generated.

    flow-print -f5 < /var/flows/myrouter/2015/2015-09/2015-09-13/ft-v05.2015-09-13.230001-0700
    Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets

    also i am not getting 404 not found for the gui
    http://server_ip/cgi-bin/FlowViewer_4.6/FlowViewer.cgi

    thanks

  16. cbiswas says:

    I’ve installed in Ubuntu but not seeing anything in gui

Leave a Reply