Tag Archives: Provider

DNS-amplification attack reflection on backhaul circuit

As many of us already know, DNS amplification attacks are a big plague for who fights every day for the sake of Internet security and service availability.

Infected hosts are instructed by botnet controllers to send DNS queries to recursive open resolvers, asking them for big zones with spoofed UDP packets containing the victim’s IP address in the source field, so that a small request would generate a big traffic toward the victim.

Small efforts are needed in order to mitigate those attacks – a proper DNS resolvers configuration to avoid open recursion, IP source validation (such as Cisco uRPF) to block source IP spoofing at the access network layer – but they may not be sufficient to immunize a network against annoying issues.

An unpleasant side effect

Even on secure networks an unpleasant side effect may occur: attack reflection against infected hosts, with the consequent backhaul circuit saturation and users’ downstream degradation.

Take, for example, the following not uncommon scenario:

DNS Amplification Reflection - Scenario

An ISP, running a properly configured DNS resolver, connects many users with a shared backhaul link between its core network and a local metro area; one or more users have infected devices responding to a botnet C&C server who aims to launch a DDoS against a given target.

A well implemented network access layer would stop spoofed packets whose source IP can not be reached through the same link on which they came from. At the same time a properly configured DNS resolver would not let recursive queries to go on by untrusted sources. The problem raises when proper DNS queries came in from trusted users and go to the ISP DNS resolver.

A not-really-failed attack attempt

Failed Attack

In the above diagram, at step 1, the botnet controller instructs the infected host to start a DNS amplification attack against the victim’s IP address 1.2.3.4. In the step 2 the malicious software tries to send a spoofed packet containing the victim’s address in the source field but something goes wrong: the operating system doesn’t let the malware to forge such a packet and rewrites it using its LAN address, or the router/firewall/CPE changes it with the WAN IP address (NAT). Anyway, at step 3, a proper DNS query comes out the user’s network and heads to the ISP DNS resolver, which in turn sends back a response with the huge DNS zone (step 4).

It’s easy to understand how this behaviour could lead to ISP internal issues regarding the backhaul link saturation and the users experience’s deterioration.

Consequences

A small upstream user’s query (65 bytes for an ANY query on isc.org) produces a big downstream response (~ 4 KB for isc.org zone), with a ~ 60x multiplicative factor. Every infected host may send many and many queries over a long period, even more than 1 query per second for many days, and many compromised hosts may be triggered at the same time by the same botnet controller.

Backhaul links may be rent from incumbent local carriers and may be characterized by an overbooking ratio calculated over the expected usage by customers who share them; high speed links which connect DNS resolvers to the core may overwhelm them when filled by UDP response packets and lead to traffic stagnation because of traffic policing operated by the carrier.

Customers also may report a bad user experience: it’s true, their links are operating at 100% of their capacity, but Facebook is slow and the VoIP is unusable.

A very big headache, even for an ISP with a properly configured network.

Symptoms

The first symptom that can be observed is an abnormal peak in resolvers bandwidth usage:

DNS resolver bandwidth usage during an attack attempt - response traffic in green

DNS resolver bandwidth usage during an attack attempt – response traffic in green

During an attack attempt the network usage (servers’ upstream) may raise up to hundreds of times higher than average.

NetFlow also may help us to identify this kind of traffic; big response UDP datagrams may be fragmented over the network and they would be shown as port-0 UDP packets in the output of nfdump or similar tools, with an high Bpp (bytes-per-packet) ratio:

Proto Src IP Addr:Port  Dst IP Addr:Port   Packets    Bytes  pps     bps   Bpp Flows
UDP    RESOLVER_1:0   ->  A.B.1.155:0        78966  106.6 M   48  519300  1350    79
UDP    RESOLVER_1:0   ->   G.H.4.73:0        35798   48.3 M   25  274100  1350    38
UDP    RESOLVER_1:53  ->  I.J.5.101:14068     7430    9.3 M    4   46712  1249   187
A 65-bytes request generated a 4157 bytes response in 3 segments - calculated at IP level

A 65-bytes request generated a 4157 bytes response in 3 segments – calculated at IP level

Mitigation

Unfortunately, as far as I know, there are still no specific implementations aimed to mitigate those kind of attack.

BIND9 has a generic rate-limit option which prevents a requestor to be told the same answer more than a specific number of times within a one-second interval, but there is no way to apply it only to a subset of responses (like the ones used in DDoS attack, such as ANY to isc.org or ripe.net). DNS RRL (Response Rate Limiting) is focused on authoritative servers, not on recursive ones.

A suitable way would be the use of the iptables recent module on recursive resolvers, but other aspects have to be considered, such as servers load and performances degradation.
A first deep-packet inspection of the incoming DNS requests would filter those DNS queries whose type has been set to ANY, then the recent module would lookup the source IP address on a local list and drop the packet if it violates the predetermined policy. For example, a policy may allow one or two queries with type = ANY every 5 seconds, so that “regular” usage would be allowed while malware initiated traffic would be dropped within few seconds.

Number of different IP addresses on the recent module's queue - peak during an attack attemp

Number of different IP addresses on the recent module’s queue – peak during an attack attemp

References

“Alert (TA13-088A) DNS Amplification Attacks”, US-Cert: http://www.us-cert.gov/ncas/alerts/TA13-088A

“DNS Response Rate Limiting (DNS RRL)”, Paul Vixie, ISC – Vernon Schryver, Rhyolite: http://ss.vix.su/~vixie/isc-tn-2012-1.txt

Graphing near realtime PPPoE/PPPoA link speed using SNMP Traffic Grapher (STG to its friends)

Sometimes it happens to me that, for troubleshooting reasons, I need to graph PPPoE or PPPoA connections speed from the NAS/BRAS side. These links are terminated on Cisco routers, where other hundreds of CPEs are connected; connections are from dialin users and I can’t have static graphs, mostly because I don’t need endusers monitoring on a fulltime basis and it would only be a huge waste of resources.

In this case a little program helps me: STG, SNMPTrafficGrapher.

STG - SNMPTrafficGrapher

STG – SNMPTrafficGrapher

It’s a small Windows utility that uses SNMP to get counters data and put them on a graph, like MRTG does. It’s easy and fast to deploy (run it, set SNMP OID and it’s ready), does not use many resources and can give you graphs updated every second.

Configuration

From the View / Settings menu you just have to set the device’s IP address and SNMP community, and then to select OID and polling frequency.

As said, users have dialin connections which go up and down and there is no way to predict their SNMP interface’s index; to obtain the right OID we can use the show snmp mib ifmib ifindex command.

Initially we get the actual Virtual-Access interface for the user we need to monitor:

Router#sh users | include MyUserName
  Vi1.195      MyUserName	   PPPoATM      -        10.11.12.13

Then we get it’s SNMP index:

Router#show snmp mib ifmib ifindex Virtual-Access 1.195
Interface = Virtual-Access1.195, Ifindex = 257

And finally we can use it to configure STG:

STG setup

Geen OID = 1.3.6.1.2.1.2.2.1.10.257 (ifInOctets.257)
Blue OID = 1.3.6.1.2.1.2.2.1.16.257 (ifOutOctets.257)

Where 257 is the dynamic SNMP ifIndex of our user’s Virtual-Access interface.

References

STG homepage: http://leonidvm.chat.ru/

GNS-3 Lab: 4-byte Autonomous System Number – interactions between 32 and 16 bit speaking BGP routers

Such as IPv4 addresses, 2-byte pool of Autonomous System (AS) Numbers is going to be exhausted soon. By the time I’m writing, it seems Internet will run out of 16-bit AS Number on 26-Sep-2012. To prevent this situation, IANA extended the AS Number field to 32 bits and, on 2007, RIRs started assigning them… or, at least, to offer them!
Of course, in order to let 4-byte ASN to work properly, a little adjustement was needed in the BGP protocol. Enter RFC 4893, BGP Support for Four-octet AS Number Space.

4-byte ASN - Lab topologyIn this post I don’t want to cover the mechanisms of this protocol extension, if you want to understand them I please you to follow some links in the References section; I prefer to show some interactions between OLD 2-byte ASN and NEW 4-byte ASN BGP speakers. I built a little GNS3/Dynamips lab using a couple of 7200s with IOS 12.2(33)SRE as NEW speakers, and a couple of 3640s as OLD peers.

In the topology every router announces any subnet to the others; BGP peering sessions follow the physical topology. Green ASs support 4-byte AS, while gray do not.
For the sake of readibility I’ll use asdot notation in this post, with the exception of some configuration blocks where I’ll use asplain just to show both notation usage.

I also uploaded two packet captures on PacketLife.net: they show BGP UPDATES with and without the NEW_AS_PATH attributes. They are 4-byte_AS_numbers_Full_Support.cap and 4-byte_AS_numbers_Mixed_Scenario.cap.

Partial 4-byte ASN support: A-B routers

Router A is on AS 10.1 / 655361 (asdot / asplain notation):

A#sh run | sec bgp
router bgp 655361
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 neighbor 172.16.3.2 remote-as 2
 no auto-summary
B#sh run | sec bgp
router bgp 2
 no synchronization
 bgp log-neighbor-changes
 network 20.0.0.0
 neighbor 172.16.3.1 remote-as 23456
 no auto-summary

As you can see, on router B neighbor statement we can’t use the real ASN to peer with A, so we have to use the AS_TRANS (23456). Let’s wait until our sessions come up, then show BGP neighbors and tables:

A#sh ip bgp neighbors 172.16.3.2
BGP neighbor is 172.16.3.2,  remote AS 2, external link
  BGP version 4, remote router ID 20.0.0.1
  BGP state = Established, up for 00:00:27
  Last read 00:00:27, last write 00:00:27, hold time is 180, keepalive interval is 60 seconds
  Neighbor sessions:
    1 active, is not multisession capable
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised
    Address family IPv4 Unicast: advertised and received
    Multisession Capability: advertised
...

Please note the Four-octets ASN Capability: advertised line: A advertised this capability but did not receive it back from B.

A#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         0.0.0.0                  0         32768 i
*> 20.0.0.0         172.16.3.2               0             0 2 i

As expected, A receives the B‘s route but…

B#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.3.1               0             0 23456 i
*> 20.0.0.0         0.0.0.0                  0         32768 i

B only sees A‘s subnet as from AS_TRANS (23456).

This is because B does not support 4-byte ASN, and router A knows this (capabilities exchange during session setup), so it just sends AS_PATH attributes containing the 16-bit AS_TRANS in place of the real 32-bit AS number.

You can find a similar UPDATE on the first capture I sent on PacketLife.net (4-byte_AS_numbers_Mixed_Scenario.cap), packet number 2.

Full 4-byte ASN support: A-D routers

Let’s setup the BGP session between routers A and D; they both support 4-byte ASN.

A#sh run | sec bgp
router bgp 655361
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 neighbor 172.16.1.2 remote-as 2621441
 neighbor 172.16.3.2 remote-as 2
 no auto-summary
D#sh run | sec bgp
router bgp 2621441
 no synchronization
 bgp log-neighbor-changes
 network 40.0.0.0
 neighbor 172.16.1.1 remote-as 655361
 no auto-summary

When sessions come up…

A#sh ip bgp neighbors 172.16.1.2
BGP neighbor is 172.16.1.2,  remote AS 40.1, external link
  BGP version 4, remote router ID 40.0.0.1
  BGP state = Established, up for 00:04:04
  Last read 00:00:46, last write 00:00:53, hold time is 180, keepalive interval is 60 seconds
  Neighbor sessions:
    1 active, is multisession capable
  Neighbor capabilities:
    Route refresh: advertised and received(new)
    Four-octets ASN Capability: advertised and received
    Address family IPv4 Unicast: advertised and received
    Multisession Capability: advertised and received
...

Here A both sent and received the 4-byte ASN capability to/by B, because it supports this feature.

A#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         0.0.0.0                  0         32768 i
*> 20.0.0.0         172.16.3.2               0             0 2 i
*> 40.0.0.0         172.16.1.2               0             0 2621441 i
D#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.1.1               0             0 655361 i
*> 20.0.0.0         172.16.1.1                             0 655361 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i

Both A and D have subnets with the full 32-bit AS number.

You can see the capture on the file 4-byte_AS_numbers_Full_Support.cap on PacketLife.net.

As I already said, let’s switch to asdot notation now, just to improve readibility:

D#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
D(config)#router bgp 40.1
D(config-router)#bgp asnotation dot
D(config-router)#do show ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.1.1               0             0 10.1 i
*> 20.0.0.0         172.16.1.1                             0 10.1 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i
A#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
A(config)#router bgp 10.1
A(config-router)#bgp asnotation dot

Ok, 10.1 is better than 655361, and 40.1 than 2621441!! :)

Mixed scenario: D-B routers

Let’s have more fun now and bring D-B session up!

D(config-router)#neighbor 172.16.2.1 remote-as 2
B(config-router)#neighbor 172.16.2.2 remote-as 23456

As we already saw, B doesn’t support 4-byte ASN, so we have to use AS_TRANS to peer with D.

D#show ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*  10.0.0.0         172.16.2.1                             0 2 10.1 i
*>                  172.16.1.1               0             0 10.1 i
*> 20.0.0.0         172.16.2.1               0             0 2 i
*                   172.16.1.1                             0 10.1 2 i
*> 40.0.0.0         0.0.0.0                  0         32768 i

The interesting thing we can see now on D is the presence of A‘s subnet (10.0.0.0) advertised by B: even if B does not support 4-octect ASN, D receives it with the real 32-bit path. When A advertises the subnet to B it makes the AS_PATH attribute up using AS_TRANS, but it also adds the transitive attribute NEW_AS_PATH, containing the full 32-bit AS numbers; when B advertises the subnet to D it adds this attribute and so D can rebuild the real 32-bit path.

As before, you can find a similar UPDATE on the capture I sent on PacketLife.net; the file is 4-byte_AS_numbers_Mixed_Scenario.cap, packet number 3.

Another interesting aspect of 4-byte AS lack of support can be seen in the following table:

B#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*  10.0.0.0         172.16.2.2                             0 23456 23456 i
*>                  172.16.3.1               0             0 23456 i
*> 20.0.0.0         0.0.0.0                  0         32768 i
*> 40.0.0.0         172.16.2.2               0             0 23456 i
*                   172.16.3.1                             0 23456 23456 i

Here B has two entries for the 10.0.0.0 subnet: they both seem to be originated from the same AS, but that’s not true: the first comes from D (AS 40.1), while the second (the selected one) comes directly from A.

Similar behaviour is for the 40.0.0.0 subnet. Of course, this is not a really big problem, but may lead to false assumptions and compromise traffic patterns analysis.

16-bit stub router

Just to complete all the possible scenarios, here is C configuration:

B#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
B(config)#router bgp 2
B(config-router)#neighbor 172.16.4.2 remote-as 3
C#sh run | sec bgp
router bgp 3
 no synchronization
 bgp log-neighbor-changes
 network 30.0.0.0
 neighbor 172.16.4.1 remote-as 2
 no auto-summary
C#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0         172.16.4.1                             0 2 23456 i
*> 20.0.0.0         172.16.4.1               0             0 2 i
*> 30.0.0.0         0.0.0.0                  0         32768 i
*> 40.0.0.0         172.16.4.1                             0 2 23456 i

It has no idea of what 4-byte ASN are, so it sees both 10.0.0.0 and 40.0.0.0 subnets as originated from the same AS.

Thanks

I have to say thanks to Marco Rizzi for his kind support on helping me to build this lab! ;-) And, of course, I suggest you to visit his blog: Marco Rizzi Blog – networking with passion!!!

Download the lab

To download the lab and configuration files click here.

References

Report on consumption of AS Numbers: http://www.potaroo.net/tools/asns/

BGP Support for Four-octet AS Number Space: RFC 4893

APRICOT 2007: 4-Byte AS Numbers (PDF)

Cisco.com: Migration Guide for Explaining 4-Byte Autonomous System

Cisco.com: Cisco IOS BGP 4-Byte ASN Support

Cisco.com: Explaining 4-Byte Autonomous System (AS) ASPLAIN and ASDOT Notation for Cisco IOS

NetFlow: installation and configuration of NFDUMP and NfSen on Debian

After the brief overview about the installation of flow-tools and FlowViewer, in this post I’d like to share my experience about the setup of a basic solution based on another pair of tools: NFDUMP and NfSen. As always on my posts, the starting point is a fresh Debian 5.0 setup.

UPDATE: you may be interested in FlowGraph too, a tool that allows to dynamically build graphs based on previously collected netflow data and to use them in a web-based front-end, adding details about Autonomous System Number holders, IPv4 and IPv6 prefixes, inet(6)num objects, netnames from RIPE Stat.

Read more …

NetFlow: how to install and configure flow-tools and FlowViewer on a fresh Debian setup

NetFlow is a very useful tool/protocol to monitor network traffic’s patterns. Many tools have been developed to collect and analyze NetFlow data, here I chose flow-tools and FlowViewer packages, and I would like to show how to get them work on a fresh Debian 5.0 (Lenny) setup.

Read more …