GNS3 Lab: Remote Triggered Black Holing

This post is part of a series about “ISP Security Tools and Techniques“; in this series I talk about some (I think) useful practices:

1. Remote Triggered Black Holing

2. BGP Customer triggered black holing

3. BGP triggered rate limiting and less-than-best-effort (LBE) with QPPB

4. Source-based RTBH with Unicast Reverse Path Forwarding (uRPF)

Stay tuned! 😉

Remote Triggered Black HolingIn this post I would like to talk about Remote Triggered Black Holing, a mechanism to protect a network by filtering malicious traffic at the edge. It’s a powerful tool ISPs can (and should) adopt to stop DDOS attacks on their networks.

UPDATE 2009-06-21: You can find a related solution to use customer triggered blackholing in this new post!

Scenario

We have a network in AS 300 with a core router connected to 3 edge routers: 2 of these have an upstream link to 2 ISPs, AS 100 and AS 200, the third has connections toward customers. In our network OSPF is running as IGP; BGP peering sessions are up with ISPs and customers.

Objectives

While providing connectivity to our customers, we want to protect our network against possible DOS coming from external attackers. We want to deploy a solution which let us to stop malicious traffic to overwhelm our core as quickly as possible.

BGP basic configuration

As first, we setup the Core router and we elect it as route-reflector…

Core:

interface Loopback0
 ip address 192.168.255.0 255.255.255.255
!
router bgp 300
 no synchronization
 bgp log-neighbor-changes
 neighbor Edge peer-group
 neighbor Edge remote-as 300
 neighbor Edge update-source Loopback0
 neighbor Edge route-reflector-client
 neighbor 192.168.1.2 peer-group Edge
 neighbor 192.168.2.2 peer-group Edge
 neighbor 192.168.3.2 peer-group Edge
 no auto-summary
!

All peers are grouped under a common peer-group and configured as route reflector clients.

Then we setup the ISPs facing edge routers: on each router we need an iBGP session with our RR, and an eBGP session with the ISP:

Edge1 (and similar on Edge2):

router bgp 300
 no synchronization
 bgp log-neighbor-changes
 neighbor 172.16.1.2 remote-as 100
 neighbor 172.16.1.2 remove-private-as
 neighbor 172.16.1.2 filter-list 1 out
 neighbor 192.168.255.0 remote-as 300
 no auto-summary
!
ip as-path access-list 1 permit ^65[0-9][0-9][0-9]$

We don’t want our AS to be a transit-AS between ISP1 and ISP2, so we just advertise locally originated prefixes on the ISPs facing routers: the filter-list matches just AS numbers in the form 65xxx, so only our customer originated prefixes will pass… and we remove the private numbers from the AS path list.

Our providers simply have an eBGP peering session with our AS, advertising their loopbacks:

ISP1 (and similar on ISP2):

router bgp 100
 no synchronization
 bgp log-neighbor-changes
 network 10.0.1.1 mask 255.255.255.255
 neighbor 172.16.1.1 remote-as 300
 no auto-summary

On Edge3, the customers facing router, we have a configuration that let us to be sure our customers will only advertise their assigned subnets or sub-prefixes:

router bgp 300
 no synchronization
 bgp log-neighbor-changes
 neighbor 192.168.0.2 remote-as 65310
 neighbor 192.168.0.2 route-map BGP-Cust1 in
 neighbor 192.168.0.6 remote-as 65320
 neighbor 192.168.0.6 route-map BGP-Cust2 in
 neighbor 192.168.255.0 remote-as 300
 no auto-summary
!
ip access-list extended BGP-Cust1
 permit ip 192.168.10.0 0.0.0.255 255.255.255.0 0.0.0.255
ip access-list extended BGP-Cust2
 permit ip 192.168.20.0 0.0.0.255 255.255.255.0 0.0.0.255
!
route-map BGP-Cust1 permit 10
 match ip address BGP-Cust1
!
route-map BGP-Cust2 permit 10
 match ip address BGP-Cust2

You can find these configurations on the “2. Routing” subdirectory within the main config dir.

Now, we are ready to deploy our solution: Remote triggered black holing.

The solution

Remote triggered black holing (RTBH) let us to prevent malicious traffic from entering our network and it can be applied in a centralized, quick and selective way. How does it work?

1) On each edge router we add a static route pointing to the null0 interface, so that all routers have the same route toward null0:

ip route 192.0.2.1 255.255.255.255 null0

2) When a host is under attack (for example 192.168.10.5/32), from a central router (the trigger) we inject a specific route for that host into BGP, with a specific BGP community used for RTBH.

3) Each edge router receives an UPDATE such this: 192.168.10.5/32 -> community 300:XXX; the router handles the UPDATE accordingly to the community and maps the prefix to the null-pointing route; in this way it forwards traffic for the attacked host to the null0, in other words it drops traffic avoiding useless resource consuption.

4) When the attack is over, we remove the route from our trigger, so edge routers will receive a withdrawal for that route, they remove the FIB entry pointing to null0 and traffic return to flow in the usual way.

In order to facilitate operations, we can extend the use of RTBH to our customers too, so that they can stop malicious traffic as soon as they detect an attack.

UPDATE 2009-06-21: You can read more about customer trigger blackholing in this new post!

RTBH policy

As first we have to decide the communities-driven policy we want to apply to RTBH: we use community 300:100 to block traffic on any edge router, 300:101 to block just traffic coming from ISP1, 300:102 to block traffic from ISP2 and 300:199 to black-hole traffic on Edge3, the customer facing router. With this kind of policy, if we know the source of the attack, we can stop it while maintaining partial service. For example, if the attack is from a network that reach us via ISP1, we can block just ISP1 traffic while maintaining traffic from ISP2. Of course, in any case we just black-hole traffic toward the attacked host/subnet, not all the traffic!

300:100 - global black-holing
300:101 - ISP1 black-holing
300:102 - ISP2 black-holing
300:199 - customers black-holing

At this time, we need to elect a router as trigger: in our example we’ll use Core router. Please note that in this example many decisions are forced by the limited number of routers we can deploy in GNS3, because of CPU and memory resources they take.

Trigger configuration

To trigger an UPDATE we use a static route manually added to the Core router. In order to let this UPDATE to go out, we need to redistribute static routes into BGP, after a little route-map handling.

The route we add to the Core router to block incoming traffic is in the form ip route ATTACKED_HOST 255.255.255.255 null0 tag XXX.

Of course, we can block a subnet instead of a host simply changing the destination IP and mask.
The tag XXX used to add the route let us to filter which routes we need to apply RTBH policy to, and also to understand what kind of policy to use.

All this stuff is handled by a route-map, applied to the static route redistribution in BGP:

Core#sh run | sec route-map
route-map RTBH permit 10
 match tag 100
 set local-preference 200
 set origin igp
 set community 19660900 no-export
route-map RTBH permit 20
 match tag 101
 set local-preference 200
 set origin igp
 set community 19660901 no-export
route-map RTBH permit 30
 match tag 102
 set local-preference 200
 set origin igp
 set community 19660902 no-export
route-map RTBH permit 40
 match tag 199
 set local-preference 200
 set origin igp
 set community 19660999 no-export
route-map RTBH deny 1000
Core(config)#router bgp 300
Core(config-router)#redistribute static route-map RTBH
Core(config-router)#neighbor Edge send-community

In the route map we match the route tag and we set the BGP community accordingly: tag 100 = community 300:100, tag 101 = community 300:101 and so on.

300:100 - tag 100 - global black-holing
300:101 - tag 101 - ISP1 black-holing
300:102 - tag 102 - ISP2 black-holing
300:199 - tag 199 - customers black-holing

We also set a higher than default local preference value (200 vs default 100) and a better origin (IGP instead of the incomplete used for redistribution): we want this prefix to be preferred to other.
It’s very important to add the no-export community too: in this way we avoid our black-holed prefix to leave our AS.

Don’t forget to configure the send-community!

Edge routers configuration

As first, we have to configure edge routers with the null-pointing route: we use the IANA TEST-NET 192.0.2.0/24 subnet.

Edge1(config)#ip route 192.0.2.1 255.255.255.255 null0
Edge2(config)#ip route 192.0.2.1 255.255.255.255 null0
Edge3(config)#ip route 192.0.2.1 255.255.255.255 null0

When a packet is dropped, an “ICMP unreachable” message is sent back to the source: to avoid a flooding of “ICMP unreachable” messages we need to add the “no ip unreachables” command to the null0 interface on the edge routers:

Edge1(config)#int null 0
Edge1(config-if)#no ip unreachables
Edge2(config)#int null 0
Edge2(config-if)#no ip unreachables
Edge3(config)#int null 0
Edge3(config-if)#no ip unreachables

Now, we have to setup the route-map used to handle incoming UPDATES.

Edge1#sh run | begin ip community-list
ip community-list 1 permit 19660900	! 300:100, always accepted (global black-holing)
ip community-list 1 permit 19660901	! 300:101, accepted just on Edge1 (ISP1)
ip community-list 2 permit 19660902	! 300:102, accepted just on Edge2 (ISP2), dropped here
ip community-list 2 permit 19660999	! 300:199, accepted just on Edge3 (customers), dropped here
ip as-path access-list 1 permit ^65[0-9][0-9][0-9]$
ip as-path access-list 1 permit ^$
!
!
route-map FROM_RR deny 10
 match community 2
!
route-map FROM_RR permit 20
 match as-path 1
 match community 1
 set community no-export no-advertise
 set ip next-hop 192.0.2.1
!
route-map FROM_RR permit 1000
Edge1(config)#router bgp 300
Edge1(config-router)#neighbor 192.168.255.0 route-map FROM_RR in

This is the route-map used on Edge1. Here we have to black-hole just prefixes tagged with communities 300:100 (global black-holing) or 300:101 (ISP1 black-holing), so as first we have to drop prefixes tagged with 300:102 (reserved for ISP2 black-holing) and 300:199 (customer-side).

In the second instance we match just prefixes coming from our AS (locally generated) or from our customers (ASs 65XYZ) matching the right communities. Here we set the next-hop for these prefixes to the null-pointing route (192.0.2.1) and set net no-export and no-advertise community.

In the last instance we permit all other prefixes.

We need a similar configuration for Edge2 and Edge3; here we just change the community-list:

Edge2#sh run | begin ip community-list
ip community-list 1 permit 19660900
ip community-list 1 permit 19660902
ip community-list 2 permit 19660901
ip community-list 2 permit 19660999
Edge3#sh run | begin ip community-list
ip community-list 1 permit 19660900
ip community-list 1 permit 19660999
ip community-list 2 permit 19660902
ip community-list 2 permit 19660901

Test the solution

Let’s suppose the host 192.168.10.1/32 on Customer10 is under attack and we don’t know what network the DOS come from.

As first, we can trigger a global black-hole:

Core#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Core(config)#ip route 192.168.10.1 255.255.255.255 null0 tag 100

Edge1 receives the new prefix and add it to the routing table:

Edge1#sh ip bgp
BGP table version is 12, local router ID is 192.168.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.1.1/32      172.16.1.2               0             0 100 i
*>i10.0.2.1/32      172.16.2.2               0    100      0 200 i
*>i192.168.10.0     192.168.0.2              0    100      0 65310 i
*>i192.168.10.1/32  192.0.2.1                0    200      0 i
*>i192.168.20.0     192.168.0.6              0    100      0 65320 i
Edge1#sh ip route 192.168.10.1
Routing entry for 192.168.10.1/32
  Known via "bgp 300", distance 200, metric 0, type internal
  Last update from 192.0.2.1 00:01:10 ago
  Routing Descriptor Blocks:
  * 192.0.2.1, from 192.168.255.0, 00:01:10 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0

A recursive lookup shows as CEF drops these packets:

Edge1#sh ip cef 192.168.10.1
192.168.10.1/32, version 30, epoch 0
0 packets, 0 bytes
  via 192.0.2.1, 0 dependencies, recursive
    next hop 192.0.2.1, Null0 via 192.0.2.1/32
    valid null (drop) adjacency

The same is on every edge router:

Edge2#sh ip route 192.168.10.1
Routing entry for 192.168.10.1/32
  Known via "bgp 300", distance 200, metric 0, type internal
  Last update from 192.0.2.1 00:03:31 ago
  Routing Descriptor Blocks:
  * 192.0.2.1, from 192.168.255.0, 00:03:31 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0
Edge3#sh ip route 192.168.10.1
Routing entry for 192.168.10.1/32
  Known via "bgp 300", distance 200, metric 0, type internal
  Last update from 192.0.2.1 00:04:13 ago
  Routing Descriptor Blocks:
  * 192.0.2.1, from 192.168.255.0, 00:04:13 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0

ISPs, as expected, don’t know anything about this new prefix:

ISP1#sh ip bgp | beg Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.1.1/32      0.0.0.0                  0         32768 i
*> 192.168.10.0     172.16.1.1                             0 300 i
*> 192.168.20.0     172.16.1.1                             0 300 i

Traffic toward the attacked host is dropped at the edge:

ISP2#ping 192.168.10.1 source lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.2.1
.....
Success rate is 0 percent (0/5)
Edge2#debug ip cef drops
IP CEF drops debugging is on
Edge2#
*Mar  1 00:28:46.947: CEF-Drop: Packet for 192.168.10.1 to null0 (silent)
*Mar  1 00:28:46.947: CEF-Drop: Packet for 192.168.10.1 -- switch to null0
*Mar  1 00:28:48.915: CEF-Drop: Packet for 192.168.10.1 to null0 (silent)
*Mar  1 00:28:48.915: CEF-Drop: Packet for 192.168.10.1 -- switch to null0
*Mar  1 00:28:50.883: CEF-Drop: Packet for 192.168.10.1 to null0 (silent)
*Mar  1 00:28:50.883: CEF-Drop: Packet for 192.168.10.1 -- switch to null0
*Mar  1 00:28:53.023: CEF-Drop: Packet for 192.168.10.1 to null0 (silent)
*Mar  1 00:28:53.023: CEF-Drop: Packet for 192.168.10.1 -- switch to null0
*Mar  1 00:28:54.963: CEF-Drop: Packet for 192.168.10.1 to null0 (silent)
*Mar  1 00:28:54.963: CEF-Drop: Packet for 192.168.10.1 -- switch to null0

When we know the attack is over, we just remove the route from the trigger…

Core(config)#no ip route 192.168.10.1 255.255.255.255 null0 tag 100

… and everything back to normal:

Edge1#sh ip cef 192.168.10.1
192.168.10.0/24, version 28, epoch 0, cached adjacency 192.168.1.1
0 packets, 0 bytes
  via 192.168.0.2, 0 dependencies, recursive
    next hop 192.168.1.1, FastEthernet0/0 via 192.168.0.0/30
    valid cached adjacency
ISP2#ping 192.168.10.1 source lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.2.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 544/651/744 ms

If we know the attack come from ISP2, we can black-hole the attacked host on the ISP2 facing router:

Core(config)#ip route 192.168.10.1 255.255.255.255 null0 tag 102

Edge1 and Edge3 reach the host as usual…

Edge1#sh ip cef 192.168.10.1
192.168.10.0/24, version 28, epoch 0, cached adjacency 192.168.1.1
0 packets, 0 bytes
  via 192.168.0.2, 0 dependencies, recursive
    next hop 192.168.1.1, FastEthernet0/0 via 192.168.0.0/30
    valid cached adjacency
Edge3#sh ip cef 192.168.10.1
192.168.10.0/24, version 31, epoch 0, cached adjacency to Serial1/0
0 packets, 0 bytes
  via 192.168.0.2, 0 dependencies, recursive
    next hop 192.168.0.2, Serial1/0 via 192.168.0.0/30
    valid cached adjacency

… while Edge2 drops traffic toward it:

Edge2#sh ip cef 192.168.10.1
192.168.10.1/32, version 33, epoch 0
0 packets, 0 bytes
  via 192.0.2.1, 0 dependencies, recursive
    next hop 192.0.2.1, Null0 via 192.0.2.1/32
    valid null (drop) adjacency

Download

You can download the GNS3 Lab here. You will find 3 subdirectories within the main config dir: “1. Interfaces and IP”, with just interfaces and their IP address, “2. Routing”, with OSPF and BGP config and “3. RTBH” with the final RTBH configuration.

Resources

Cisco.com: Remotely Triggered Black Hole Filtering

Marco d’Itri: BGP-triggered blackholing

The following two tabs change content below.
Italian, born in 1980, I started working in the IT area in the late '90s; I'm now a system and network administrator with a deep knowledge of the global Internet and its core architectures.

6 Comments

  1. […] to trigger black holing for their prefixes. What I will write is based on my previous post GNS3 Lab: Remote Triggered Black Holing: same scenario, same startup config (the final one of that […]

  2. fliprich says:

    This lab is real cool and everything works except for a small issue that’s probably taken care of in the other techniques.

    I noticed that when trying to blackhole routes, like the /32 mentioned in the example, the routes are distributed correctly as mentioned but traffic from the non-violating ISP still gets dropped at the Core (trigger router) because of the trigger route. I’m sure there’s a way around it by using an additional redundant Core router but does anyone know any ways around the issue with using the topology shown? I’m thinking something with source-based routing, but that seems too cumbersome for an ISP. Maybe there’s some ways to do it with BGP communities?

  3. pierky says:

    Hi,
    you may deploy an external RTBH route server, even a small linux box running Quagga should be ok. You have to setup your edge routers in order to peer with the route server, then you can use it to inject blackholed prefixes.

Leave a Reply