This post is part of a series about “ISP Security Tools and Techniques“; in this series I talk about some (I think) useful practices:
1. Remote Triggered Black Holing
2. BGP Customer triggered black holing
3. BGP triggered rate limiting and less-than-best-effort (LBE) with QPPB
4. Source-based RTBH with Unicast Reverse Path Forwarding (uRPF)
Stay tuned! 😉
In this post I would like to talk about Remote Triggered Black Holing, a mechanism to protect a network by filtering malicious traffic at the edge. It’s a powerful tool ISPs can (and should) adopt to stop DDOS attacks on their networks.
UPDATE 2009-06-21: You can find a related solution to use customer triggered blackholing in this new post!
Scenario
We have a network in AS 300 with a core router connected to 3 edge routers: 2 of these have an upstream link to 2 ISPs, AS 100 and AS 200, the third has connections toward customers. In our network OSPF is running as IGP; BGP peering sessions are up with ISPs and customers.
Objectives
While providing connectivity to our customers, we want to protect our network against possible DOS coming from external attackers. We want to deploy a solution which let us to stop malicious traffic to overwhelm our core as quickly as possible.
BGP basic configuration
As first, we setup the Core router and we elect it as route-reflector…
Core:
interface Loopback0 ip address 192.168.255.0 255.255.255.255 ! router bgp 300 no synchronization bgp log-neighbor-changes neighbor Edge peer-group neighbor Edge remote-as 300 neighbor Edge update-source Loopback0 neighbor Edge route-reflector-client neighbor 192.168.1.2 peer-group Edge neighbor 192.168.2.2 peer-group Edge neighbor 192.168.3.2 peer-group Edge no auto-summary !
All peers are grouped under a common peer-group and configured as route reflector clients.
Then we setup the ISPs facing edge routers: on each router we need an iBGP session with our RR, and an eBGP session with the ISP:
Edge1 (and similar on Edge2):
router bgp 300 no synchronization bgp log-neighbor-changes neighbor 172.16.1.2 remote-as 100 neighbor 172.16.1.2 remove-private-as neighbor 172.16.1.2 filter-list 1 out neighbor 192.168.255.0 remote-as 300 no auto-summary ! ip as-path access-list 1 permit ^65[0-9][0-9][0-9]$
We don’t want our AS to be a transit-AS between ISP1 and ISP2, so we just advertise locally originated prefixes on the ISPs facing routers: the filter-list matches just AS numbers in the form 65xxx, so only our customer originated prefixes will pass… and we remove the private numbers from the AS path list.
Our providers simply have an eBGP peering session with our AS, advertising their loopbacks:
ISP1 (and similar on ISP2):
router bgp 100 no synchronization bgp log-neighbor-changes network 10.0.1.1 mask 255.255.255.255 neighbor 172.16.1.1 remote-as 300 no auto-summary
On Edge3, the customers facing router, we have a configuration that let us to be sure our customers will only advertise their assigned subnets or sub-prefixes:
router bgp 300 no synchronization bgp log-neighbor-changes neighbor 192.168.0.2 remote-as 65310 neighbor 192.168.0.2 route-map BGP-Cust1 in neighbor 192.168.0.6 remote-as 65320 neighbor 192.168.0.6 route-map BGP-Cust2 in neighbor 192.168.255.0 remote-as 300 no auto-summary ! ip access-list extended BGP-Cust1 permit ip 192.168.10.0 0.0.0.255 255.255.255.0 0.0.0.255 ip access-list extended BGP-Cust2 permit ip 192.168.20.0 0.0.0.255 255.255.255.0 0.0.0.255 ! route-map BGP-Cust1 permit 10 match ip address BGP-Cust1 ! route-map BGP-Cust2 permit 10 match ip address BGP-Cust2
You can find these configurations on the “2. Routing” subdirectory within the main config dir.
Now, we are ready to deploy our solution: Remote triggered black holing.
The solution
Remote triggered black holing (RTBH) let us to prevent malicious traffic from entering our network and it can be applied in a centralized, quick and selective way. How does it work?
1) On each edge router we add a static route pointing to the null0 interface, so that all routers have the same route toward null0:
ip route 192.0.2.1 255.255.255.255 null0
2) When a host is under attack (for example 192.168.10.5/32), from a central router (the trigger) we inject a specific route for that host into BGP, with a specific BGP community used for RTBH.
3) Each edge router receives an UPDATE such this: 192.168.10.5/32 -> community 300:XXX; the router handles the UPDATE accordingly to the community and maps the prefix to the null-pointing route; in this way it forwards traffic for the attacked host to the null0, in other words it drops traffic avoiding useless resource consuption.
4) When the attack is over, we remove the route from our trigger, so edge routers will receive a withdrawal for that route, they remove the FIB entry pointing to null0 and traffic return to flow in the usual way.
In order to facilitate operations, we can extend the use of RTBH to our customers too, so that they can stop malicious traffic as soon as they detect an attack.
UPDATE 2009-06-21: You can read more about customer trigger blackholing in this new post!
RTBH policy
As first we have to decide the communities-driven policy we want to apply to RTBH: we use community 300:100 to block traffic on any edge router, 300:101 to block just traffic coming from ISP1, 300:102 to block traffic from ISP2 and 300:199 to black-hole traffic on Edge3, the customer facing router. With this kind of policy, if we know the source of the attack, we can stop it while maintaining partial service. For example, if the attack is from a network that reach us via ISP1, we can block just ISP1 traffic while maintaining traffic from ISP2. Of course, in any case we just black-hole traffic toward the attacked host/subnet, not all the traffic!
300:100 - global black-holing 300:101 - ISP1 black-holing 300:102 - ISP2 black-holing 300:199 - customers black-holing
At this time, we need to elect a router as trigger: in our example we’ll use Core router. Please note that in this example many decisions are forced by the limited number of routers we can deploy in GNS3, because of CPU and memory resources they take.
Trigger configuration
To trigger an UPDATE we use a static route manually added to the Core router. In order to let this UPDATE to go out, we need to redistribute static routes into BGP, after a little route-map handling.
The route we add to the Core router to block incoming traffic is in the form ip route ATTACKED_HOST 255.255.255.255 null0 tag XXX.
Of course, we can block a subnet instead of a host simply changing the destination IP and mask.
The tag XXX used to add the route let us to filter which routes we need to apply RTBH policy to, and also to understand what kind of policy to use.
All this stuff is handled by a route-map, applied to the static route redistribution in BGP:
Core#sh run | sec route-map route-map RTBH permit 10 match tag 100 set local-preference 200 set origin igp set community 19660900 no-export route-map RTBH permit 20 match tag 101 set local-preference 200 set origin igp set community 19660901 no-export route-map RTBH permit 30 match tag 102 set local-preference 200 set origin igp set community 19660902 no-export route-map RTBH permit 40 match tag 199 set local-preference 200 set origin igp set community 19660999 no-export route-map RTBH deny 1000
Core(config)#router bgp 300 Core(config-router)#redistribute static route-map RTBH Core(config-router)#neighbor Edge send-community
In the route map we match the route tag and we set the BGP community accordingly: tag 100 = community 300:100, tag 101 = community 300:101 and so on.
300:100 - tag 100 - global black-holing 300:101 - tag 101 - ISP1 black-holing 300:102 - tag 102 - ISP2 black-holing 300:199 - tag 199 - customers black-holing
We also set a higher than default local preference value (200 vs default 100) and a better origin (IGP instead of the incomplete used for redistribution): we want this prefix to be preferred to other.
It’s very important to add the no-export community too: in this way we avoid our black-holed prefix to leave our AS.
Don’t forget to configure the send-community!
Edge routers configuration
As first, we have to configure edge routers with the null-pointing route: we use the IANA TEST-NET 192.0.2.0/24 subnet.
Edge1(config)#ip route 192.0.2.1 255.255.255.255 null0 Edge2(config)#ip route 192.0.2.1 255.255.255.255 null0 Edge3(config)#ip route 192.0.2.1 255.255.255.255 null0
When a packet is dropped, an “ICMP unreachable” message is sent back to the source: to avoid a flooding of “ICMP unreachable” messages we need to add the “no ip unreachables” command to the null0 interface on the edge routers:
Edge1(config)#int null 0 Edge1(config-if)#no ip unreachables
Edge2(config)#int null 0 Edge2(config-if)#no ip unreachables
Edge3(config)#int null 0 Edge3(config-if)#no ip unreachables
Now, we have to setup the route-map used to handle incoming UPDATES.
Edge1#sh run | begin ip community-list ip community-list 1 permit 19660900 ! 300:100, always accepted (global black-holing) ip community-list 1 permit 19660901 ! 300:101, accepted just on Edge1 (ISP1) ip community-list 2 permit 19660902 ! 300:102, accepted just on Edge2 (ISP2), dropped here ip community-list 2 permit 19660999 ! 300:199, accepted just on Edge3 (customers), dropped here ip as-path access-list 1 permit ^65[0-9][0-9][0-9]$ ip as-path access-list 1 permit ^$ ! ! route-map FROM_RR deny 10 match community 2 ! route-map FROM_RR permit 20 match as-path 1 match community 1 set community no-export no-advertise set ip next-hop 192.0.2.1 ! route-map FROM_RR permit 1000
Edge1(config)#router bgp 300 Edge1(config-router)#neighbor 192.168.255.0 route-map FROM_RR in
This is the route-map used on Edge1. Here we have to black-hole just prefixes tagged with communities 300:100 (global black-holing) or 300:101 (ISP1 black-holing), so as first we have to drop prefixes tagged with 300:102 (reserved for ISP2 black-holing) and 300:199 (customer-side).
In the second instance we match just prefixes coming from our AS (locally generated) or from our customers (ASs 65XYZ) matching the right communities. Here we set the next-hop for these prefixes to the null-pointing route (192.0.2.1) and set net no-export and no-advertise community.
In the last instance we permit all other prefixes.
We need a similar configuration for Edge2 and Edge3; here we just change the community-list:
Edge2#sh run | begin ip community-list ip community-list 1 permit 19660900 ip community-list 1 permit 19660902 ip community-list 2 permit 19660901 ip community-list 2 permit 19660999
Edge3#sh run | begin ip community-list ip community-list 1 permit 19660900 ip community-list 1 permit 19660999 ip community-list 2 permit 19660902 ip community-list 2 permit 19660901
Test the solution
Let’s suppose the host 192.168.10.1/32 on Customer10 is under attack and we don’t know what network the DOS come from.
As first, we can trigger a global black-hole:
Core#conf t Enter configuration commands, one per line. End with CNTL/Z. Core(config)#ip route 192.168.10.1 255.255.255.255 null0 tag 100
Edge1 receives the new prefix and add it to the routing table:
Edge1#sh ip bgp
BGP table version is 12, local router ID is 192.168.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.0.1.1/32 172.16.1.2 0 0 100 i
*>i10.0.2.1/32 172.16.2.2 0 100 0 200 i
*>i192.168.10.0 192.168.0.2 0 100 0 65310 i
*>i192.168.10.1/32 192.0.2.1 0 200 0 i
*>i192.168.20.0 192.168.0.6 0 100 0 65320 i
Edge1#sh ip route 192.168.10.1
Routing entry for 192.168.10.1/32
Known via "bgp 300", distance 200, metric 0, type internal
Last update from 192.0.2.1 00:01:10 ago
Routing Descriptor Blocks:
* 192.0.2.1, from 192.168.255.0, 00:01:10 ago
Route metric is 0, traffic share count is 1
AS Hops 0
A recursive lookup shows as CEF drops these packets:
Edge1#sh ip cef 192.168.10.1
192.168.10.1/32, version 30, epoch 0
0 packets, 0 bytes
via 192.0.2.1, 0 dependencies, recursive
next hop 192.0.2.1, Null0 via 192.0.2.1/32
valid null (drop) adjacency
The same is on every edge router:
Edge2#sh ip route 192.168.10.1
Routing entry for 192.168.10.1/32
Known via "bgp 300", distance 200, metric 0, type internal
Last update from 192.0.2.1 00:03:31 ago
Routing Descriptor Blocks:
* 192.0.2.1, from 192.168.255.0, 00:03:31 ago
Route metric is 0, traffic share count is 1
AS Hops 0
Edge3#sh ip route 192.168.10.1
Routing entry for 192.168.10.1/32
Known via "bgp 300", distance 200, metric 0, type internal
Last update from 192.0.2.1 00:04:13 ago
Routing Descriptor Blocks:
* 192.0.2.1, from 192.168.255.0, 00:04:13 ago
Route metric is 0, traffic share count is 1
AS Hops 0
ISPs, as expected, don’t know anything about this new prefix:
ISP1#sh ip bgp | beg Network Network Next Hop Metric LocPrf Weight Path *> 10.0.1.1/32 0.0.0.0 0 32768 i *> 192.168.10.0 172.16.1.1 0 300 i *> 192.168.20.0 172.16.1.1 0 300 i
Traffic toward the attacked host is dropped at the edge:
ISP2#ping 192.168.10.1 source lo0 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds: Packet sent with a source address of 10.0.2.1 ..... Success rate is 0 percent (0/5)
Edge2#debug ip cef drops IP CEF drops debugging is on Edge2# *Mar 1 00:28:46.947: CEF-Drop: Packet for 192.168.10.1 to null0 (silent) *Mar 1 00:28:46.947: CEF-Drop: Packet for 192.168.10.1 -- switch to null0 *Mar 1 00:28:48.915: CEF-Drop: Packet for 192.168.10.1 to null0 (silent) *Mar 1 00:28:48.915: CEF-Drop: Packet for 192.168.10.1 -- switch to null0 *Mar 1 00:28:50.883: CEF-Drop: Packet for 192.168.10.1 to null0 (silent) *Mar 1 00:28:50.883: CEF-Drop: Packet for 192.168.10.1 -- switch to null0 *Mar 1 00:28:53.023: CEF-Drop: Packet for 192.168.10.1 to null0 (silent) *Mar 1 00:28:53.023: CEF-Drop: Packet for 192.168.10.1 -- switch to null0 *Mar 1 00:28:54.963: CEF-Drop: Packet for 192.168.10.1 to null0 (silent) *Mar 1 00:28:54.963: CEF-Drop: Packet for 192.168.10.1 -- switch to null0
When we know the attack is over, we just remove the route from the trigger…
Core(config)#no ip route 192.168.10.1 255.255.255.255 null0 tag 100
… and everything back to normal:
Edge1#sh ip cef 192.168.10.1
192.168.10.0/24, version 28, epoch 0, cached adjacency 192.168.1.1
0 packets, 0 bytes
via 192.168.0.2, 0 dependencies, recursive
next hop 192.168.1.1, FastEthernet0/0 via 192.168.0.0/30
valid cached adjacency
ISP2#ping 192.168.10.1 source lo0 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds: Packet sent with a source address of 10.0.2.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 544/651/744 ms
If we know the attack come from ISP2, we can black-hole the attacked host on the ISP2 facing router:
Core(config)#ip route 192.168.10.1 255.255.255.255 null0 tag 102
Edge1 and Edge3 reach the host as usual…
Edge1#sh ip cef 192.168.10.1
192.168.10.0/24, version 28, epoch 0, cached adjacency 192.168.1.1
0 packets, 0 bytes
via 192.168.0.2, 0 dependencies, recursive
next hop 192.168.1.1, FastEthernet0/0 via 192.168.0.0/30
valid cached adjacency
Edge3#sh ip cef 192.168.10.1
192.168.10.0/24, version 31, epoch 0, cached adjacency to Serial1/0
0 packets, 0 bytes
via 192.168.0.2, 0 dependencies, recursive
next hop 192.168.0.2, Serial1/0 via 192.168.0.0/30
valid cached adjacency
… while Edge2 drops traffic toward it:
Edge2#sh ip cef 192.168.10.1
192.168.10.1/32, version 33, epoch 0
0 packets, 0 bytes
via 192.0.2.1, 0 dependencies, recursive
next hop 192.0.2.1, Null0 via 192.0.2.1/32
valid null (drop) adjacency
Download
You can download the GNS3 Lab here. You will find 3 subdirectories within the main config dir: “1. Interfaces and IP”, with just interfaces and their IP address, “2. Routing”, with OSPF and BGP config and “3. RTBH” with the final RTBH configuration.
Resources
Cisco.com: Remotely Triggered Black Hole Filtering
Marco d’Itri: BGP-triggered blackholing
Latest posts by Pier Carlo Chiodi (see all)
- Good MANRS for IXPs route servers made easier - 11 December 2020
- Route server feature-rich and automatic configuration - 13 February 2017
- Large BGP Communities playground - 15 September 2016
[…] to trigger black holing for their prefixes. What I will write is based on my previous post GNS3 Lab: Remote Triggered Black Holing: same scenario, same startup config (the final one of that […]
[…] 1. Remote Triggered Black Holing […]
[…] 1. Remote Triggered Black Holing […]
This lab is real cool and everything works except for a small issue that’s probably taken care of in the other techniques.
I noticed that when trying to blackhole routes, like the /32 mentioned in the example, the routes are distributed correctly as mentioned but traffic from the non-violating ISP still gets dropped at the Core (trigger router) because of the trigger route. I’m sure there’s a way around it by using an additional redundant Core router but does anyone know any ways around the issue with using the topology shown? I’m thinking something with source-based routing, but that seems too cumbersome for an ISP. Maybe there’s some ways to do it with BGP communities?
Hi,
you may deploy an external RTBH route server, even a small linux box running Quagga should be ok. You have to setup your edge routers in order to peer with the route server, then you can use it to inject blackholed prefixes.
[…] 1. Remote Triggered Black Holing […]