This post is part of a series about “ISP Security Tools and Techniques“; in this series I talk about some (I think) useful practices:
1. Remote Triggered Black Holing
2. BGP Customer triggered black holing
3. BGP triggered rate limiting and less-than-best-effort (LBE) with QPPB
4. Source-based RTBH with Unicast Reverse Path Forwarding (uRPF)
Stay tuned! 😉
As I already wrote in my previous posts, an ISP can provide their customers some useful tools to mitigate (D)DoS attacks: Remote Triggered Black Holing and its NOC-independent version, Customer triggered black holing are tools that, once identified attacked hosts or networks, let us to stop malicious traffic at the provider’s edge.
Anyway, when we drop traffic toward attacked hosts, we can’t investigate the attack anymore; we would need a tool which allowed us to analyze traffic and, in the meantime, that would avoid wasting network resources. Our (dear) provider could provide it by implementing rate limiting and less-than-best-effort services using QoS Policy Propagation via BGP (QPPB).
Scenario and goals
The scenario I will use in this post is the same I used for previous posts I already mentioned before. We are the AS 300 provider, we have 2 customers and 2 upstream providers connected with BGP sessions. Edge1 and Edge2 are routers toward the upstream providers, Edge3 is the router our customers are connected to.
Startup configurations are the same I left on my last post, “Customer triggered black holing”.
As said, the goal is to provide a tool customers can use to rate-limit traffic toward attacked hosts, in order to let them to investigate attacks. This tool should be used by customers avoiding NOC intervention. Scalability is a must.
The solution and how it works
The solution proposed here is based on QoS Policy Propagation via BGP (QPPB).
Cisco defines it as a feature that “allows you to classify packets by IP precedence based on BGP community lists, BGP autonomous system paths, and access control lists” (see Cisco 10000 Series Router Quality of Service Configuration Guide). And, of course, that is all! There’s not much more to say!
How does it work? A BGP router running QPPB receives a prefix and matches it using a route-map, then it sets that prefix’s IP precedence or QoS-group accordingly; router’s interfaces are configured to match traffic’s source or destination address against the prefix and to classify packets; once classified, packets can be matched against normal QoS class-matches and policies. Take a look at the picture for a diagram.
Be aware, no attributes or other infos are added to BGP UPDATEs; it’s just a local mechanism to mark routes and classify packets.
Now that we know how QPPB works we can use it to implement rate-limiting QoS policies and to trigger them via BGP.
As first, we have to define on each edge router a QoS policy with two class-maps: one used to rate-limit traffic, and the other used to mark the traffic as less-than-best-effort (LBE): lets say we’ll use QoS-group 1 to rate-limit traffic at 8Kbps and QoS-group 2 to mark traffic as LBE.
Edge1 and Edge2:
class-map match-all QPPB-QoSGroup-1 match qos-group 1 class-map match-all QPPB-QoSGroup-2 match qos-group 2 ! policy-map QPPB class QPPB-QoSGroup-1 police cir 8000 conform-action transmit exceed-action drop class QPPB-QoSGroup-2 set dscp cs1
Then we can apply our policy to the core facing interfaces, for outgoing traffic (traffic from upstream providers toward our core/customers):
Edge1 and Edge2:
interface FastEthernet0/0 service-policy output QPPB
Now, we have to define BGP communities to map QoS groups: we will use the following mapping:
300:201 - QoS group 1 (rate-limit to 8 Kbps) 300:202 - QoS group 2 (LBE marking)
Once defined, we have to implement communities in BGP:
Edge1 and Edge2:
ip community-list 3 permit 300:201 ip community-list 4 permit 300:202 ! route-map QPPB permit 10 match community 3 set ip qos-group 1 route-map QPPB permit 20 match community 4 set ip qos-group 2 route-map QPPB permit 1000 ! router bgp 300 table-map QPPB
As you can see, the route-map used for QPPB can’t be the same used for the inbound BGP UPDATEs. We must add another route-map to the BGP process using the table-map subcommand: as Cisco says in the Command Lookup Tool this command is used “to modify metric and tag values when the IP routing table is updated with BGP learned routes”.
Now we can enable QPPB on upstream ISPs facing interfaces to classify incoming traffic based on its destination IP address:
Edge1 and Edge2:
interface Serial1/0 bgp-policy destination ip-qos-map
For the sake of completion, I say you can use this command to match source or destination address of a packet, and to use IP precedence or QoS-groups for classification. Here we use destination based matching and QoS-groups.
Now, let our customer routers to trigger the services: as usual, we will use a tagged static route redistributed in BGP. We already configured the route-map and redistribution in previous posts, so we just need to add some entries to it:
Cust10 and Cust20:
route-map RTBH permit 100 match tag 201 set community 300:201 ! route-map RTBH permit 110 match tag 202 set community 300:202
To test the solution we just have to add a static route toward the prefix we want to rate-limit (or to mark as LBE) on the customer router:
Cust10(config)#ip route 192.168.10.20 255.255.255.255 fa1/0 tag 201
On the edge router we have the new /32 prefix with the expected community:
Edge1#sh ip bgp 192.168.10.20 BGP routing table entry for 192.168.10.20/32, version 12 Paths: (1 available, best #1, table Default-IP-Routing-Table) Flag: 0x820 Advertised to update-groups: 2 65310 192.168.0.2 (metric 66) from 192.168.255.0 (192.168.255.0) Origin incomplete, metric 0, localpref 100, valid, internal, best Community: 19661001 Originator: 192.168.3.2, Cluster list: 192.168.255.0
and we also have a route and a CEF entry tagged with QoS-group 1:
Edge1#sh ip route 192.168.10.20 Routing entry for 192.168.10.20/32 Known via "bgp 300", distance 200, metric 0 Tag 65310, qos-group 1, type internal Last update from 192.168.0.2 00:00:55 ago Routing Descriptor Blocks: * 192.168.0.2, from 192.168.255.0, 00:00:55 ago Route metric is 0, traffic share count is 1 AS Hops 1 Route tag 65310 Edge1#sh ip cef 192.168.10.20 192.168.10.20/32, version 34, epoch 0, cached adjacency 192.168.1.1 0 packets, 0 bytes, qos-group 1 via 192.168.0.2, 0 dependencies, recursive next hop 192.168.1.1, FastEthernet0/0 via 192.168.0.0/30 valid cached adjacency
Now, we ping 192.168.10.20 from ISP1 (don’t expect echo replies, there is not a host at that address, but we just need traffic going toward it):
ISP1#ping 192.168.10.20 source lo0 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.10.20, timeout is 2 seconds: Packet sent with a source address of 10.0.1.1 ..... Success rate is 0 percent (0/5)
On Edge1, fa0/0 output policy counters show 5 packets classified as QPPB-QoSGroup-1 class-match:
Edge1#sh policy-map interface fa0/0 output FastEthernet0/0 Service-policy output: QPPB Class-map: QPPB-QoSGroup-1 (match-all) 5 packets, 570 bytes 5 minute offered rate 0 bps, drop rate 0 bps Match: qos-group 1 police: cir 8000 bps, bc 1500 bytes conformed 5 packets, 570 bytes; actions: transmit exceeded 0 packets, 0 bytes; actions: drop conformed 0 bps, exceed 0 bps [cut]
Let’s try with Cust20 and LBE service:
Cust20(config)#ip route 192.168.20.40 255.255.255.255 fa1/0 tag 202
The prefix is on Edge2, right community, right route tag:
Edge2#sh ip bgp 192.168.20.40 BGP routing table entry for 192.168.20.40/32, version 15 Paths: (1 available, best #1, table Default-IP-Routing-Table) Flag: 0x820 Advertised to update-groups: 2 65320 192.168.0.6 (metric 66) from 192.168.255.0 (192.168.255.0) Origin incomplete, metric 0, localpref 100, valid, internal, best Community: 19661002 Originator: 192.168.3.2, Cluster list: 192.168.255.0 Edge2#sh ip route 192.168.20.40 Routing entry for 192.168.20.40/32 Known via "bgp 300", distance 200, metric 0 Tag 65320, qos-group 2, type internal Last update from 192.168.0.6 00:01:15 ago Routing Descriptor Blocks: * 192.168.0.6, from 192.168.255.0, 00:01:16 ago Route metric is 0, traffic share count is 1 AS Hops 1 Route tag 65320 Edge2#sh ip cef 192.168.20.40 192.168.20.40/32, version 32, epoch 0, cached adjacency 192.168.2.1 0 packets, 0 bytes, qos-group 2 via 192.168.0.6, 0 dependencies, recursive next hop 192.168.2.1, FastEthernet0/0 via 192.168.0.4/30 valid cached adjacency
We try to ping the host from ISP2…
ISP2#ping 192.168.20.40 so lo0 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.20.40, timeout is 2 seconds: Packet sent with a source address of 10.0.2.1 ..... Success rate is 0 percent (0/5)
… and Edge2 policy counters go up for QPPB-QoSGroup-2:
Edge2#sh policy-map interface fa0/0 output FastEthernet0/0 Service-policy output: QPPB Class-map: QPPB-QoSGroup-1 (match-all) 0 packets, 0 bytes 5 minute offered rate 0 bps, drop rate 0 bps Match: qos-group 1 police: cir 8000 bps, bc 1500 bytes conformed 0 packets, 0 bytes; actions: transmit exceeded 0 packets, 0 bytes; actions: drop conformed 0 bps, exceed 0 bps Class-map: QPPB-QoSGroup-2 (match-all) 5 packets, 570 bytes 5 minute offered rate 0 bps, drop rate 0 bps Match: qos-group 2 QoS Set dscp cs1 Packets marked 5 [cut]
In the RateLimit_LBE.zip file you can find the GNS3 Lab with previous and current configuration files. On the RTBH_Configs directory you will find the “5. Rate-limit and LBE” subdirectory with the configuration discussed in this post.
QoS Policy Propagation with BGP (QPPB) on Informit: http://www.informit.com/content/images/9781587201240/appendix/QPPBSection.pdf
Cisco 10000 Series Router Quality of Service Configuration Guide: http://www.cisco.com/en/US/docs/routers/10000/10008/configuration/guides/qos/10qqppb.html
Latest posts by Pier Carlo Chiodi (see all)
- Good MANRS for IXPs route servers made easier - 11 December 2020
- Route server feature-rich and automatic configuration - 13 February 2017
- Large BGP Communities playground - 15 September 2016
[…] Zabbix: how to monitor Radius (and other services) with external check items and netcat (nc) GNS3 Labs: BGP triggered rate limiting and less-than-best-effort (LBE) with QPPB […]
[…] GNS3 Labs: BGP triggered rate limiting and less-than-best-effort (LBE) with QPPB « Pierky̵… said June 29, 2009 at 9:40 am […]
I am kind of confused after checking the bgp tables.
The community attribute of bgp entry for 192.168.10.20 at core, edge1, edge2, edge3 and customer-10 contains 300:201. But the community attribute of the same entry at IPS1, ISP2, and customer-20 does not contain 300:201.
It seems like when the edge routers advertise 192.168.10.20 to its external neighbors, the community attribute was removed. Do you understand why?
— James Huang
those neighbors have not the “send-community” command in the config:
If you want to send communities you have to add the following:
If you enable the send-community you have to take care of what communities you really want to send; on this series posts I just wanted to show how the techniques work, and I didn’t take care of communities propagation enough. If you want to use these techniques in a production environment you have to strengthen the use of no-export community and, for example, to be sure to not announce /32 rate-limited prefixes to your upstream providers, but just your aggregate.
Sorry for missing somethig this obvious.
— James Huang
Are you aware of SNMP monitoring facilities while using QPPB?
I am investigating QPPB to deploy a solution of two distinct kinds of IP traffic through one same WAN circuit and same BGP peering to a client, but the management wants to be sure that traffic monitoring is possible for each kind of traffic.
you may take a look at my post Cisco Class-Based QoS SNMP MIB and statistics monitor for NMS.
I never tried SNMP monitoring of QPPB policies, but I guess it should work fine as standard QoS. Please, let me know if you try it! 😉
[…] 3. BGP triggered rate limiting and less-than-best-effort (LBE) with QPPB […]