Avoid Cisco FIB/TCAM exhaustion on full BGP table feed

The number of IPv4 prefixes in the global BGP table is approaching the limit of many Cisco products, such as 7600/6500 RSP720/Sup720 and some ASR1000, which may hold a maximum of ~500K routes in their FIB (the Forwarding Information Base, where only best paths are stored).

These routers can usually handle a bigger load of prefixes, they can also be used to receive the full BGP table from many upstream providers concurrently, but they can’t manage more than 500/512K entries in their RIB or FIB.

Routing protocols to FIB


On 6500/7600 platforms, the default PFC3BXL/PFC3CXL TCAM configuration provides 512K entries for IPv4 and 256K for IPv6; the show mls cef maximum-routes command allows to show the running TCAM layout, that can be changed by entering mls cef maximum-routes ip … (it requires a reload).

7600(config)#mls cef maximum-routes ip 768
Maximum routes set to 786432. Configuration will be effective on reboot.

7600#show mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
 IPv4 + MPLS         - 512k (default)
 IPv6 + IP Multicast - 256k (default)

User configured :-
---------------
 IPv4                - 768k
 MPLS                - 16k (default)
 IPv6 + IP Multicast - 120k (default)

Upon reboot :-
-----------
 IPv4 + MPLS         - 512k (default)
 IPv6 + IP Multicast - 256k (default)

While this method allows to patch the immediate requirement for more IPv4 FIB space, it has the side effect of reducing the IPv6 resources, which are supposed to be more and more needed in the future.

Some ASR1000 platforms support up to 500K IPv4 routes and, as far as I know, they don’t allow RIB/FIB repartition.

In some scenarios, in order to continue using these routers while avoiding performance impairments, the BGP – Selective Route Download feature may help.

Routing protocols - table-map - FIB

The BGP—Selective Route Download feature allows a network administrator to selectively download some or none of the BGP routes into the Routing Information Base (RIB). (Cisco.com)

Sample scenario

Suppose to have a border router which receives 3 BGP feeds: some prefixes from a multihomed customer, some from an Internet exchange and the full table from a transit provider. This border router is also connected to a core router, where other border routers and the rest of the network is connected too. Configuring the table-map … filter option on the border router allows to prevent its FIB to be loaded with a lot of useless prefixes, which may be summarized with a default route toward the transit provider.

BGP prefixes filtering with table-map

Filtering may be based on many parameters; the one I used here is the next-hop address of each learned prefix. The final configuration looks like this:

router bgp xxx
 address-family ipv4
  table-map BGP-Table-IPv4 filter
!
route-map BGP-Table-IPv4 permit 10
 match ip next-hop BGP-NextHop-OK
route-map BGP-Table-IPv4 deny 10000
!
ip access-list standard BGP-NextHop-OK
 deny Transit_Provider_IP
 permit any

In order to minimize the changes to the standard configuration, an access-list is created which only denies the transit provider’s next-hop address.
Preferred prefixes learned from the Internet exchange and from the multihomed customer are permitted (not denied) by the access-list and, consequently, by the BGP-Table-IPv4 route-map referenced in the table-map command. Other prefixes are denied by the route-map and prevented from entering into the FIB, but they are anyway kept in the BGP database and announced toward the core router. Local-pref and other features may be configured on the routers to implement usual routing policies. Of course, in this example the core router must have a big enough FIB to handle the full BGP!

Please note that before configuring this feature, specific considerations are needed about local routing policies and traffic engineering mechanisms; special attention should be paid to locally generated prefixes announced to the border router by the rest of the network. As always, do it at your risk!

References

CIDR Report: Summary of total route table size for the past 7 days

Cisco.com: Cisco 7600 Series Route Switch Processor 720 Data Sheet

Cisco.com: Cisco ASR 1000 Series Embedded Services Processors Data Sheet

Cisco.com: BGP – Selective Route Download

The following two tabs change content below.
Italian, born in 1980, I started working in the IT area in the late '90s; I'm now a system and network administrator with a deep knowledge of the global Internet and its core architectures.

One Comment

  1. Awesome article. I thought I had tried the DENY clause on the route map and found that it didn’t work. I was using the Sup720 and taking a full BGP load, so I was in the same situation you were in.

    I ended up setting the metric…

    route-map LIMIT_RIB permit 20
    set metric 4294967295

    That’s the max metric and when set to 4294967295 it doesn’t even get inserted into the RIB. From there it doesn’t get inserted into the FIB/TCAM (of course), which was the main source of the problem.

    Glad to see someone else found this! We were getting this…
    %MLSCEF-DFC1-4-FIB_EXCEPTION_THRESHOLD: Hardware CEF entry usage is at 95% capacity for IPv4 unicast protocol.

    We dropped down to 34,554 CEF entries by simply eliminating prefixes that were more than three AS hops away. Anything more than three AS hops away we just supplied a default route that was “good enough”. As soon as the packet got to the next router it has a full BGP table so the routing resumed as normal. 34,554 easily fit into the RIB/FIB/TCAM so we were back up to wire-rate!

Leave a Reply