Understanding NCS5500 Resources (S01E02)
You can find more content related to NCS5500 including routing memory management, VRF, URPF, Netflow following this link.
In IOS XR 7.3.1, we will decommission the “internet-optimized” mode, please check this article: https://xrdocs.io/ncs5500/tutorials/decommissioning-internet-optimized-mode/
S01E02 IPv4 Prefixes
Previously on “Understanding NCS5500 Resources”
In the previous post, we introduced the different routers and line cards in NCS5500 portfolio. We classified them into two categories: with or without external TCAM (eTCAM). And we introduced the different databases available to store information, inside and outside the Forwarding ASIC (FA).
All the principles described below and the examples used to illustrate them were validated in August 2017 with Jericho-based systems, using scale (with eTCAM) and base (without eTCAM) line cards and running the two IOS XR releases available: 6.1.4 and 6.2.2. Jericho+ based systems will be used in a follow up post in the same series (season 2 ;)
IPv4 routes and FIB Profiles
A quick refresh will be very useful to understand how routes are stored in NCS5500:
- LPM: Longest Prefix Match Database (sometimes referred to as KAPS for KBP Assisted Prefix Search, KBP being itself Knowledge Based Processor) is an SRAM used to store IPv4 and IPv6 prefixes. Scale: variable from 128k to 400k entries. We can perform variable length prefix lookup in LPM.
- LEM: Large Exact Match Database also used to store specific IPv4 and IPv6 routes, plus MAC addresses and MPLS labels. Scale: 786k entries. We perform exact match lookup in LEM.
- eTCAM: external TCAMs, only present in the -SE “scale” line cards and systems. As the name implies, they are not a resource inside the Forwarding ASIC, it’s an additional memory used to extend unicast route and ACL / classifiers scale. Scale: 2M IPv4 entries. We can also perform variable length prefix lookup in eTCAM.
The origin of the prefixes is not relevant. They can be received from OSPF, ISIS, BGP but also static routes. It doesn’t influence which database will be used to store them. Only the address-family (IPv4 in this discussion) and the subnet length of the prefix will be used in the decision process.
Hardware programming is done through an abstraction layer: Data-Plane Agent (DPA)
Also, it’s important to remember we are not talking about BGP paths here but about FIB entries: if we have 10 internet transit providers advertising more or less the same 700k-ish routes (with 10 next-hop addresses), we don’t have 7M entries in the FIB but 700k. Few exceptions exist (like using different VRFs for each transit provider) but they are out of the scope of this post.
Originally, IPv4/32 are going in LEM and all other prefix lengths (IPv4/31-/0) are stored in LPM. We changed this default behavior by implementing FIB profiles: Host-optimized or Internet-Optimized.
RP/0/RP0/CPU0:NCS5500(config)#hw-module fib ipv4 scale ?
host-optimized-disable Configure Host optimization by default
internet-optimized Configure Intetrnet optimized
RP/0/RP0/CPU0:NCS5500(config)#
Host-optimized is the default option. Committing a change in the configuration will prompt you to reload the line-cards or chassis to enable the new profile.
Note: in IOS XR 7.3.1, we will decommission the “internet-optimized” mode, please check this article: https://xrdocs.io/ncs5500/tutorials/decommissioning-internet-optimized-mode/
For a base line card (those without -SE in the product ID), we will have the following order of operation:
When a packet is received, the FA performs a lookup on the destination address:
- first lookup is performed in the LEM searching for a IPv4/32 exact match
- second lookup is accessing the LPM searching for a variable length match between IPv4/31 and IPv4/25
- third lookup is done in the LEM again, searching for a IPv4/24 exact match
- finally, the fourth lookup is checking the LPM a second time searching for a variable length match between IPv4/23 and /0
All is done in one single clock tick, it doesn’t require any kind of recirculation and doesn’t impact the performance (in bandwidth or in packet per second).
This mode is particularly useful with a large number of IPv4/32 and IPv4/24 in the routing table. It could be the case for hosting companies or data centers.
Using the configuration above, you can decide to enable the Internet-optimized mode. This is a feature activated globally and not per line card. After reload, you will see a very different order of operation and prefix distribution in the various databases with base line cards and systems:
The order of operation LEM/LPM/LEM/LPM is now replaced by an LPM/LEM/LEM/LPM approach.
- first lookup is in LPM searching for a match between IPv4/32 and IPv4/25
- second lookup is performed in LEM for an exact match on IPv4/24 and IPv4/23.
- third lookup is done in LEM too, and this time for also an exact match on IPv4/20
- fourth and final step, a variable length lookup is executed in LPM for everything between IPv4/22 and /0
Here again, everything is performed in one cycle and the activation of the Internet Optimized mode doesn’t impact the forwarding performance.
As the name implies, this profile has been optimized to move the largest route population present on the Internet (IPv4/24, IPv4/23, IPv4/20) into the largest memory database: the LEM. If you followed carefully, you noticed that a couple of improvement are needed to implement this sequence of lookup.
First, the match on LEM needs to be on an exact prefix length but the step two is done on IPv4/24 and IPv4/23. Indeed a function in DPA splits all IPv4/23 received from the upper FIB process in two. Each IPv4/23 is programmed as two sub-sequent IPv4/24s in hardware. We will illustrate this case with an example in the lab later in this post, advertising 300,000 IPv4/23.
Second, the exact match in step 3 for IPv4/20 in LEM is only possible if we don’t have any IPv4/22 or IPv4/21 prefixes overlapping with this IPv4/20. This implies the system performs another pro-active check to verify we don’t have overlap. If an overlap happens, the IPv4/20 prefix is moved from LEM to LPM dynamically. We will illustrate this mechanism later in the post, advertising 100,000 IPv4/20 first, then advertising 100,000 IPv4/21 overlapping on the IPv4/20. We will see the IPv4/20 moved into LPM automatically.
With this Internet Optimized profile activated, it’s possible to store a full internet view on base systems and line cards (we will present a couple of examples at the end of the documents).
- LEM is 786k large
- LPM scales from 256k to 350-400k (depending on the internet distribution, this algorithmic memory is dynamically optimized)
- Total IPv4 scale for base systems is 786k + 350k = 1,136k routes
What about the scale line cards and routers (NCS5501-SE, NCS5502-SE and all the -SE line cards) ?
The two optimized profiles described earlier don’t impact the lookup process on scale systems, which will always follow this order of operation:
Just a two-step lookup here:
- first lookup is in LEM for an exact match on IPv4/32
- second and last lookup in the large eTCAM for everything between IPv4/31 and /0
Needless to say, it is all done in one single operation in the Forwarding ASIC.
- LEM is 786k large
- eTCAM can offer up to 2M IPv4 entries
- Total IPv4 scale for scale systems is 786k + 2M = 2,786k routes
Lab verification
Let’s try to illustrate it in the lab, injecting different types of IPv4 routes.
On NCS5500, the IOS XR CLI to verify the resource utilization is “show controller npu resources all location 0/x/CPU0”.
We will advertise prefixes from a test device and check the memory utilization. It’s not an ideal approach because, for the sake of simplicity, the routes advertised through BGP are contiguous:
RP/0/RP0/CPU0:NCS5500#sh route bgp
B 2.0.0.0/32 [20/0] via 192.168.1.2, 04:13:13
B 2.0.0.1/32 [20/0] via 192.168.1.2, 04:13:13
B 2.0.0.2/32 [20/0] via 192.168.1.2, 04:13:13
B 2.0.0.3/32 [20/0] via 192.168.1.2, 04:13:13
B 2.0.0.4/32 [20/0] via 192.168.1.2, 04:13:13
B 2.0.0.5/32 [20/0] via 192.168.1.2, 04:13:13
B 2.0.0.6/32 [20/0] via 192.168.1.2, 04:13:13
[...]
RP/0/RP0/CPU0:NCS5500#
Despite common belief, it’s not an ideal situation. On the contrary, algorithmic memories (like LPM) will be capable of much higher scale with real internet prefix-length distribution. Nevertheless, it’s still an ok approach to demonstrate where the prefixes are stored (based on the subnet length).
We will take a look at two systems using scale line cards (24H12F) in slot 0/6 and base line cards (18H18F) in slot 0/0, and running two different IOS XR releases (6.1.4 and 6.2.2).
200k IPv4 /32 routes
Let’s get started with the advertisement of 200,000 IPv4/32 prefixes.
On base line cards running Host-optimized profile, IPv4/32 routes are going to LEM.
RP/0/RP0/CPU0:NCS5500-614#sh route sum
Route Source Routes Backup Deleted Memory(bytes)
connected 6 1 0 1680
local 7 0 0 1680
static 2 0 0 480
ospf 100 0 0 0 0
dagr 0 0 0 0
bgp 100 200000 0 0 48000000
Total 200015 1 0 48003840
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /32 | utility wc -l
200000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
[...]
Current Usage
NPU-0
Total In-Use : 200131 (25 %)
iproute : 200029 (25 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
[...]
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 87036
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 148 (0 %)
iproute : 5 (0 %)
ip6route : 117 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
Estimated Max Entries (and the Current Usage percentage derived from it) are only estimations provided by the Forwarding ASIC based on the current memory occupation and prefix distribution. It’s not always linear and should be taken with a grain of salt.
On base line cards running Internet-optimized profile, IPv4/32 routes are going to LPM:
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /32 | utility wc -l
200000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 107 (0 %)
iproute : 5 (0 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 323057
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 200171 (62 %)
iproute : 200029 (62 %)
ip6route : 116 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
Finaly, on scale line card, regardless of the profile enabled, the IPv4/32 are stored in LEM and not eTCAM:
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/6/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 200127 (25 %)
iproute : 200024 (25 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources exttcamipv4 location 0/6/CPU0
HW Resource Information
Name : ext_tcam_ipv4
OOR Information
NPU-0
Estimated Max Entries : 2048000
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 10 (0 %)
iproute : 10 (0 %)
ipmcroute : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
500k IPv4 /24 routes
In this second example, we announce 500,000 IPv4/24 prefixes. With both host-optimized and internet-optimized profiles on base line cards, we will see these prefixes moved into LEM.
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /24 | utility wc -l
500000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 500131 (64 %)
iproute : 500029 (64 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 87036
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 148 (0 %)
iproute : 5 (0 %)
ip6route : 117 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
On scale line cards, only IPv4/32s are going to LEM. The rest (that includes our 500,000 IPv4/24s) will be pushed to the external TCAM:
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/6/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 127 (0 %)
iproute : 24 (0 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/6/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 118638
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 144 (0 %)
iproute : 0 (0 %)
ip6route : 117 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources exttcamipv4 location 0/6/CPU0
HW Resource Information
Name : ext_tcam_ipv4
OOR Information
NPU-0
Estimated Max Entries : 2048000
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 500010 (24 %)
iproute : 500010 (24 %)
ipmcroute : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
300k IPv4 /23 routes
In this third example, we announce 300,000 IPv4/23 prefixes. With the Host-optimized profiles on base line cards, they will be moved to the LPM.
RP/0/RP0/CPU0:NCS5500-614#sh route sum
Route Source Routes Backup Deleted Memory(bytes)
connected 6 1 0 1680
local 7 0 0 1680
static 2 0 0 480
ospf 100 0 0 0 0
dagr 0 0 0 0
bgp 100 300000 0 0 72000000
Total 300015 1 0 72003840
RP/0/RP0/CPU0:NCS5500-614#sh route bgp
B 110.0.0.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.2.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.4.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.6.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.8.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.10.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.12.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.14.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.16.0/23 [20/0] via 192.168.1.2, 00:18:17
B 110.0.18.0/23 [20/0] via 192.168.1.2, 00:18:17
^c
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /23 | utility wc -l
300000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 131 (0 %)
iproute : 29 (0 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 261968
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Red
OOR State Change Time : 2017.Aug.04 06:59:54 UTC
Current Usage
NPU-0
Total In-Use : 261714 (100 %)
iproute : 261571 (100 %)
ip6route : 117 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
Only 261k IPv4/23 prefixes out of the 300k were programmed. Then, we reached the max of the memory capacity (we removed the error messages reporting that extra entries have not been programmed in hardware because the LPM capacity was exceeded).
Let’s enable the Internet-optimized profile (and reload).
This time, the 300,000 IPv4/23 will be split into two, creating 600,000 IPv4/24. And we will move them into LEM. Note: routes are not split into RIB/FIB but just when programmed into the hardware. That’s why a show route will display 300,000 entries and the show contr npu resource will display 600,000.
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /23 | utility wc -l
300000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 600107 (76 %)
iproute : 600005 (76 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 140729
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 171 (0 %)
iproute : 29 (0 %)
ip6route : 116 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
The same example with scale line cards will not be dependant on the optimized profile activated, all the IPv4/23 routes will be stored in external TCAM:
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/6/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 127 (0 %)
iproute : 24 (0 %)
ip6route : 0 (0 %)
mplslabel : 102 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/6/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 117819
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 143 (0 %)
iproute : 0 (0 %)
ip6route : 116 (0 %)
ipmcroute : 50 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources exttcamipv4 location 0/6/CPU0
HW Resource Information
Name : ext_tcam_ipv4
OOR Information
NPU-0
Estimated Max Entries : 2048000
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 300010 (15 %)
iproute : 300010 (15 %)
ipmcroute : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
100k IPv4 /20 routes
In this last example, we announce 100,000 IPv4/20 prefixes. You got it, so no need to describe:
- the host-optimized profile on base line cards where these 100k will be stored in LPM
- the scale cards where these routes will be pushed to the external TCAM
Let’s focus on the behavior with base line cards running an Internet-optimized profile. We only advertise IPv4/20 routes and no overlapping routes, they will be all stored in LEM.
RP/0/RP0/CPU0:NCS5500-614#sh bgp sum
[...]
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.11.12.1 0 100 13287 1466 4300005 0 0 13:11:23 100000
RP/0/RP0/CPU0:NCS5500-614#sh route bgp
B 3.0.0.0/20 [200/0] via 192.168.1.2, 10:05:09
B 3.0.16.0/20 [200/0] via 192.168.1.2, 10:05:09
B 3.0.32.0/20 [200/0] via 192.168.1.2, 10:05:09
B 3.0.48.0/20 [200/0] via 192.168.1.2, 10:05:09
B 3.0.64.0/20 [200/0] via 192.168.1.2, 10:05:09
B 3.0.80.0/20 [200/0] via 192.168.1.2, 10:05:09
B 3.0.96.0/20 [200/0] via 192.168.1.2, 10:05:09
[...]
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /20 | utility wc -l
100000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 100002 (13 %)
iproute : 100002 (13 %)
ip6route : 0 (0 %)
mplslabel : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 148883
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 153 (0 %)
iproute : 19 (0 %)
ip6route : 113 (0 %)
ipmcroute : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
Now, we advertise 100,000 new IPv4/21 routes. All are overlapping the IPv4/20 we announced earlier. The IPv4/20 will no longer be stored in LEM but will be moved into LPM, for a total of 200,000 entries:
RP/0/RP0/CPU0:NCS5500-614#sh bgp sum
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.11.12.1 0 100 13901 1474 4600005 0 0 13:19:24 200000
RP/0/RP0/CPU0:NCS5500-614#sh route bgp
Sat Aug 5 00:15:38.987 UTC
B 3.0.0.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.0.0/21 [200/0] via 192.168.1.2, 00:02:27
B 3.0.16.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.16.0/21 [200/0] via 192.168.1.2, 00:02:27
B 3.0.32.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.32.0/21 [200/0] via 192.168.1.2, 00:02:27
B 3.0.48.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.48.0/21 [200/0] via 192.168.1.2, 00:02:27
B 3.0.64.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.64.0/21 [200/0] via 192.168.1.2, 00:02:27
B 3.0.80.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.80.0/21 [200/0] via 192.168.1.2, 00:02:27
B 3.0.96.0/20 [200/0] via 192.168.1.2, 00:00:02
B 3.0.96.0/21 [200/0] via 192.168.1.2, 00:02:27
[...]
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /20 | utility wc -l
100000
RP/0/RP0/CPU0:NCS5500-614#sh route bgp | i /21 | utility wc -l
100000
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lem location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 2 (0 %)
iproute : 2 (0 %)
ip6route : 0 (0 %)
mplslabel : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#sh contr npu resources lpm location 0/0/CPU0
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 539335
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 200153 (37 %)
iproute : 200019 (37 %)
ip6route : 113 (0 %)
ipmcroute : 0 (0 %)
RP/0/RP0/CPU0:NCS5500-614#
Real use-cases
To conclude, let’s illustrate with real but anonymized use-cases.
On a base system running IOS XR 6.2.2 with internet-optimized profile and a “small” internet table.
RP/0/RP0/CPU0:5501#show route sum
Route Source Routes Backup Deleted Memory(bytes)
connected 2 3 0 1200
local 5 0 0 1200
local LSPV 1 0 0 240
static 2 0 0 480
ospf 1 677 2 0 163072
bgp xxxx 615680 10 0 147765600
dagr 0 0 0 0
Total 616367 15 0 147931792
RP/0/RP0/CPU0:5501#show dpa resources iproute location 0/0/CPU0
"iproute" DPA Table (Id: 17, Scope: Global)
--------------------------------------------------
IPv4 Prefix len distribution
Prefix Actual Prefix Actual
/0 1 /1 0
/2 0 /3 0
/4 1 /5 0
/6 0 /7 0
/8 15 /9 9
/10 35 /11 102
/12 277 /13 527
/14 955 /15 1703
/16 12966 /17 7325
/18 12874 /19 23469
/20 35743 /21 39283
/22 72797 /23 60852
/24 346773 /25 3
/26 19 /27 21
/28 17 /29 13
/30 229 /31 0
/32 368
[...]
RP/0/RP0/CPU0:5501#show contr npu resources all location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 498080 (63 %)
iproute : 507304 (65 %)
ip6route : 12818 (2 %)
mplslabel : 677 (0 %)
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 510070
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 192543 (38 %)
iproute : 176254 (35 %)
ip6route : 15583 (3 %)
ipmcroute : 0 (0 %)
Same route distribution on a scale system running IOS XR 6.2.2:
RP/0/RP0/CPU0:5501-SE#show route sum
Route Source Routes Backup Deleted Memory(bytes)
connected 4 3 0 1680
local 7 0 0 1680
local LSPV 1 0 0 240
static 2 0 0 480
ospf 1 677 2 0 163072
bgp xxxx 615681 10 0 147765840
dagr 0 0 0 0
Total 616372 15 0 147932992
RP/0/RP0/CPU0:5501-SE#show dpa resources iproute location 0/0/CPU0
"iproute" DPA Table (Id: 17, Scope: Global)
--------------------------------------------------
IPv4 Prefix len distribution
Prefix Actual Capacity Prefix Actual Capacity
/0 1 20 /1 0 20
/2 0 20 /3 0 20
/4 1 20 /5 0 20
/6 0 20 /7 0 20
/8 15 20 /9 9 20
/10 35 205 /11 102 409
/12 277 818 /13 527 1636
/14 955 3275 /15 1703 5731
/16 12966 42368 /17 7325 25379
/18 12874 42571 /19 23469 86576
/20 35743 127308 /21 39283 141634
/22 72797 231894 /23 60852 207107
/24 346773 1105235 /25 4 4298
/26 19 4503 /27 21 3275
/28 17 2865 /29 13 6959
/30 231 2865 /31 0 205
/32 376 20
[…]
RP/0/RP0/CPU0:5501-SE#show contr npu resources all location 0/0/CPU0
HW Resource Information
Name : lem
OOR Information
NPU-0
Estimated Max Entries : 786432
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 13887 (2 %)
iproute : 376 (0 %)
ip6route : 12827 (2 %)
mplslabel : 677 (0 %)
HW Resource Information
Name : lpm
OOR Information
NPU-0
Estimated Max Entries : 551346
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 15612 (3 %)
iproute : 0 (0 %)
ip6route : 15589 (3 %)
ipmcroute : 0 (0 %)
HW Resource Information
Name : ext_tcam_ipv4
OOR Information
NPU-0
Estimated Max Entries : 2048000
Red Threshold : 95 %
Yellow Threshold : 80 %
OOR State : Green
Current Usage
NPU-0
Total In-Use : 616012 (30 %)
iproute : 616012 (30 %)
ipmcroute : 0 (0 %)
Examples above show it’s possible to store a full internet view in a base system.
With a relatively small table of 616k routes, we have LEM used at approximatively 65%. But it’s frequent to see larger internet tables (closer to 700k in August 2017), with many peering routes and internal routes. It still fits in but doesn’t give much room for future growth.
We advise to prefer scale line cards and systems for such use-cases.
In the next episode, we will cover IPv6 prefixes. Stay tuned.
Leave a Comment