NCS5500 Routing Resource with 2020 Internet (and Future)

27 minutes read

NCS 5500 INTERNAL RESOURCES.png

You can find more content related to NCS5500 including routing memory management, VRF, URPF, Netflow, QoS, EVPN, Flowspec implementation following this link.

Introduction

Between September 2017 and March 2018, we published five articles to answer recurrent questions on the routing memory utilization. The NCS5500 adoption was ramping up, and users were learning about the LEM, LPM and eTCAM used to store the routing information.
1 - NCS5500 Resources focusing on IPv4 prefixes:
https://xrdocs.io/ncs5500/tutorials/2017-08-03-understanding-ncs5500-resources-s01e02/
2 - NCS5500 Resources focusing on IPv6 prefixes:
https://xrdocs.io/ncs5500/tutorials/2017-08-07-understanding-ncs5500-resources-s01e03/
3 - full internet support on non-eTCAM systems:
https://xrdocs.io/ncs5500/tutorials/2017-12-30-full-internet-view-on-base-ncs-5500-systems-s01e04/
4 - very large routing table in eTCAM systems:
https://xrdocs.io/ncs5500/tutorials/2018-01-25-s01e05-large-routing-tables-on-scale-ncs-5500-systems/
5 - NCS5500 Jericho+ Systems and their Scalability:
https://xrdocs.io/ncs5500/tutorials/Understanding-ncs5500-jericho-plus-systems/

In the third article, we demonstrated a feature used to optimize the route distribution between LEM and LPM to internet distribution. It was updated with a disclaimer in September 2019, we don’t recommend to use non-SE systems (Jericho or Jericho+) for full internet view. It will be different with Jericho2, but we will keep it for a dedicated post.

We have now three remaining options:

  • Jericho / Qumran-MX with NL eTCAM
  • Jericho+ with OP eTCAM
  • Jericho+ with large LPM

Let’s try to predict the future (with some really bad science)

Based on the routing table growth during the last 6 years, we can try to “project” the evolution of the table in the near future (let’s say, 9 years).
From a pure scientific perspective, it is not worth much. I admit it. So let’s take it for what it is, a wet finger estimation.
Let me explain the “method” used to get the number we will eventually use to estimate the resources utilization.

Sources: routes as seen in AS6447:
https://bgp.potaroo.net/as6447/
and
https://bgp.potaroo.net/v6/as6447/

Also, we used the following data:

And to convert epoch / UNIX timestamps in human readable dates: https://www.epochconverter.com/

I started writing the article the 10th of July, so I collected the figures from the sources above the 10th of July of each year.

YearIPv4 totalIPv6 totalv4/24v6/48
2014522,31319,025275,8057,978
2015587,10524,651310,94910,607
2016645,83333,495351,52714,438
2017710,97643,249390,41318,727
2018761,17059,330422,45425,383
2019817,79974,613464,43334,508
2020865,27491,133500,53044,876

Let’s focus on the IPv4 part first, and more particularly to the /24 population.

YearIPv4 totalGrowthv4/24GrowthGrowth Increasenon v4/24GrowthGrowth Increase
2014522,313-275,805--246,508--
2015587,10564,792310,94935,144-276,15629,648-
2016645,83358,728351,52740,5785,434294,30618,150-11,498
2017710,97665,143390,41338,886-1,692320,56326,2578,107
2018761,17050,194422,45432,041-6,845338,71618,153-8,104
2019817,79956,629464,43341,9799,938353,36614,650-3,503
2020865,27447,475500,53036,097-5,882364,74411,378-3,272

The “Growth” is showing the difference between two sub-sequent years and the “Growth Increase” shows the difference between two sub-sequent growth rates. The second number should help identifying a linear or an algorithmic progression.

IPv4/24 growth trend: it doesn’t seem we have a clear trend here, between the different years we see various numbers scattered from -6,845 to +9,938.

Conclusion: we will consider it’s a linear growth and we will estimate the number of new IPv4/24 per year: +42,000 prefixes.

Non-IPv4 growth: except in 2017, it shows the growth is progressively slowing down.

Conclusion: here it will be a totally arbitrary decision to continue this trend with lower and lower numbers (starting from -2,000 in 2021 to -200 in 2029).

The results of this projection are the following:

YearIPv4 totalGrowthv4/24GrowthGrowth Increasenon v4/24GrowthGrowth Increase
2019817,79956,629464,43341,9799,938353,36614,650-3,503
2020865,27447,475500,53036,097-5,882364,74411,378-3,272
2021919,27454,000542,53042,0005,903376,7449,378-2,000
2022969,15249,878584,53042,0000384,6227,878-1,500
20231,018,03048,878626,53042,0000391,5006,878-1,000
20241,066,10848,078668,53042,0000397,5786,078-800
20251,113,58647,478710,53042,0000403,0565,478-600
20261,160,56446,978752,53042,0000408,0344,978-500
20271,207,14246,578794,53042,0000412,6124,578-400
20281,253,42046,278836,53042,0000416,8904,278-300
20291,299,49846,078878,53042,0000420,9684,078-200

IPv4-projection.png

Checking other sources on the web, like this APNIC article from Geoff Huston:
https://blog.apnic.net/2020/01/14/bgp-in-2019-the-bgp-table/
They predict 1,079,000 routes for Jan 2025, so it matches our “model”, predicting something between 1,066,108 and 1,113,586 IPv4 prefixes between July 2024 and July 2025.

Let’s study the IPv6 internet table evolution. Due to the smallest size of this table, it’s more hazardous to create a projection, but it won’t prevent me from making arbitrary asumptions to build my model ;)

YearIPv6 totalGrowthGrowth Increasev6/48GrowthGrowth Increasenon v6/48GrowthGrowth Increase
201419,025--7,978--11,047--
201524,6515,626-10,6072,629-14,0442,997-
201633,4958,8443,21814,4383,8311,20219,0575,0132,016
201743,2499,75491018,7274,28945824,5225,465452
201859,33016,0816,32725,3836,6562,36733,9479,4253,960
201974,61315,283-79834,5089,1252,46940,1056,158-3,267
202091,13316,5201,2374,487610,3681,24346,2576,152-6

More particularly, we will pay attention to the /48 prefixes and their progression.

IPv6/48 growth trend: The “growth increase” varies from +458 to +2,469 during the last 5 years.

Conclusion: Let’s take the highest number for the projection, rounded to +2500.

Non-IPv6/48 growth trend: The “growth” varies from 2,997 to 9,425 but with a majority of the years around 6,000.

Conclusion: For this one, we will take a totaly arbitrary growth of 6,000 new IPv6 prefixes (non /48) per year.

The results of this projection are the following:

YearIPv6 totalGrowthGrowth Increasev6/48GrowthGrowth Increasenon v6/48GrowthGrowth Increase
201974,61315,283-79834,5089,1252,46940,1056,158-3,267
202091,13316,5201,23744,87610,3681,24346,2576,152-6
2021110,00118,8682,34857,74412,8682,50052,2576,000-152
2022131,36921,3682,50073,11215,3682,50058,2576,0000
2023155,23723,8682,50090,98017,8682,50064,2576,0000
2024181,60526,3682,500111,34820,368250070,2576,0000
2025210,47328,8682,500134,21622,8682,50076,2576,0000
2026241,84131,3682 500159,58425,3682,50082,2576,0000
2027275,70933,8682,500187,45227,8682,50088,2576,0000
2028312,07736,3682,500217,82030,3682,50094,2576,0000
2029350,94538,8682,500250,68832,8682,500100,2576,0000

IPv6-projection.png

Here again we can compare this guesstimation with Geoff’s projection
https://blog.apnic.net/2020/01/14/bgp-in-2019-the-bgp-table/
In January 2025, they present a range starting from 160,000 for the linear model and 318,000 for the algorithmic one, while we plan for something between 181,605 and 210,473.

Admittedly, it’s pretty vague ;)

Let’s take these numbers nevertheless and see where these routes will be stored depending on the different products.

In summary, we project the following distribution:

Date (10th of July of each year)IPv4 totalv4/24v4 non/24IPv6 totalv6/48v6 non/48
2020865,274500,530364,74491,13344,87646,257
2021919,274542,530376,744110,00157,74452,257
2022969,152584,530384,622131,36973,11258,257
20231,018,030626,530391,500155,23790,98064,257
20241,066,108668,530397,578181,605111,34870,257
20251,113,586710,530403,056210,473134,21676,257
20261,160,564752,530408,034241,841159,58482,257
20271,207,142794,530412,612275,709187,45288,257
20281,253,420836,530416,890312,077217,82094,257
20291,299,498878,530420,968350,945250,688100,257

Products and ASICs

In summary:

Jericho / Qumran-MX with NL eTCAMJericho+ with OP eTCAMJericho+ with large LPM (and no eTCAM)
NCS5501-SENCS55A1-36H-SE-SNCS55A1-24H
NCS5502-SENCS55A2-MOD-SE-SNCS55A1-48Q6H
NC55-24X100G-SENC55-A-36X100-SE-SNCS55A1-24H6H-SS

Jericho / Qumran-MX with NL eTCAM

Products:

  • NCS5501-SE
  • NCS5502-SE
  • NC55-24X100G-SE

JerichoNL12k.png

Jericho+ with OP eTCAM

Products:

  • NCS55A1-36H-SE-S
  • NCS55A2-MOD-SE-S
  • NC55-A-36X100-SE-S

Jplus-OP.png

Jericho+ with large LPM (and no eTCAM)

Products:

  • NCS55A1-24H
  • NCS55A1-48Q6H
  • NCS55A1-24H6H-SS

Case 1: Default configuration is Host Optimized

LargeLPM-host-opt.png

Case 2: user changed to host-optimized-disable

LargeLPM-host-disable.png

Lab and Test

In this section, we will inject a real table in the routers and collect utilization statistics for the different resources (LEM, LPM and potentially external TCAM). Then, we will inject v4/24s, v6/48s and other routes following the estimated progression described above, and we will see the impact on resources, year after year.

It certainly very fun (is it?) but the real purpose of this exercice is to extract valuable and actionable information out of these tests. For example, what hw-module profile we should use in the future depending on the ASIC type.

Starting point

We use a public v4/v6 view collected in 2019. It shows:

  • 790,780 IPv4 prefixes
    • 445,773 /24s
    • 345,007 non-/24s
  • 72,949 IPv6 prefixes
    • 35,009 /48s
    • 37,940 non-/48s

Jericho / Qumran-MX with NL eTCAM

RP/0/RP0/CPU0:5508-2-702#sh bgp sum
BGP router identifier 1.3.5.8, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 4539177
BGP main routing table version 4539177
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker         4539177    4539177    4539177    4539177     4539177           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
192.168.100.153   0   100 1719117      16  4539177    0    0 00:13:25     790771

RP/0/RP0/CPU0:5508-2-702#
RP/0/RP0/CPU0:5508-2-702#sh bgp ipv6 un sum
BGP router identifier 1.3.5.8, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0800000   RD version: 72953
BGP main routing table version 72953
BGP NSR Initial initsync version 4 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker           72953      72953      72953      72953       72953           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
2001:111::151     0   100   72965      16    72953    0    0 00:14:01      72948

RP/0/RP0/CPU0:5508-2-702#
RP/0/RP0/CPU0:5508-2-702#sh dpa resources iproute loc 0/7/CPU0

"iproute" OFA Table (Id: 41, Scope: Global)
--------------------------------------------------
IPv4 Prefix len distribution
Prefix   Actual            Capacity    Prefix   Actual            Capacity
 /0       26                20           /1       0                 20
 /2       4                 20           /3       7                 20
 /4       26                20           /5       0                 20
 /6       0                 20           /7       0                 20
 /8       11                20           /9       12                20
 /10      36                204          /11      97                409
 /12      285               818          /13      571               1636
 /14      1143              3275         /15      1913              5732
 /16      13184             42381        /17      7901              25387
 /18      13534             42585        /19      25210             86603
 /20      39182             127348       /21      47039             141679
 /22      100821            231968       /23      78898             207173
 /24      445838            1105590      /25      144               4299
 /26      211               4504         /27      383               3275
 /28      537               2866         /29      721               6961
 /30      3241              2866         /31      440               204
 /32      9657              20

OFA Infra Stats Summary
                 Create Requests: 1846483
                 Delete Requests: 1055411
                 Update Requests: 241726
                    Get Requests: 0

                          Errors
                Resolve Failures: 0
                 Not Found in DB: 0
                    Exists in DB: 0
                 No Memory in DB: 0
               Reserve Resources: 0
               Release Resources: 0
                Update Resources: 0

                          NPU ID: NPU-0                  NPU-1                  NPU-2                  NPU-3
           Create Server API Err: 0                      0                      0                      0
           Update Server API Err: 0                      0                      0                      0
           Delete Server API Err: 0                      0                      0                      0

RP/0/RP0/CPU0:5508-2-702#sh dpa resources ip6route loc 0/7/CPU0

"ip6route" OFA Table (Id: 42, Scope: Global)
--------------------------------------------------
IPv6 Prefix len distribution
Prefix   Actual       Prefix   Actual
 /0       25           /1       0
 /2       0            /3       0
 /4       0            /5       0
 /6       0            /7       0
 /8       0            /9       0
 /10      25           /11      0
 /12      0            /13      0
 /14      0            /15      0
 /16      76           /17      0
 /18      0            /19      2
 /20      12           /21      4
 /22      7            /23      5
 /24      23           /25      7
 /26      13           /27      18
 /28      94           /29      2696
 /30      418          /31      179
 /32      12686        /33      1060
 /34      814          /35      517
 /36      2887         /37      501
 /38      908          /39      282
 /40      3689         /41      544
 /42      888          /43      144
 /44      4720         /45      465
 /46      2223         /47      1352
 /48      35009        /49      0
 /50      0            /51      1
 /52      0            /53      0
 /54      0            /55      0
 /56      219          /57      2
 /58      0            /59      0
 /60      16           /61      0
 /62      0            /63      3
 /64      87           /65      0
 /66      0            /67      0
 /68      0            /69      0
 /70      0            /71      0
 /72      0            /73      0
 /74      0            /75      0
 /76      0            /77      0
 /78      0            /79      0
 /80      0            /81      0
 /82      0            /83      0
 /84      0            /85      0
 /86      0            /87      0
 /88      0            /89      0
 /90      0            /91      0
 /92      0            /93      0
 /94      0            /95      0
 /96      0            /97      0
 /98      0            /99      0
 /100     0            /101     0
 /102     0            /103     0
 /104     25           /105     0
 /106     0            /107     0
 /108     0            /109     0
 /110     0            /111     0
 /112     0            /113     0
 /114     0            /115     0
 /116     0            /117     0
 /118     0            /119     0
 /120     2            /121     0
 /122     0            /123     0
 /124     1            /125     8
 /126     432          /127     16
 /128     97

OFA Infra Stats Summary
                 Create Requests: 73268
                 Delete Requests: 66
                 Update Requests: 0
                    Get Requests: 0

                          Errors
                Resolve Failures: 0
                 Not Found in DB: 0
                    Exists in DB: 0
                 No Memory in DB: 0
               Reserve Resources: 0
               Release Resources: 0
                Update Resources: 0

                          NPU ID: NPU-0                  NPU-1                  NPU-2                  NPU-3
           Create Server API Err: 0                      0                      0                      0
           Update Server API Err: 0                      0                      0                      0
           Delete Server API Err: 0                      0                      0                      0

RP/0/RP0/CPU0:5508-2-702#

That places us somewhere between 2018 and 2019 on the estimation we built from the potaroo info. We will start from 2019 and calculate what needs to be advertised on top of our full views.

A simple substraction will tell us how many routes “extra” we need to advertise to simulate the growth along the years:

Yearv4/24Extra /24v4 non/24Extra othersv6/48Extra /48v6 non/48Extra others
2020500,53054,757364,74419,73744,8769,86746,2578,317
2021542,53096,757376,74431,73757,74422,73552,25714,317
2022584,530138,757384,62239,61573,11238,10358,25720,317
2023626,530180,757391,50046,49390,98055,97164,25726,317
2024668,530222,757397,57852,571111,34876,33970,25732,317
2025710,530264,757403,05658,049134,21699,20776,25738,317
2026752,530306,757408,03463,027159,584124,57582,25744,317
2027794,530348,757412,61267,605187,452152,44388,25750,317
2028836,530390,757416,89071,883217,820182,81194,25756,317
2029878,530432,757420,96875,961250,688215,679100,25762,317

At the starting point we have:

Jericho with NL eTCAM

RP/0/RP0/CPU0:5501-SE-6625#sh platf
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/RP0/CPU0        NCS-5501-SE(Active)        IOS XR RUN        NSHUT
0/RP0/NPU0        Slice                      UP
0/FT0             NCS-1RU-FAN-FW             OPERATIONAL       NSHUT
0/PM1             NCS-1100W-ACFW             OPERATIONAL       NSHUT
RP/0/RP0/CPU0:5501-SE-6625#sh contr npu resources lem loc 0/0/CPU0
HW Resource Information
    Name                            : lem

OOR Information
    NPU-0
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green

Current Usage
    NPU-0
        Total In-Use                : 45013    (6 %)
        iproute                     : 10006    (1 %)
        ip6route                    : 35009    (4 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)

RP/0/RP0/CPU0:5501-SE-6625#
RP/0/RP0/CPU0:5501-SE-6625#sh contr npu resources lpm loc 0/0/CPU0
HW Resource Information
    Name                            : lpm

OOR Information
    NPU-0
        Estimated Max Entries       : 534746
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green

Current Usage
    NPU-0
        Total In-Use                : 38969    (7 %)
        iproute                     : 0        (0 %)
        ip6route                    : 39733    (7 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)

RP/0/RP0/CPU0:5501-SE-6625#
RP/0/RP0/CPU0:5501-SE-6625#sh contr npu resources exttcamipv4 loc 0/0/CPU0
HW Resource Information
    Name                            : ext_tcam_ipv4

OOR Information
    NPU-0
        Estimated Max Entries       : 2048000
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green

Current Usage
    NPU-0
        Total In-Use                : 781297   (38 %)
        iproute                     : 782062   (38 %)

RP/0/RP0/CPU0:5501-SE-6625#

Jericho+ with OP eTCAM

RP/0/RP0/CPU0:5508-2-702#sh platf 0/0
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/0/CPU0          NC55-36X100G-A-SE          IOS XR RUN        NSHUT
RP/0/RP0/CPU0:5508-2-702#sh contr npu resources exttcamipv4 loc 0/0/CPU0
HW Resource Information
    Name                            : ext_tcam_ipv4
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 4000000
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 790992   (20 %)
        iproute                     : 791072   (20 %)

...
RP/0/RP0/CPU0:5508-2-702#sh contr npu resources exttcamipv6 loc 0/0/CPU0
HW Resource Information
    Name                            : ext_tcam_ipv6
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 2000000
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 73052    (4 %)
        ip6route                    : 73202    (4 %)

...
RP/0/RP0/CPU0:5508-2-702#

Jericho+ with Large LPM and no eTCAM

RP/0/RP0/CPU0:24H-1-701#sh platf
Node              Type                       State             Config state
--------------------------------------------------------------------------------
0/RP0/CPU0        NCS-55A1-24H(Active)       IOS XR RUN        NSHUT
0/RP0/NPU0        Slice                      UP
0/RP0/NPU1        Slice                      UP
0/FT0             NC55-A1-FAN-FW             OPERATIONAL       NSHUT
0/FT1             NC55-A1-FAN-FW             OPERATIONAL       NSHUT
0/PM0             NCS-1100W-ACFW             FAILED            NSHUT
0/PM1             NCS-1100W-ACFW             OPERATIONAL       NSHUT
RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lem loc 0/0/CPU0
HW Resource Information
    Name                            : lem
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 490276   (62 %)
        iproute                     : 455268   (58 %)
        ip6route                    : 35009    (4 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)
...
RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lpm loc 0/0/CPU0
HW Resource Information
    Name                            : lpm
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 1563508
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 373486   (24 %)
        iproute                     : 335526   (21 %)
        ip6route                    : 37956    (2 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)
...
RP/0/RP0/CPU0:24H-1-701#

Year 2020

We advertise the extra routes through a new peer (/24 + /23 for IPv4 and /48 + /47 for IPv6).

On the route generator:

[email protected]:~/routem$ more extra-v4-24H.2020
router bgp 100
bgp_id 192.168.100.153
neighbor 192.168.100.200 remote-as 100
neighbor 192.168.100.200 update-source 192.168.100.152
capability ipv4 unicast
capability refresh
network 1 11.1.1.0/24 54757
aspath 1 random 5
locpref 1 120
metric 1 5
network 2 51.1.1.0/23 19737
aspath 2 random 5
locpref 2 110
metric 2 10
sendall

[email protected]:~/routem$ more extra-v6-24H.2020
router bgp 152
bgp_id 192.168.100.11
neighbor 2001:111::200 remote-as 100
neighbor 2001:111::200 update-source 2001:111::152
capability ipv6 unicast
capability refresh
network 1 100:1:1:1/48 9867
aspath 1 random 5
locpref 1 120
metric 1 5
network 2 102:1:1:1/47 8317
aspath 2 random 5
locpref 2 110
metric 2 15
sendall
[email protected]:~/routem

And on the IOS XR router:

RP/0/RP0/CPU0:24H-1-701#sh bgp sum
BGP router identifier 1.3.5.9, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 1735636
BGP main routing table version 1735636
BGP NSR Initial initsync version 6 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker         1735636    1735636    1735636    1735636     1735636           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
192.168.100.151   0   100 1721352   12009  1735636    0    0    1d13h     790771
192.168.100.152   0   100     163      12  1735636    0    0 00:01:28      74494

RP/0/RP0/CPU0:24H-1-701#sh bgp ipv6 un sum
BGP router identifier 1.3.5.9, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0800000   RD version: 273405
BGP main routing table version 273405
BGP NSR Initial initsync version 6 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker          273405     273405     273405     273405      273405           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
2001:111::151     0   100  148151   12245   273405    0    0 21:04:39      72948
2001:111::152     0   152     110     128   273405    0    0 00:00:07      18184

RP/0/RP0/CPU0:24H-1-701#sh dpa resources iproute loc 0/0/CPU0 | i /24
 /24      500530       /25      144
RP/0/RP0/CPU0:24H-1-701#sh dpa resources ip6route loc 0/0/CPU0 | i /48
 /48      44876        /49      0
RP/0/RP0/CPU0:24H-1-701#

Jericho with NL eTCAM

RP/0/RP0/CPU0:5501-SE-6625#sh contr npu resources lem loc 0/0/CPU0
HW Resource Information
    Name                            : lem

OOR Information
    NPU-0
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green

Current Usage
    NPU-0
        Total In-Use                : 54881    (7 %)
        iproute                     : 10007    (1 %)
        ip6route                    : 44876    (6 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)

RP/0/RP0/CPU0:5501-SE-6625#sh contr npu resources lpm loc 0/0/CPU0
HW Resource Information
    Name                            : lpm

OOR Information
    NPU-0
        Estimated Max Entries       : 549919
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green

Current Usage
    NPU-0
        Total In-Use                : 47287    (9 %)
        iproute                     : 0        (0 %)
        ip6route                    : 48051    (9 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)

RP/0/RP0/CPU0:5501-SE-6625#sh contr npu resources exttcamipv4 loc 0/0/CPU0
HW Resource Information
    Name                            : ext_tcam_ipv4

OOR Information
    NPU-0
        Estimated Max Entries       : 2048000
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green

Current Usage
    NPU-0
        Total In-Use                : 855743   (42 %)
        iproute                     : 856508   (42 %)

RP/0/RP0/CPU0:5501-SE-6625#

Jericho+ with OP eTCAM

RP/0/RP0/CPU0:5508-2-702#sh contr npu resources exttcamipv4 loc 0/0/CPU0
HW Resource Information
    Name                            : ext_tcam_ipv4
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 4000000
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 865438   (22 %)
        iproute                     : 865518   (22 %)
...
RP/0/RP0/CPU0:5508-2-702#sh contr npu resources exttcamipv6 loc 0/0/CPU0
HW Resource Information
    Name                            : ext_tcam_ipv6
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 2000000
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 91237    (5 %)
        ip6route                    : 91387    (5 %)

...

Jericho+ with Large LPM and no eTCAM

RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lem loc 0/0/CPU0
HW Resource Information
    Name                            : lem
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 554898   (71 %)
        iproute                     : 510026   (65 %)
        ip6route                    : 44876    (6 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)

...
RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lpm loc 0/0/CPU0
HW Resource Information
    Name                            : lpm
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 1559724
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 401492   (26 %)
        iproute                     : 355214   (23 %)
        ip6route                    : 46274    (3 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)

...
RP/0/RP0/CPU0:24H-1-701#

Year 2021 to year 2029

We continue to increase the amount of “extra” routes step by step and we gather the results in following charts.

Jericho w/ NL12K eTCAM

YearLEM MaxLEM in-useLPM MaxLPM in-useeTCAM MaxeTCAM in-use
Starting Point78643245013534746389692048000781297
202078643254881549919472872048000855743
202178643267749554073532872048000908919
202278643283113558189592872048000957666
20237864321009855620756528720480001006238
20247864321213535664897128720480001053379
20257864321442215641167728720480001099520
20267864321695815683078328720480001146447
20277864321974495655088928720480001193007
20287864322278175698919528720480001239282

LPM-Jericho-NL12k.png

LEM-Jericho-NL12k.png

eTCAM-Jericho-NL12k.png

Conclusion: these devices can handle the internet growth with no concern or limitation.

Jericho+ w/ OP eTCAM

YeareTCAM Maxv4 in-usev6 in-use
Starting Point4M+2M79099273052
20204M+2M86543891237
20214M+2M918614110105
20224M+2M967361131473
20234M+2M1015933155341
20244M+2M1063074181709
20254M+2M1109215210577
20264M+2M1156142241945
20274M+2M1202702275813
20284M+2M1248977312181

Jplus OP.png

Conclusion: these devices can handle the internet growth with zero concern or limitation, we have a lot of available space in the OP eTCAM.

Jericho+ with Large LPM and host-optimized (default mode)

YearLEM MaxLEM in-useLPM MaxLPM in-use
Starting Point7864324902761563508373486
20207864325548981559724401492
20217864326089601575645419475
20227864326653241573918433222
20237864327249441572299446038
20247864327816321570948458084
2025----

LPM-LargeLPM-hostOptEnable.png

LEM-LargeLPM-hostOptEnable.png

We will cover what’s happening in 2024 in the next section.

Jericho+ with Large LPM and host-optimized-disable

YearLEM MaxLEM in-useLPM MaxLPM in-use
Starting Point786432445091349849819259
2020786432543731355608902022
2021786432672411386730961198
20227864328260914181571015945
202378643210047314229561070517
202478643212084114231071123656
202578643214370514125511175799
202678643216907714005641228726
202778643219694513881031281286
202878643222731313738801333561

LPM-LargeLPM-hostOptDisable.png

LEM-LargeLPM-hostOptDisable.png

Special case of the Jericho+ with Large LPM

When simulating year 2024, we hit the first bottleneck with the J+ with large LPM systems:

RP/0/RP0/CPU0:24H-1-701#sh bgp sum
BGP router identifier 1.3.5.9, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 3153690
BGP main routing table version 3153690
BGP NSR Initial initsync version 6 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker         3153690    3153690    3153690    3153690     3153690           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
192.168.100.151   0   100 1721534   12191  3153690    0    0    1d16h     790771
192.168.100.152   0   100    1118     155  3153690    0    0 00:00:14     275328

RP/0/RP0/CPU0:24H-1-701#
RP/0/RP0/CPU0:24H-1-701#sh bgp ipv6 un sum
BGP router identifier 1.3.5.9, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0800000   RD version: 755765
BGP main routing table version 755765
BGP NSR Initial initsync version 6 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker          755765     755765     755765     755765      755765           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
2001:111::151     0   100  148333   14655   755765    0    0    1d00h      72948
2001:111::152     0   152    1063     764   755765    0    0 00:00:33     108656

RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lem loc 0/0/CPU0
HW Resource Information
    Name                            : lem
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Red
        OOR State Change Time       : 2020.Jul.27 08:23:12 PDT


Current Usage
        Total In-Use                : 781632   (99 %)
        iproute                     : 670293   (85 %)
        ip6route                    : 111348   (14 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)


NPU-1
OOR Summary
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Red
        OOR State Change Time       : 2020.Jul.27 08:23:12 PDT


Current Usage
        Total In-Use                : 781632   (99 %)
        iproute                     : 670293   (85 %)
        ip6route                    : 111348   (14 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)


RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lpm loc 0/0/CPU0
HW Resource Information
    Name                            : lpm
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 1570948
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 458084   (29 %)
        iproute                     : 387806   (25 %)
        ip6route                    : 70274    (4 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)


NPU-1
OOR Summary
        Estimated Max Entries       : 1570948
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 458084   (29 %)
        iproute                     : 387806   (25 %)
        ip6route                    : 70274    (4 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)


RP/0/RP0/CPU0:24H-1-701#
RP/0/RP0/CPU0:24H-1-701#sh dpa resources iproute loc 0/0/CPU0

"iproute" OFA Table (Id: 37, Scope: Global)
--------------------------------------------------
IPv4 Prefix len distribution
Prefix   Actual       Prefix   Actual
 /0       1            /1       0
 /2       4            /3       7
 /4       1            /5       0
 /6       0            /7       0
 /8       10           /9       12
 /10      36           /11      97
 /12      285          /13      571
 /14      1143         /15      1913
 /16      13184        /17      7901
 /18      13534        /19      25210
 /20      39182        /21      47039
 /22      100821       /23      131178
 /24      665574       /25      144
 /26      211          /27      383
 /28      537          /29      721
 /30      3241         /31      440
 /32      9496

OFA Infra Stats Summary
                 Create Requests: 1740765
                 Delete Requests: 677889
                 Update Requests: 13819

                          Errors
                Resolve Failures: 0
                 Not Found in DB: 0
                    Exists in DB: 0
                 No Memory in DB: 0
               Reserve Resources: 0
               Release Resources: 0
                Update Resources: 0

                          NPU ID: NPU-0                  NPU-1
           Create Server API Err: 4777                   4777
           Update Server API Err: 0                      0
           Delete Server API Err: 0                      0

RP/0/RP0/CPU0:24H-1-701#

Starting from this point, the LEM is saturated while LPM is only used 29%.
It would be profitable to disable the default “host-optimized” mode. With this configuration (requiring a reload of the product), we store the IPv4/24 routes in LPM, which is the largest memory of the system.

RP/0/RP0/CPU0:24H-1-701#conf
RP/0/RP0/CPU0:24H-1-701(config)#hw-module fib ?
  dlb   Destination Based Load balancing
  ipv4  Configure ipv4 protocol
  ipv6  Configure ipv6 protocol
  mpls  Configure mpls protocol
RP/0/RP0/CPU0:24H-1-701(config)#hw-module fib ipv4 ?
  scale  Configure scale mode for no-TCAM card
RP/0/RP0/CPU0:24H-1-701(config)#hw-module fib ipv4 scale ?
  host-optimized-disable  Configure Host optimization by default
  internet-optimized      Configure Intetrnet optimized
RP/0/RP0/CPU0:24H-1-701(config)#hw-module fib ipv4 scale host-optimized-disable
In order to activate this new scale, you must manually reload the chassis/all line cards
RP/0/RP0/CPU0:24H-1-701(config)#commit
RP/0/RP0/CPU0:24H-1-701(config)#end
RP/0/RP0/CPU0:24H-1-701#

After reloading the router, we verify LEM and LPM:

RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lem loc 0/0/CPU0

HW Resource Information
    Name                            : lem
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 120841   (15 %)
        iproute                     : 9496     (1 %)
        ip6route                    : 111348   (14 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)

...
RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lp

HW Resource Information
    Name                            : lpm
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 1423107
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 1123656  (79 %)
        iproute                     : 1053378  (74 %)
        ip6route                    : 70275    (5 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)

LEM is now very lightly used (15%) and we will use massively the LPM for most of our prefixes. That’s why it’s now loaded at 79% but keep in mind it’s the largest memory of the system.

We continue advertising more and more routes and in 2028, we are getting very close to the limit for this chipset:

RP/0/RP0/CPU0:24H-1-701#sh bgp sum
BGP router identifier 1.3.5.9, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 5960575
BGP main routing table version 5960575
BGP NSR Initial initsync version 629974 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker         5960575    5960575    5960575    5960575     5960575           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
192.168.100.151   0   100 1719200      99  5960575    0    0 01:36:51     790771
192.168.100.152   0   100    2525      63  5960575    0    0 00:00:14     462640

RP/0/RP0/CPU0:24H-1-701#sh bgp ipv6 un sum
BGP router identifier 1.3.5.9, local AS number 100
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0800000   RD version: 1939637
BGP main routing table version 1939637
BGP NSR Initial initsync version 181612 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer
Speaker         1939637    1939637    1939637    1939637     1939637           0

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
2001:111::151     0   100   73048    8961  1939637    0    0 01:37:08      72949
2001:111::152     0   152    3031    1169  1939637    0    0 00:00:41     239128

RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lem loc 0/0/CPU0
HW Resource Information
    Name                            : lem
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 786432
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Green


Current Usage
        Total In-Use                : 227313   (29 %)
        iproute                     : 9496     (1 %)
        ip6route                    : 217820   (28 %)
        mplslabel                   : 0        (0 %)
        l2brmac                     : 0        (0 %)

...
RP/0/RP0/CPU0:24H-1-701#sh contr npu resources lpm loc 0/0/CPU0
HW Resource Information
    Name                            : lpm
    Asic Type                       : Jericho Plus

NPU-0
OOR Summary
        Estimated Max Entries       : 1373880
        Red Threshold               : 95
        Yellow Threshold            : 80
        OOR State                   : Red
        OOR State Change Time       : 2020.Jul.27 10:16:25 PDT


Current Usage
        Total In-Use                : 1333561  (97 %)
        iproute                     : 1239283  (90 %)
        ip6route                    : 94275    (7 %)
        ipmcroute                   : 1        (0 %)
        ip6mcroute                  : 0        (0 %)
        ip6mc_comp_grp              : 0        (0 %)


...
RP/0/RP0/CPU0:24H-1-701#

Conclusion: it’s advisable to disable the default profile “host-optimized”, it will extend significatively the router capability when used with full internet view.

Conclusion

In this post, we created a prevision model for the internet size progression.
No doubt it can be refined. For example, we only considered the “non-IPv4/24 routes” and “non-IPv6/48 routes” as a block. Also because we advertised only subsequent /23s and /47s to create the “extra prefixes”. Much better can be done here (hopefully by someone else ;)).
Take all this with a grain of salt. Also, if you are aware of more precise prediction models (with an evolution for individual prefix length), please let us know. In the lab, we simulated internet routing from 2020 to 2028 and we examined each memory consumption (LEM, LPM and when present, eTCAM).
The systems with external TCAM are clearly showing a ton of free space for internet, even in 8 years.
The systems based on Jericho+ with large LPM can also be used for internet peering, but it may be required to disable the “host-optimized” mode in a couple of years (around 2024) to leverage the large size of the LPM and offer 4+ more years of growth.

Next episode: Jericho2. Stay tuned.

Leave a Comment