NCS5500 QoS Part 2 - Verifying Buffering in Lab and Live Networks

You can find more content related to NCS5500 including routing memory management, VRF, URPF, ACLs, Netflow following this link.

Also you can find the first part of this post here:
https://xrdocs.io/ncs5500/tutorials/ncs5500-qos-part-1-understanding-packet-buffering/

Checking Buffering in action

This second blog post will take concrete examples to illustrate the concepts covered in the first part.
The NCS5500 is based on a VOQ-only, single-lookup and ingress-buffering forwarding architecture.
We will use a lab example to illustrate how the system handles bursts, then we will present the monitoring tools / counters we can use to measure where packets are buffered, and finally we will present the data collected on 500+ NPUs in production.
This should answer frequently asked questions and clarify all potential doubts.

Video

We recommend to start watching this short Youtube video first:
https://www.youtube.com/watch?v=1qXD70_cLK8

Lab test

For this first part, and following customer request, we built a large test bed:

• NCS5508 with two line cards 36x100G-A-SE (each card is made of 4x NPU Jericho+, each one handling 9 ports 100G)
• 27 tester ports 100GE (Spirent) connected to 27 router ports
  • 9 ports on LC 4 NPU 0
  • 9 ports on LC 4 NPU 1
  • 9 ports on LC 6 NPU 0

We generate a background / constant traffic of 80% line rate (80Gbps on each port) between two NPUs. This bi-directional traffic is displayed in purple in the diagram above.
Then, we will use the remaining 9 ports to generate peaks of traffic targeted to the ports Hu0/6/0/0-8 (shown in red in the diagram).
These bursts are lasting 100ms, every second.

On the tester, we didn’t make any specific PPM adjustment and used internal clock.

On the router, no specific configuration either. Interfaces are simply configured with IPv4 addresses (no QoS).

The tests performed are the following:

• test 1: 80% background and all the ports bursting at 20%. That means:
  • 900ms at 80% LR
  • 100ms at 100% LR
    We verify no packets are dropped on the background or the bursts, and we also make sure with the counters that packets are exclusely handled in the OCB.
• test 2 :
  • 80% background
  • other ports bursting at 20%
  • one single port bursts at 25%, creating a 5Gbps saturation for 100ms
    Here again, we verify no packets are dropped on the background or the bursts, but also we verify that packets are sent to the DRAM.
    It’s expected since one queue exceeds the threshold and is evicted to the external buffer.

This test is basic but has been requested by several customers to verify we had no drop in such situations. It proves it’s not the case, as designed.

Metrology

In the former test, we used specific counters to verify the buffering behavior.
Let’s review them.

We will collect the following Broadcom counters:

• IQM_EnqueuePktCnt: total number of packets handled by the NPU
• IDR_MMU_CREDITS: total number of packets moved to DRAM
• IQM_EnqueueDscrdPktCnt: total number of packets dropped because of taildrop
• IQM_RejectDramIneligiblePktCnt: total number of packets dropped because DRAM was not accessible in read, typically when the bandwidth to DRAM is saturated
• and potentially also IDR_FullDramRejectPktsCnt and IDR_PartialDramRejectPktsCnt

Form CLI “show controller npu stats counters-all instance all location all” we can extract: ENQUEUE_PKT_CNT, MMU_IDR_PACKET_COUNTER and ENQ_DISCARDED_PACKET_COUNTER

RP/0/RP0/CPU0:ROUTER#show controller npu stats counters-all instance all location all

FIA Statistics Rack: 0, Slot: 0, Asic instance: 0

Per Block Statistics:

Ingress:

NBI RX:
  RX_TOTAL_BYTE_COUNTER          = 161392268790033002
  RX_TOTAL_PKT_COUNTER           = 164628460653364

IRE:
  CPU_PACKET_COUNTER             = 0
  NIF_PACKET_COUNTER             = 164628460651867
  OAMP_PACKET_COUNTER            = 32771143
  OLP_PACKET_COUNTER             = 4787508
  RCY_PACKET_COUNTER             = 67452938
  IRE_FDT_INTRFACE_CNT           = 192

IDR:
  MMU_IDR_PACKET_COUNTER         = 697231761913
  IDR_OCB_PACKET_COUNTER         = 1

IQM:
  ENQUEUE_PKT_CNT                = 164640311902277
  DEQUEUE_PKT_CNT                = 164640311902198
  DELETED_PKT_CNT                = 0
  ENQ_DISCARDED_PACKET_COUNTER   = 90015441

To get the DRAM reject counters, we will use:

• show contr npu stats counters-all detail instance all location all
  or if the IOS XR version doesn’t support the “detail” option, use the following instead:
• show controllers fia diagshell 0 “diag counters” loc 0/x/CPU0

RP/0/RP0/CPU0:ROUTER#show contr npu stats counters-all detail instance all location all | i Dram

  IDR FullDramRejectPktsCnt            :                0
  IDR FullDramRejectBytesCnt           :                0
  IDR PartialDramRejectPktsCnt         :                0
  IDR PartialDramRejectBytesCnt        :                0
  IQM0 RjctDramIneligiblePktCnt        :                0
  IQM1 RjctDramIneligiblePktCnt        :                0
  IDR FullDramRejectPktsCnt            :                0
  IDR FullDramRejectBytesCnt           :                0
  IDR PartialDramRejectPktsCnt         :                0
  IDR PartialDramRejectBytesCnt        :                0
  IQM0 RjctDramIneligiblePktCnt        :                0
  IQM1 RjctDramIneligiblePktCnt        :                0

--%--SNIP--%--SNIP--%--
  

None of these counters are available through SNMP / MIB but instead you can use streaming telemetry:

From https://github.com/YangModels/yang/blob/master/vendor/cisco/xr/653/Cisco-IOS-XR-fretta-bcm-dpa-hw-resources-oper-sub2.yang

You’ll found:

ENQUEUE_PKT_CNT: iqm-enqueue-pkt-cnt

    leaf iqm-enqueue-pkt-cnt {
      type uint64;
      description "Counts enqueued packets";

MMU_IDR_PACKET_COUNTER: idr-mmu-if-cnt

    leaf idr-mmu-if-cnt {
      type uint64;
      description "Performance counter of the MMU interface";

ENQ_DISCARDED_PACKET_COUNTER: iqm-enq-discarded-pkt-cnt

    leaf iqm-enq-discarded-pkt-cnt {
      type uint64;
      description "Counts all packets discarded at the ENQ pipe";

At the moment (Apr 2019), RjctDramIneligiblePktCnt / FullDramRejectPktsCnt / PartialDramRejectPktsCnt are not available in the data models and therefor, can’t be streamed.

Auditing real production routers

We have the counters available and we asked multiple customers (25+) to collect data from their production routers.
In total, we had information for 550 NPUs transporting live traffic in multiple network positions:

• IP core
• MPLS core (P/LSR)
• Internet border (transit / peering)
• CDN (connected to FB, Akamai, Google Cache, Netflix, …)
• PE (L2VPN and L3VPN)
• Aggregation
• SPDC / ToR leaf

The data aggregated is helpful since it gives a vision of what is happening in reality.
The total amount of traffic measured is tremendous: 24,526,679,839,376,100 packets!!!
Not in lab, not in academic models / simulations, but in real routers.

With the show commands described in former section, we extracted:

• ENQUEUE_PKT_CNT: packets transmitted in the NPU
• MMU_IDR_PACKET_COUNTER: packets passed to DRAM
• ENQ_DISCARDED_PACKET_COUNTER: packets taildropped
• RjctDramIneligiblePktCnt: packets drop because of DRAM bandwidth

Dividing MMU_IDR_PACKET_COUNTER by ENQUEUE_PKT_CNT, we can compute the ratio of packets moved to DRAM.
–> 0,151%
This number is an average value and should be considered as such. It shows that indeed, the vast majority of the traffic is handled in OCB (inside the NPU).

Dividing ENQ_DISCARDED_PACKET_COUNTER by ENQUEUE_PKT_CNT, we can compute the ratio of packets taildropped.
–> 0,0358%
Having drops is normal in the life of a router. Multiple reasons here, from TCP windowing to temporary congestion situations.

Finally, RjctDramIneligiblePktCnt will tell us if the link from the NPU to the DRAM can get saturated and drops packets with production traffic.
–> not a single packet discarded in such scenario.

LAPTOP: nicolas$ grep RjctDramIneligiblePktCnt * | wc -l
    1570
LAPTOP: nicolas$ grep RjctDramIneligiblePktCnt * | grep " 0" | wc -l
    1570
LAPTOP: nicolas$ grep RjctDramIneligiblePktCnt * | grep -v " 0" | wc -l
       0
LAPTOP: nicolas$

In this chart, we sort by numbers of ENQUEUE_PKT_CNT: it represents the most active ASICs in term of packets handled.

For the most busiest NPUs collected, we see the DRAM ratio and taildrop ratio being actually much smaller than aggregated numbers.

How to read these numbers?

First of all, it demonstrates clearly that most of the packets are handled inside the ASIC, only a very small portion of the traffic being evicted to DRAM.

Second, with RjctDramIneligiblePktCnt being zero in EVERY data collection, we prove that bandwidth from NPU to DRAM (900Gbps unidirectional) is correctly dimensionned. It handles the real burstiness of the traffic without a single drop.

Last, the data collected represents a snapshot. It is recommended to collect these counters regularly and to analyze them with the network activity during the interval.
Having higher numbers in your network may be correlated to a particular outage or specific situation.
Having small numbers, in the other hand, is much easier to read (no drops being… no drops).

Conclusion

In conclusion, the ingress-buffering / VOQ-only model is well adapted for real networks.

We have seen “academic” studies trying to prove the contrary, but the numbers are talking here.

A sandbox, or an imaginary model are not relevant approach.

Production networks deployed all around the world, in different positions/roles, transporting Petabytes of traffic for multiple years, prove the efficiency of this architecture.