Pathchecker: iperf + netconf for OSPF path failover

12 minutes read

Introduction

If you haven’t checked out the XR toolbox Series, then you can do so here:

XR Toolbox Series

This series is meant to help a beginner get started with application-hosting on IOS-XR.

In this tutorial we intend to utilize almost all the techniques learnt in the above series to solve a path remediation problem:

  • Set up a couple of paths between two routers. Bring up OSPF neighborship on both links. One link is forced to be the reference link by increasing the ospf cost of the other link.

  • Use a monitoring technique to determine the bandwidth, jitter, packet loss etc. parameters along the active traffic path. In this example, we utilize a python app called pathchecker that in turn uses iperf to measure link health.

  • Simulate network degradation to force pathchecker (running inside an LXC) to initiate failover by changing the OSPF path cost over a netconf session.

This is illustrated below:

pathchecker-topo

Understand the topology

As illustrated above, there are 3 nodes in the topology:

  • rtr1 : The router on the left. This is the origin of the traffic. We run the pathchecker code inside an ubuntu container on this router. The path failover happens rtr1 interfaces as needed.

  • devbox : This node serves two purposes. We use it to create our ubuntu LXC tar ball with the pathchecker code before deploying it to the router. It also houses two bridge networks (one for each path) so that we can create very granular impairment on each path to test our app.

  • rtr2 : This is the destination router. pathchecker uses an iperf client on rtr1 to get a health estimate of the active path. You need an iperf server running on rtr2 for the pathchecker app to talk to.

Pre-requisites

  • Make sure you have Vagrant and Virtualbox installed on your system.

  • The system must have 9-10G RAM available.

  • Go through the Vagrant quick-start tutorial, if you haven’t already, to learn how to use Vagrant with IOS-XR: IOS-XR vagrant quick-start

  • It would be beneficial for the user to go through the XR Toolbox Series. But it is not a hard requirement. Following the steps in this tutorial should work out just fine for this demo.

Once you have everything set up, you should be able to see the IOS-XRv vagrant box in the vagrant box list command:

  AKSHSHAR-M-K0DS:~ akshshar$ vagrant box list
  IOS-XRv (virtualbox, 0)
  AKSHSHAR-M-K0DS:~ akshshar$ 

Clone the git repo

The entire environment can be replicated on any environment running vagrant provided around 9-10G RAM is available. The topology will include 2 IOS-XR routers (8G RAM) and an ubuntu instance (around 512 MB RAM).

Clone the pathchecker code from here: https://github.com/ios-xr/pathchecker


AKSHSHAR-M-K0DS:~ akshshar$ git clone https://github.com/ios-xr/pathchecker.git 
Cloning into 'pathchecker'...
remote: Counting objects: 46, done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 46 (delta 8), reused 0 (delta 0), pack-reused 18
Unpacking objects: 100% (46/46), done.
Checking connectivity... done.
AKSHSHAR-M-K0DS:~ akshshar$ 

Spin up the devbox

Before we spin up the routers, we need to create the container tar ball for the pathchecker code. The way I’ve set up the launch scripts for rtr1, the bringup will fail without the container tar ball in the directory.

Move to the Vagrant directory and launch only the devbox node:


AKSHSHAR-M-K0DS:~ akshshar$ cd pathchecker/
AKSHSHAR-M-K0DS:pathchecker akshshar$ cd vagrant/
AKSHSHAR-M-K0DS:vagrant akshshar$ pwd
/Users/akshshar/pathchecker/vagrant
AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant up devbox 
Bringing machine 'devbox' up with 'virtualbox' provider...
==> devbox: Importing base box 'ubuntu/trusty64'...

---------------------------- snip output ---------------------------------

==> devbox: Running provisioner: file...
AKSHSHAR-M-K0DS:vagrant akshshar$ 
AKSHSHAR-M-K0DS:vagrant akshshar$ 
AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant status 
Current machine states:

rtr1                      not created (virtualbox)
devbox                    running (virtualbox)
rtr2                      not created (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
AKSHSHAR-M-K0DS:vagrant akshshar$ 

Create the Pathchecker LXC tar ball

Launch an Ubuntu LXC inside devbox

SSH into “devbox”:


vagrant ssh devbox

Create the pathchecker lxc template:



[email protected]:~$  sudo lxc-create -t ubuntu --name pathchecker 
Checking cache download in /var/cache/lxc/trusty/rootfs-amd64 ... 
Installing packages in template: ssh,vim,language-pack-en
Downloading ubuntu trusty minimal ...
I: Retrieving Release 
I: Retrieving Release.gpg 
I: Checking Release signature
------------------------------ snip output ------------------------------------

Start the container. You will be dropped into the console once boot is complete.

Username: ubuntu
Password: ubuntu


[email protected]:~$ sudo lxc-start --name pathchecker 
<4>init: hostname main process (3) terminated with status 1
<4>init: plymouth-upstart-bridge main process (5) terminated with status 1
<4>init: plymouth-upstart-bridge main process ended, respawning


Ubuntu 14.04.4 LTS nc_iperf console

pathchecker login: ubuntu
Password:      
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-87-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

[email protected]:~$ 

Install Application dependencies inside LXC

Install iperf and all the dependencies required to install ncclient inside the container. We’ll also install git, will need it to fetch our app.

sudo apt-get -y install python-pip python-lxml python-dev libffi-dev libssl-dev iperf git

Install the latest ncclient code and jinja2 code using pip (required for our app). We also downgrade the cryptography package to 1.2.1 to circumvent a current bug in the package.

sudo pip install ncclient jinja2 cryptography==1.2.1

Perfect, all the dependencies for our app are now installed.

Fetch the application code from Github

Fetch our app from Github:

[email protected]:~$ git clone https://github.com/ios-xr/pathchecker.git
Cloning into 'pathchecker'...
remote: Counting objects: 46, done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 46 (delta 8), reused 0 (delta 0), pack-reused 18
Unpacking objects: 100% (46/46), done.
Checking connectivity... done.
ub[email protected]:~$ 

Change SSH port inside the container

When we deploy the container to IOS-XR, we will share XR’s network namespace. Since IOS-XR already uses up port 22 and port 57722 for its own purposes, we need to pick some other port for our container.

P.S. If you check the Vagrantfile, we intend to expose port 58822 to the user’s laptop directly, on rtr1.

Let’s change the SSH port to 58822:


[email protected]:~$ sudo sed -i s/Port\ 22/Port\ 58822/ /etc/ssh/sshd_config 
[email protected]:~$ 

Check that your port was updated successfully:

[email protected]:~$ cat /etc/ssh/sshd_config | grep Port
Port 58822
[email protected]:~$ 

We’re good!

Package up the LXC

Now, shutdown the container:


[email protected]:~$ sudo shutdown -h now 
[email protected]:~$ 
Broadcast message from [email protected]
	(/dev/lxc/console) at 10:24 ...

The system is going down for halt NOW!
------------------------------ snip output ------------------------------------

You’re back on devbox.

Become root and package up your container tar ball

sudo -s

cd /var/lib/lxc/pathchecker/rootfs/

tar -czvf /vagrant/pathchecker_rootfs.tar.gz *

See what we did there? We packaged up the container tar ball as pathchecker_rootfs.tar.gz under /vagrant directory. Why is this important?
Well, Vagrant also automatically shares a certain directory with your laptop (for most types of guest operating systems). So the /vagrant is automatically mapped to the directory in which you launched your vagrant instance. To check this, let’s get out of our vagrant instance and issue an ls in your launch directory:


[email protected]:~$ 
[email protected]:~$ exit
logout
Connection to 127.0.0.1 closed.
AKSHSHAR-M-K0DS:vagrant akshshar$ 
AKSHSHAR-M-K0DS:vagrant akshshar$ pwd 
/Users/akshshar/pathchecker/vagrant
AKSHSHAR-M-K0DS:vagrant akshshar$ ls -l pathchecker_rootfs.tar.gz  
-rw-r--r--  1 akshshar  staff  301262995 Jul 18 07:57 pathchecker_rootfs.tar.gz
AKSHSHAR-M-K0DS:vagrant akshshar$ 

Launch Router Topology

To launch the two routers in the topology, make sure you are in the vagrant directory under pathchecker and issue a vagrant up


AKSHSHAR-M-K0DS:vagrant akshshar$ pwd
/Users/akshshar/pathchecker/vagrant
AKSHSHAR-M-K0DS:vagrant akshshar$ 
AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant up
Bringing machine 'rtr1' up with 'virtualbox' provider...
Bringing machine 'devbox' up with 'virtualbox' provider...
Bringing machine 'rtr2' up with 'virtualbox' provider...

-------------------------------- snip output --------------------------------------


Once everything is up, you should see the three nodes running:  

AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant status
Current machine states:

rtr1                      running (virtualbox)
devbox                    running (virtualbox)
rtr2                      running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

We’re all set! Let’s test out our application.

Test out pathchecker!

Before we begin, let’s dump some configuration outputs on rtr1:

Check current OSPF cost/path state


AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant port rtr1
The forwarded ports for the machine are listed below. Please note that
these values may differ from values configured in the Vagrantfile if the
provider supports automatic port collision detection and resolution.

22 (guest) => 2223 (host)
 57722 (guest) => 2200 (host)
 58822 (guest) => 58822 (host)
AKSHSHAR-M-K0DS:vagrant akshshar$ ssh -p 2223 [email protected] 
The authenticity of host '[localhost]:2223 ([127.0.0.1]:2223)' can't be established.
RSA key fingerprint is b1:c1:5e:a5:7e:e7:c0:4f:32:ef:85:f9:3d:27:36:0f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2223' (RSA) to the list of known hosts.
[email protected]'s password: 


RP/0/RP0/CPU0:rtr1#
RP/0/RP0/CPU0:rtr1#show  running-config  router ospf 
Mon Jul 18 15:25:53.875 UTC
router ospf apphost
 area 0
  interface Loopback0
  !
  interface GigabitEthernet0/0/0/0
  !
  interface GigabitEthernet0/0/0/1
    cost 20
  !
 !
!

RP/0/RP0/CPU0:rtr1#show route 2.2.2.2 
Mon Jul 18 15:26:03.576 UTC

Routing entry for 2.2.2.2/32
  Known via "ospf apphost", distance 110, metric 2, type intra area
  Installed Jul 18 15:18:28.218 for 00:07:35
  Routing Descriptor Blocks
   10.1.1.20, from 2.2.2.2, via GigabitEthernet0/0/0/0
      Route metric is 2
  No advertising protos. 
RP/0/RP0/CPU0:rtr1#

We can see that the current OSPF cost on Gig0/0/0/1 is 20, higher than Gig0/0/0/0. Hence as the route to 2.2.2.2 (loopback 0 of rtr2) shows, the current path selected is through Gig0/0/0/0

Start iperf server on rtr2

iperf was already installed on rtr2 as a native application (more on native apps here: XR toolbox part 5: Running a native WRL7 App) during the vagrant up process.

Start iperf server on rtr2 and set it up to accept UDP packets:


AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant ssh rtr2 
Last login: Mon Jul 18 15:57:05 2016 from 10.0.2.2
xr-vm_node0_RP0_CPU0:~$ 
xr-vm_node0_RP0_CPU0:~$ iperf -s -u 
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 MByte (default)
------------------------------------------------------------




Start pathchecker on rtr1 (LXC)

SSH into the pathchecker ubuntu container (already brought up as part of vagrant up process) by using port 58822 on your laptop:

Password for user “ubuntu” : ubuntu


AKSHSHAR-M-K0DS:vagrant akshshar$ 
AKSHSHAR-M-K0DS:vagrant akshshar$ ssh -p 58822 [email protected]
The authenticity of host '[localhost]:58822 ([127.0.0.1]:58822)' can't be established.
RSA key fingerprint is 19:54:83:a9:7a:9f:0a:18:62:d1:f3:91:87:3c:e9:0b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:58822' (RSA) to the list of known hosts.
[email protected]'s password: 
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.14.23-WR7.0.0.2_standard x86_64)

 * Documentation:  https://help.ubuntu.com/
Last login: Mon Jul 18 15:19:45 2016 from 10.0.2.2
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ 

The pc_run.sh script simply runs the pathchecker.py application with a few sample parameters:


[email protected]:~$ 
[email protected]:~$ cat ./pathchecker/pc_run.sh 
#!/bin/bash

./pathchecker.py --host 6.6.6.6 -u vagrant -p vagrant --port 830 -c 10 -o apphost -a 0 -i GigabitEthernet0/0/0/0 -s 2.2.2.2  -j 4 -l 5 -f -t 10
[email protected]:~$ 

Based on above output, the “-l” option represents the threshold for packet loss and has been set to 5% for this run. Similarly, jitter has a threshold value of 4. Start the pathchecker app by running the pc_run.sh script in the pathchecker repository:


[email protected]:~$ cd pathchecker/ 
[email protected]:~/pathchecker$ ./pc_run.sh 
Error while opening state file, let's assume low cost state
Currently, on reference link GigabitEthernet0/0/0/0 
Starting an iperf run.....
20160718162513,1.1.1.1,62786,2.2.2.2,5001,6,0.0-10.0,1311240,1048992
20160718162513,1.1.1.1,62786,2.2.2.2,5001,6,0.0-10.0,1312710,1048474
20160718162513,2.2.2.2,5001,1.1.1.1,62786,6,0.0-10.0,1312710,1048679,2.453,0,892,0.000,1

bw is
1025.5546875
jitter is
2.453
pkt_loss is
0.000
verdict is
False
Currently, on reference link GigabitEthernet0/0/0/0
Starting an iperf run.....



Perfect! The App seems to be running fine on the reference link Gig0/0/0/0.

Create impairment on Active path

With the app running, let’s scoot over to “devbox” which will also act as our impairment node.


AKSHSHAR-M-K0DS:vagrant akshshar$ vagrant ssh devbox 
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-87-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Mon Jul 18 16:38:49 UTC 2016

  System load:  0.0               Processes:             76
  Usage of /:   6.3% of 39.34GB   Users logged in:       0
  Memory usage: 32%               IP address for eth0:   10.0.2.15
  Swap usage:   0%                IP address for lxcbr0: 10.0.3.1

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud


Last login: Mon Jul 18 16:38:50 2016 from 10.0.2.2
[email protected]:~$ 
[email protected]:~$ ls
impair_backup.sh  impair_reference.sh  stop_impair.sh
[email protected]:~$ 
[email protected]:~$ cat impair_reference.sh 
#!/bin/bash
echo "Stopping all current impairments"
sudo tc qdisc del dev eth3 root &> /dev/null
sudo tc qdisc del dev eth4 root &> /dev/null
echo "Starting packet loss on reference link"
sudo tc qdisc add dev eth3 root netem loss 7% 
[email protected]:~$ 
[email protected]:~$ ./impair_reference.sh
Stopping all current impairments
Starting packet loss on reference link
[email protected]:~$ 

As we can see, the reference impairment script creates a packet loss of 7% on the reference link

Take a look at the running pathchecker application on rtr1. It should switch to the backup link once it detects an increase in packet loss beyond 5% (as specified in the pc_run.sh file):


Currently, on reference link GigabitEthernet0/0/0/0
Starting an iperf run.....
20160718164745,1.1.1.1,60318,2.2.2.2,5001,6,0.0-10.0,1311240,1048992
20160718164745,1.1.1.1,60318,2.2.2.2,5001,6,0.0-10.0,1312710,1048516
20160718164745,2.2.2.2,5001,1.1.1.1,60318,6,0.0-573.0,1312710,18328,5.215,0,892,0.000,1

bw is
1025.5546875
jitter is
5.215
pkt_loss is
0.000
verdict is
True
Woah! iperf run reported discrepancy, increase cost of reference link !
Increasing cost of the reference link GigabitEthernet0/0/0/0
Currently, on backup link 
Starting an iperf run.....
20160718164755,1.1.1.1,61649,2.2.2.2,5001,6,0.0-10.0,1311240,1048992
20160718164755,1.1.1.1,61649,2.2.2.2,5001,6,0.0-10.0,1312710,1048577
20160718164755,2.2.2.2,5001,1.1.1.1,61649,6,0.0-583.3,1312710,18002,1.627,0,893,0.000,0

bw is
1025.5546875
jitter is
1.627
pkt_loss is
0.000
verdict is
False
Currently, on backup link
Starting an iperf run.....
20160718164805,1.1.1.1,59343,2.2.2.2,5001,6,0.0-10.0,1311240,1048992
20160718164805,1.1.1.1,59343,2.2.2.2,5001,6,0.0-10.0,1312710,1048520
20160718164805,2.2.2.2,5001,1.1.1.1,59343,6,0.0-593.4,1312710,17697,2.038,0,893,0.000,0

The app initiated the failover! Let’s see how the router responded.

Verify the Failover was successful


AKSHSHAR-M-K0DS:vagrant akshshar$ ssh -p 2223 [email protected]
[email protected]'s password: 


RP/0/RP0/CPU0:rtr1#
RP/0/RP0/CPU0:rtr1#
RP/0/RP0/CPU0:rtr1#show  running-config  router ospf
Mon Jul 18 17:50:47.851 UTC
router ospf apphost
 area 0
  interface Loopback0
  !
  interface GigabitEthernet0/0/0/0
   cost 30
  !
  interface GigabitEthernet0/0/0/1
   cost 20
  !
 !
!

RP/0/RP0/CPU0:rtr1#

Great! The Cost of the Gig0/0/0/0 (reference) interface has been increased to 30, greater than the cost of Gig0/0/0/1. This forces the failover to happen to the Gig0/0/0/1 for the iperf traffic (or any traffic destined to rtr2).


RP/0/RP0/CPU0:rtr1#show route 2.2.2.2
Mon Jul 18 18:01:49.297 UTC

Routing entry for 2.2.2.2/32
  Known via "ospf apphost", distance 110, metric 21, type intra area
  Installed Jul 18 16:47:45.705 for 01:14:03
  Routing Descriptor Blocks
  11.1.1.20, from 2.2.2.2, via GigabitEthernet0/0/0/1
      Route metric is 21
  No advertising protos. 
RP/0/RP0/CPU0:rtr1#

It works! The failover happened and the next hop for 2.2.2.2 (loopback0 of rtr2) is now 11.1.1.20 through Gig0/0/0/1 (the backup link).

We leave it upto the reader to try and impair the backup link now and see the App switch the path back to the reference interface.

Leave a Comment