Cisco 7609 Bugs Galore

The Cisco IOS is, I read somewhere, not unlike women :-). Just when you think that you know her, you see a new side that you hadn't seen before. Oh Yeah! :-)

The Nayatel Core team has been able to create two new Bug IDs in the Cisco 7609s IOS. The first one is CSCsh60112. This bug is a real nasty one. Imagine this: all internet traffic entering the Autonomous System stops. You get a call telling you that the Internet is down. You login and check the BGP neighbor relationship and find that all neighbors are up and reachable. Next, you check the IGP. Everything seems to be in order. ISIS is working perfectly. Hmmm. You dialup and reach the internet through a 2nd provider and check the routes on the Internet. Voila! BGP is not advertising the subnets anymore!.

A quick show ip route on the Internet gateway confirms that the nailed-down /24s which were being redistributed into the IGP have disappeared. You login to the PE router on which the redistribution was taking place and check the routes using the show running-config command. Well, the routes ARE shown. But they are not being redistributed anymore!!!!.

Solution: You decide to remove the static routes to Null0 and put them back again. This solves the problem and since the Gateway now learns the /24s from the IGP, BGP starts advertising those routes again. When 622Mbps of Internet connectivity goes down, keeping one's cool is the best thing to do :-). And that reminds me of the decade old Sysadmin rule: Rule #1. Don't Panic. Rule #2. Don't Panic. Rule #3. Don't Panic.... Words to Live By!


Here's the official Bug Detail: After SSO failover, static Null0 routes are not seen in the RIB but are still seen in the running config!


As if this wasn't enough along came Bug CSCsm32555. The Bug description is "Unable to Route traffic across MPLS VRF to GRE peer". The Customer had multiple sites connected via Layer 3 MPLS VPNs. A remote site did not have MPLS-enabled ISP so I decided to bring the traffic via a GRE tunnel and make it a part of the Customers VPN. Easier said than done. When the GRE tunnel was made part of the VRF, connectivity to the remote site (from where the GRE tunnel originated) was broken.

I terminated the tunnel on a 7206-VXR to ensure that this was an IOS bug. On the 7206, things were fine and it was proven that this was a 7609 IOS bug.

This bug is now resolved in 12.2(33.0.6)SRC!.

PTA to install NARUS Secure Suite at TWA1 Landing Station

In a bid to monitor H.323 and SIP traffic entering and exiting Pakistan, Pakistan Telecommunications Authority will be commissioning the NARUS-based monitoring system on May 1st, 2008 at TWA1's landing station in Karachi. A Juniper M320 is the router used for terminating trans-ocean links at the landing station. The basic purpose of this system, as we are told, is to ensure that no 'gray' voice traffic ends up through the internet in Pakistan. Such traffic should be terminated through the LDIs (Burraq Telecom, PTCL, and Wateen among others) who are licensed to do so.

This also raises serious concern regarding privacy of users legitimate voice traffic carried using Yahoo Messenger, MSN Messenger, SKYPE and other popular programs in addition to a users Internet activity in general.

Here are some of NARUS 'features' as taken from the Narus website:

* Real-time data capture, classification and normalization at speeds from 100baseT to 10G/OC192 and Narus Virtual-Analyzer (IP data from network elements, e.g. SNMP data, flow data from Cisco or Juniper routers, GTP streams from SGSN and GGSN elements, log files such as syslog files from network hosts, etc.)

* Information on all IP traffic, regardless of the protocol or network type

* Utilizes full traffic flow and all network layers of visibility for accurate detection

Note that the NARUS system is already enabled on PTCL's ITI Infrastructure. TWA1 will be under the same monitoring now beginning May 1st 2008. Passive monitoring, that NARUS supports is fine as long as the intent is to stop the gray voice traffic. However, the same system can also be used to spy on anyone's Internet activity as long as the IP Address of the user is identified.

Phil Zimmerman wrote in the user guide of his famous PGP program that

If Privacy is outlawed, only outlaws will have privacy

Phil's essay Why I wrote PGP is a must read.

Meanwhile, I just ensured that encryption on my Internet connection is working fine. As Phil wrote: Its Personal. Its Private. And it's none of anyone's business!