Welcome to this week’s Ticket of the Week segment or TOW for short. TOW is where I cover a range of hot topics and trends from within the industry that include actual examples of tickets from previous employers and clients edited only for confidentiality reasons. Have a suggestion please use the contact page and someone will get back to you promptly and you may be featured in the segment.
Editor’s Note: For my technical readers excuse the obvious acronyms as I am trying to ensure that a bigger audience is able to connect with our world via my blog.
I saw a simple “Destination Unreachable” ticket in the que and didn’t think twice about it. For me that’s about as easy as it comes. People tend to make these networks bigger than what they are and forget about the whole KISS principle. At the end of the day someone is trying to get from point A to point B. In this particular example point A was your typical 10.x.x.x (SIP for source IP) address and point B was a 166.x.x.x (DIP for destination IP) or whatever private to public IP you’d like it to be. Right away you know two things. One there’s some sort of NAT (network address translation) and two your path will traverse a FW and traverse your WAN or edge router.
I started with the 10.x.x.x address and verified it was reachable initially via ping and then via traceroute to see the path. Not an issue there. With the 166.x.x.x address I executed a traceroute and as expected the ICMP replies ceased due to the particular rules in place at the firewall. I logged into the last device on the traceroute and verified my thoughts. Show ip route 166.x.x.x revealed a static route to a next hop address of let’s say 10.2.1.1. Show ip arp gave me a MAC address of 021c.7f3d.8019, and a show mac address-table address 021c.7f3d.8019 provided an interface of port-channel 50. Show interface port-channel 50 indicated via it’s description that it was indeed connected to a firewall and it had one gigabit Ethernet interface as a member. Int eth11/25 so I verified that this description did indeed match the description of the port-channel.
As I imagined everything was routine up until that point and since I was in a pretty good mood that night I even contacted the firewall team for them to verify traffic was indeed traversing on its way out to the world wide web. Here things took an unexpected turn. The security engineer informed me that the only traffic he saw was from my machine when I was performing the pings and traceroutes. I immediately became a bit suspicious of this ticket and contacted the user to have them produce traffic. Luckily the switch in question already had monitors set up so I pulled a packet capture and verified that I did see traffic on the ingress interface but not on the specified egress interface which mind you was statically (read manually/permanently) assigned. Meaning it will always choose that path whether the link is available or not and it is one of the main reasons we prefer dynamic routing protocols.
There’s a website, for those of you who aren’t engineers/techs, ARIN.net that will give you information on just about any IP address. So I popped in the DIP and when the owner showed something completely different than it should’ve I really knew there was a problem. Turns out someone had configured this connection on the backup network, meaning throw everything above out the window, with addresses from an entirely different organizations public IP range. Pretty much all of us were thrown for a loop by the discovery. One emergency change control later Operations corrected Engineering’s mistake and all systems were normal.
Have a story of your own? Leave a comment or use the contact page and you may see it here during TOW!