The Riverbed Blog (testing)

A blog in search of a tagline

We’ll Hang with You Until It’s Fixed

Posted by riverbedtest on September 19, 2011

Today's Guest Blogger is Kary Rogers, a member of Riverbed's Support Team since 2008.

When you install a Riverbed product into your network, you may have questions or run into problems and need some help. Riverbed Support is available 24 hours a day, seven days a week to answer your questions and to help solve your network problems when you have a current support agreement.

One of Riverbed Support's tenets is that we don't pass the buck or point fingers; we endeavor to be a good partner with everyone. If in the course of troubleshooting, the root cause appears to be another device in your network, we'll help to troubleshoot the issue as best we can and stay with you until the issue is resolved.

I had a case in which the customer reported that a particular application was breaking when optimized Retrans_small[1] but only at one remote location; it worked fine everywhere else. This was a case in which we required traffic captures to perform a deeper analysis. Quickly, we were able to determine that shortly after the client's application began talking to the server, the WAN optimized traffic was getting dropped as we saw retransmits in the WAN data. Many times this turns out to be an MTU issue, but this was not the case here.

Traffic analysis is all about pattern recognition and knowing the protocol – i.e. expected behavior. The more you look at traffic data, the quicker and easier it becomes to spot the pattern or the breaking of a pattern. In this case, I noticed that the beginning of the conversation had no DCSP marking in the IP header, but after a short time the traffic was being marked with a DSCP value of 8. Any traffic that had this marking was unacknowledged by the receiver and retransmitted.  So we found why the application was breaking, but why was this traffic being marked?

Dscp8[1] I gave my analysis to the customer, and he was at a loss to explain why this traffic was coming into the Steelhead marked with DSCP 8. He checked with the local network administrator and found that he had enabled a feature on the LAN device called Adaptive QoS, a feature that is designed to limit high bandwidth connections when you have a slow WAN link that might become saturated. Once a connection reaches the bandwidth trigger threshold, it gets marked as DSCP 8, and then the WAN router drops the packets into a scavenger class. You don't want to know what happens to packets that end up there; it is not pretty.

Now that we understood that the customer's network equipment was doing this, he wanted to know why it only happened when the application was optimized by the Steelhead. The answer is one of the main reasons for installing Steelheads: the sending computer couldn't reach the throughput required to trigger adaptive QoS without a Steelhead optimizing the traffic!

Adaptive QoS, rogue DHCP servers, or solar flares; we'll hang with you until it's fixed.

2 Responses to “We’ll Hang with You Until It’s Fixed”

  1. Daran said

    Hi. Quick question. We have just put in two x 5050 Steelheads. The next day we have certain applications (Oracle and an in-house app) having issues. As a test we moved a catch all pass-thru un-optimized filter at the top of the in-path settings to see if the Steelheads were root cause problem with added latency. It appears the units are still optimizing traffic. Do I need to save the new configuration or are they like Cisco switches and applied to running-config immediately. Sorry, newbie question – but we are newbies to them! Any help appreciated.

  2. The best way to get an answer to that question is to post it to our Community forums. We even have a forum called Help the Newbies. I’d post the question for you, but then you’d never find out when a response was posted because the question wouldn’t be able to notify you.
    Check out

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: