The Riverbed Blog (testing)

A blog in search of a tagline

Riverbed thanks Cisco for their great QA work on the Steelhead

Posted by riverbedtest on September 17, 2009

At Riverbed, we are committed to improving the Steelhead product by adding new features and capabilities.  As with any software product, an unfortunate side-effect of our efforts to improve our product is that software bugs can result. Our QA and engineering teams do their best to identify and resolve every known bug in the Steelhead software before it is released, but on occasion bugs can be discovered by our customers in their network environments.  After all, the Steelhead solution is tested daily in very demanding environments by thousands of Riverbed customers, including in some of the largest and most complex IP networks in the world. 

When bugs are reported to Riverbed, we take them very seriously.  However, imagine our surprise earlier in the summer of this year when we received a report of a bug that was discovered not by a Riverbed customer, but by Cisco Systems!  Well, like I said, we're happy to improve our software, even if by taking a clue from a competitor, so we took this bug report very seriously.

Perhaps a little bit of background is in order here.  Although the bug was discovered by Cisco, it was actually reported to Riverbed by a customer who was being shown a demonstration of the bug over a webex session.  You see, Cisco was trying to convince the customer that Riverbed's product couldn't handle high traffic loads because of a failure that occurred when the Steelhead 6020 is exposed to 5000 TCP connections generated by a Spirent tester using a specific traffic profile (i.e., HTTP connections).  Cisco ignored the fact that Riverbed already has hundreds of customers using the same Steelhead 6020 to optimize as many as 40,000 TCP connections, and they attempted to characterize this problem with the 5000 connections not as a bug, but as an overall Riverbed limitation.  The customer was suspicious, and alerted us as to what Cisco was doing.

We received enough clues to reconstruct how Cisco contrived the test, and were able to replicate the bug event in our own labs.  Here are some additional details on what we found about the bug:

  • No Riverbed customers have reported this bug to Riverbed.  It doesn't show up in real-life networks.

  • The bug ONLY occurs with HTTP connections, because it is specific to Riverbed's HTTP optimization feature, whereas Cisco WAAS has no comparable HTTP optimization feature.

  • The bug only occurs when a large number (~5000) of HTTP connections all simultaneously issue GET operations for large web objects, in this case web objects that are 1MB in size.  In a normal real-life environment, web objects are typically only a few KB in size.

  • For those environments that do involve large web objects (e.g., Sharepoint files), the bug will not be triggered as long as GET requests do not happen simultaneously across all 5000 active HTTP sessions.  Since in real-life applications do not exhibit this behavior, no Riverbed customers have complained about this bug.

  • If the HTTP optimization blade is disabled, then the bug will not occur.  Any customers who are concerned could simply disable the HTTP feature (and Cisco WAAS has no equivalent feature), and the bug will not appear.  They can re-enable HTTP optimization when the Steelhead is updated with the bug fix from the Riverbed support website.

I would like to inform concerned parties–including our competitor Cisco–that the bug has been resolved, and the fix is now available on the Riverbed support website, in the recently-available RiOS 5.5.5 release.  Following our stringent quality standards, we have spent the last several weeks testing and validating the fix, and are now comfortable-enough to release it.  But once again, the credit goes to Cisco for finding this obscure and difficult-to-find bug, and we thank them for their diligent efforts.  For those who don't intend to run Cisco's contrived tests, this software upgrade is purely optional and not required.

In an attempt to legitimize their contrived test, Cisco has "commissioned" Miercom to document this obscure Riverbed bug.  The Miercom document (document number 090815B) can be accessed here:

http://www.miercom.com/dl.html?fid=20090915&type=report

For anyone who has any illusions that Miercom's reports are objective in any way, you may also want to consider other Miercom reports that favor other WAN optimization solutions:

Citrix WANScaler: http://www.citrix.com/site/resources/dynamic/salesdocs/CitrixSystems_8800.pdf

Silver Peak: http://www.miercom.com/dl.html?fid=20051207&type=report&PHPSESSID=c8e84098c29b5e9d1f687962c2833950

Blue Coat (formerly Packeteer): 

http://www.miercom.com/dl.html?fid=20070914&type=report&PHPSESSID=507081d1bfb92c4dfdfedf78268a4258

With all of the above reports from Miercom each recommending a different vendor, how is one to determine which is truly the vendor that they recommend for WAN optimization?  Rather, it appears that Miercom recommends any vendor that has recently paid them a "commission." 

At Riverbed, we're flattered by all the attention Cisco is devoting to their chief competitor in WAN optimization.  And we appreciate the help they are giving us in finding bugs to improve the Steelhead solution. But on the other hand I'm sure Cisco's existing WAAS customers might be somewhat dismayed by all of the resources that Cisco is placing into their competition's product.  Why is Cisco spending their precious QA resources testing and identifying bugs in their competitor's product, when their own customers suffer and struggle with the poor software quality that afflicts WAAS? And why is Cisco focusing on finding obscure bugs in simulated WANs when their own WAAS customers are suffering from WAAS bugs and that happen in real WAN environments? 

Click on the following link to view Riverbed's full response to Cisco's paid-for Miercom report:

>>Download Riverbed response to Miercom report 090815B<<

 

Advertisements

6 Responses to “Riverbed thanks Cisco for their great QA work on the Steelhead”

  1. Jerry said

    Miercom tesing is “independent” only to the extent that they are not the company making the product.
    Also, persuant to the information/invitation in Miercom’s latest Cisco report I asked for more detail about the test and have not had a response. Does that mean something?

  2. Arun said

    Well, I read this Miercom report and its not about credibility of Miercom…for sure..
    Its about what they found out….
    One thing is clear Cisco got some architectural weakness in your solution. The point is not about HTTP bug but the point is, a HTTP bug in your software can bring the whole system down and can cause disruption of traffic in datacenter..!!!!! How should I satisfy my customer that it won’t happen again..??Its scary what these guys found out….
    There may be some CIFS or MAPI bug next which can again brought down your system..who knows….How you can guarantee that this will not happen again? Its a flaw in your solution and I really suspect if your solution is enterprise ready.
    To make this point more stronger, Miercom shows some screen shot…showing many “Sport crash” bug fixes in your software release notes, that means their assumption of every single bug in Riverbed RiOS is a critical bug may be right.
    Also after hearing your response on reporting, I really can’t credibly believe on Riverbed solution.
    You are doing TCP application optimization and ignoring TCP headers in your statistics…aahh!!
    You should document it in your documentation.
    I really love your solution Josh but I have seen many start-ups company fool customer by showing a wrong picture of their RoI. Its about credibility..You guys let me down.!!

  3. Josh Tseng said

    Arun,
    Your logic doesn’t quite make sense. Why use any product at all if it might have bugs that can affect the data center? That rationale is certainly more applicable to Cisco WAAS than it is to Riverbed. I personally have spoken with many WAAS customers who have had their network operations severely disrupted by WAAS. Many of these customers had to remove WAAS because it was so disruptive, and a sizeable number of them are now Riverbed customers. While Riverbed is far from perfect, I’m not aware of a single implementation that we’ve had to abort or cancel as a result of software quality problems.
    As far as Steelhead reporting, just look at page 264 in the Steelhead Management Console Users Guide, which states:
    What This Report Tells You
    The Data Reduction report answers the following questions:
    􀂄 What was the total reduction in the amount of data that can be transmitted for each application?
    􀂄 What was the peak reduction in the amount of data transmitted for each application?
    􀂄 What was the total increase of data transmitted for the application and time period specified?
    You can look through the entire manual if you like, and you will see that we NEVER made any claim to count packet headers. As stated in the Riverbed manual, we only count data sent by the application for data reduction reporting. For normal-sized packets of >1000 bytes, the difference is less than 2% anyway. That’s why Cisco/Miercom had to contrive their test by forcing the Steelhead to send very small packets of ~61 bytes. I don’t understand why you make a big fuss out of this anyway, other than it being a dubious attempt to mislead the unwary.
    As far as Riverbed ROI, it goes far beyond the statistics that are reported on the Steelhead management GUI. We have a lineup of customers who will tell you how they’ve saved $millions as a result of Riverbed. If you contact your Riverbed representative, we can let you talk with some of them. These are real customers who have realized real dollar savings as a result of removing hundreds or thousands of servers from branch offices, and shutting down unneeded data centers. I’m sure Cisco’s internal IT organization would love to be able to do that too, but too bad…they’re stuck with WAAS.
    Josh

  4. Just a point of factual clarification, Cisco WAAS does have an HTTP optimization module, it’s been there since WAAS version 4.1
    Cheers,
    Brad

  5. Josh Tseng said

    Hi Brad,
    Thanks for the comment. To be more clear, Cisco WAAS’s so-called “HTTP optimization” feature is really just a limited TCP connection re-use feature. Essentially, it is a weaker version of Riverbed’s TCP connection pooling feature–limited because the WAAS connection reuse feature can only be used for HTTP connections (while Riverbed applies TCP connection pooling for all TCP-based applications when in correct addressing mode), and also because it cannot set up connections pre-emptively due to the WAAS transparency mode.
    More specifically, the bug that was triggered in the Cisco/Miercom test is found in an HTTP optimization feature that WAAS does NOT have. I am talking specifically about Riverbed’s HTTP object parse and pre-fetch feature. By simultaneously downloading 5000 x 1MB web objects in their test, Cisco/Miercom were able to force the Riverbed HTTP optimization module to consume 5GB of memory within an instant, and this caused the Steelhead to panic as it came under memory pressure. In other words, they were exploiting a weakness in a feature that they don’t offer in their own WAAS product.
    In any case, that weakness has now been fixed. Furthermore, no one who has been using Riverbed’s HTTP parse & prefetch feature ever encountered this bug in their real-life networks anyway. The bug can only be triggered under the extreme torture-test conditions found in the Cisco/Miercom test.
    Josh

  6. Riverbed is annoying said

    Riverbed and this garbage they splutter out is so annoying. Went through a recent review of WAN accel recently and all other vendors had ethics and stuck to their product not how crap the other vendors were. The crap Riverbed puts you through just to buy their product is beyond annoying and it just makes Riverbed seem so insecure and biased.
    Just to indicate my bias we did go with Riverbed in the end, primarily due to their vmware specific roadmap.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: