The Riverbed Blog (testing)

A blog in search of a tagline

Why do competitors resort to paying “independent” testing firms to rig unfair tests?

Posted by riverbedtest on June 1, 2011

 
Bribe
 

Riverbed competitors have a huge problem–no matter how hard they try, they just can't manage to beat Riverbed in product evaluations performed by prospective customers.  The problem for these vendors is that customers are just too insistent on using fair and objective evaluation procedures.  When WAN optimization products are judged using such customer-supplied evaluation criteria, that usually means Riverbed comes out on top. 

A number of WAN optimization vendors have been frustrated by customer tendencies to ignore their suggestions on how to conduct product evaluations. And so they have taken matters into their own hands by paying "independent" testing companies to design their own set of test procedures.  Once paid, the testing firms loyally follow orders to engineer tests that hide any weaknesses in the sponsoring vendor's product while creating a perception of a problem in the Riverbed product.

The following diagram is an obvious example of what I'm talking about.  The "independent" testing firm asserts that Riverbed's symmetric optimization technology cannot be used in public cloud environments, ignoring the existence of our Cloud Steelhead product and recent Akamai announcement.  A truly objective evaluation would have documented the differences between asymmetric forward caching (a technology in wide use since the 1990's) and the new generation of symmetric dual-ended WAN optimization technologies, and simply noted that Riverbed does not natively support the former.  But of course, the sponsoring vendor is not interested in objectivity, and so they order the testing firm to place the Steelhead product into the asymmetric configuration for which it was not designed (without the benefit of Squid running on RSP), and to run through the testing procedure as if it were a meaningful test. 

Below:  "Independent" tester uses misconfigured Steelhead to demonstrate superiority of the sponsoring vendor's product

  Test

While the above shows a recent example, the fact is that over the years, a number of different Riverbed competitors have resorted to paying "independent" testing firms to rig unfair tests.  In the past years, such bogus tests have not prevented sponsoring vendors from continuing to lose market share in WAN optimization.  And yet Riverbed competitors continue to sponsor bogus tests such as the example above.  What is more puzzling is that these sponsoring vendors seem to think they can actually fool a portion of their prospective customers with such dubious reports.  It all just goes to show the level of regard they have for their customers.

Download White Paper – Riverbed Rebuttal of Blue Coat Testing Report

Advertisements

12 Responses to “Why do competitors resort to paying “independent” testing firms to rig unfair tests?”

  1. EIP said

    I was talking to a customer a couple of years back, and I gave him some printed copies of ‘neutral testing’ for the product I was positioning (It was not related to WAN Optimization). He immediately told me that anyone can ‘buy such tests’!
    So I guess customers do not take such results on the face value, but still read such articles so that they can know the technical factors/ features with which they can evaluate such products…

  2. Josh Tseng said

    I agree may be some value in reading test reports for many types of IT products, even if those reports are sponsored by a vendor. However, the reports I’ve seen in the WAN optimization market–most recently the one issued by Blue Coat–are so blatantly lopsided that sponsoring vendor must think their target customer is very gullible and easy to fool.
    If I were to create an “independent” test report designed to benefit my products, I would be a bit less obvious by making the test results a bit more believable.
    Josh

  3. Peter said

    This is true to an extent, although not all competitors use unfair testing conditions.

  4. Johnny Juniper said

    This is why we never use Gartner, they are paid hacks

  5. Steve Broadhead said

    Josh.
    Apologies for the delayed response to this blog, but it has only just come to my attention.
    I am the founder of Broadband-Testing and the author behind the report you are attacking. I have already responded – over two weeks ago – to another Riverbed blog (http://www.linkedin.com/groups/Published-Report-BroadBand-Testing-showing-151524.S.56049378?view=&srchtype=discussedNews&gid=151524&item=56049378&type=member&trk=eml-anet_dig-b_pd-ttl-cn) and I have responded accordingly to this one.
    If you would care to visit our site at this link:
    http://www.broadband-testing.co.uk/download/BBT_Response_to_Riverbed.pdf
    you can read a full explanation (too long for a blog comment) as to why you are so wide of the mark in your comments it’s scary (as well as being personally insulting) and as to how you are factually incorrect in many ways, including your claims as to what Blue Coat cannot do, but that’s for them to respond to (as I believe they have).
    And, BTW, reading some of the other blog entries, I thought you ought to know that the “no one ever got sacked for…” phrase originates with IBM; Cisco only borrowed it.
    I should also add that, as a jobbing journalist along with my product testing duties with Broadband-Testing, it is not generally considered smart behaviour for a vendor to upset a journalist. Just polite advice…
    Cheers – Steve Broadhead, Broadband-Testing

  6. Josh Tseng said

    Steve,
    Thanks for your post. You claim my comments are “so wide of the mark,” but your response document does little more than restate the thesis of the original report that Blue Coat paid you to put together.
    What puzzles me more than anything is that after all that has been discussed, you continue to portray yourself as an a neutral 3rd party. I’m sorry, but if you are paid by one party to conduct the test, then that disqualifies you from making any believable claim of objectivity. I suggest you not insult anyone’s intelligence by presuming otherwise–I know it, and I think most readers know it. In the end, this practice reflects badly on the vendor offering you the bucks. Perhaps that is what is really bothering you–these observations threaten your business model. Maybe Blue Coat and other vendors are now less inclined to give you more bucks to do more “independent” tests.
    Josh

  7. Josh.
    Thanks for taking the time to check out the response.
    Please note, we weren’t hiding anything from anyone here; a press release put out by Blue Coat, that I vetted, confirmed that the company had engaged me directly in the first paragraph. The last thing I would ever try to do is mislead anyone.
    All I can assue you – and everyone – is that the testing was carried out and published in exactly the same way as would be a single product/vendor test (which almost all our public tests are). There is nothing I can do, of course, if you choose to disbelieve that.
    The other point I was trying to make to Riverbed and Blue Coat is to avoid complacency; there are lots of new kids on the block and the competition will get more intense. The need to support new application types natively (rather than via 3rd party relationships) is, I believe, essential. Hence the reason for noting the likes of Voipex (who I understand had talks with R’bed last year), and – as I said – watch out for some new faces appearing this summer (which is very busy for us).
    Cheers – SteveB

  8. Josh Tseng said

    >>>All I can assue you – and everyone – is that the testing was carried out and published in exactly the same way as would be a single product/vendor test (which almost all our public tests are). There is nothing I can do, of course, if you choose to disbelieve that.<<<
    Steve,
    I have read your response document, but I really wonder if you have read or considered Riverbed's response to your report. If you haven't read it, then please do so, because I don't want to repeat it here.
    Your documents all echo the Blue Coat marketing message. We have competed against Blue Coat for 4 years, and there is nothing new or surprising in your reports offer that is different from the standard Blue Coat marketing pitch from the last 4 years. Blue Coat's video capabilities date from the year 2000 (back when Real Networks was the standard for Internet video), and their asymmetric web caching capabilities (i.e., the public cloud test in your report) date from 1996 when Blue Coat was known as Cacheflow (heck, I did some work with Cacheflow when I was an HP employee back in the late 1990's…).
    That being the case, I am still puzzled as to how you can expect people to believe you are acting in an objective manner. Your report was a hack job paid for by Blue Coat to tout their marketing message. It emphasizes legacy capabilities available in many other products today (some are free), while ignoring and de-emphasizing the newest generation of WAN optimization capabilities first pioneered by Riverbed.
    Josh

  9. Josh.
    This is just going round in circles now, so final comment from me: I made the point that I’ve read the R’bed response, hence my note that, while some of it makes sense, other aspects are fundamentally incorrect. And that’s being kind.
    And, as to the “nothing new” argument. I have made the point several times that I have seen far better technology from clients in the past than anything that was on show in the recent test project; but that particular technology wasn’t given marketing $$$$ and hence is no longer still around. The UK/EMEA produces most of the leading technologies but never stands a chance against US investment – that’s the reality we live with in Europe.
    But in terms of being able to support every type of acceleration in a single device; well if you guys don’t think that a one-box solution that accelerates Video, VoIP, SaaS etc is important, with The Cloud looking to dominate and outsourcing going crazy, then “there may be trouble ahead”, in the immortal words of Irving Berlin. I’ve already made the point that many SPs/MNSPs etc in EMEA won’t even consider WAN optimisation currently as it’s already seen as too complicated? So are they going to sign up for a solution that requires multiple devices/servers/services? I don’t think so…

  10. Josh Tseng said

    >>>But in terms of being able to support every type of acceleration in a single device; well if you guys don’t think that a one-box solution that accelerates Video, VoIP, SaaS etc is important, …So are they going to sign up for a solution that requires multiple devices/servers/services? I don’t think so…<<<
    Steve,
    Yes, you claimed to have read the Riverbed response, but you seem to be ignorant of what it says. That is why I suspected you had not read it. If you had read it, then you would have known that Riverbed does offer a one-box solution that can run any and all of Blue Coat capabilities and more. It's known as Riverbed Services Platform. A sizeable number of Riverbed customers are using it to install Blue Coat's competitors (Websense, McAfee, Squid, etc.) on a Steelhead box to obtain security, video, web caching (i.e., your SaaS acceleration), as well as a number of other capabilities not available from Blue Coat. But no need to repeat the material here–go read the Riverbed response if you want to understand it.
    I suspect you are a newbie to WAN optimization. If that's the case, I suggest you not obtain your technology survey by just talking to one vendor. Just as an enterprise customer investigating a new technology would, you should talk to multiple vendors, in order to get a better perspective of the capabilities and features of various products out there. In this case, the one vendor you are talking to is not even in the top 3 in terms of WAN optimization market share, according to Gartner. They have been losing share for quite some time.
    Josh

  11. Bert-Jan said

    Gents, while you’re in the contest of proving which one is right or which one is wrong I, as a true independent tester can tell you one large flaw in the Bluecoat asymmetric sided caching, it is faster of course but also a large risk of serving stale content. We have many financial institutes as customers and believe me they do NOT want to get served stale content like old exchange rates or other stale content that they need to rely on to make financial or business decisions! Many providers do not tag their content properly to identify a newer version is available. Hence you will have to put cache bypass rules in for each and any site with the result that the advantage is gone completely. Serving stale content is a nightmare and also management wise difficult to handle! So I would strongly suggest to use symmetric caching to ensure the correct content is ALWAYS served. Same goes for documents (CIFS).
    Coming back to the validity of the tests, these kind of drawbacks are NOT taken into account when testing which in my opinion makes the tests indeed invalid and not relevant.
    Furthermore I cannot believe that the baseline and the warm Riverbed test have the same result. This obviously looks like the tester did something wrong with the configuration or the test is indeed rigged. I would gladly take up the challenge with the tester to redo the test properly and see if the results are still the same.
    Last note: There is more than performance that will make a good WAN Accelerator! If customers really want the best solution on WAN acceleration, they should contact us (www.ipknowledge.net) as we have tested them all independently in customer environments and not in labs that do not represent the true world!

  12. Mark Day said

    One of the many interesting aspects of WAN optimization is that it triggers passionate debates, with a level of intensity that is surprising to many observers who expect IT discussions to be colorless and boring. The downside of the passion is that sometimes people in these arguments get carried away. As sometimes happens in political discussions, the two sides dig in, stop listening, and talk past each other. Unfortunately I think that’s what happened here.
    To be clear about the part of the argument that has merit on each side:
    • it’s completely reasonable for Steve Broadhead to insist that his firm clearly disclosed that they were paid by Blue Coat, and to counter any impression that the relationship to Blue Coat was hidden.
    • And it’s also reasonable for Steve to reiterate that his firm really performed the tests as described.
    • Likewise, I think it’s completely reasonable for Josh Tseng to point out that it is intrinsically unconvincing for Blue Coat (or anyone else) to engage a firm like Broadband-Testing for this kind of testing and describe them as an “independent” testing firm, as though they had performed an even-handed test or were completely unbiased with respect to the desired results.
    • And it’s also reasonable for Josh to point out specifics of which tests were chosen, as a concrete demonstration that the Broadband-Testing work was deliberately tilted in favor of the sponsor. Pointing to flaws in the testing does not mean that Steve Broadhead is dishonest: it means that he was an honest agent working on behalf of his clearly-disclosed sponsor – which is quite different from being a good source of comparative information.
    Unfortunately, both sides then escalated the argument to a place where some dubious things were said on both sides and the overall tone is not good. We have corporate standards and policies on blogging, and this thread fell short of the high quality and professional demeanor that we aim for. For the Riverbed side I’d like to apologize for those deficiencies, while underscoring the correctness of the technical and analytical points that Josh made.
    Sincerely,
    Mark Stuart Day
    Chief Scientist, Riverbed Technology

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: