The Riverbed Blog (testing)

A blog in search of a tagline

Is lab testing enough?

Posted by riverbedtest on June 25, 2008

Just imagine this scenario:  You are now five months into the deployment process rolling out devices from your chosen WAN optimization vendor.  The project was supposed to have been completed months ago, but it has been stalled for the last four months. The first 8 to 10 sites went relatively smoothly, and WDS devices were deployed to those initial sites in a matter of weeks.  But then problems started occuring when you attempted to deploy additional WAN optimization devices to your remaining sites.  As the number of sites having the devices grew beyond the initial few, application sessions started dropping.  Connections started slowing down.  End-users started complaining. You are concerned and frustrated, especially since the last 4 months have been spent bickering with the vendor about the original promises they made to you.

What went wrong?  During the evaluation process, you did extensive lab testing to prove that the WDS vendor’s products worked as claimed.  You even used a WAN simulator to inject the latency and delay that you knew existed in your real-life WAN.  And yes, you observed 30X performance improvement for transfering files using CIFS between your lab machines.  Yes, you observed 75% bandwidth reduction as promised by the WDS vendor.  Everything seemed to work as advertised, so at the time you had no reason to question the vendor’s claims that the solution would easily scale to the 48 branch offices connected to your company’s WAN infrastructure.

The unfortunate fact is that many customers have endured similar scenarios to that described above, as I can personally attest.  Their mistake–relying only on lab testing to validate vendor claims for a WDS solution that is to be deployed at large scale, to many sites, with large numbers of users.  The problem is that just about all WDS solutions can perform adequately…in limited and isolated lab environments, where they are shielded from real-life network conditions found in real-life WAN environments.  They might even work in a real-life network of limited size, with a small number of sites each with a few dozen users.  But when faced with a large and complex production network environment, including heavy traffic loads and an unknown and diverse mix of different applications, many WDS solutions will fail.

So what would have been a better approach to avoid the above outcome?  One measure to consider is to do a live pilot test over your own WAN infrastructure, using the WDS vendor’s devices deployed to your real branch office sites.  The primary benefit with this approach is it allows you to observe the vendor’s products in action optimizing transfers of the actual file and application data used by your company’s employees.

But even then, a pilot test is rarely practical beyond more than a small number of sites.  Most WDS vendors aren’t going to lend you more than a few devices to conduct your pilot testing.  With the limited scale of the pilot there’s no way you can be sure that what works for 2-3 sites is going to work for all 48 sites in your production network.

That’s where the advice in my previous blog comes in–make sure you talk to references of other customers who have deployed the WDS vendor’s product.  Don’t accept just any reference–it must be a reference of a customer who has deployed the WDS solution at a similar scale to what you envision for your network.  Furthermore, it is important to understand if the reference customer’s network environment and applications are similar to that of your own.

Identifying the proper WDS solution is an extremely important process, and there is too much at stake to undertake this process casually.  Selecting wrong WDS vendor solution, especially one that is incapable of scaling for your environment, will result in months of wasted effort and frustration.  The resulting loss to your corporation goes far beyond the monetary investment for the unusable equipment.

One Response to “Is lab testing enough?”

  1. Frankie04 said

    The article is same as what I think. It is good to see such netural document in a vendor’s blog.
    Particular to this document, I think it is not only hard to ask vendor to lend multiple sites WDS device, but also hard to get a realistic customer reference. So, it is still be a very hard situation to stimulate a close enough environment to the live network of our own.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: