Why VMware over Netapp NFS

1. While today the majority of VMware ESX servers connect to their datastores using either Fiber Channel or iSCSI protocol. I believe that using the NFS protocol is a significantly better way to access your ESX datastores. Here's why...
   
   An ESX datastore is simply a place to store your virtual machine files. Yes files, nothing more than files like a word document. So all that’s needed from an ESX host is a way to read and write to these files.
   
   You have two options for datastores: VMFS or NFS. If you want to use the advanced features like VMotion you need a datastore that is shared across all your ESX hosts. Initially VMware only supported VMFS for datastores and hence the reason for the high number of FC implementations, however NFS was added in ESX 3.0 (August 2006) and is starting to catch on.
   
   NFS is not really a filesystem, but a protocol to access files on a remote file server and has been in use since 1984. The remote file server is where the filesystem lives and is really where all the magic happens. In the case of Netapp, the filesystem is called WAFL, with Windows it’s called NTFS and Linux it may be ESX3 or some other filesystem.
   
   So basically, Vmware not only has a significant burden with managing all the components of virtualization, it also has to maintain VMFS. With NFS, the burden shifts to the NFS vendor which also has the freedom to add features as long as it adheres to the NFS protocol.
   
   Here are some reasons to use the Netapp implementation of NFS for VMware instead of using VMFS volumes over FC or iSCSI:

• You get thin provisioning by default with NFS. FC and iSCSI VMDKs are thick. This can save 50% of your disk space.


• Adding NFS datastores are simple. Mount the NFS volume using the GUI and start creating VMs


• Adding additional Netapp filers for datastores requires no down and no cabling changes.


• You can have large datastores that span many disks. 16TB for Netapp.


• You can use A-SIS to de-duplicate your datastores for a 50-80% reduction in disk space


• You can expand AND decrease NFS volumes on the fly


• You can use snapshots of the volumes to restore individual VMs


• You can use snapmirror to backup VMware volumes to a DR site over a WAN


• You don't have to deal with VMFS or RDMs


• You don't have to deal with FC switches, zones, lun sizing, HBAs, and identical LUN IDs


• You can restore multiple VMs, individual VMs, or files within VMs.


• You can instantaneously clone (Netapp Flexclone), a single VM, or multiple VMs


• You can also backup whole VMs, or files within VMs using NDMP or any other backup software


• ESX server I/O is small block and extremely random which means that bandwidth matters little


• No single disk I/O queue, so your performance is strictly dependent upon the size of the pipe and the disk array.
• Failover to your SnapMirrored copies can be done in minutes. iSCSi/FC requires LUN resignaturing.
  
• In the near future, you will be able to clone a single VM or create 100’s of VMs from a template in seconds
  
  

Some background… The previous information is based on our experience, and not just some theory.

In August 2006 when NFS was announced we were in the planning stage for a major upgrade to our VMware infrastructure. Our VMware infrastructure then consisted of about 20 ESX hosts with about 750 VMs all using Fiber Channel to Hitachi SAN. We are also a heavy user of Netapp filers and knowing the benefits of NFS over SAN we decided to investigate the possibility of using Netapp over NFS. I’m pretty sure we were the first Netapp customer to do this…

Of course the first hurdle was performance. Fortunately, we had more than a years worth of VMware performance data on our SAN. After looking very close at the numbers, we discovered that the throughput to the SAN was extremely low. Somewhere in the 10-15MB/s on average across all 20 ESX host, and the spikes were well under 50MB/s. Since the migration to NFS is so simple, we decide to move several test servers to NFS. All we had to do is mount a NFS share on the current ESX hosts and start moving the VMs. After migrating several 100 VMs to NFS over 6 months, we decided to implement our new Infrastructure completely on NFS.

We purchased 2 dedicated Netapp 3070s and several new 8way ESX hosts for the new project. We also used an existing Netapp R200 to keep 21 says of snapshots for online backups. The R200 also serves as a failover I case of complete corruption of our primary system. Within 6 months we had completely migrated all of our VM’s off SAN and onto Netapp NFS. We now run almost 1000 VMs in this environment.

With our current Netapp IO load on the 3070’s, we estimate that we could add 2000 or more VMs to this configuration by simply adding ESX hosts. The Netapp 3070c IO is running 4MB on average with a few 30MB spikes during the day. Not one IO performance issue has arisen. Our VMware administrators says it’s even faster than our SAN when performing OS Builds, VMotion and Cloning.

We currently don’t run Exchange or Sqlserver VMs, however with 10Gbit and Infiniband solutions on the way I believe that soon all real servers will be virtual.

So I stand my initial statement, however I should say that today it’s really Netapp and not NFS that makes the difference. In the future however, I expect to see other vendors catch up with Netapp and all their added value to the VMware Infrastructure environment.


Additional Links to NFS for Vmware

• Netapp Building Vmware with NFS and POC


• NETWORK APPLIANCE AND VMWARE VIRTUAL INFRASTRUCTURE 3 STORAGE


• VMware over NFS


• VMware Server and NFS? Am I alone
  
  NFS Saved By VMware?