Isilon and Vmware ESX

I've done limited VMware ESX testing using Isilon as datastores, and so far no issues.

Isilon has the ability to fail NFS mounts from one head to another without any inflight read or write loss which is impressive. They also provide up to N+4 protection which is the highest that I've seen. N+4 means you can have up to 4 heads fail without data loss.

Storage (up to 12TB in 2U) can literally be added in 30 seconds. Plug in power, ethernet, and infiniband cables and power on. The front panel asks "Join cluster xxxx Y/N". Select Y and 30 seconds later you have another 12TB.

Isilon also has the ability to dynamically load balance NFS clients across the cluster, which will allow you to load balance your ESX host using multiple NFS mounts. Hopefully Vmware will soon support dynamic DNS IPs for cluster storage, which would allow only one NFS mount.

Performance using one ESX hosts is just as good the other NAS solutions I've tested. Each Isilon node adds CPU and cache, so a 6 node cluster is like having 6 NAS heads! So basically the cluster can grow up to 1.6PB and 96 nodes and performance gets better as you add nodes.

Isilon does Point in Time snapshots at any directory level which is also unique to the NAS market.

Isilon currently only does file replication to a secondary cluster, which means that if a file changes the whole file needs to be transferred to the second cluster. Block level replication is in the works...

Isilon has some real potential and some very unique features with OneFS, and with 10G support early next year Isilon might be a preferred solution soon.



I've started a Isilon Yahoo group here: Isilon at Yahoo! Groups








dp

VMware ESX 3.5 Released Today

VMWARE ESX Server 3.5 was released today: 12/10/2007.

VMware ESX Server 3.5
Latest Version: 3.5 12/10/2007 Build: 64607 Release Notes

ESX 3.5 Server is a major release that will deliver several new features.

As I see it, Storage Vmotion is the largest feature, however it's not included in these downloads and must be downloaded as a separate component. Migrations using Storage VMotion must be administered through the Remote Command Line Interface (Remote CLI), which is available for download here.

Here's a short list of Vmware ESX 3.5 New Features



Storage VMotion
InfiniBand network cards
10Gbit Ethernet network cards
Update Manager
SATA storage devices
Swapfiles-less VMotion
200 hosts and 2000 virtual machines
128GB RAM per host 64GB RAM per virtual machine
AMD Rapid Virtualization Indexing
para-virtualized Linux guest OSes
Open Virtual Machine Format (OVF)

Neterion's 10 Gigabit Ethernet Driver Included in VMware ESX 3.5

Neterion's Xframe V-NIC is First 10 GbE Adapter to Be Supported in New Release of VMware 3.5, enabling IT Managers to Virtualize I/O-Intensive Applications for the First Time.


Now if we can get the network storage vendors on the ball with 10Gbit support!

Why VMware over Netapp NFS

1. While today the majority of VMware ESX servers connect to their datastores using either Fiber Channel or iSCSI protocol. I believe that using the NFS protocol is a significantly better way to access your ESX datastores. Here's why...
   
   An ESX datastore is simply a place to store your virtual machine files. Yes files, nothing more than files like a word document. So all that’s needed from an ESX host is a way to read and write to these files.
   
   You have two options for datastores: VMFS or NFS. If you want to use the advanced features like VMotion you need a datastore that is shared across all your ESX hosts. Initially VMware only supported VMFS for datastores and hence the reason for the high number of FC implementations, however NFS was added in ESX 3.0 (August 2006) and is starting to catch on.
   
   NFS is not really a filesystem, but a protocol to access files on a remote file server and has been in use since 1984. The remote file server is where the filesystem lives and is really where all the magic happens. In the case of Netapp, the filesystem is called WAFL, with Windows it’s called NTFS and Linux it may be ESX3 or some other filesystem.
   
   So basically, Vmware not only has a significant burden with managing all the components of virtualization, it also has to maintain VMFS. With NFS, the burden shifts to the NFS vendor which also has the freedom to add features as long as it adheres to the NFS protocol.
   
   Here are some reasons to use the Netapp implementation of NFS for VMware instead of using VMFS volumes over FC or iSCSI:

• You get thin provisioning by default with NFS. FC and iSCSI VMDKs are thick. This can save 50% of your disk space.


• Adding NFS datastores are simple. Mount the NFS volume using the GUI and start creating VMs


• Adding additional Netapp filers for datastores requires no down and no cabling changes.


• You can have large datastores that span many disks. 16TB for Netapp.


• You can use A-SIS to de-duplicate your datastores for a 50-80% reduction in disk space


• You can expand AND decrease NFS volumes on the fly


• You can use snapshots of the volumes to restore individual VMs


• You can use snapmirror to backup VMware volumes to a DR site over a WAN


• You don't have to deal with VMFS or RDMs


• You don't have to deal with FC switches, zones, lun sizing, HBAs, and identical LUN IDs


• You can restore multiple VMs, individual VMs, or files within VMs.


• You can instantaneously clone (Netapp Flexclone), a single VM, or multiple VMs


• You can also backup whole VMs, or files within VMs using NDMP or any other backup software


• ESX server I/O is small block and extremely random which means that bandwidth matters little


• No single disk I/O queue, so your performance is strictly dependent upon the size of the pipe and the disk array.
• Failover to your SnapMirrored copies can be done in minutes. iSCSi/FC requires LUN resignaturing.
  
• In the near future, you will be able to clone a single VM or create 100’s of VMs from a template in seconds
  
  

Some background… The previous information is based on our experience, and not just some theory.

In August 2006 when NFS was announced we were in the planning stage for a major upgrade to our VMware infrastructure. Our VMware infrastructure then consisted of about 20 ESX hosts with about 750 VMs all using Fiber Channel to Hitachi SAN. We are also a heavy user of Netapp filers and knowing the benefits of NFS over SAN we decided to investigate the possibility of using Netapp over NFS. I’m pretty sure we were the first Netapp customer to do this…

Of course the first hurdle was performance. Fortunately, we had more than a years worth of VMware performance data on our SAN. After looking very close at the numbers, we discovered that the throughput to the SAN was extremely low. Somewhere in the 10-15MB/s on average across all 20 ESX host, and the spikes were well under 50MB/s. Since the migration to NFS is so simple, we decide to move several test servers to NFS. All we had to do is mount a NFS share on the current ESX hosts and start moving the VMs. After migrating several 100 VMs to NFS over 6 months, we decided to implement our new Infrastructure completely on NFS.

We purchased 2 dedicated Netapp 3070s and several new 8way ESX hosts for the new project. We also used an existing Netapp R200 to keep 21 says of snapshots for online backups. The R200 also serves as a failover I case of complete corruption of our primary system. Within 6 months we had completely migrated all of our VM’s off SAN and onto Netapp NFS. We now run almost 1000 VMs in this environment.

With our current Netapp IO load on the 3070’s, we estimate that we could add 2000 or more VMs to this configuration by simply adding ESX hosts. The Netapp 3070c IO is running 4MB on average with a few 30MB spikes during the day. Not one IO performance issue has arisen. Our VMware administrators says it’s even faster than our SAN when performing OS Builds, VMotion and Cloning.

We currently don’t run Exchange or Sqlserver VMs, however with 10Gbit and Infiniband solutions on the way I believe that soon all real servers will be virtual.

So I stand my initial statement, however I should say that today it’s really Netapp and not NFS that makes the difference. In the future however, I expect to see other vendors catch up with Netapp and all their added value to the VMware Infrastructure environment.


Additional Links to NFS for Vmware

• Netapp Building Vmware with NFS and POC


• NETWORK APPLIANCE AND VMWARE VIRTUAL INFRASTRUCTURE 3 STORAGE


• VMware over NFS


• VMware Server and NFS? Am I alone
  
  NFS Saved By VMware?
  
  
  
  
  

Advanced Single Instance Storage (A-SIS)

In May 2007, NetApp announced a new deduplication technology that can significantly increase the amount of data stored in a set amount of disk space: Advanced Single Instance Storage (A-SIS) deduplication. This technology is available (at no charge!) for NetApp.

How much space can you save? It depends on the data set and the amount of duplication it contains. Here are a couple of examples of the savings that NetApp customers have seen:

• An investment management company reduced backups copies of their VMware images by 90%.
• A test and measurements manufacturer realized a 98% space savings on daily database backups.
• A global oil and gas company achieved a 35% space savings for its home directory storage.

Here's a great TechTalk A-SIS and a Technical Report Here

A-SIS is part of the Nearstore license which I believe is under $3K, but don't quote me... Your rep should be able to give you a quote. You need 1 license per filer and no other cost.
You vmware volumes will be reduced (calculate) by 50-80+% which will more than pay for the license...

Microsoft Virtual Machine Manager (VMM)

Here are some first impressions:

http://4sysops.com/archives/microsoft-virtual-machine-managers-vmm-bugs/

Here is a video overview of VMM:

http://www.microsoft.com/systemcenter/scvmm/flash/SCVMM.html

Oracle VM

Oracle is entering the virtual arena with its own Oracle VM hypervisor based on Xen called Oracle VM. The new product includes a web management console and will be released for free (with optional support agreements) on November 14.

Oracle VM will available as a free download here Wednesday.

What is Oracle thinking? I don't get it. Maybe it will be clear on Wednesday.

Oracle VM is server virtualization software which fully supports both Oracle and non-Oracle applications, and is three times more efficient than other server virtualization products. Backed by Oracle's world-class support organization, customers now have a single point of enterprise-class support for their entire virtualization environments, including the Linux operating system, Oracle Database, Fusion Middleware, and Application software—all of which are supported with Oracle VM.

VDI and Netapp: 100VMs, 10 Minutes, 10Gb total!

If you need to create 100's of VMs or VMs of the same type in seconds using only the size of 1 VM, then Netapp has your answer.

Netapp Flexclones are writable clones of a parent volume that use no additional space. Flexclones can be used to create clones of VMs in seconds.

I have been talking about the power of using Netapp for your datastores. Here is yet another feature that no other vendor can do: 100VMs in 10 Minute DEMO

RamSan-500 2TB Solid State Raid

The photo speaks for itself.. This 2TB flash array has 8FC ports, 64GB of DDR cache memory and hot swapable flash modules in a raid configuration. The management interface allows for quick configuration of LUNS and front lcd panel is used for setting the IP address and name. List price is $300K, or $150K/TB

Texas Memory Systems 2TB Flash Array

CAPACITY: 1 to 2 TB Flash 16 to 64 GB DDR Cache

INTERFACES: 4Gb FC 2Gb 4x InfiniBand

PERFORMANCE: 100,000 IOPS 2 GB/s r/w

The Flash Memory (1-2TB usable capacity) is arrayed in nine RAID-5 protected hot swappable modules. The reliability of each module is further enhanced with ECC memory layouts, wear-leveling, and bad-block retirement. The RamSan-500 uses NAND-SLC flash memory, the highest quality of flash memory available on the market. The RamSan-500 is designed from the ground-up to protect data from the problems inherent to traditional flash storage.

FusionIO: The Next Big Thing in Storage

Update: 12/28/07 -- FusionIO updated their web site... They have a new girl ( I like the old one) and an listed price of $2,400 for the 80GB model which is exactly $30/GB

The enterprise storage game just changed today. "Power of a SAN in the palm of your hand"

If FusionIO is for real, enterprise storage has just shifted gears. While storage vendors have been focusing on larger disk based systems with all the fancy features, it appears that out of nowhere a new storage solution was born today.

Demo Says

ioMemory & the ioDrive may prove to be among the most important products
ever to launch at DEMO. As our volumes of data and digital media soars, ioMemory addresses well the new demands of the digital content age. Yet this
high-performance, high-capacity, small-footprint, low-impact storage technology
may have tremendous implications for the design of data centers and their impact
on energy consumption, as much as for the product’s capability as a high-speed
storage solution.

A new flash storage card could make huge storage area networks go the way of the floppy disk. The company’s ‘ioDrive’ combines hundreds of gigabytes of flash storage onto a small computer card and company officials claim that the tiny card could replace banks of hard drives.

The card will initially have 80-640 GB of NAND flash on ONE PCI card and will scale up to 1.2 TB by the end of next year.

Price: FusionIO says ~$30/GB

Available: 1Q 2008

Performance? Remember when you moved from floppy to a hard drive? FusionIO is taking us to the next level of performance just like the hard drive did when it replaced the floppy disk.

Here are the performance numbers:

VMWARE and FUSIONIO. WOW!

VMworld 2007 Highlights

ESX 3i , Storage Vmotion and Site Recovery Manager (SRM)

ESX Server 3i is the industry’s first hardware-integrated hypervisor built on a next-generation thin architecture, giving your organization an efficient foundation for building a dynamic, automated data center.

Storage Vmotion gives the admin the ability to move the backend storage of a VM without taking the virtual machine down. It's still unclear exactly how this will be implemented, but basically the vmdk files will be copied to a new location then vmware will automagically swap to the new files. If you have a VM that needs faster disks, SVmotion it!

Site Recovery Manager will allow the admin to control how a system fails over to another site. The backend storage will do the syncing of the data, and virtual center will allow the control of switching to the new site. Business Recovery predefied! Nice!


VMware ESX Server 3.1.0 / VirtualCenter 2.1.0 features list - Updated with full details

Virtual Optics. The Beginning

I'm not a writer or pretent to be one that has a grasp on the english language. (see my point yet?) But I do have ideas and i hope someone someday might find them here.



Virtual Optics: The ability see the future of computing. Virtually...

Vmware started an revolution in enterprise computing. Not since the beginning of the IBM PC in 1982 has such a product stired so many business roots..


What a better way to start a new blog but on the eve of the next generation of enterprise computing. VMworld!

VMworld 2007 starts tomorrow and with all the posting online, I thought I would put my 2 cent into the pot.

My goal is to track all aspects of enterprise virtualization. Wish me luck.