Recently, I found myself in a position where I needed to write some technical recommendations, apart from my job for a project. In the end, I decided not to do the project, but I spent a great deal of time writing up this document. I felt like it would be a shame to waste all that time I put into it, so I’m going to post it here on my blog. It’s probably nothing that most of my readers care about, but since I put time into it, I wanted to do something with it.

It’s rough, and needs a bit more editing, but if this is something you find useful, feel free to share it, with proper attribution, and let me know that it found a life outside of my Google Drive folder.

Challenges and opportunities with VMware VSAN technology

Jamison Swift

Introduction

With the goal of the fully virtualized datacenter now firmly entrenched in most organizations, the gateway has been opened up to many new paradigms for enterprise technology. Virtualization of the operating system is only the beginning, and has opened the door to virtualization of other layers of the technology stack. No longer do organizations need to be bound to physical hardware limitations when implementing new networking features, as they can simply implement a newer virtual image of a firewall or switch. The same concepts apply to storage which no longer needs to be bound to specific hardware devices and technologies. Instead, virtualized storage can provide a way for organizations to utilize commodity hardware to horizontally scale their architecture, without large capital investments in dedicated storage arrays. One of the market leaders in this space is VMware with their Virtual SAN (VSAN) product.

Overview of Virtual SAN and Software Defined Storage

Software Defined Storage is a recent paradigm in the enterprise technology world. Simply put, software defined storage means that your storage layer is abstracted from your hardware layer, through the use of specialized software. SDS allows storage administrators to present storage to operating systems in a unified manner, without the operating systems knowledge or understanding of the hardware configuration that is presenting the storage.

For storage administrators, this paradigm requires a deeper knowledge of how their storage is being presented and utilized to the end system. No longer can they simply provide access to a LUN (Logical Unit Number) and let the operating system take over from there. Instead, the storage administrator now has much finer grained control of what they can offer to the client operating system. Various physical storage arrays can be grouped together to create virtual LUNs that can be combined and given to clients as a single storage array.

Behind the scenes, storage administrators can alter the physical arrays to better maximize usage and control the amount of storage that they need to purchase. This can have a positive effect on capital costs, as storage can be more tightly controlled and managed centrally. Storage administrators can dynamically change the amount of storage that is presented to host operating systems, without the host operating system knowing where that storage came from. It could be that small portions of that storage came from other parts of the storage array that are not known to the host operating systems, but SDS abstracts those details and simplifies life for system administrators who are watching their systems storage needs grow over time.

Enterprises may already be familiar with a type of Software Defined Storage if they currently utilize Netapp Clustered Data OnTAP. This type of SDS is vendor specific and built into the Netapp cDOT operating system. The Netapp cDOT system allows admins to present various LIFs (logical interfaces) that abstract which Netapp FAS controllers and storage aggregates the host operating system is utilizing. This also allows storage administrators to move storage volumes between various aggregates in an effort to balance capacity and utilization.

This type of SDS gives a measure of flexibility, however it has it’s limitations. It is based solely on the implementation on the storage array hardware itself. Whenever additional storage is needed, additional disk trays must be purchased, and if an organization has reached capacity on its storage controllers, an entire new set of controllers must be purchased. The cost investment in large storage arrays often gives sticker shock to CIOs especially when trying to manage incremental growth.

Consider the following scenario: A Netapp FAS 8020 has a maximum capacity of 1,920TB and 480 drive units. If an organization needs to grow beyond these bounds, even in a small way, the procurement of more controllers, or larger controllers, can be cost prohibitive. Additionally, large scale hardware implementations often take crucial resources away from other work for multi-month projects of procuring, installing, and configuring new controllers and disk shelves. For these type of scenarios, VMware’s VSAN option can present a different paradigm that may be more appropriate for many enterprises.

VMware Virtual SAN is a part of the VMware product suite that comes with its vSphere virtual environment software. VSAN lives within the hypervisor itself, and presents storage to the guest operating systems the same way as standalone arrays. The difference is how that storage is acquired behind the scenes. With VSAN, clusters of commodity hardware, with on-board storage, is utilized to create a large pool of available storage that can be managed and presented dynamically to operating systems.

Just as with other SDS solutions, the receiving operating system is ambivalent as to the source of the storage. Storage administrators can manage the presented volumes much like traditional arrays, but everything is contained within the hypervisor environment. This means that both system administrators and storage admins can utilize a single hyper converged environment and toolset to manage systems from end-to-end.

VMware VSAN provides all of the core features of traditional arrays. Data deduplication and compression help to reduce storage growth, especially among scaled out systems with many identical blocks (such as VDI). Data deduplication and compression can be turned on at an array level with a simple toggle switch, however switching deduplication on will require a dynamic reformating of the arrays which can result in processing overhead. A maintenance window would be recommended if this feature is not utilized at system installation. Additionally, deduplication does not occur in the cache layer, but only in the capacity tier, due to frequent access of blocks in the cache layer and limiting performance issues for production systems.

VSAN can be implemented with two different configurations. A hybrid solution which utilizes flash storage for the cache layer, while relying on 10K spinning disks for the main capacity layer. This option can provide a cost savings to the enterprise, as often times there may be large data sets that are not accessed frequently and can be stored on the slower, cheaper traditional hard drive. The second option is an all-flash solution that utilizes flash disk throughout the entire workflow. This is obviously a more expensive proposition, but as the cost of flash storage continues to drop, there is no reason to doubt that this model will be the preferred option in the future.

In performance tests provided by VMware, both types of arrays performed well in both sequential and random access tests. As can be expected the all-flash arrays were able to provide much higher levels of performance, beating the hybrid solutions with 200-300% improvements in latency. VMware provides extensive tuning information to clients to help them tune their arrays based upon their needs.

Opportunities with Virtual SAN and SDS

For an organization that is already invested in VMware, utilizing Virtual SAN is quick win opportunity. One of the biggest advantages of VSAN the ability to have a single pane of glass for management of storage and systems. Using a single tool, both system admins and storage administrators can get a big-picture view of a set of enterprise systems. VMware vSphere stretched clusters allow admins to manage not just a single set of devices, but an entire enterprise, in multiple locations, with a single console. Storage that is allocated through VSAN can be spread across multiple clusters for even more flexibility, and to help conform to disaster recovery specifications.

Another advantage of Netapp VSAN is its reliance on flash based storage as a key component of the architecture. To create a cluster with VSAN you need at least three hosts (although full redundancy requires at least 4), and each node must have at least one SSD drive available to it. This increases read performance dramatically and helps to overcome the stigma that virtual environments have about I/O performance. When moving to an all-flash architecture with VSAN these performance improvements continue to improve as the capacity layer is able to perform mixed workload operations at a comparable speed to the caching layer. With all-flash poised to be the future of most transactional storage, VSAN is positioned as a strong option for organizations with a robust VMware ecosystem.

Commodity hardware is yet another opportunity with VMware Virtual SAN. Unlike traditional storage arrays, VSAN allows organizations to procure commodity servers from various manufacturers at a lower overall cost than dedicated storage arrays. VMware provides a ‘Virtual SAN Ready Nodes configurator tool’ on their website to help organizations match their needs to vendors who can provide systems that are ready to work with VSAN. Companies such as Dell, HP and Lenovo are all included as options for VSAN ready nodes that can fit into an organization’s existing enterprise hardware footprint. The reliance on commodity hardware means more choices for the enterprise than large storage arrays from a few select vendors.

The reliance on commodity hardware also gives VMware VSAN an advantage in horizontal scaling. As an organization grows, horizontal scaling becomes a simple math equation of adding more resources. Many traditional storage arrays allow expansion through disk shelves, but when the controller is maxed out, additional expensive controllers must be purchased, resulting in a more vertically scale out paradigm. VSAN allows organizations to continue to grow dynamically based upon needs. Additionally, virtual machine capacity expands along with VSAN capacity. As more nodes are added, CPU and memory capacity increase as well. This means a more integrated growth and lifecycle paradigm than with segmented storage and compute resources.

It is worth noting that VMware continues to expand and build upon VSAN. With the recent release of 6.2, features such as deduplication and compression have arrived. Additionally RAID 5/6 is now available as an option, allowing for a lower storage requirement  while still maintaining a robust Failure to Tolerate (FTT) level (i.e. at least 2). Admins must be use caution when sizing these however, and should be taking the entire cluster size into account along with their FTT level, and not simply using a 10% cache allocation rule.  

Challenges of Virtual SAN and SDS

VMware Virtual SAN, and SDS, is not without its challenges, and it is not a panacea for all situations. In particular SDS increases the scope of knowledge required of storage engineers. A storage engineer who must manage both traditional arrays as well as VSAN, require additional training and architectural guidance. Although many of the concepts, such as iOPS, compression, and flash cache management are similar, the SDS layer adds additional complexities and new tools that must be learned and maintained. One of the opportunities of SDS is the ability for a product to add new features without changing the underlying hardware layer. This can be a struggle for engineers to constantly keep up with a changing software layer on top of their physical storage layer.

Perhaps the largest issue with VSAN is its dependence on the VMware hypervisor to support it. This should not be an issue for organizations that already have a homogeneous environment that only consists of VMware deployments. Despite a desire on the part of many organizations to become 100% virtualized, the truth remains that many systems are simply not compatible with the virtualized server paradigm. Legacy systems, or systems that require interfacing with proprietary hardware, are often left on physical platforms due to the large cost of migration, or general incompatibility with a virtual environment. For these physical devices, the only option is still more traditional storage array systems. However, if these outlier systems are well managed, and limited in scope, a smaller dedicated set of storage arrays can be managed alongside a VSAN environment. Many organizations could still see a significant cost savings, and better scalability with a VSAN platform, and should be able to manage additional standalone storage arrays without too much concern. However, enterprises should go into any decision about VSAN with their eyes wide open as to how feasible a full 100% deployment really is.

Another related challenge with VSAN is that organizations are locked into the VMware ecosystem. There are many tools in the marketplace that will help organizations migrate VMware VMDK systems to other platforms, such as Microsoft Hyper-V. These tools do not address the storage arrays that are left behind with VSAN, and a migration to another platform could result in complicated and lengthy migration mitigations. If an organization chooses to go down the path of VMware’s hyper converged paradigm, they should be prepared to adopt it fully and completely. The VMware ecosystem works best when all of the components are managed in harmony with one another. Even the act of turning on VSAN is simply a switch inside of the vSphere console.

Finally, because the VMware Virtual SAN product brings together system management and storage management, it may tempt organizations to jettison their traditional storage administrators, with the hope that regular system admins can manage the storage layer as well. If an organization is smaller, with a limited technology footprint (500 or less virtual hosts), this paradigm may be plausible. However, for larger organizations the benefit of a dedicated storage team to manage and lifecycle virtual storage along with any physical storage arrays, is invaluable. Dedicated storage engineers can focus on the larger picture of the entire storage installation, and can make judgements about performance tuning and capacity management that might escape system admins who are more focused on individual system performance.

Recommendation

There are two architectural scenarios that should strongly consider VMware Virtual SAN for their storage arrays. The first of these are environments that are already fully embedded in the VMware ecosystem. If an organization is already virtualized and committed to VMware, then utilizing VSAN is a winning situation. Capital expenditures on costly storage arrays can be minimized while at the same time server environments enjoy the benefits of highly available storage. Commodity hardware can be utilized for ease of procurement, and horizontal scaling means a VMware environment can expand as a business grows.

The second big win is for small shops, or small teams with dedicated resources. VMware Virtual SAN brings enterprise level storage array features to environments that cannot afford (or do not require) large expensive arrays. Small clusters of servers, or hyper converged appliances can be deployed to small teams to provide them with fault tolerant storage than can be managed within a single pane of glass.

VMware Virtual SAN provides Software Defined Storage solutions for multiple enterprise paradigms, and gives organizations more options than investing in more large scale dedicated storage arrays. Full integration with the VMware vSphere environment gives the benefit of a single pane of glass for managing both storage and compute resources. This pays off with lower costs and investment in dedicated storage equipment, and ease of growth through a horizontally scalable platform.