Coraid Blog

Coraid + VMware Storage Virtualization Make the Grade in Ohio

We have a number of joint customers with VMware and the City of North Canton Schools in Ohio is a great example of that (link words "great example" to full case study NCantonCaseStudy.pdf). The school district, like most local governments this year, is facing some budget challenges. AoE-based EtherDrive storage provided a for fast and affordable storage.

SearchStorage's Todd Erickson has the story here (LINK TO Full Story). He talks to storage administrator John Fano about the Ohio school's move to AoE-based EtherDrive storage from Coraid. Fano goes through experiences setting up the system to details on how they use EtherDrive storage system and VMware for the district's file services, and how data growth already has the district expanding the storage in a simple manner.

What is Canton using?

"... two Coraid Inc. EtherDrive SR1521 storage appliances, one EtherDrive VS21 virtual storage appliance, and five EtherDrive SAN HBA 2 Gbps PCI-E cards. Each EtherDrive SR1521 shelf has 15 7,200 rpm 250 GB SATA disks chunked into RAID 5 arrays...."

Read the entire article here:  Full Story.

New 2 TB Drives

It appears the hard drive manufacturers may be jumping directly to the 2TB enterprise class drives, thereby skipping the 1.5TB drives altogether.

Hard Drive manufacturers have a tendency to release a hard drive's specifications and market them shortly before the drive is actually available for purchase from distributors or resellers. For those customers who need the extra drive capacity urgently, patience is advised. But why should a customer wait to purchase when they so desperately need the capacity?

Initially manufacturers have a limited supply for a new model of hard drive. The reasons for this are varied. From CORAID's experience it can take from two to three months before drive quantities are available for SAN type purchases. New hard drives in the market are more likely to have firmware issues.  Therefore, the risk of getting a brand new hard drive with the first version of firmware is considered high risk from a SAN perspective. However, once problems are identified, the hard drive manufacturers are normally very quick to make fixes and provide firmware upgrades to customers.

At CORAID we want to ensure all of our EtherDrive products using any of our certified drives meet our strict requirements of SAN performance & reliability.  Our supplier(s) do their own tests on new hard drives. Afterwards CORAID receives several hard drive samples where the drives go through rigorous testing.  Next, attention to the hard drive firmware is scrutinized in the CORAID test lab and we constantly scan the web for any reports of firmware issues.  Once CORAID determines the hard drive firmware is stable and the drive is able to pass our testing, we then add the hard drive to the disk compatibility table.

Ultimately, this means a customer needs to wait because firmware needs to be stable, drives need to be tested and certified, and quantities must be available from the manufacturer. As always, please ONLY use enterprise class hard drives for any SAN, not desktop class. If you would like to be notified when new 2TB (or other size drives) hard drives are added to the CORAID Disk Compatibility Table, please contact sales@coraid.com.

Why is ATA-over-Ethernet faster?

AoE has some real advantages over the other storage area network protocols in regards to scalability and performance.  The biggest advantage with AoE lies in the fact that it is datagram based and not connection based.  Fibre Channel and iSCSI are both based on a connection from the storage initiator and the target.  This made sense when network technology was limited and a single 100 million bit per second network was all anyone could afford.  At Fibre Channel prices you would only buy one host bus adapter for your server.  In those days a single connection from the server through the single adapter through a single path in the switch fabric to a single port on the target was as fast as you could go.

But now times have changed.  Ethernet technology has surpassed Fibre Channel in speeds, it now has 10 billion bits per second, and has made one billion bits per second an inexpensive commodity.  Current price for a 1Gb switch is about $15 per port compared to a 4Gb Fibre Channel switch from about $800 per port.  Host interfaces to Fibre Channel are about $1,600, with the cheapest off-brand still being as high as $1,000.  For VMware ESX servers, a CORAID EtherDrive Host Bus Adapter costs only $99 and include two 1Gb ports.

This all makes using multiple links to the storage attractive.  By putting more Ethernet ports on a server, which can easily be done with dual and quad port EtherDrive HBAs, storage traffic can use  all the available links.  This can't be done with storage protocols that are based on a connection between the server and the storage.  Fibre Channel is a five layer protocol based on named ports in the storage and the switching fabric.  A complicated scheme called multi-pathing is used to allow standby routes to the storage in the event of a failed link.  The iSCSI protocol rides on top of TCP/IP, which provides a connection between two IP host ports.  It also has a single path through the network to the host, meaning that it is limited to the speed of a single port on the initiating machine.

AoE, however, is a datagram based protocol.  That means that each request and response packet that is not part of a connection and can use any port available to access the storage.  If an initiating machine has four one billion bits per second ports requests are sent out each simultaneously making a couple of $99 HBA run as fast as a $1000+ Fibre Channel card.  Fibre Channel and iSCSI can't do that, no matter how many HBAs you put in the initiating system.

For information on CORAID EtherDrive SAN visit www.coraid.com

Affordability and Performance - The Ultimate Challenge of SAN Technology

Once the Gigabit Ethernet standard was announced in the late 1990's, it became clear to the CORAID co-founder, who is also the inventor of the Cisco PIX firewall, a new alternative may exist for network storage without the high costs and limiting factors of Fibre Channel and iSCSI.  A goal of making high performance network storage that is also affordable became a reality with the open network protocol designated AoE, short for ATA-over-Ethernet. AoE was pioneered by CORAID and is supported by Linux and is available as part of all major Linux distributions. CORAID's AoE is a competing technology to Fibre Channel and iSCSI. AoE is freely available to anyone. CORAID EtherDrive storage appliances, all based on AoE, are available at a fraction of the cost of comparable storage products and solutions, thus enabling VMware ESX customers a needed option for network storage which is high performance and very affordable.

When the decision was made to use Ethernet to enable network storage, it quickly became apparent that the true speeds of Ethernet had to be fully realized and could not be by using TCP/IP. In networking there are several layers to consider when designing a network protocol, starting with the physical layer, the fastest, all the way to the software driver or application layer.  These layers too often depend on generic handling of networking details at lower layers causing too much overhead and processing, thus resulting in much slower speeds and performance as a result. Unfortunately TCP/IP fits this slow case when trying to enjoy the speeds available with Gigabit Ethernet. Therefore the design goal became clear; develop the network protocol as close to the hardware layer as possible, yet have it be manageable enough so as not to be tied to one piece of Ethernet hardware.  Ethernet datagrams fit this design goal and became the cornerstone of the yet unnamed protocol. As it happens, using only datagrams for a protocol, translates into enormous scaling and performance benefits when the protocol is developed in a pure, clean manner.

Keeping in mind the design goal of maximum Ethernet speed, it became imperative to have the proper, efficient message structure and commands to make Ethernet based network storage a reality. Embedding ATA commands into the Ethernet datagrams fit this design goal exceedingly well. The ATA commands completed the protocol, aptly named "ATA-over-Ethernet", or AoE. It is important to note that because AoE is a true protocol, it does not matter what type of hard drive is used when implementing the protocol and corresponding driver. CORAID develops its EtherDrive storage appliances for the most affordable and reliable hard drives on the market to benefit users controlling costs while providing ease of storage growth and management.

For information on CORAID EtherDrive SAN visit www.coraid.com

Virtualization - The Promise (Part II)

Access to network storage, the myriad of network connections and paths used to get to the data, grows in a non-linear manner with high demand and full utilization of the network. Hence, network storage is really a networking challenge, not a storage challenge. Fibre Channel and iSCSI were designed by storage experts.  ATA-over-Ethernet (AoE) was designed by network protocol experts.
In the early days of Ethernet, network speeds were sufficient for basic communication between computers, but certainly not for network storage. The original Ethernet standard(10Mbit/s) evolved into Fast Ethernet(100Mbit/s) and next into Gigabit Ethernet(1Gbit/s), with an even more recent 10 Gigabit Ethernet(10Gbit/s). Once Gigabit Ethernet arrived in the market it had widespread adoption, therefore produced in mass quantities making the NIC's very affordable for everyone as commodity products. Finally Ethernet had the speed at 1Gbit/s that could theoretically match the media transfer rate of hard drives at 1Gbit/s or higher. The challenge now became how to fully realize the performance speeds of Ethernet and to make the market aware of how today's Ethernet can readily handle the demanding networking load required by network storage.

The inventor of the Cisco PIX firewall and CORAID co-founder wrestled with the challenge to fully realize the performance capabilities of Ethernet for network storage by designing an open network protocol called ATA-over-Ethernet (AoE). The significance of AoE is its very lightweight footprint, with a specification of only 11 pages.  AoE is the enabling framework that allows EtherDrive SAN to be high performance and affordable. Because AoE only relies on Ethernet, all of the components to create and manage an Ethernet based network are available as very affordable, commodity networking hardware. Taking this approach has allowed CORAID to create high performance storage appliances by assembling commodity technologies, thus keeping cost per GB of storage at $0.50, a very small fraction of the cost of other storage hardware in the market. VMware ESX users now have the much needed and unheard of combination of "High Performance and Affordability" for their network storage.

The robust and flexible nature of AoE has allowed CORAID to develop EtherDrive SAN storage with an almost unlimited amount of expansion. Each CORAID EtherDrive storage appliance is deemed a "shelf". AoE allows addressing, or access, to 65k of these AoE shelves. Each shelf in turn can host 255 targets. Each target supports 48-bit logical block addressing, or 144 Petabytes. Currently, available drive capacity and hardware limitations only allow for 10's of Terabytes per shelf. With over 16.7 Million (65k Shelves * 255 targets per shelf) targets available on a single network, each with a theoretical capacity of up to 144 Petabytes, it becomes clear why CORAID's almost unlimited expansion capabilities become critical to storage growth.

Just as important, if not more important than storage growth and capacity, is access to network storage. Without scalable network access, network storage quickly becomes a bottleneck for the entire system. For VMware ESX, CORAID helps solve this dilemma by simply adding one or more 2-gigabit (dual 1-Gig) EtherDrive HBA to get more bandwidth and leverage available network capacity. Fibre Channel and iSCSI cannot scale in this manner and are therefore doomed to become network storage bottlenecks because they are not robust enough to scale to existing network capacity cost effectively.  By today’s standards, the ultimate performance is achieved by using one or more 20-gigabit (dual 10-Gig) EtherDrive HBA. This enables VMware ESX to fully leverage high performance network storage at maximum network capacity with the benefit of the most affordable price.

For information on CORAID EtherDrive SAN visit www.coraid.com

Virtualization - The Promise (Part I)

CORAID is committed to providing very high performance at the most affordable price. The EtherDrive Host Bus Adapter is a doorway for many users to enjoy the benefits of a storage area network affordably for the first time in conjunction with VMware's ESX virtual environment capabilities. The EtherDrive HBA allows ESX users to use EtherDrive SAN storage and is very inexpensive. Network storage can now be virtually assigned and managed for maximum utilization with the combined power of VMware ESX and CORAID's EtherDrive SAN.

Disk capacity used to be very limited, therefore allowing all disks to remain part of the server. Once hard drive capacity and speed increased there was no longer room for hard drives on a single server. Additionally, more applications and users began to require more and more access to the same data, so the data had to become available in a centralized, more easily managed method. Speed and the ability to scale capacity became more and more important as the need and availability of centralized data became mission critical.

At the time, the available networking technologies, including Ethernet, were not capable of the high throughput required to make network storage a reality. The problem was viewed as more of a storage problem, however, and not a network dilemma. Fibre Channel, originally designed for simplifying connections and increasing distances, was co-opted as the solution to provide the speed and reliability required to make network storage a reality. Fibre Channel evolved into a rebuild of all network layers from the ground up. This makes Fibre Channel hardware more complex and proprietary. This combined with the fact only relatively small volumes of Fibre Channel hardware are sold, makes Fibre Channel very expensive to own and manage with no hope of it becoming affordable at commodity prices.
Another unfortunate drawback of Fibre Channel (iSCSI too) is that it is connection based, meaning two well defined end points must be configured and managed tightly. Although connection based protocols are secure, reliable and can have decent speed, they do not really allow for leveraging the existing network infrastructure.  With ATA-over-Ethernet (AoE), increasing the overall speed and throughput is accomplished simply by adding another host bus adapter to get more overall speed and throughput. Without increased (scaled) network capacity it is impossible to grow storage capacity seamlessly and without exorbitant costs. Another very real issue is how different applications, users and systems need access to network storage in demanding and often times unseen ways. Storage needs grow in a linear fashion with capacity planning being straight forward. Access to network storage, the myriad of network connections and paths used to get to the data, grows in a non-linear manner with high demand and full utilization of the network. Hence, network storage is really a networking challenge, not a storage challenge.

For information on CORAID EtherDrive SAN visit www.coraid.com

Virtualization - Why bother?

Virtualization, a once almost unheard of method for managing IT resources is quickly becoming a standard IT practice using proven technologies such as VMware's ESX product family. The ability to fully utilize hardware and consolidate server, network and especially storage resources, has lead to unmatched efficiencies of IT resource management and cost savings. Taking unused CPU cycles and RAM combined with better management of fast growing disk space needs and network connectivity in an abstract, virtual manner enables efficient economies of scale. This in turn creates opportunities for easier management and allocation of infrastructure, backup, redeployment and re-location of IT rollouts. Once configured, the virtualization benefits are simple, yet impacting. One can combine and run OS's and applications on the same machine or across a set of machines fluidly where Virtual Machine (VM) configuration and inspection is almost trivial. Production up-time, backups, disaster recovery and hardware failures are more easily handled with no or little impact to production systems when deployed correctly. Decoupling hardware and resources leads to easier aggregation and management of these same resources for more efficient and impacting utilization.

To experience the full benefits of virtualization, an IT environment is required to use network storage. Moving storage out from the servers and onto the network is critical for sharing, managing and growing storage resources.  Until now, network storage was either slow or expensive with performance and scalability as key concerns. Now there is a very high performance option that is very fast, affordable and scalable giving virtualization environments a much needed network storage advantage. Enter the CORAID EtherDrive SAN Host Bus Adapter. The EtherDrive HBA is designed specifically for VMware ESX 3.5 environments allowing users to enjoy the benefits of CORAID EtherDrive family of ATA-over-Ethernet (AoE) based storage area network products. For more information, visit www.coraid.com

VMWorld 2008 Virtual Exhibitor website

We had a great showing at VMWorld 2008 in Las Vegas.  Thanks for visiting our booth.  Our Virtual Exhibitor website will be displayed throughout the year and has additional information of our VMware ESX HBA and compatibility plans.  http://www.vmworld.com/vmworld/community/exhibitors/coraid/

App note for AoE WAN Tunnel now available

In one of our previous blog posts, we demonstrated a proof-of-concept of an AoE WAN tunnel.  We have now produced an application note delineating the process used in the test.  Here is the link, which will also be listed on Coraid's recourses page.

http://www.coraid.com/site/co-pdfs/AoE%20WAN%20with%20GRE%20and%20IPSEC.pdf

AoE WAN Tunnel

On July 23rd, Coraid successfully demonstrated that AoE LUNs can be accessed over an IP network, using the public Internet and over leased-line IP connections. To do this, AoE frames were passed through a GRE tunnel created using standard router equipment.

The Proof of Concept (POC) connection was made from the Coraid offices in Athens, GA to a customer site in Fort Worth, TX. Although the T1 line used in the POC was too small for bulk data transfer, it was more than adequate to support bi-directional remote management of SR disk shelves via a CEC connection.

Coraid Engineering is working on a new release of the storage virtualization software for the VS-series appliances that will allow asynchronous LUN replication. With the news that AoE can be transported over WAN links, we are hopeful that given sufficient bandwidth, replication will be possible to remote site storage shelves.

Potential Application examples -

1) A large database needs to be replicated to a remote site. Users would initially create a baseline synchronization of the database to a local EtherDrive RAID LUN, then remove the disks from the storage appliance and ship them to the remote backup site and install them in a similar EtherDrive storage appliance. This first step is easy because Coraid EtherDrive appliances store all the RAID configuration on the RAIDed disks and the remote storage chassis learns the configuration from the disks when they are installed at the remote location.

Once the remote AoE device is online, a GRE tunnel can be used between the primary and backup site to allow the primary and backup LUNs to be kept synchronized by directly using AoE without using software like rsync or drbd.

This will be possible by using the VS-21 EtherDrive VirtualStorage appliance. A feature called LUN replication will asynchronously synchronize with the remote site. As long as the rate of change in the database does not create an excessive time lag caused by the bandwidth of the AoE tunnel, we believe useful remote asynchronous LUN replication is possible.

2) The AoE tunnel may also be useful for WAN access to the console port on a remote EtherDrive storage appliance using the Coraid Ethernet Console (CEC) program.

We will be posting setup details on the web site Resources Tab, App Notes page. Your questions and comments are welcome.

Coraid Engineering Staff