HDS Geek Day 0.9 – We Should Totally Virtualize That!

More often than not, the first product that comes to mind when someone says Hitachi Data Systems is their flagship USP-V platform, also sold as the HP XP24000 and formerly as the Sun 9980V. Some folks may also think of the HDS USP-VM, which is like a mini-USP-V. But while people may know of the products, there’s a real knowledge gap in what they do and how they do it. It’s something that came up frequently during HDS Geek Day; that Hitachi has generally done a less than great job of communicating products and features.

The other point that came up frequently is that Hitachi Data Systems has historically been a technology company. This is part of why you’ll find me among the USP-V fans; technical superiority. Not because HDS flew me out to talk about it. Sorry, I was sold on the USP-V (well, technically, the USP-VM) as soon as I dragged a good description of it out of the VAR.If you’re not already familiar with the USP-V and USP-VM, allow me to provide you a quick overview of the USP-V. It’s one big, bad box comprised of a central complex that houses the controllers and ports and a handful of disks, and up to four expansion frames which connect additional disk. It also provides vendor neutral virtualization, thin provisioning, automatic load balancing, non-disruptive disk migration, data at rest encryption, non-disruptive data rebalancing, remote replication, and you know what?

There’s no way to give you a quick summary of the USP-V without leaving out key features. There are just too many. So first, I’ll link you to HDS’ Product Page for the USP-V. Then what I’m going to recommend is that you review the community created USP-V manual (PDF) maintained by one @nigelpoulton. The USP-V is despite a short and easy to remember name, an incredibly complex and fully featured storage system which is not easy to understand or explain in just one post. And strategy is also a separate post out of necessity.

As for the USP-VM? It’s a USP-V that can go in a standard rack, has less ports, less virtual storage machines, but maintains software feature parity. In other words, it’s a fractional USP-V squarely aimed at folks who have grown past midrange such as the AMS2000, but haven’t grown to the point of needing 224 fiber channel ports. Which does of course, include vendor neutral virtualization. (The entire crowd just went “holy crap, Phil doesn’t love SVC any more?!” I do. We’re getting to that. This is a long post.)

The best demonstration of what USP-V does, or can do, is to illustrate with pictures. Because they’re worth a thousand words give or take. This is a picture of what your typical heterogeneous large enterprise environment might look like:

Typical Large Enterprise Environment

Here, we have a mainframe, a legacy iSeries, POWER running AIX and Linux, HP Superdome running HP-UX and Windows, Sun E15k, and Windows, Linux and VMware systems in blades and racked servers. Storage is provided to the mainframe and iSeries by an IBM DS8k over FICON and ESCON, while open systems and x86/x64 are fed from a mix of HDS Lightning 9900V, EMC DMX, and IBM DS4k for fiber channel along with HP P4000 and IBM DS3k for iSCSI, EMC NAS, probably some NetApp, you get the idea. It’s a tangled mess of cables physically and logically, requiring a team of 5 or more full time storage administrators. These storage administrators have to work with HDP, IBM TPC, Navisphere, SymCLI, OnTAP, and the list goes on for over a dozen separate software packages that I didn’t draw because there’s just too many.

Thankfully, it’s time for tech refresh at this particular shop, and they’ve chosen to work with their HDS VAR on a USP-V install and a heavy focus on cost savings and efficiencies. So here’s what that same environment looks like when we introduce USP-V as a consolidating system:

Heterogeneous environment consolidated with USP-V

You may notice a subtle difference in this picture (besides the Big Black Press Embargo Box™.) Like suddenly, things got very very simple. This is partly design, and mostly reality. Everybody’s environment will vary, but as you can clearly see, this environment is now presented to hosts purely as HDS for storage – including FICON and ESCON. So yes, that old iSeries is actually ESCON connected.  They’re handling several hundred terabytes using thin provisioning, dynamic load balancing, and more than anything else, the USP-V’s virtualization feature. This allows them to put AMS2k’s behind their USP-V, managed by the same software with the same interface, up to the maximum single frame limit of 247PB. Not pictured is the competitor’s storage being reused as necessary behind the USP-V using it’s virtualization engine. Yep, they didn’t buy all new – that Symmetrix is lurking, so is that DS8100 they upgraded to a DS8700. And that’s not a typo – 247 petabytes maximum per frame. If you’re feeling a touch crazy, that means you could put about 128 EMC VMAX’s configured for maximum capacity with 1TB SATA disks behind a single USP-V.

Perhaps an even better illustration of what makes the USP-V a true stand-out product in storage is a feature comparison between competitors. Remember here that the USP-V competes in the large enterprise space. To qualify for my comparison chart, it had to be a standalone disk system with no additional attached products (such as NAS gateways) and could only use features and options available as of June 21, 2010 that directly installed or integrated into the base controller. (So separate NAS heads, right out!)

Feature Comparison Chart for USP-V

* – For HDS and EMC, the installed cache is a misleading number. Cache installed is halved by mirroring requirements on EMC and Cache Duplexing on HDS. This means that the USP-V has <=256GB effective, USP-VM has <=64GB effective, and VMAX has <=512GB effective. IBM uses a very different caching strategy and algorithms and does not perform 1:1 mirroring.
** – The IBM DS8700’s POWER6 based controllers can actually perform DIMM level isolation of cache faults. However, a memory controller failure will result in a controller outage.
+ – Prior releases and hardware supported “LPAR Mode” which performed CPU and cache partitioning and limiting functions on DS8100 and DS8300. However, this has been removed from the DS8700. It received a red mark because users upgrading from DS8100 or DS8300 to DS8700 need to be very aware of that fact.
— EMC gets a red mark on Distance Mirroring not because they don’t have superior technology – they do. They have too many options though, each with unique and very strict caveats and other complexities, making it difficult to decide and implement, and most require hardware above and beyond a copy of what you’re mirroring.

All that said, this isn’t your standard comparison chart; green means “clear advantage, stand out feature, or something nobody else has”, light green means “not bad at all!”, brown means “most of you won’t care about this”, and red means “there is a definite disadvantage or limitation you need to be aware of here.” Just because somebody got a red mark doesn’t mean it’s not a good product – it means that you need to take notice of that limitation.
There’s also a reason I included the USP-VM, which probably isn’t very clear. As I said above; it’s a slice of a USP-V, and it’s provided to illustrate the architectural advantages of the USP design in certain areas. (Also that USP-V and USP-VM are the only current products that have ESCON, for those of you stuck with legacy AS/400.) It also serves as an illustration as to why you’d choose USP-V over USP-VM; that being that the USP-V architecture does not lend itself well to scaling downward from a fault tolerance standpoint.

Scaling down a USP-V results in two key single points of failure; the USP-VM has only a single crossbar switch, and only one back-end controller – the 1 in the (3/1). This back-end controller is what handles your attached disk; I’d rather see 2 of those to match the USP-V’s 2-per-expansion layout. It also has a much lower port count, but that’s to be expected. Think of USP-VM as the “gateway drug,” because from a software standpoint, it provides everything the USP-V does. Just don’t try to compare the two on performance or price; they really are very different products targeted at different areas.

Staying on this track, let’s talk USP-V fault tolerance and why they got a lot of green. The USP-V is designed nothing like DS8700 or VMax – it’s designed like a mainframe on steroids. Four crossbar switches internally mean a two switch failure is only a performance impact. Up to 4 back-end controllers (out of 8) can fail without disruption. The most common non-disk problem – cache failure – is extremely isolated and can be resolved with no service interruption or performance impact. A single cache module failure will result in the loss of 16GB of effective cache, with no controller impact. Advanced software features in the USP-V allow any of the 14 front-end controllers to fail with a loss of all associated ports and non-disruptively fail over to another FE controller (dependent on Hitachi Dynamic Link Manager Advanced for some systems, which is separately licensed software.) In other words, it is extremely hard not only to kill a USP-V, but to even make it blink.
By comparison, the DS8700 has superior but still limited isolation; individual components within each controller can sometimes be isolated to permit repairs during scheduled maintenance windows – such as cache modules. Some components such as port cards can be replaced hot. But certain component failures will cause severe impact – for example, the loss of a GX2 I/O link can have a severe performance impact or cause partial loss of disk access. The VMax is pretty opaque, like most EMC products, but it isn’t hard to spot it’s weak points. The loss of a single CPU in an “engine” will almost certainly cause the engine to go offline; it’s a limitation of an x86-based architecture. Because distance and frame mirroring require specific dedicated ports, these are another weak point. Having two RapidIO switches is better than the USP-VM’s single crossbar here, but the loss of one is demonstrated to potentially cause catastrophic performance impact.

So it’s also blunt statement time: absolutely nobody, and I do mean nobody, can touch the USP-V’s single frame total capacity (that’s internal disk plus external virtualized storage). VMax requires special order and RPQ just to have 2.4PB of raw disk, with less than 1.6TB usable. Less than 2% of USP-VM’s usable capacity – which is only 40% of USP-V’s.  When it comes to capacity across every vendor out there, HDS is out playing baseball while everyone else is watching capacity stagnate^W^W paint dry. Seriously; even the IBM SVC taps out at 2.4PB – <1% of USP-V!

Changing gears, let’s talk about our simplified sample environment above and why large enterprises would choose HDS and the USP-V for their consolidation projects, or just their storage problems period. Listen to InfoSmack #55 and you’ll hear why from the man himself, Hu Yoshida – openness. HDS makes no bones about it – they are committed to an entirely open architecture from the hardware to the software. What this means is that USP-V can not only attach to FC-AL, FICON and ESCON, it means that it will talk to NetApp, EMC, IBM, Xiotech Emprise, Sun, FreeBSD operating in FC Target mode on some old PC, that ancient ESS Shark you have lying around, Joe’s – you know what? You get the idea.

Now expanding on that as Hu did, that also applies to software. I’ll cover software at length elsewhere, but management and monitoring of your USP-V generally should be done through Hitachi Device Manager and Hitachi Dynamic Provisioning (HDM and HDP). But if you’re an IBM shop with an investment in TotalStorage Productivity Center, the USP-V speaks SMI-S. It will work with TPC. Or you’re an OpenView shop; same thing. USP-V has an incredibly open software architecture, which gives it the unique ability to slot into any production environment you care to throw it at. This is something no other storage vendor does – sorry folks, lip service doesn’t count. Having to roll your own abstraction is right out, too.

And this is another key area people just don’t know about. HDS has realized and now knows and understands this, but it’s always been a communication issue. People gripe about the software, but the fact is that Hitachi produces some excellent software. Moreover, HDS produces a lot of software. Their goal is to be your one stop shop for storage, just like everyone else, but they also focus on having something to offer if they can’t be your sole provider.

Why am I mentioning software? Hitachi’s migration software is considered to be the gold standard, and is a big part of a typical USP-V deployment. Using Hitachi’s very open migration software, you can quickly and (more) easily migrate data from your old storage system onto the USP-V, bring your old storage system under the USP-V’s virtualization architecture, and use the same software to migrate the pages (USP-V operates with 42MB pages) back onto the old storage. Most of the software can be tightly integrated with HDM and HDP, so that instead of thick provisioning in step three above? You can leverage HDP to migrate the data onto a thin provisioned LUN.

And now we get into the USP-V’s very real weak points. (Bolded so folks realize I’m talking about the bad. Just in case.) First and foremost; USP-V is expensive. Very expensive. Hitachi targets large enterprise, and that means large enterprise price tags. Not just on the USP-V but the USP-VM. Where we run into problem two; only HDS can sell you the USP-VM, while HP can sell you the USP-V branded as the XP24k. Finding an HDS VAR can be a challenge, this going back to HDS’ issues with communicating features and products to end users. No matter how you slice it and no matter how you buy it, the USP-V is one big expensive box.

Second is that the USP-V is frankly, a storage system first and foremost and virtualization second. This doesn’t mean that the USP-V is bad at storage virtualization; it means that it’s a different virtualization. First local disk with wide striping, then external storage virtualization. It’s a few thousand lines of code on the storage engines, versus SVC’s tens of thousands if not hundreds of thousands. Which means far less chance for bugs to sneak in, and far less CPU spent on it. But that’s because the USP-V is a storage system and not a virtualization system. Seriously, if you were to ask anyone at Hitachi – which for the record I did! – they’ll agree with the following statement: “USP-V is a high end enterprise storage system that also can do virtualization of external storage.” That it virtualizes and wide-stripes internal disk is a little besides the point; the point is that you have the internal disk. Most other virtualization systems are appliances; not storage systems themselves.

If your primary concern or interest is in virtualizing external storage and not in an enterprise storage system that can also virtualize your external storage, USP-V is the wrong choice. You’re talking about a system which has proven that without external storage it can break numerous SPC records easily. The USP-V as illustrated, competes with DS8700 and VMAX; it does not compete with IBM SVC, Oracle Unified Storage, EMC Invista and so on. You get the idea.

The other key weakness of USP-V is a lack of familiarity. Most of my readers actually have never touched HDS products, much less HDM/HDP. There is a learning curve, it is steep, and in an economy like we have now, that can be a major impediment. There’s also a lack of familiarity up the chain through C-level executives, because Hitachi has been historically bad at getting the word out about their products. They’re a technology company, not a marketing company. People who know them, love them. People who don’t know them, still don’t know them. And part of what we gained and what HDS gained at Geek Day 0.9 was an understanding that this is a problem, and a promise that HDS is actively working on it. Even as I write this. (Yeah, I’m part of the solution! Shocking, innit?)

So, what’s the takeaway here?

What’s the grand summary and reason for the Big Black Press Embargo Box™? Well, honestly – you have to come back tomorrow. It sounds like a cheesy way to get more hits, I know, but hits cost me money in essence – I don’t get paid. You have to come back tomorrow, because I can’t give you the total picture until then. But I can give you part of it, and a reason why you need to come back tomorrow:

If you are a large enterprise customer, why are you not looking at USP-V yet? And don’t say you’re looking at XP24k – it is incredibly important to understand that while the USP-V and XP24k are architecturally identical, they are not sold by the same people, they are not sold at the same prices, and they do not have the same support structures. Both of them are excellent ways to get your hands on the best HDS has to offer, which makes it even more important for you to evaluate them as individual products as part of a total offering – service, support, software. HP also doesn’t sell the AMS2000 family, something to consider from a management software standpoint.

If virtualization, tiering, capacity, scale-out, mid-range, consolidation or multiprotocol matters to you: you need to come back tomorrow. It’s more than worth it.

Hitachi has also been undergoing what I’d describe as a transformation. Traditionally, they’ve been able to offer advanced services like implementation, special design, and so on only to their Japan customers, because that’s where the bulk of the work is done. That’s changing. What HDS could offer you 12 months ago is not what HDS can offer you today. What HDS could offer you 6 months ago is not what they can offer you today.

If you are not a large enterprise customer or someone with a DS8700 or VMax sized budget, then investigating USP-V is purely an educational exercise and somewhat of a time waste. However, you should be looking at USP-VM as part of your overall evaluations if you’re in the niche I call mid-range – that being everything between IBM DS3k / HP P4000 / etc. and DS8k / VMax / etc. You also need to come back tomorrow for the mid-range goodies. I promise they are worth it.

If your primary interest is in just virtualizing storage you already have? Give USP-V a pass for now, unless you intend to couple it with consolidation and/or a storage refresh. USP-V is going to cost you far too much and sure, you gain a tremendous amount from USP-V, but there are cheaper and better options for just virtualization. You definitely should give USP-VM a look, but consider the other storage virtualization options you’re looking at. If IBM SVC or NetApp V-Series or HP SVSP topped the list, or if you’re looking to virtualize storage like IBM DS4k, IBM DS5k, or AMS2000 chances are that the USP is actually the wrong product for you. It’s also way more important for you to come back tomorrow. Mid-range goodies await you which apply to IBM SVC, NetApp V-Series, and so on.

Addendum! All the neat stuff; Zero-Page Reclaim, Page Structures, LDEVs, Pools, thin provisioning, etcetera. You know. The technical stuff. That will be covered in additional blog posts later. Promise. (This one came in at over 3000 words. Trust me. You want them in separate posts.)

  1. No Comments