Dismantling More ‘badBIOS’ Hyperbole and Explaining How TAO Works

So, I just watched Jacob Applebaum’s presentation at CCC (I’m catching up) and frankly, I haven’t seen a more shameful display of zealotry and laziness in quite some time. That’s not security expertise – that’s mostly pitching policy using iffy examples, which just undermines the political arguments. Open source does not magically make things more secure – never has, never will. Just because you can ‘inspect’ code doesn’t magically fix other problems or prevent that code from being full of holes.

If you want to argue this with me, Rule #1 is ‘bring technical facts and knowledge.’ If you can’t do extensive modifications of a PC BIOS without breaking it and/or write a device driver, go get more knowledge. There’s a whole Internet out there to learn from. And unlike so many of the supposed ‘security experts’ I actually bothered to take the time to learn how the PC BIOS works back in the 90’s, instead of pulling conspiracy theories out of my butt that can and have been completely disproved by basic forensic analysis.

Continue reading ‘Dismantling More ‘badBIOS’ Hyperbole and Explaining How TAO Works’

The PC BIOS is Insecure As Hell. WHY?

There’s a whole cadre of people out there who are utterly convinced that modifying the PC BIOS in ways both good and evil is somehow new. It is not at all possible to be further from the truth – modifying the BIOS dates back to the original IBM PC. There were official IBM kits to upgrade the BIOS by replacing UEPROM (UV Erasable Programmable ROM) or PROM (Programmable ROM), and unofficial kits that you soldered into the system to alter behavior.

In other words, BIOS modifying has been around a very, very long time. So has BIOS upgrading. But in the old days, it required actual hardware access to either pull chips or expose the erasure window on the UEPROM. And it was fraught with risks – if you broke the window, the UEPROM was ruined. If you didn’t get the sticker back on right, you’d suddenly find your system broken as the BIOS was erased. So people started looking for ways to solve these problems. And as technology advanced, it became a textbook case in reducing security.

Continue reading ‘The PC BIOS is Insecure As Hell. WHY?’

The badBIOS Analysis Is Wrong.

NOTE2: Holy crap lots of comments and I find out that the comment setup isn’t threading them properly so it’s a BEAR to try and make sense of. I am truly and terribly sorry about that. Talk about counter-productive to a conversation! Trying to see if there’s some way to fix this.

NOTE: Approving comments as I get time. If I didn’t do it yet, I just haven’t gotten to it. If I don’t approve yours, it’s either because A) you made it clear you didn’t read or have no clue what you’re talking about, and therefore detract from the conversation or B) you’re trolling / blindly defending / blindly attacking to derail the conversation. In either case, rebutting basic ignorance of technology, complete lack of reading comprehension or someone just being an ass is a waste of everyone’s time. Including your own.

Look, I’m not known for pulling punches and I’m not about to start now. The fact is that everything I have read about #badBIOS is completely and utterly wrong; from the supposed “escaping air gap” to well.. everything. And I should know. I’ve dealt with malicious BIOS and firmware loads in the past. I’ve also dealt with BIOS development and modification for two decades. It’s a very important skill to have when you regularly build systems that are well outside manufacturer ‘recommended’ areas.

The whole of the analysis would be laughable if people weren’t actually taking it seriously and believing it because they’ve seen edge cases or very specific examples. And the result is that they’re looking in the wrong place.

Continue reading ‘The badBIOS Analysis Is Wrong.’

My Thoughts on the Whiptail Buy

As you’ve certainly heard by now, Cisco is buying Whiptail, who makes storage. This comes right on the heels of VMware essentially parking NSX on John Chambers front lawn, which comes after Cisco EOL’d the Nexus 1000v, and you get the idea. There’s been a lot of very transparent product-sniping going on with no signs of stopping.

This Is Not Good For Customers

Customers are not stupid. Customers are tired of the vendors parroting “increased competition! Good for customers!” No, it is not always good for customers. In the real world, most shops buy their IT equipment on a 36 to 48 month cycle. That applies to hardware and software – they expect their software to be supported for the duration, and they buy support contracts accordingly. 36 months at a time.

The Nexus 1000v is a powerful example of why this can be bad for customers. Cisco introduced the 1010 appliance in 2011 – and cancelled any further development on September 14, 2012. Just a smidge over 12 months later. That Cisco will continue to support it is absolutely irrelevant – customers expect and deserve software updates. Instead, they bought 24+ months of best effort support. This is for a product that was promoted as a cornerstone of VCE.

I’ve heard several arguments that “oh, we didn’t cancel/discontinue” and “oh but it’s in the 1100V which is better.” First of all, that’s not what Cisco says. (And again: that’s all I can go by from a customer perspective.) Secondly, the 1100 Cloud Services Platform requires the purchase of additional hardware and new licenses. It’s a hardware appliance.
@TheJasonNash: @rootwyrm @jdooley_clt @theronconrey But that’s just it.  It’s not canned.  It still runs current VSM just as it did before.
@TheJasonNash: @rootwyrm @jdooley_clt @theronconrey The difference is that the 1100 has newer CPUs therefore more VSBs.  That’s it.
Will that continue to be the case? I don’t know and nobody can say. Again: it’s a question customers have to ask. What I see as a prospective customer is that a product was introduced and very quickly cancelled.

So how does Whiptail fit into this?

Could The Distrust Be More Obvious?

Cisco is buying a flash storage company – that’s all Whiptail does is flash. Right on the heels of EMC announcing new VNX family products which are available as all flash. Again: customers are not stupid. They can read between the lines. Does it have to be true? No – only possible and reasonable to believe. What customers are hearing here? Is that either Cisco wants to cut out EMC or doesn’t believe EMC has the chops.

And given everything else that’s going on, it’s a reasonable conclusion – either one of them. Especially with Cisco loudly and publicly denouncing NSX in company blogs and through unofficial channels. (Disclaimer: I have no opinion on NSX at this stage. Talk to me after I’ve used it, as with all products.) Then to less than a week later turn around and buy Whiptail? Everybody knows VMware and EMC are joined at the hip.

And this goes back to being bad for customers. Because it is now very reasonable for them to doubt the capabilities of EMC’s offerings – after all, Cisco just bought a storage company! Why would they do that unless there was something wrong with the E in VCE? Whether or not there actually is, the question has been raised – by the vendor. The customer now has to do their due diligence – adding to the workload and time – to figure out what it is or isn’t true.

Disclaimer: I have no information on VNX-F that you don’t. Does it have the chops? I don’t know. Do I doubt VNX-F’s capabilities? Absent the Whiptail buy I had no reason to. The point is that there are now reasons to doubt, introduced by Cisco.

So What Does This Mean For VCE?

I’m not going to pretend I have special knowledge about what’s going on, because I don’t, and because that’s not the point of this post. The point of this post is what it looks like from the customer perspective. What customers are seeing is suddenly Cisco is bashing VMware and buying a company that competes directly with their partner EMC.

Not only is it a company that directly competes with their partner, it’s a company that says the way their partner does storage is fundamentally wrong. Now if you’re not seeing that as a shot across the bow or potential breakup of the partnership, you’re not doing your proper due diligence. Especially when you consider that less than a week ago EMC announced the availability of the VNX-F in Milan. (At a very lavish event, no less.)

Customers are and should be very concerned by this move on Cisco’s part – because like it or not, it does directly affect them if they’re considering Vblock. Do I think this is going to tear apart VCE or that it’s doomed or such? Hardly. I don’t see any evidence or reason why it would. I can only prove that there is a possibility it will have a negative impact on sales. Not that it is having, has had, or will have. Just that it might.

What’s The Visible Risk?

Like it or not, there’s also a very visible risk to customers with regards to the vendor side of things. As it sits, when you buy a Vblock, you’re getting a packaged deal which involves three companies – VMware, Cisco and EMC = VCE. But as it is, Cisco already controls 66% of the hardware, as they dictate the server (UCS) and network element (Nexus/MDS).

With the purchase of Whiptail, Cisco now has the ability to exert 100% control over the hardware. Cisco servers, Cisco network, Cisco storage. (It also enables Cisco to completely dump MDS as they’ve been trying to for a while now.) Customers like the ‘single throat to choke’ approach, till they find out it often means paying a lot closer to list price on everything.

And why wouldn’t Cisco want 100% control of the hardware? Why would they not want to offer a solution which is comprised entirely of their hardware, so they could sell more of it? There’s no reason for them to not do that. They don’t compete in the hypervisor space, so they could just as easily turn VCE into VCC, Hyper-V+CC, Citrix+CC, Xen+CC and still be perfectly content. The only software they’re selling here is the software you have to buy (UCS management elements, Nexus licenses, etcetera) and that’s forced purchase with no alternatives.

People like to recite the mantra that “competition is good for customers.” And often it’s true. But this is one of those cases where customers should be extremely wary, because not all competition is actually competition. This is one of those cases – it may well be a vendor attempting to cut off competition by controlling as much of the stack as they can. (You may remember this from the IBM S/390, AS/400 and zSeries.)

What’s to stop Cisco from saying in 12 months that they’ll no longer support EMC attached to UCS/Nexus or won’t support VNX-F with UCS? Not a thing, really. Will they? I don’t know. But they might.

The Conclusion, For Customers.

There is no set in stone conclusion here, no absolute facts, no “ah-HA!” moment. There is, but it’s entirely “ah-HA! I should ask about THAT!” and the facts are all ‘with X it is possible to do Y’ rather than proof that Y is being done.  This very public series of tiffs is damaging to trust and confidence in vendors, which makes answering those questions even harder.

Is Cisco going to try and kick out EMC? I don’t know. Will they offer a full-stack that competes? Maybe. Will we see a “Vblock” running Hyper-V instead? Could happen. But these are only possibilities, not facts. The only absolute facts we know are as follows:

The real point is that repeated public spats like this, especially coupled to clearly strategic buys, does and should introduce serious doubt on the part of customers. There are now a lot of very hard questions to ask, in large part because the VCE alliance is clearly less in-step with each other than thought.

Cisco could be planning Whiptail as an adjunct to VNX-F or for an unrelated VDI or Unified Messaging bid for all we know. But now customers have to extend their investigation and research processes asking these questions, and wondering if the answers will be the same 12 months down the road. Confidence and trust have been eroded substantially now – and all the members of VCE are going to have a hard road to earn it back.

Since it seems unclear to the Internet…

One, I hate writer’s block. So this post may be slightly confusing. Two, I could, you know, blah blah “140”  but you know what? That’s a cop out sometimes, and this is one of those times. And the Internet at large seems to be rather… confused and/or not understanding my stance on discrimination. Not “discrimination” in the sense of “oh well X isn’t actually qualified for Y”, but discrimination in the sense of “X can’t have Y because Z” where Z is pretty much everything one Paul “I’m Not A Bigot” Graham uses an excuse.

So let me be absolutely crystal clear about my stance:


Please feel free to place this same message on your blog, website, whatever. Just as long as it’s not your useless lying “equal opportunity” page. And stop claiming you aren’t lying about it, Silly-con Valley “bros.” You’re so full of it, there are farmers in Oklahoma who want to spread you on their fields. And stop pretending that Paul Graham isn’t a racist and bigot and doesn’t actively encourage discrimination, bias and bigotry.

No. Seriously. Stop. Because I have honestly reached the point where I’m going to go for the Educational Cinder Blocks and a trebuchet the next time I hear it.

Enough of the outright lying where privileged idle rich twits like PG conflate “qualification” with “discrimination.” It’s a complete and utter lie, the same as the claims that it’s a “meritocracy.” If it was an actual meritocracy, there wouldn’t be hundreds of failed startups. If it was an actual meritocracy, more than <1% – yes, less than 1% – of founders would be African American. Want to know how many non-white kids Paul Graham has given money to? Two. And HN tries to justify it because they “live in poverty” and “few tech founders come out of poverty”. And if that entire thread doesn’t make you sick, seriously, what the hell is wrong with you?

Continue reading ‘Since it seems unclear to the Internet…’

Availability in the Modern World, Part 1

So lately we’ve been talking about two of my favorite words, stability and resilience. And mostly how my stance is that you can’t have one without the other – because that’s been my experience, without exception. When you take away stability, the resilience goes away because multiple components fail in parallel. When you take away resilience, a single fault takes the whole thing out.

There’s an excellent discussion going on with myself, @jamesurquhart over here, some good stuff from @mthiele10 over here, and so on.

But the whole point of stability and resilience is availability. Because the fact is that availability is paramount to all, period, no exceptions. It doesn’t matter if you’re public facing like Netflix or it’s an internal application. If people can’t use it, it’s a doing nothing more than burning cash.

The idea that cloud somehow changes the equation is, in fact, completely false. Cloud doesn’t change these basic concepts – it changes how you achieve them. That’s all. Doesn’t matter if your solution is hosted or in house; if it’s down, it’s worthless. If it’s down regularly, it’s worthless. But to really understand things, first we need to understand availability and start debunking a lot of the traditional “enterprise” cruft that’s been wrong for as long as I’ve been in IT. (That’s a long time.)

Continue reading ‘Availability in the Modern World, Part 1′

Stability + Resilience, not Stability|Resilience

So @AndiMann @jamesurquhart @f3ew and I have been having a bit of a discussion on Twitter regarding resilience, change control, deployment cycles and such and I really can’t fit my thoughts on the matter into 140 characters. It is a bit of a complicated topic, but not as complicated as folks keep saying in various outlets.

First of all, the most important thing to recognize is that almost any company pushing it as an OR rather than an AND is trying to sell you hardware, software or services. Period. That’s just a truism and it’s unavoidable and something they don’t like me calling them out on. Especially when I point out that they’re trying to sell you these things whether or not you actually need them. But come on – we all already know they’re going to do that.

But that’s not what we’re here to look at as much as the AND versus OR argument. There’s a lot of folks who have gone completely overboard with this idea that if you don’t do continuous deployment, you’re doing it wrong. And the simple fact of the matter is that they’re wrong. IT is not a zero sum game, nor is it strictly OR operations. Most organizations don’t want or need continuous deployment. And many organizations (e.g. Google who likes to break their infrastructure at the expense of paying customers and products) are doing it completely wrong. Continue reading ‘Stability + Resilience, not Stability|Resilience’

Lies and the Larrys Who Tell Them

Another Oracle Open World keynote, another pack of lies from Larry Ellison. Bad enough that the still blind (dangit, why does my prescription have to suck so much?!) guy is actually writing a blog post about it. Before people start spreading more lies as truth. Mind, I’m not there, so this is an incomplete dismantling of the falsehoods. But there’s some real ugly ones.

And yeah, being a Sun guy for as long as I have been, well. I feel obligated to share the real facts of things, so folks aren’t reliant on spin, revisionism and unicorns.

Continue reading ‘Lies and the Larrys Who Tell Them’

Neil Armstrong – a true hero

I really still don’t know how to put into words the sadness I felt when I heard of Neil Armstrong’s passing. I can write 5000+ words about a storage system, over 3000 about a single game, but I don’t think I’ll ever be able to put into words what I felt and still feel.

People these days misuse and abuse the phrase ‘my hero.’ You know what a hero is? A “person who, in the opinion of others, has heroic qualities or has performed a heroic act and is regarded as a model or ideal”. Neil Armstrong has been my hero for pretty much my entire life. Often misquoted, Neil’s first words when he stepped on the moon were and have always been: “That’s one small step for “a” man, one giant leap for mankind.” Frankly, the grammar arguments are a pointless and stupid distraction. That’s what he said, and that’s what he meant.

He was just a man. The first man to set foot on another planet, but still just a man. He never pretended to be anything else. He never put on airs, pretended to be someone or something he wasn’t, or went seeking fame and glory. He didn’t go to the moon because he wanted to be famous. He did it because he earned a BS in aeronautical engineering from Purdue under the Holloway Plan. He went to the moon because it was the greatest engineering and technical challenge we had ever undertaken. He went to the moon not because of fame, not because of politics, but because he was the right man for the job.

Anyone who says Neil Armstrong was not the right man for the job, doesn’t know Neil Armstrong. He could have swapped Buzz Aldrin for fame-hound Lovell; he declined, saying that he felt Jim Lovell deserved his own command. He could have gone back to the moon. He could have had anything he wanted. Instead, he stepped aside so that others could shine.

Dozens, if not hundreds of companies sought Neil Armstrong as a spokesman – most failed. Chrysler succeeded because he believed they had a strong engineering division and wanted to help a company in financial difficulties. He never brought up the moon landing, he never brought up being an astronaut, and he became part of the research and development team.
To quote the man himself: “I’ve taken the position that, if the right situation came along, where I thought I could be of significant help .. and it would not jeopardize my honesty.” He was reputed to be a good businessman, but more importantly, an honest one.

He could have been famous; he could have parlayed that into a career in politics, in science, in anything at all he wanted. And that’s exactly what he did. He did what he wanted. When he was pressed by reporters about his lack of publicized appearances, he responded “[w]ell, I was pleased doing the things I was doing. That’s the sum and substance of it.”

He stopped giving autographs in 1994 when he found that people were selling them for large amounts of money and people were circulating forgeries. He sued Hallmark when they used his name without permission, and his barber when he tried to sell hair clippings. He settled both cases, refusing to accept any money for himself – every last cent went to charity.

He could have lived anywhere in the world. He could have made millions selling autographs and memorabilia. He could have had anything he wanted. Instead, he returned to Ohio. He lived quietly, privately, and modestly. He did what made him happy.

President Obama said he carried the aspirations of the entire United States on that fateful day in 1969; I refuse to accept that. Neil Armstrong was the living example of the aspirations, hopes, and dreams of the entire human race. He was the embodiment of the best our species has to offer. On July 21, 1969, he carried the hopes for every man, woman and child on this planet to the lunar surface. The flag he planted was American, but the footprints he left were for all mankind.

Space flight has never interested me, and probably never will. For me, it’s always been about the engineering challenges, thanks to him. The first computer system I properly engineered, I named “Eagle.” For Neil.

Neil Armstrong is my hero. He embodied all the qualities and ideals I aspire to myself. And he always will.

The OpenStack Foundation and Individual Board Elections

(Update’s at the bottom, so you can skip ahead.)
A corporate member of OpenStack basically threatened Shanley Kane
, who had been nominated as an individual that happened to work for a company they apparently consider a competitor – though that’s speculation on my part. (It’s the most obvious motive, but we don’t know what their true motive is.) This member corporation apparently stated that they would block her nomination to the Individual Board, even though they had no ability to, unless she gave them detailed information about her employer’s plans to the organization making threats – basically, conducting corporate espionage in exchange for not interfering with the allegedly independent board election process.

That’s beyond shameful, and blatantly illegal – it’s called attempted blackmail. The fact that the company had the gall to do so speaks to a total lack of ethics and governance at the organization making these threats. As to OpenStack – there is no possible way I could or would support an organization which tolerates members of any sort using threats and intimidation to influence the decision making process. And the fact that any corporate member would feel they could conduct themselves in this way calls into question governance capability and ethics of the OpenStack Foundation itself. A question which must be answered if their legitimacy is to be preserved.

And we have no idea if this is the first time or the fifth time. How many other members have they blackmailed or threatened in order to get the result they want? How many other companies are doing this? This conduct MUST be investigated, to find out just how widespread it is, and how many decisions have been enforced or changed through these unethical means.

Unless I missed a memo, the whole point of the OpenStack Foundation individual board was to provide a body representative of the community, independent from the “Platinum” and “Gold” (sponsoring companies) to prevent undue or excessive influence from companies with more money. This incident would seem to indicate otherwise.

As a result, I have no choice but to recommend and request that all individuals and corporations demand external counsel be hired to investigate this matter, whether or not other decisions have been influenced in this manner, and that all OpenStack Foundation elections and nominations be halted pending the results of the investigation. If this incident is true (and I have no reason to disbelieve Shanley’s statements,) it must be determined how widespread it has been, corrective action must be taken, and the members who have engaged in this conduct must be expelled immediately, permanently, and publicly.


Folks have more or less been making a LOT of baseless accusations and slinging a lot of unjustifiable mud in Shanley’s direction because she has not publicly named the parties involved or said any more about it. First and foremost; this is in the hands of the lawyers. Why? Because there really is and was no choice in the matter. As Shanley has said: she’s cooperating with the OpenStack board as much as she’s able, and continuing to do so. That does not mean publicly naming the parties involved, nor are people entitled to that information.

Secondly, frankly, this is not a matter between the accused and Shanley, and never has been. There are four parties – Shanley Kane, the accused, OpenStack, and Shanley’s employer. Why? Because the allegations include a demand for proprietary information from her employer. That’s not a “personal” problem, even when standing for a position as an individual. That’s a corporate espionage problem. The kind of problem where if something gets leaked, stolen, or broken into? It can and will fall back on you if you don’t report it immediately. Forget losing your job – you will not only get sued out of existence, you will probably go to jail too.

Now are you going to tell me that if someone demanded you engage in corporate espionage against your employer, you wouldn’t immediately report it to HR? Hint: I worked at a company where corporate espionage was a real concern and had occurred before. Failing to report it meant you were A) fired immediately B) sued in civil court C) prosecuted criminally when possible D) the company allegedly requesting it got sued E) miscellaneous things like blacklisting to ensure you never work in the industry again. I know of absolutely no company out there that does not follow the same pattern.

So no. Nobody is entitled or owed anything beyond what Shanley has said publicly, and it would be incredibly unethical and irresponsible (and potentially slanderous) for Shanley or anyone else to go around naming names. The matter is under investigation, and in the hands of people who know how to do that the right way – as in lawyers who understand the law. Bashing her and making accusations that it’s “made up” or she’s lying because she’s not saying more is incredibly disrespectful and completely wrong; and the same goes for OpenStack. It would be absolutely unethical of them – and potentially illegal – to release the names of anybody involved until the investigation is completed. And that holds true regardless of the result.

So yes, I absolutely support the course of action OpenStack has taken and I applaud the board for their quick action. However, I am still troubled by the possibility that this is not an isolated incident – I do not know whether or not OpenStack is investigating this aspect; and obviously I will reserve judgment on the actions taken for when those actions are taken, and not before.

So knock the conspiracy theories the hell off and just be patient. Until proven otherwise, I believe that everyone involved is doing their best to resolve this in a fair and ethical manner, and to ensure it does not happen ever again. And there’s nothing more you can ask of them at this point.