Millennia IT Blog

Strange things are afoot in the IT industry, and I can’t quite get my finger on what it is that is happening.

We seem to be having the best year we have ever had in the industry, and that includes the so-called “boom” years before 2007. I say so-called because the IT industry probably had two mini recessions between 1999 and 2007, so it didn’t leave a lot of good years anyway.

It is not that there is particularly any sense from the customer base that things are picking up, and indeed we have seen the smaller end of our customer list shrink in the last couple of years as tight market conditions bit, but the amount of work that is building up in the mid to large enterprise space is quite astonishing.

I suppose that there comes a point where you can’t put essential IT work off any longer, or the consequences of that outweigh the potential hit you get spending in financially uncertain times; and today IT is pretty much the lifeblood of any reasonably sized organisation.

So from 6 months ago when we were wondering if Europe would blow up (it still might) and what consequences it would have for our business, we are now in the happy – albeit quite stressful – position of having enough work and potential orders to see us out through this year and into next.

Quite how amongst all the doom and gloom on TV we can stage such a miraculous turnaround mystifies me, but perhaps all the years of hard work and keeping ourselves financially stable through the banking crisis is now paying dividends.

Not only are we still here, we are doing better than ever!

“Official” statistics now put Consumer Price Inflation (CPI) at around 4.5% in the UK and a little under 4% in the US as measured by the CPI-U data. However no matter who you talk to they never can quite understand why these published figures never seem to match the relentless upward march in prices they see in the real world.

It is almost impossible to track how prices have changed over the years because successive Governments on both sides of the pond had continually tinkered with the figures, generally in the downward direction.

This is beneficial for the debt-laden Governments because they are able to hide real world inflation away from the masses while quietly inflating away their debt bubbles. This, as ever, rewards the profligate at the expense of the prudent and results in the purchasing power of fiat paper currency being eroded more rapidly than one would care to consider.

The Gold Standard

In order to clearly demonstrate how far we have diverged  from reality in the last 30 years then we merely have to look at precious metals such as gold and silver. These both had a big run up in prices in the turbulent 70s and only really perform well in times of inflation and uncertainty. Because of constant tinkering with the former and copious BS from central bankers making us think the latter was a thing of the past, these commodities basically flat lined until just a few years ago.

In fact the most famous indicator of the turning point was the sale of 395 tons of gold by Gordon Brown between 1999 and 2002. The price has never been lower since that time, and it also coincided with the Great Top in western stock markets.

Gold has recently made huge gains from the $250 - $300 an ounce Gordon sold for, and recently topped $1,900 an ounce. This has led many to say a new all time top has been reached, but in terms of inflation-adjusted real value is this actually the case? This is where a comparison between real and “official” inflation becomes so stark.

Because gold is priced in US$ I’m going to use inflation statistics of the US to indicate the point. Official figures for CPI (CPI-U) are available from the Bureau of Labor Statistics, but an alternative “real” CPI has also been produced by John Williams at ShadowStats.

ShadowStats basically takes the CPI as it was during Jimmy Carter’s tenure as president, and continues to calculate CPI in same way as it was done back then. No fiddling, no “adjustments”, just the same base calculations year on year. This allows us to compare where gold may peak if was to reach the same inflation adjusted price as it did in 1980, when it briefly hit $850 an ounce.

Where are we now?

If you take the official CPI-U figures from then to now you would arrive at a figure of around $2300 - $2400 per ounce. This is pretty near to where we have got to and certainly seems to back the calls for a top in the gold market. However when the ShadowStats CPI figures are applied to the peak gold price of 1980 it shows that today, to equal this record top in the price of gold, it would have to exceed $15,000 an ounce!

From this it is not hard to see where the purchasing power of your savings has gone, and is actually worth merely a seventh of even what the US Govt admits has been eroded away in the last 30 years.

So it’s not hard to see why houses have leapt from £10,000 to £200,000 in that time, and it does clearly show that deposits in a bank, whatever the headline interest, bleed away into the debt mountain the Govt creates over time. At the end of the day capitalism is designed for the few, to be paid for by the many.

All that Glisters…

Despite all the hype around gold perhaps the biggest move yet to be made is by its poor relation, silver. Silver also participated in speculation and in 1979 reached nearly $50 an ounce. It is unlikely that such an equivalent price would be reached again today as this was done on huge margin, and the resulting unwinding of the price in 1980 lead to the price crashing – wiping out the Hunt brothers in the process on Silver Thursday (27 March 1980).

However the price is currently sitting around $40 again, so on an inflation adjusted basis just how far COULD it run?

Well, based on CPI-U this would be around $130 to $140 an ounce – more than triple where it is currently – and on the ShadowStats index it would be approaching $350 an ounce.

So the “poor man’s gold” may yet have its day.

In conclusion

Never, EVER, believe predictions of deflation – unless it is consumer electronics or something similar that benefits from technological development and mass production. Deflation is an invented instrument used by central banks and governments to hide their practice of inflating away the ever increasing debt pile. Unfortunately even this scheme cannot last forever, and something really dramatic will be needed to dig us out of the hole we’ve all collectively stuck our heads in.

Flash storage has been around for a while now, but to date it has remained too expensive for most organisations – even though the performance benefits offered are so compelling. However it now seems that with MLC prices continuing to fall, and MLC developing in lifetime and reliability, the time of flash entering the mainstream SMB market may now be upon us.

Problems faced by traditional HDD

Hard disk drives (HDD) are electro-mechanical devices subject to the laws of physics. They are spinning disks of magnetic media that have hit the wall at an rpm of 15,000 – a speed which has been common in high end SAS drives now for years and is unlikely to be increased. There is really no way forward to go in terms of spindle speed, so in order to extract the very last bit of performance from them data has become more tightly packed as capacity has increased and this has lead to better throughput.

However this only works for sequential reads and writes, and any randomness of I/O soon chops the performance off at the knees for a spinning disk, and what is the biggest source of random I/O today? It’s virtualisation, the direction everybody is now moving en masse.

On top of this HDD require a lot of power to maintain those spin speeds, and although some “green” models can spin down when not busy it is likely under RAID and virtualisation that they never really get any quiet time. Which of course brings us to yet more overhead poor old HDD has to cope with, RAID.

RAID accelerates throughput and I/O by binding multiple disks to act as one, but only when things are normal. The increasing capacities of those disks highlights an increasing problem with RAID sets in that when a disk fails the RAID has to rebuild a spare disk from the parity information, and on a RAID 5 group containing 2TB drives rebuild times can sometimes be talked of in terms of DAYS, let alone hours! Heaven help you if another drive fails in that time…

This has lead to an increase in the use of double-parity RAID 6 to help mitigate the dreaded error during rebuild scenario, but this in itself increases rebuild times due to the extra parity information – as well as slowing writes further even during optimal operation.

Virtualisation killed the magnetic disk

Ok, bit extreme, the death of tape has been called for years and it still exists. But the rapid uptake of server virtualisation created the initial problems with an increase in random I/O, and now desktop virtualisation – with its far higher IOPS requirement -  is really showing the age of spinning disks. Many VDI implementations, some for very large companies and costing millions, have failed to scale as expected and nearly all of them were due to poor storage subsystem performance.

No matter how good the storage subsystem it all comes down to the magnetic disk properties eventually, and you can’t get past physical laws with fancy programming – no matter what a vendor may tell you.

One of the big issues with virtualisation is LUN over subscribing. This is where differing workloads all have calls on the storage subsystem at the same time, and if a mission-critical VM wants to write to a LUN while a lazy lower priority VM is reading from it then the mission-critical VM may have to wait because the storage often has no idea about priorities. VMware has features such as disk shares and I/O control, but this is still often not granular enough to be completely effective.

If you happen to be using SATA drives with shortened buffers and queues because the SAS alternatives were just too expensive to get through the project board… well, how expensive is a storage timeout on your database?

Workarounds to keep HDD going

The usual answer to “We have a problem” is “Well fix it!”. So we have to enact a number of work-arounds to keep our systems going.

The most common way to do this is just throw more and more disks at the problem. This way more performance is derived from bigger RAID arrays, however the law of diminishing returns states that each extra HDD adds less each time it is added until you can be throwing bucketfuls of expensive hardware at a 10% I/O increase. Who has the budget for that these days?

Some systems use short-stroking, which is basically using just the outer, faster portion of each disk to increase performance. However this is incredibly wasteful and increases the problem of HDD sprawl even more.

These are just sticking plasters, we need a new way…

Along comes Flash – but oh the expense!

Flash Solid State Disks (SSD) have the performance. With no moving parts they are just as efficient at random I/O as they are at sequential, and with lower seek times and latency they can deliver far higher performance than any spinning disk could hope to match. Single-level cell (SLC) drives are the cream of the crop, with very high lifetimes, performance, and a price tag to match – between 10 and 20 times a 15k HDD on a per TeraByte basis.

Because of this they are often only used as cache to provide a low latency staging for the disks behind. The problem with cache is that if the data isn’t in it when required it has to come from HDD, and in the case of writes the amount of data written can fill the cache and then performance plummets while the cache is flushed to disk.

So cache has a place but again is really just a sticking plaster over the old problem.

Another method is to tier the data by required performance and install SSD as Tier 0 level with cascading levels of performance further down the tiers in the form of SAS and SATA HDD in varying spindle speeds down to 5.4k. This provides a hybrid solution balancing performance with capacity, but then requires a lot more management. You also have to get your data sizing just right or you find all that expense going out the window as the system bottlenecks again.

Of course all this management means specific systems designed to automate this process, and often you will find all the storage has to be from one vendor and comes at a premium price.

MLC Flash comes of age

A better solution comes in the form of multi-level cell (MLC) drives, which are cheaper to produce but are generally of lower performance than SLC and have a lower lifetime due to the amount of times a cell can be re-written. This is improving but the way this media is used is the key to launching it into the mainstream marketplace as a storage media of choice.

There are many vendors that have developed this media in such a way that it has lifetimes that match conventional disk. The most common is by deduplicating inline – to reduce the actual amount of data stored, and then serialising the data through the storage interface so it can be intelligently written in such a way as to minimise media wear and tear though “wear levelling algorithms”. WhipTail has had such a device out for a while, but where one leads others are bound to follow.

One such vendor that has exploded onto the market recently is Nimbus, with their new line of S-class 100% flash based storage. The Nimbus comes as fully modular shelves of drives with 100GB, 200GB and 400Gb capacity, giving shelf capacities of around 2.5TB, 5TB, and 10TB respectively. Usable capacities due to system configuration may be quite a bit less, but this is also offset by inline deduplication of up to 10x depending on the base data.

The maximum capacity shelf can produce 800,000 RANDOM IOPS and a full rack up to 1.65 million random IOPS. The maximum generally constrained by the ability to transmit that many transactions in a single second – although I’m sure that will yet increase.

Aha, but what about the cost, I’ll bet that is budget busting?

Their entry level 2.5TB usable shelf starts at just $24,995 – about the same price and capacity as a Dell EqualLogic PS6000XV in RAID-10 format, with over 50x the performance.

Still interested?

How the heck?....

Yep, I wondered too but it is all about the way the system is built and used.

Nimbus have designed their own Extended multi-level Cell (EMLC) “flash blades”. These have 10 times the erase cycle lifetime of MLC drives at 30,000 cycles, or about 1/3 of that of SLC drives.

For those worried that this might still not be enough for Enterprise use the way Nimbus use these drives means that to burn them out in 5 years would require you to write over 7TB an hour to them, continuously! Of course as drives get cheaper even the threat of 5 years lifetime becomes a moot point, how many of you have swapped a regular HDD within 2 or 3 years?

The on board HALO OS controls the wear-levelling algorithms and garbage collection to provide consistent system performance, as well as delivering all the Enterprise functionality such as: inline deduplication, storage virtualisation, thin provisioning, snapshots, mirroring (sync/async), and multi-protocol access from iSCSI to Infiniband (unfortunately not AoE – I may work on them about that one ;) ).

Oh, and remember the agonising rebuild times with HDD in the event you do have a drive failure? With Nimbus you are back into minutes even with RAID 6 protection as standard.

Power to the people

One area often overlooked when doing a lifetime ROI study on a storage system is running costs. SSD consume far less power because of the lack of moving parts and the removal of the need for HDD sprawl to achieve performance requirements. As such, power consumption can be as much as 90% lower than traditional 15K disks at around 5W per TB. This will become incredibly important as energy costs only seem to go one way these days, especially in Rip Off Britain.

The run to server virtualisation has made great gains in removing power hungry servers that weren’t delivering anything but still consuming a lot of power. Some of this saving was cut by power gains in storage systems needed to service this virtualised world, but now here too we have a chance to make radical cuts.

A combined 200 VM virtualised system running on flash storage can now extend the power savings beyond 85% in both power delivery and resulting cooling requirements, compared to the same 200 physical servers with local storage.

Data centre emissions reduction target – tick.

When considered alongside reducing purchase costs an all flash system can actually be significantly cheaper than its HDD competitor over a 5 year life cycle, already, and this can only get better.

In conclusion

Original source: Nimbus (www.nimbusdata.com)

Finally it seems that flash is ready to enter the arena as mainstream storage, perfectly placed for the next major upswing in virtualisation caused by Cloud uptake and VDI gaining more traction. Once a significant amount of organisations have seen what is possible, and at such a reasonable cost, the development of this technology will only bring prices lower.

Traditional HDD still has a long time to run, in the areas of high capacity and serial I/O such as backup and archiving it is well placed to provide the complement to flash – which has a long way to go before Petabytes of archive storage look sensible on this platform. However the cost of flash has now been removed as a barrier to entry in the mainstream market for general production use.

Now, at last, storage can catch up with the Moore’s law developments that CPU and RAM have undergone in the last decade or so.

Flash has arrived – make it so.

For more info contact us at info@millennia.it, and we can discuss your storage requirements and design a SSD/HDD balanced system that meets your needs, and your pocket.

It still amazes me what a range of prices and service levels there are in the IT support industry today. You would think that after so many years there would be some sort of standardisation so that there would be little to choose between different suppliers, but if anything the range has become ever wider. This must be a pretty confusing issue for the eventual consumer of such services, as they try and figure out what they need and who is not only likely to supply it, but at what price and what quality.

Two such experiences I have had recently throw this into sharp relief, an example of incredibly poor service and an example of stunningly high prices. For obvious reasons names will be excluded from these tales.

My first experience involves the rapidly growing cloud services market. To some extent I can forgive companies that have trouble setting realistic prices, as their intial investment is high and they are pretty keen to start getting that money back in quickly. However, looking at a supplier for Hosted Exchange made me wonder if the company involved was staffed by people fresh out of investment banking, because the level of prices was around 5x more than anything I had seen before. This made me wonder what sort of companies are out there that would pay these sort of figures, and why the heck they were never customers of mine (just kidding, a bit)!

The whole point about cloud services as far as I am concerned is that they give flexibility, and by concentrating resources and then sharing them out between clients they can use an economy of scale to make it more cost effective than the client doing it themselves. In this case the client would have been able to build their own system to support 3 times as many users for the same cost as the cloud provider was charging – what the heck is the point of that?

The problem here is that issues like the recent Amazon fubar have not helped the perception of cloud services, and if the costs also seem to be excessive then a lot of people are going to turn away – making the ROI for those already heavily invested even more problematical. Considering I am in the process of regenerating my own virtualised hosting model to run under the new VMware vCloud model I could well do without this.

On the upside of course I know I can be a heck of a lot more competetive on price than the larger providers, which is counter-intuitive when you think about it.

The second experience is altogether more common, but no less upsetting because of that – poor service. This is a classic example of a company that had a service contract and quoted a price that was generally below market to get the business, which of course colours the opinion of the client to more realistic prices, but then totally fails to deliver any sort of reasonable service.

In this case the company in question had been tasked with setting up a VMware cluster. This is not an easy task but these days anybody advertising VMware services on their website pretty damn well should have VCPs on their staff, in which case it should be a straightforward operation. However following this support company into a client and doing a health check on their systems uncovered an incredibly amateur installation, resulting in the inability to activate HA because of incorrect network configurations. This was the whole reason VMware had been brought in as with only 7 VMs there was no cost benefit over hardware servers. It also appears the support company had boosted its low support charges by selling the client a version of vSphere way in excess of what they required, Enterprise on 3 hosts (for 7 VMs!) via a mid-size acceleration pack instead of Essentials Plus. This in itself must have cost £15000 to £20000 more than it needed to, and it wasn’t even installed properly.

So the client had been left with a non functioning VMware cluster that actually exposed their systems to more risk due to lack of internal knowledge of VMware and incompetent installation and support. This doesn’t sell virtualisation when a reboot of one of their hosts takes down vCenter and it doesn’t move to a different host. The reaction of the support company to “install vCenter on a physical server” shows the complete lack of knowledge of this organisation.

This of course leaves me with the unenviable task of not only managing the client’s expectations on a reasonable cost for taking over support, but also that I won’t be equally as useless when it comes to supporting their systems. Fixing a vCenter service start problem during the health check (5 minutes, they had been waiting 2 weeks since it failed and it was a misconfigured virtual network) probably helped, but the finanace director may see things differently at the bottom line.

It is very hard to make headway in the IT Services sector as a small business with so much competition. The fact we can survive and have customers on the books for several years shows it is possible to do business even in a crowded market, but businesses are still a long way from getting the service they need at a price that provides value. Inventing new paradigms such as cloud will not help the end customer if all it does is cause more confusion and more ways to deliver questionable service at ridiculous prices.