jump to navigation

DR tests too costly? 3 February, 2009

Posted by JohnMurrayUK in Planning.
Tags: , , , , , , ,
add a comment

I was speaking with a colleague last week about UK businesses scaling back their budgets for disaster recovery and business continuity provision. It seems that while some firms have decided to take the risk of not having a plan at all, others are trying to find shortcuts to reduce their spending. By far the most obvious piece of the jigsaw to remove, for most businesses, are the test invocations.

Test invocations form a crucial part of all disaster recovery plans, but often it is the most expensive component of the solution. Test invocations are frequently overlooked at the outset of a business continuity plan, as service providers and manufacturers proclaim ‘ease of recovery’. Only when the first test is carried out does the extent of the hidden costs become apparent. Even simple tape restore testing can be time consuming and therefore expensive (and often outside the desired Recovery Time Objective or RTO). Worse still, if the test fails, further staff time must be dedicated to investigation and documentation updates. When job losses are on the horizon, and teams are running on empty, just sparing the staff to fulfil the project may not be an option.

Some DR processes have an even higher cost due to bad design, and can only be carried out at the expense of uptime. HP ServerPhysical servers sometimes need to be moved, or shutdown to carry out all the environment or application testing. Some business continuity advisors get it right and ask service providers to ‘bundle’ test invocations into the service contract. That is fine as far as it goes, but it still frequently does not account for the hidden costs like resource, transport, and documentation updates.

It seems fair then to reduce or postpone test invocations as part of a budget cutting directive, but at what cost? When times are good, and business is booming, cashflow is rarely a problem. IT budgets increase as stakeholders recognise the need for business continuity plans and related insurance strategies. In reality, during such times, the organisation may be able to recover from the impact of a couple of days IT downtime. Sure, some customers will switch to your competitors, some of those will never come back, but your order book, and cashflow will be strong enough to carry the business through. In contrast, during a recession, when order books are small, and cashflow is tight, the same period of IT downtime, and resultant loss of business, could be enough to break the camel’s back. Hence, economic recession makes a working business continuity plan even more crucial.

Some service providers, like virtualDCS (but there are others), have engaged with their customers to find a solution to this dichotomy. It is possible, given the right approach, to leave the invocation process to the service provider. The service provider maintains a detailed documentation process, and provides both the equipment and the manpower to invoke the solution independently, with no impact on the client’s live running IT operation, or the team supporting it. Once the solution is fully invoked, the business can carry out specific application tests, before leaving the service provider to dismantle the invocation test again, and update the documentation.

This sounds like a shift to wholly outsourcing the disaster recovery solution to a service provider, and it is. It also sounds very expensive, but it isn’t. The recovery team at virtualDCS (I can’t speak for other service providers), perform test invocations every day of the year. Fortunately live invocations are rare, but test invocations happen on a regular basis. Because the test invocation is a routine action, and highly automated, the costs are kept small, and more importantly, included in the contract. With contracts starting at around £50/week for a server with 60GB of data, and an achievable Recovery Point Objective (RPO) of near zero, why would you do it yourself?

Sphere:Related Content


A Disaster Recovery Plan that’s more than just a tick in a box. 30 October, 2008

Posted by rmayuk in Planning, Replication, Snapshots, Technology.
Tags: , , ,
add a comment

Having a disaster recovery plan is the first step to ensuring some degree of business continuity. However, if your plan is to go to the nearest IT retailer and buy some servers, or get a ship to site hardware contract then spend days recovering your systems from tape. Think will it really work when you need it, can you really afford to be without your systems for days.

In most cases having a plan will be sufficient, it will keep customers and investors happy right until the point you have a disaster, a fire strikes, or servers and PCs are stolen, or a natural disaster occurs. At these times, businesses using tradition disaster recovery procedure reach for their backup tapes and pray. Will the restore process work, how much data have I lost since the last backup, how will I continue to operate whilst my servers are being rebuilt.

The speed at which business applications and data can be restored is vital to the survival of a company following a disaster. 32% of businesses state that just four hours downtime, or less, could be fatal to their organisation (source: Continuity Central – 2006)

At 1pm on a Monday the air conditioning unit in your computer room stops working, 30 minutes later your 4 main servers overheat and shut themselves down. Having opened a few windows you try to restart them, 2 of them won’t restart, your mail server and your database server. Time to invoke your Disasters Recovery Plan; you call your provider request the servers on your contract to be brought to your site. You go and get your tapes from the fire safe for Friday’s backup and wait. 6 Hours later the servers arrive and you start the recovery process. You’re dedicated IT team work through the night recovering your servers by lunch time the next day your systems are back up and running. What have you lost?

· You have now not received or sent any email for 24 hours.

· All your customer emails have been bounced back to the send.

· You have not been able to ship any orders for 24 hours, or take many either.

· All the orders you took since your last backup on Friday night have been lost.

· The orders you took on Saturday and Shipped Monday morning now don’t even exist on your system.

· You have lost money, credibility and customers.

At 1pm on a Monday the air conditioning unit in your computer room stops working, 30 minutes later your 4 main servers overheat and shut themselves down. Having opened a few windows you try to restart them, 2 of them won’t restart, your mail server and your database server. Time to invoke your Disasters Recovery Plan; you call virtualDCS your servers are all covered by our rapid recover service. Within 2 hours your copies of your servers are available from our data centre, your users gain connectivity again. What have you lost?

· Your mail was temporally interrupted however all mail was held for you until your mail system became available again.

· You were not able to ship orders for 2 hours.

· You did not need to really on you backup so you only lost 1 order.

· It was incontinent but customers understood the very brief interruption.

Many companies are starting to take advantage of cheap storage solutions, off site backup solution, data replication solutions, using the internet or other larger network connections. These solutions are all floored, and there is a better way of doing it. Our Solution:

· does not require you to buy two of everything just in case your live servers go wrong.

· is not reliant on your IT department, what if there not there when disaster strikes.

· will maintain copies of your whole server, every update, every patch, and every email.

· can have your systems available to your users in minutes not days.

· will restore the service back to your site when you are ready.

Using the latest in virtualisation and replications technologies we have created a solution that can provide a cost effective answer to your Business Continuity needs.

One step forward, two steps back 3 October, 2008

Posted by JohnMurrayUK in Replication, Snapshots.
Tags: , , , , , , , ,
1 comment so far

When thinking about IT recovery, most people start by looking at their backup tape protection.  With modern real-time data replication solutions the RPO all but vanishes, but so does an oft forgotten failsafe – point in time recovery.

With real-time replication, a corruption to a file in your live site, will also be copied to your standby site.  Failover in this scenario gets you nowhere, and all of a sudden last nights backup tape looks worth it’s weight in silicon.

There are additional safeguards you should consider when looking at replicated solutions.  Those that are based on transactional recovery points (like SQL Log Shipping) allow you to ‘playback’ the days activities upto the point of the corruption, but this takes skill to manage, and can dramatically extend your RTO.

Other solutions take advantage of snapshot technology at the failover site.  By taking a point in time snapshot (think of it like a disk backup), at regular intervals, you can choose when failing over whether to fail to your latest RPO, or a point in time earlier.  With data corruptions, this ability can be a lifesaver, and depending on the replication technology you choose, it need not be complex.

Double-Take replication adds the selection of snapshot to the failover management tool, conbining both solutions into one interface.  This enables simple point in time recovery for those hard to fathom data corruptions.

Perfect planning prevents …. 2 October, 2008

Posted by JohnMurrayUK in Planning.
Tags: , , , , ,
1 comment so far

Business Continuity Plans are a good starting point to help you assess your ability to respond to an operational ‘disaster’.  Concentrate on your business processes; where do my orders come from? how do I deliver my product/service? etc and see if you can fulfill these requirements without access to your building or your IT infrastructure.

Once you’ve completed that task, it’s time to start putting some numbers against these activities; how much money/customers would you lose if you were unable to take an order for a day?  what business impact would there be if you were unable to supply your product? etc.

The above excercises should have focussed your mind on the key critical components to your business, and also provides a context for your budget.

Don’t forget, if you need help with putting your plans together, just give us a call.