Doleski Fuji June 08 2011 This Is A DR Declaration Not A .

This is a Disaster DeclarationThis is not a test!Presented by:Richard DolewskiWTS, Inc.

If this FUJIFILM event were a 007 MovieFrom Boulder ( I mean Broomfield ) with LoveOn Fuji’s Secret ServiceThe man with the Golden TapeThe LTO tape that Loved meLTO5 is not EnoughLicense to TapeNever say Downtime AgainNever say VTL AgainLive and Let SaveFor Your Backups Onlyi Fuji

Case Study - Disaster Declarations !!2 Disasters to walk through real life eventsBoth companies are in the Manufacturing business with thecorporate datacenter supporting several sales offices, distributionfacilities and manufacturing plants scattered throughout the USWe will refer to these as the Smith Corporation and the ZedCompany3

We are all not Safe fromWeather Related Disasters !!!

6

8

9

10

11

12

Definition of a DisasterA sudden, Unplanned Event that causes great damage or loss to anorganizationThe time factor determines whether an interruption in service is aninconvenience or a disaster. The time factor varies from organization toorganization13

What is a Disaster ?ANYTHING !!!!That stops your business from functioning & that cannot becorrected within an acceptable amount of time .14

What constitutes a disaster inyour organization ?Disaster Recovery deliverables are based on:RPO – Recovery Point ObjectivesRTO - Recovery Time Objectives15

WksDaysHrsMinsSecsRecovery PointRECOVERY POINTOBJECTIVERecover data to last knowngood point before outageSecsMinsHrsDaysWksRecovery TimeRECOVERY TIMEOBJECTIVEThe amount of time ittakes to recoverproduction system toresume its operation16

Do you have a fully documented andtested DR Plan ?

DR Planscome in manyflavors

No / Incomplete Disaster Recovery Plan Guarantees: Confusion Lack of direction Conflict Lost Customers Lost Time19

Disaster Recovery PlanningCannot be approached casuallyThe Plan must be .9 Well organized9 Action Oriented9 ComprehensiveObjective:Total restoration of Services in a timely manner20

The Products of a DR PlanWho will execute recovery actionsWhat is needed to continue, resume, recover or restorebusiness functionsWhen business functions and operations must resumeWhere to go to resume corporate, business and operationalfunctionsHow detailed procedures for continuity, resumption, recovery orrestorationCLASSIC: WHO – WHAT – WHEN – WHERE - HOW21

Common Issues Has your plan kept up to date with your IT integrations Expectations of Plan are un-realistic I no longer have the staff Implement DR into your Change Control Process22

The Smith Corporation Profile9 Last DR test 1 year ago9 100% successful - 14 hours9 Test annually with 4 IT Staff9 Trained in recovery of the system9 Tapes sent offsite daily9RTO & RPO - 24 hours23

The Smith Corporation Profile24

Smith Corp - Disaster¾ Water Damage under computer room floor on the 32nd Floor¾Severe power failure – Wednesday night 7 pm¾Site expected to be down well past the weekend.¾Disaster Declared ! 11pm4 Hours25

Smith Corp - Disaster¾ Person declaring disaster “ Not authorized “ and unfamiliar withpassword activation procedures.¾Hot-site begins preparation of hotsite, but requests authorizedindividual to call within the next hour4 hours26

Smith Corp - Startup Issues¾ Hot-Site equipment is available 4 hours after call¾Smith Corp still trying to locate authorized staff.( Alternates not available or no longer employed )¾Disaster Declaration fee of 7500.00. What’s this ?8 hours27

Smith Corp - Startup Issues¾Staff unable to recall tapes en-route to hot-site.¾Offsite Storage Vendor will not accept phone request.“ You must signon to Web site to recall tape media “28

Smith Corp - Startup Issues¾ Utilize Hot-Site infrastructure - Recall tapes¾iSeries - Full 21 save available & Wednesday nightbackup which was successful¾Intel – Last full and incremental backups successful¾Tapes are requested for Emergency Service Delivery¾All required tape media arrives - 90 Minutes9.5 hours29

Smith Corp - Startup Issues¾ Intel backup procedure includes:¾System state¾All filesystem volumes¾All databasesIssue - Tapes arrive in locked box30

Smith Corp - Startup Issues¾ iSeries Daily backup procedure includes:¾SAVSECDTA¾SAVCFG¾SAVLIB * ALLUSR ( save while active )¾SAVDLO¾SAVIssue - Tapes arrive in locked box31

Smith Corp - Startup Issues¾ Who has the Key ?¾Decision is made! Don’t wait just break the locks!¾30 minutes is lost before decision is made.10 hours32

Smith Corp - Startup Issues Issue - SAVSYS on 3590 format LIC and Operating system cannot be restored from 3590 format tape. Daily procedures saved on brand new LTO4 Hot-Site staff switch tape drives to accommodate daily save restoreon older tape technology12 hours33

Physical Servers to a Virtual SolutionSmith CorpHot-siteElectronicBackupSolutionImport virtualmachinesP – V Images34

Smith Corp - Startup Issues¾Electronic Backup Solution ( VTL ) physically removedfrom data center and transported to hot-site35

Smith Corp - Off to the races¾ Procedure execution for iSeries *ALLUSR recovery starts¾Procedure execution for Wintel Data Layer recovery starts¾Notice goes out to the business of an estimatedrestoration completion of 14 hours from NOW !16 hours36

Smith Corp - Update¾ Tape restore halted due to a message¾Step omitted *HOLDQSYSOPR message queueApproximately 45 minutes lost¾¾Called all 3rd party vendors for keys.New notice is sent to Business for restoration completion14 hours from NOW !17 hours37

Smith Corp - System ready¾ System is restored in 8 hours. Available Early !!¾Systems are checked. All files are restored.¾Non critical data omitted¾All 3rd party keys applied except for one as procedure ismissing and waiting for vendor call back.¾¾Testers have given the green light.We are back in business25 hours38

Report Card - Smith Corp9Need to keep Authorized staff for Declaration up to date9Know your Offsite Vendor limits9Locked Offsite Storage Boxes9Keep Hot-site in the know of hardware changes9Utilize one tape format for backups ( Enterprise )9Electronic Backup Solution ( VTL Onsite – Major Gap )39

Report Card - Smith Corp9 Save virtual machines images to Tape & store offsite9Missed Step - Human error9The entire staff was very well trained92 Staffers were on holidays - the other 2 came through9Third party procedures included key retrieval & activation40

The Zed Company Profile9 Last DR test 8 months ago.9 100% successful - 17 hours9 A special save to facilitate DR test.9 Have never tested a mid week recovery.9 Recently switched to a new software backup solution9 No Generator ( Not in Budget )41

Murphy's LawA typical summer day .85 degrees . Blue Skies.Weekend forecast of Sun, Sun & more Sun.The best laid out plans were now ready.Only Questions Remains .How quick can I leave the office Friday ?Then the cell phone Rings !.It’s the Data Center.Do you answer the phone ?42

This is a Disaster !!43

The Zed Company Profile9Primary Technical Owner of DR conductedlast 2 DR tests in Phoenix.9Alternate person ran operations back at the office.9 Tapes sent offsite daily9RTO 48 hours9 RPO 24 hours44

Zed Company - Startup Issues¾ Called in to Declare disaster but fails to show up !¾Waiting on staff to arrive – Unable to locateclient - Voice mail. ( Dead cell Phone )5 hours¾Client returns call 2 hours later !Power is slowly coming on in parts of the cityHe has heard his will come shortly.7 hours45

Zed Company - Startup Issues¾ Hot-site suggests tapes get shipped to startrecovery just in case. Client agrees.¾Client does not know which tapes he needs.¾All backup software recovery reports are kept on the system.¾Suggest sending last full & all of this weeksdailies. Hot-site staff try to recover.9 hours46

Zed Company - Startup Issues¾Tape media arrivesA box of 14 - LTO tapes in NO particular order.¾No media report arrived with tapes¾Which one contains the catalog ?¾Staff loads all tapes into library and inventories using backup software¾DR Plan missing at hot site- Alternate person goesback to site to obtain the plan10.5 hours47

Zed Company - Startup Issues¾Signed into Offsite Tape Vendor web site to determinewhat backup sets arrived at hot site¾Found tape catalog as labeled in offsite tape vendor inventory¾Unable to restore tape catalog – “invalid catalog error”¾Manually cataloged each tape - 6.5 hours wasted!17 hours48

Zed Company - Startup Issues¾ Management are very uptight Name calling ¾Every action/ or decision is now being questioned.¾Pressure is probably at a peak as Plant locationsare asking when ?49

Zed Company - Startup Issues¾Primary Technical Staff person now en-routeto hot-site to assist.DR Procedures arrive with Primary person21 hours50

Zed Company - Startup Issues¾The correct backup sets requested from offsite vendor¾Tapes were cataloged, fortunately the data was found¾Start recovery of rest of the system29 hours51

Zed Company - Startup Issues¾Seniormanagement ( not IT ) summons lead Technicalperson back as power is now coming on acrossthe street. Power is Expected anytime now.¾Declaration is HALTED & tapes are requested back.Against recommendations !31 hours52

Zed Company - Back on Again¾ Client calls back 4 hours later. Requesting therestore to continue. Power still an issue.¾Hot-Site informs client the system has been scratchedand they must start all over again.¾Client will bring tapes to begin reload.34 hours53

Zed Company - Back on Again¾No staff are available. Exhausted at this point.36 hours54

Zed Company - Back on Again¾Business wants a guarantee the system will beup in 8 hours. Business orders severely affected.¾Issues - Client didn’t maintain their backup environmentFirst test since new backup software installed55

Zed Company - Back on Again¾Restore starts immediately. This time with tapes already catalogedand DR documentation in hand¾System is restored in 17 hours¾Startup procedures incomplete, but manage to get ERPservices up.53 hours56

Report Card - Zed Company¾ Client Declares and Stops communicating with team¾Indecisive¾Backup policies not documented. A mystery to all.¾New Backup Software recovery never tested¾Backup Software reporting not sent offsite¾Tapes shipped. Documentation incomplete57

Report Card - Zed Company¾ Wrong person sent to recover system¾DR Plan unavailable¾Senior Management Pressure¾Staff in a state of flux - Coming & going¾Halted Recovery¾Restart all over again.¾Lack of staff delegation & training¾No Generator ( Now in Budget )58

The ZED CompanyDoes Your Business Have a Say!!Do the bean counters knowwhat IT can deliver ?59

Consider the Business impact of Down time !60

Report Card - Zed Company¾ Management Shocked that the IT folks lost all orders¾24 Hour RPO - Who agreed to this ?¾System was not available for 2 days¾48 Hour RTO - Who agreed to this ?61

Questions for Company ManagementInclude: Manufacturing, Finance, Purchasing, Sales, Warehousing etc.9Ask them about their Business.9What Services do you provide them!!9Is IT aligned with the business ?62

What about your email? Zed CompanyEmail has become Mission Critical ! Email represents the way you communicate with customers,partners, and employees. In a disaster situation, this may be a key component of yourcommunications strategy. Is Email listed as a critical application for your organization andincluded in your disaster recovery plan?63

Testing SummaryRecovering from a Disaster is overwhelmingThis can lead to frustration & loss of motivationFrequent Testing results in:9999People that are comfortable in familiar situationsVisually see the recovery goal & effortsConfidence already in placeWill staff be available when you need them most ?9Audit your backup policies .NOW !!!!64

Your Company Data is safe with meIt’s eleven o’clockDo you know where your data is ?

Recipe for DisasterIngredients:One Average, everyday filing cabinetAll your important business documentsOne Average business fire66