WHITEPAPER RESOURCE Active Directory Disaster Recovery - Semperis
WHITEPAPER RESOURCEActive DirectoryDisaster RecoveryWritten byRussell SmithSPONSORED BYA BWW Media Group Brand
EXECUTIVE SUMMARYAs the cornerstone of most enterpriseIT systems, Active Directory has grownboth in importance and complexity inrecent years. Enterprise IT environmentshave evolved with the rise of the mobileworkforce and cloud-based applications,and as a result, businesses have becomeincreasingly dependent on Active Directoryfor authentication and authorization.The new Active Directory usage landscapehas introduced greater complexity to theenterprise IT environment, raising the riskof AD disasters tied to human error andcyberattack. More and more frequently,attackers are using Active Directory as anattack vector to compromise enterprisesand, in some severe cases, wiping out theentire IT environment.2Active Directory Disaster Recovery hasalways been an extremely complicatedprocess, requiring lengthy preparation,planning and testing. Depending onthe size of the forest, and source of ADfailure, restoring Active Directory can takedays or more, rendering businesses nonfunctional during the recovery process.This white paper examines the complexityof Active Directory recovery, outlinespotential Active Directory failures andsolutions, and proves the necessity for anActive Directory Disaster Recovery plan.THE COMPLEXITY OF ACTIVEDIRECTORY RECOVERYActive Directory is not immune to disastersand recovering AD in the event of a disasterrequires in-depth knowledge of how itworks. Active Directory is designed for
distributed networks and uses a multimaster replication model to ensure thatthe directory can be updated and queriedefficiently in any location. As a multimaster replicated database, it is subjectto replication timing constraints with thepotential for different directory viewsdepending on the Domain Controller(DC) to which you are connected.The key challenge in Active Directoryrecovery is that you cannot simply restore asingle domain controller from a backup andhope the environment is back to normal.DCs work together to form a topologyand provide a set of services across anorganization - this topology is built into themetadata of an AD forest and that metadatais retained within the AD’s backup. Whenyou restore a domain controller, youmust ensure that the metadata of therestored environment is consistent withthe servers that are available—not thosethat used to be available. Otherwise,client systems will be unable to correctlyleverage your newly restored environment.In addition, the restoration itself mustbe carefully orchestrated. Root domainservices must be brought up beforechildren, Flexible Single Master Operation(FSMO) roles must be restored andthe Global Catalog must be re-built.If you throw DNS into the mix, clientsystems will not find the right servicesif DNS, a critical piece of AD health,does not reflect the actual environmentas it exists right now (as opposed toreflecting the previous environment).In order to properly restore the existingenvironment, at least one domain controllerin each domain should be restored frombackup in isolation, and then reconnectedto recreate the forest. Only once privilegedaccounts have had their passwordsreset and any issues that were presentbefore the restore operation have beencorrected, should the remaining domaincontrollers be redeployed, and the newdirectory database allowed to replicate.The bottom line is that the orchestration ofthe recovery of Active Directory is just asimportant as having backups of your DCs,and this complexity can greatly prolong therecovery process, if being done manually.The Microsoft Active Directory forestrecovery guide only provides genericinstructions that need to be adapted foreach unique restore operation and requirea lot of manual effort, meaning that ActiveDirectory could be unavailable for a fewdays if you need to restore a full forest.The complexity of the recovery processwill depend on what caused the disaster,so it’s critical to understand the root causeof the failure prior to performing a restore.WHEN DISASTER STRIKESInformation systems rely on Active Directoryfor user authentication and security, so anyoutage can be catastrophic. Some commonevents that cause Active Directory to fail,or actions that are irreversible, include:3
Database corruptionAccidental or intentional deletion ofobjectsPlanned schema changesUnplanned or unsanctioned schemachangesRaising the functional level of thedomain or forestPermission changesDisk and memory errors can causedatabase corruption, which often resultsin lsass.exe errors in the System event logand the Active Directory Domain Servicesto halt. If you have two or more domaincontrollers in each site, the temporaryunavailability of a single domaincontroller shouldn’t be critical. But ifphysical or logical corruption spreadsto more than one domain controller,then it might be necessary to perform acomplete forest restore.For example, when a British HospitalTrust suffered a complete ActiveDirectory failure in 2013, it took theIT team days to diagnose and repairthe failure. The outage was caused bydatabase corruption which happenedover a long holiday weekend and wentunnoticed until the following Tuesdaymorning. Experts from Microsoft and anIT consultancy worked for two days torestore the Trust’s Active Directory.The outage delayed the treatment of 706patients, and new appointments were4recorded on paper for the duration of theoutage and then entered manually oncesystems were back online.RANSOMWARE & MALICIOUS ACTSRansomware, and other types ofmalware, infect end-user devices, but ITinfrastructure is increasingly the target.End-user devices are usually the firsttarget because they are not secured to thesame level as domain controllers. Hackerscan harvest privileged Active Directoryaccount password hashes and Kerberostickets from users’ PCs to stealthily accessdomain controllers without needing toknow an account password. This so called“pass the hash” attack on privileged ActiveDirectory credentials gives hackers accessto domain controllers and any systemsthat rely on Active Directory for security.But ransomware isn’t the only danger.Insiders can intentionally or accidentallycompromise Active Directory. Especiallyin situations where security best practicesare not followed. IT staff are commonlygranted privileged access to ActiveDirectory on a permanent basis, whichmakes a hacker’s job easier. Furthermore,separation of administration roles is rarelypracticed, and security dependencies arecreated between highly-trusted systems,like domain controllers, and systemswith lower trust, like end-user devices.Automation technologies, like PowerShellscripts, can make large numbers of changes
to Active Directory that quickly propagate.But poorly tested code can result infailures of production systems. Malicioussoftware can also find its way onto domaincontrollers. But it only takes a single changeto cause a failure that prevents domaincontrollers servicing logon requests, breaksreplication, prevents additional domaincontrollers being added to the domain,or changes being made to the directory.Because of these threats, organizations needto protect Active Directory and prepare forworst-case scenarios where the only optionis to perform a complete forest restore.PLANNED & MALICIOUS CHANGESRegardless of how much planning andtesting you carry out, applications andsystems in your production environmentcould be affected by changes to ActiveDirectory. Changes sometimes happenaccidentally or are the result of maliciousactivity. Strict change control procedurescan prevent unwanted changes, butunsanctioned changes could be carried outby a malicious actor, a disgruntled insider,or accidentally by a system administrator.Schema changes, and raising the forestand domain function levels, are bothirreversible actions. Forest and domainfunctional levels determine the level ofcompatibility for the forest and domainrespectively with domain controllersrunning older versions of WindowsServer. When the domain and forestfunctional levels are raised, all domaincontrollers in the forest and domain mustbe running a version of Windows Serverthat is at least the same version as thefunctional level of the forest and domain.Raising domain and forest functional levelsis a safe operation if all domain controllersare running the required version ofWindows Server to support the newfunctional level. Schema changes can bemore problematic and should be tested ina pre-production lab environment beforebeing approved for release in productionbecause there’s no supported methodfor backing out of schema changes.If schema or functional level changesneed to be reversed, the only option isto perform a complete forest restore.OBJECT DELETIONDeleting directory objects, or changes topermissions on objects, can cause ActiveDirectory to fail. Strict change controlprocedures, and adhering to securitybest practices, are the best ways to avoidaccidental object deletion or modification.Active Directory also includes a flag thatcan be set on important objects to preventusers deleting them with one click. Toenable the flag on every OrganizationalUnit (OU) in a domain, use the GetADOrganizationalUnit and Set-ADObjectPowershell cmdlets as shown below.5
Get-ADOrganizationalUnit-filter * Set-ADObject-ProtectedFromAccidentalDeletion: trueThe Active Directory Recycle Bin can beused to restore deleted objects but it isn’tenabled by default. The forest functionallevel must be set to Windows Server 2008R2 (or higher) and it is an irreversiblechange. Starting with the administrationtools for Windows Server 2012, deletedobjects can be restored using ActiveDirectory Administrative Center (ADAC).Using the Recycle Bin is preferableto restoring objects from backup orreanimating tombstoned objects.Performing an authoritative restorerequires booting a domain controllerinto Directory Services Restore Mode.Note that removed link-valued attributes,such as groups, and cleared non-linkvalued attributes, are not restored whenyou reanimate tombstoned objects.Some organizations implement lag sitesas a recovery solution and for restoringdeleted objects. A lag site is an ActiveDirectory site which has delayed replicationfrom other sites in the domain. If objectsare deleted from the directory, the lagsite can be used to restore them. But lagsites shouldn’t be used as a completerecovery solution for several reasons.Microsoft doesn’t support lag sites asa recovery solution. In the event of a6malicious attack, AD can be configured sothat objects are replicated immediatelyto the lag site. Additionally, lag sites area security threat when objects deletedin the main site remain in the lag site.Consider a situation where a user accountis deleted but still exists in the lag site.If the Netlogon service is enabled ondomain controllers in the lag site, adeleted user might still be able to log on.BOUNCING BACKRECOVERING A SINGLE DOMAINCONTROLLERCorruption problems can sometimes berepaired in Directory Services RestoreMode (DSRM) using ntdsutil, a builtin command-line tool. DSRM is a safemode for Active Directory that allowsadministrators to carry out repairs whilethe database is offline. In a worst-casescenario, where the database can’t berepaired and only one domain controlleris affected, the server can be removedfrom the domain and re-promoted.If one domain controller needs to beremoved from the domain, move or seize(depending upon the state of the domaincontroller) any FSMO roles it holds andthen remove the domain controllerfrom the domain using the UninstallAddsDomainController PowerShell cmdletor Server Manager. If the domain controlleris Windows Server 2016, Windows Server2012 R2, or Windows Server 2012, the
demoted domain controller’s metadata isautomatically removed from the directoryproviding that during removal, Force theremoval of this domain controller is notselected, or the -forceremoval parameterisn’t set to true when using PowerShell.Reinstall Active Directory on the sameor different hardware and then let thedirectory partitions replicate to it. If youdecide to use the same server hardware,it is important to determine the root causeof the failure before reinstating the server.PERFORMING A FORESTRESTOREIn the event of a complete outage,security breach, or irreversible changeto Active Directory, you should performa forest restore to bring back all thedomains in a forest. Restoring a forestis a complicated process that involvesrestoring Active Directory from full serverbackups of one domain controller ineach domain, connecting the restoreddomains on an isolated network, and thenadding the remaining domain controllers.Performing a forest restore involves manysteps that mean Active Directory could beunavailable for a couple of days. Microsoftonly provides generic forest recoveryinstructions that need to be adaptedfor each unique restore operation. Youcan download the white paper here.One domain controller in each domain mustbe restored from backup in isolation, andthen reconnected to recreate the forest.Only once privileged accounts have hadtheir passwords reset and any issues thatwere present before the restore operationhave been corrected, should the remainingdomain controllers be redeployed, and thenew directory database allowed to replicate.SELECTING A TRUSTED BACKUPMicrosoft recommends that you use atrusted backup that is a few days old toavoid restoring a copy of the databasethat reintroduces the problem that causedthe failure, unless you can pinpoint exactlywhen the problem was introduced into thedirectory, with the help of the Windowsevent logs. In the case of a malicious attack,or complete forest melt down, the eventlogs might not be available unless they areregularly shipped to a server that doesn’trely on Active Directory for its security.Using a backup that is a few days old willmean that the restored domains won’tinclude changes made to the directory inthe days before the outage. But the effortrequired to reinstate these changes canoffset the time lost in restoring domains thatdon’t resolve the issues present before theoutage occurred. All group membershipsshould be reviewed after restoration,and this process will identify a significantnumber of the changes made post backup.Starting in Windows Server 2008, the7
Active Directory database mounting tool(Dsamain.exe) can be used to mountthe Active Directory database frombackups made using ntdsutil, WindowsBackup, or a backup tool that supportsActive Directory. A mounted databasecan be viewed using ldp.exe or ActiveDirectory Users and Computers (ADUC).The ability to view the database in thisway is useful when determining whichbackup to use for a restore operation. Inolder versions of Windows Server, it wasnecessary to restore a domain controllerto view the Active Directory database.FULL SERVER RESTOREWindows Server 2008 (and later) doesn’tsupport restoring a server using the systemstate to a new installation of Windows,regardless of whether installed on thesame or new hardware. Therefore, youshould make full server backups of domaincontrollers, perform full server restores,and only perform a system state restoreafter a full server restore to mark SYSVOL asauthoritative if the restored server is the firstwriteable domain controller in the domain.At least two writeable domain controllersshould be backed up in each domain.If you don’t want to make two backups foreach domain controller, i.e. a full serverbackup and a system state backup, thenSYSVOL can be marked authoritative byediting the msDFSR-Options horitativesynchronization-fo) in Active Directoryif SYSVOL is replicated using DFRS. If it isreplicated using FRS, then you will needto stop the FRS service, edit the BurFlagsregistry key replication-servi), and restart the service.A writeable domain controller shouldbe restored in the forest root first tomake sure that the Schema Admins andEnterprise Admins groups are presentbefore other domains are restored andto make sure that the trust hierarchy isn’tbroken during the restore process. Unlessthe forest consists of a single domain, thedomain controller you restore should notbe a Global Catalog. If you have no choicebut to restore a domain controller thatwas a Global Catalog, disable the GlobalCatalog after the restore operation iscomplete to prevent lingering objects.You should perform a non-authoritativerestore of Active Directory ‘DirectoryServices’ and an authoritative restore ofthe SYSVOL share so that when additionaldomain controllers are added to thedomain, they synchronize the contentsof SYSVOL from a server that has beenset as authoritative. You can performa restore using the built-in WindowsBackup tool or a third-party backupsolution that supports Active Directory.Once the forest root domain is in place,you can begin to recover other domainssimultaneously, providing that parent
domains are always restored beforechild domains. The last step is to makethe domain controller in the forest roota Global Catalog. Once all the domainsare restored, you can check that theyare working using the dcdiag, nltest, andrepadmin tools on an isolated network.from the failed domain. If forest failurewas caused by something outside ofActive Directory, like ransomware, thenyou must reinstall Windows Server.Before connecting the restored forestback to the production network andredeploying other domain controllers,you should clean up the metadata for allother writeable domain controllers in thedomain. This will make sure that NTDSsettings objects are not duplicated, andunnecessary replication links are notcreated. Furthermore, if restored domaincontrollers held the RID master FSMO rolebefore recovery, it won’t be able to createnew relative IDs (RIDs) until the metadatafor all other writeable domain controllersis removed. RIDs form part of the uniquesecurity identifier (SID) that is assigned toeach new Active Directory security principal.Well known for their high-performancegraphics cards, NVIDIA has embraced VRImplementing security best practices,and the latest technologies in WindowsServer 2016 and Windows 10, helpsreduce the likelihood of a successfulransomware attack. But systems cannever be one hundred percent secure,so a disaster recovery plan for ActiveDirectory is essential. Performing animpact assessment for Active Directoryinvolves mapping security dependencies todetermine which critical business systemsrely on Active Directory for security.Once these dependencies have beenestablished, you will be able to identify allthe systems that rely on Active Directory.Bringing Active Directory online as quicklyas possible after a failure requires atested disaster recovery plan. The detailsof the plan will depend on many factors:RESTORING THE REMAININGDOMAIN CONTROLLERSIf you are sure that the forest failurewasn’t cause by something outside ofActive Directory, i.e. a hardware failureor security breach, you can connectthe restored forest to the productionnetwork and add the remaining domaincontrollers to each domain withoutreinstalling Windows Server. Before youadd the domain controllers back to therestored forest, forcibly remove themRISK AND IMPACTASSESSMENT which version of Windows Server eachdomain controller is runningwhether domain controllers areinstalled on physical or virtual hostshow you will determine which is thelatest trusted backup for each domainwhether domain controllers will beinstalled to the same or new hardware9
Regulatory standards and service levelagreements (SLAs) may also impactdecisions in how you plan for disasterrecovery. The recovery process can bespeeded up by using full server restoresinstead of system state backups. Butorchestrating a full forest restore is difficultusing the standard tools because theyare not designed for automation. As partof designing a recovery plan, you shoulddetermine which domain controllers arerequired to get line-of-business systemsback online even if performance is impacted.A further concern for companieswith a hybrid cloud solution is AzureActive Directory, which in largerorganizations is almost alwayssynchronized with on-premise ActiveDirectory. Extending on-premise ActiveDirectory to the cloud introduces anadditional risk and complexity to themanagement and the recovery process.A forest-wide Active Directory failure cancause a complete outage of all businesssystems, and recovery can be complexand time-consuming. Following bestpractice advice from Microsoft is anessential step in ensuring that ActiveDirectory is protected. But nothing canreplace a proven disaster recovery plan.SUMMARYIn recent years, businesses have becomeincreasingly dependent on Active Directory,expanding their reliance on AD DirectoryServices to include authentication and10authorization of mobile workforce andcloud-based applications. This increaseddependency has led to greater complexityin the enterprise IT environment andraised the risk of Active Directory disasterstied to ransomware, malicious acts ormisconfigurations, and human error.While some Active Directory failurescan be repaired manually, recoveringActive Directory in case of a disaster isa long, cumbersome process that canleave businesses offline for days. Theonly way to ensure continued businessoperations is by making sure thatActive Directory is truly protected anda solid disaster recovery plan is in place.
Semperis is an enterprise identity protection company that enables organizations to quicklyrecover from accidental or malicious changes and disasters that compromise Active Directory,on-premises and on cloud. The Semperis Directory Services Protection Platform providesenterprises with the capabilities to automatically restore an entire Active Directory forest,quickly recover thousands of objects or a single crucial attribute, and instantly revert to aprevious Active Directory state. Semperis customers include Fortune 500 companies andenterprises spanning financial, healthcare, government and other industries worldwide.SPONSORED BYTable 1 - Recommended VR Configurations for DELL Precision Towers11
that rely on Active Directory for security. But ransomware isn't the only danger. Insiders can intentionally or accidentally compromise Active Directory. Especially in situations where security best practices are not followed. IT staff are commonly granted privileged access to Active Directory on a permanent basis, which
DNS is a requirement for Active Directory. Active Directory clients such as users computers) use DNS to find each other and locate services advertised in Active Directory by the Active Directory domain controllers. You must decide whether DNS will be integrated with Active Directory or not. It is easier to get Active Directory up and
An Active Directory forest is a collection of one or more Active Directory domains that share a common Active Directory schema . Most Active Directory environments exist with one Active Directory domain in its own Active Directory forest .
Active Directory Recovery Planning Chewy Chong Senior Consultant Systems Engineering Practice Avanade Australia SVR302 . Key Takeaways . Backup utility, DNS Manager, Active Directory Domains and Trusts Microsoft Management Console snap-in, Active Directory Installation Wizard, Active Directory Schema snap-in, Active Directory Sites and .
Module 4: Principles of Active Directory Integration This module explains how Active Directory can be integrated and used with other Active Directory Forests, X.500 Realms, LDAP services and Cloud services. Lessons Active Directory and The loud _ User Principle Names, Authentication and Active Directory Federated Services
Introducing Active Directory Countless books, articles, and presentations have been written on the subject of Active Directory, and it is not the intention of this book to repeat them. However, it is important to review a few basic terms and concepts inherent in Active Directory. Figure 3-1 illustrates the concepts that make up an Active Directory.
It is not the actual backup, which you would use to restore the Active Directory or Group Policies in the case of emergency when the server or Active Directory has crashed. A snapshot is used to restore the state of Active Directory, Group Policies, or their objects individually but only if Active Directory or the Server is in working condition.
Active Directory and Non Microsoft DNS: Facts and Fiction Jeremy Moskowitz, Group Policy MVP 6 The case for non-Microsoft DNS Active Directory administrators naturally want Active Directory to perform at its highest capabilities. The key activities that Active Directory and its domain controllers should be performing are: Authentication
c. AEMCLRP accreditation for automotive EMC testing 2) The Regulatory requirements of the European Automotive EMC Directive 2008/104/EC, and UN ECE REG 10 (which is used worldwide) 3) Doing EMC to control Functional Safety risks. The design, risk assessment, verification and validation techniques that should be done in addition to the above tests, for ESAs that can affect vehicle safety, to .
