Approximate Dynamic Programming (ADP) Methods For
IntroductionOptimal ControlADP ApproachesResults and ConclusionsApproximate Dynamic Programming (ADP)Methods for Optimal Control ofCardiovascular Risk in Patients with Type 2DiabetesJennifer MasonPhD CandidateEdward P. Fitts Department of Industrial & Systems EngineeringNorth Carolina State UniversityRaleigh, NCINFORMS Annual MeetingCharlotte, NCNovember 14, 2011ADP Methods for Optimal ControlJenn Mason, NC State1/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsCollaboratorsBrian DentonEdward P. Fitts Department of Industrial & Systems EngineeringNorth Carolina State UniversityNilay ShahDepartment of Health Sciences ResearchMayo ClinicSteven SmithDepartment of EndocrinologyMayo ClinicThis project was funded in part by the National ScienceFoundation under grant CMMI-0969885.ADP Methods for Optimal ControlJenn Mason, NC State2/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsOutlineIDiabetes BackgroundIMarkov Decision Process (MDP) for Optimal ControlIApproximate Dynamic Programming MethodsIIAggregate MDPIBasis Function ApproximationNumerical Results and ConclusionsADP Methods for Optimal ControlJenn Mason, NC State3/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsChronic Disease ManagementIChronic diseases are the leading cause of death in theU.S. and other countriesIFor many chronic diseases there are treatment options tomanage the disease and reduce the risk of adverse eventsIOptimal control of treatment can prolong lives, improvequality of life, and reduce costsADP Methods for Optimal ControlJenn Mason, NC State4/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsDiabetesI23.6 million people in the U.S. havediabetesITwo out of three deaths are caused bystroke or coronary heart disease (CHD)IBlood pressure and cholesterolmedications are often part of treatmentplans for diabetes patientsADP Methods for Optimal ControlJenn Mason, NC State5/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsMedicationsIBlood Pressure MedicationsIIIIIBeta BlockersACE Inhibitors / ARBsThiazidesCalcium Channel BlockersCholesterol MedicationsIIStatinsFibratesADP Methods for Optimal ControlJenn Mason, NC State6/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsCurrent U.S. GuidelinesIJNC 71 :IITreatment Goal: SBP/DBP 130/80 mmHgATP III2 :ITreatment Goal: LDL 100 mg/dL1: The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of HighBlood Pressure, NIH Publication No. 03-5233, 2003.2: Third Report on the National Cholesterol Education Program Expert Panel on Detection, Evaluation, andTreatment of High Blood Cholesterol in Adults (Adult Treatment Panel III), NIH Publication No. 01-3670, 2001.ADP Methods for Optimal ControlJenn Mason, NC State7/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsBounded, Continuous State SpaceADP Methods for Optimal ControlJenn Mason, NC State8/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsMarkov Decision Process (MDP)Time HorizonIt {1, 2, . . . , T }StatesIIhealth states:Imaxlipid ratio (LR): LR]t LLR [0, LRIsystolic blood pressure (SBP): SBP LSBP [0, SBPmax ]tmedication states:M {mt (m1,t , m2,t , . . . , mn,t ) mi,t {0, 1}}Actions for medication iA( LR , SBP ,mi,t )tADP Methods for Optimal Controlt({Ii , Wi } {Wi }if mi,t 0if mi,t 1Jenn Mason, NC State9/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsRewardsSocietal Perspective: , CHD, mt ) R q( StrokettLR SBPStroker ( t , t , mt ) , CHD, mt )if the patient is alive C( tt 0otherwisePatient Perspective: q(mt ) if the patient is aliveSBPand has not had any eventsr ( LR, mt ) t , t 0otherwiseADP Methods for Optimal ControlJenn Mason, NC State10/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsOptimality Equations t 1, . . . , T 1:SBPvt ( LR, mt )t , tZ λZ X maxa A LR SBP( , ,mt )ttimmediate reward z} {SBPr ( LR, mt )t , t SBPLR SBPSBPSBPLRpa ( LR, mt )vt 1 ( LRt 1 , t 1 , mt 1 t , tt 1 , t 1 , mt 1 )d t 1 d t 1mt 1SBP LRt 1 t 1 {zexpected discounted value to go}Boundary Condition for t T :SBPLR SBPvT ( LRT , T , mT ) µ( T , T , mT )ADP Methods for Optimal ControlJenn Mason, NC State11/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP ApproachesIUniform AggregationIBasis Function ApproximationADP Methods for Optimal ControlJenn Mason, NC State12/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 1: Uniform AggregationFixed Finite GridADP Methods for Optimal ControlJenn Mason, NC State13/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 1: Uniform AggregationIA mean value is associated with each discrete stateIExample: LRµ(S1LR )0 LR t UB(S1 ) LRLRLR µ(S2 ) UB(S1 ) t UB(S2LR )LR LRg ( t ) . . LRLR µ(S ) UB(S ) LR LRmaxqtq 1IThe approximate MDP is solved using backwards inductionADP Methods for Optimal ControlJenn Mason, NC State14/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 1: Uniform AggregationState Transition DiagramNo MedicationsLøMøHøVøAdverse Eventor DeathADP Methods for Optimal ControlJenn Mason, NC State15/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 1: Uniform AggregationState Transition DiagramStatinsLSMSHSVSNo MedicationsACE InhibitorsLøMøHøVøLAMAHAVAStatins ACE InhibitorsAdverse Eventor DeathADP Methods for Optimal ControlLS,AMS,AHS,AVS,AJenn Mason, NC State16/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 2: Basis Function ApproximationSBPṽt ( LR, mt )t , t K XXSBPwt,k ,mt bt,k ( LR, mt )t , tk 1 mtSBP , m ) is weighted bywhere each basis function bt,k ( LRtt , twt,k ,mtISBP , m ): the patient’s annual probability of nobt,1 ( LRtt , tCHD eventISBP , m ): the patient’s annual probability of nobt,2 ( LRtt , tstrokeADP Methods for Optimal ControlJenn Mason, NC State17/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 2: Basis Function ApproximationAnnual Probability of No CHD Event10.995Female, Medium SBP, Low LR0.99Female, High SBP, Medium LR0.985Male, Medium SBP, Low LR0.98Male, High SBP, Medium LR0.9750.974041ADP Methods for Optimal Control424344Patient Age4546Jenn Mason, NC State18/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsADP Approach 2: Basis Function ApproximationLinear Program to Estimate Basis Function Weightsmin z T XKXXmt t 1 k 1s.t.KXXXSBPbt,k ( LR, mt )t , t LR SBPttSBPwt,k ,mt bt,k ( LR, mt ) λt , tXX X LRtk 1 wt,k ,mtKXSBPLR SBPpa ( LR, mt )t 1 , t 1 , mt 1 t , tmt 1 SBPtSBPLR SBPwt 1,k ,mt 1 bt 1,k ( LR, mt ),t 1 , t 1 , mt 1 ) r ( t , tk 1SBP t 1, . . . , T 1, a A( LR , SBP ,m ) , LR LSBP , mt M,t LLR , ttKXttSBPLR SBPLRSBPwT ,k ,mT bT ,k ( LR LSBP , mT M,T , T , mT ) µ( T , T , mT ), T LLR , Tk 1wt,k ,mt 0, k 1, . . . , K , t 1, . . . , T , mt .ADP Methods for Optimal ControlJenn Mason, NC State19/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsNumerical ExperimentsIComparison of ADP methodsIComparison of near-optimal policies to internationalguidelinesADP Methods for Optimal ControlJenn Mason, NC State20/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsData SourcesInputTransitions among health statesProbabilities of CHD or StrokeProbability of death from other causesMedication Costs and QALY estimatesADP Methods for Optimal ControlSourceMayo Clinic EMR and DEMSUKPDS Risk EquationsCDC Mortality TablesHealth Services LiteratureJenn Mason, NC State21/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsSimulationIAll models were coded in C/C IThe basis function LP wassolved with CPLEX usingConcert TechnologyIComputation time on a 2.83GHzPC with 8GB of RAM:IIADP Methods for Optimal ControlSimulation: 10 seconds foreach instanceSolution to MDP: 18 minutesJenn Mason, NC State22/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsPatient Perspective: Comparison of ADP MethodsExpected QALYs before a stroke or CHD event: (N 60,000)MalesSimulation of Basis Function PolicySimulation of Aggregate MDP PolicyAggregate MDP ResultsQALYs68.65868.86268.72395% CI(68.559, 68.758)(68.765, 68.959)FemalesSimulation of Basis Function PolicySimulation of Aggregate MDP PolicyAggregate MDP ResultsQALYs73.16173.61072.97495% CI(73.061, 73.261)(73.511, 73.708)ADP Methods for Optimal ControlJenn Mason, NC State23/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsSocietal Perspective: Male Results75.7Maximum QALYs75.6CanadaOptimal Tradeoff CurveUS I75.5EuropeanUnionAustraliaJoint BritishUS IIExpected QALYs (yrs.)75.475.375.275.17574.974.874.7No scounted Medication and Hospitalization Costs ( )ADP Methods for Optimal ControlJenn Mason, NC State24/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsSocietal Perspective: Female Results77.9Maximum QALYs77.8Optimal Tradeoff CurveExpected QALYs (yrs.)77.7EuropeanCanada UnionAustraliaUS I Joint BritishUS II77.677.577.477.3No scounted Medication and Hospitalization Costs ( )ADP Methods for Optimal ControlJenn Mason, NC State25/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsConclusionsIState aggregation is superior to basis functionapproximation of the value functionICoordinated treatment of blood pressure and cholesterol inpatients with diabetes substantially lowers costs andincreases quality-adjusted lifespanADP Methods for Optimal ControlJenn Mason, NC State26/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsFuture WorkIFurther experimentation with basis functions to achievebetter policiesIIdentification of easy-to-implement and near-optimalheuristicsADP Methods for Optimal ControlJenn Mason, NC State27/28
IntroductionOptimal ControlADP ApproachesResults and ConclusionsQuestions?Jennifer E. MasonPhD CandidateEdward P. Fitts Department of Industrial & SystemsEngineeringNorth Carolina State Universityjemason2@ncsu.eduADP Methods for Optimal ControlJenn Mason, NC State28/28
ParametersMedicationACE Inhibitors / ARBsThiazidesβ BlockersCalcium Channel BlockersStatinsFibratesADP Methods for Optimal ControlAnnual Cost 48 48 48 866 212 652QALY Decrement0.0050.0050.0050.0050.0030.003Jenn Mason, NC State29/28
ParametersParameterInitial hospitalization for stroke (C S )Initial hospitalization for CHD (C CHD )Follow-up for stroke (CF S )Value 13,204 18,590 1,664Follow-up for CHD (CF CHD ) 2,576Willingness-to-pay Factor (R0 )Discount Factor (λ) 100,0000.97CHD decrement (d CHD )0.07Stroke decrement (d S )0.21ADP Methods for Optimal ControlSourceNationwide Inpatient Sample 2006Nationwide Inpatient Sample 2006Thom et al. 2006Russell et al. 1998Thom et al. 2006Rascati 2006Gold et al. 1996Clarke et al. 2002Tsevat et al. 1993Clarke et al. 2002Tengs et al. 2001Tengs et al. 2003Jenn Mason, NC State30/28
Approximate Dynamic Programming (ADP) Methods for Optimal Control of Cardiovascular Risk in Patients with Type 2 Diabetes Jennifer Mason PhD Candidate Edward P. Fitts Department of Industrial & Systems Engineering North Carolina State University Raleigh,
operations, operational design, the elements of combat power, and the operations process as described in ADP 3-0 and addressed in ADP 2-0, ADP 3-37, ADP 4-0, ADP 5-0, ADP 6-0, and ADP 6-22. Readers must be familiar with ADP 3-07, ADP 3-28, and ADP 3-90. Leaders must understand how offensive, defensive, and stability or defense support of civil authorities (DSCA) operations complement each .
adp totalsource fl xxix, inc. 1 adp blvd # ms 433, roseland, nj, usa, 07068-1728 xx-xxx6440 adp totalsource i inc 1 adp blvd # ms 433, roseland, nj, usa, 07068-1728 xx-xxx4217 adp totalsource ii inc 1 adp blvd # ms 433, roseland, nj, usa, 07068-1728 xx-xxx8526 adp totalsource nh xxviii, inc. 1 adp blvd # ms 433, roseland, nj, usa, 07068-1728 xx .
adp payroll fees adp-fees 13hmd 5416805 ebiiipay bill pymt verizon electronic pmt-web american express elec remit 110422060208750 ccd debit adp tx/fincl svc adp-tax 517032344452hmd ccd debit adp tx/fincl svc adp-tax 517032344451hmd ccd debit adp tx/
The ADP compiler automatically generates C code for ADP algorithms Our new result is the extension of the ADP compiler, such that it generates CUDA code for Nvidia graphic cards 2 GPU Parallelization of ADP. Dynamic Programming (DP) Dynamic Programming (DP) is useful in Sequence comparison
ADP Trademarks ADP , the ADP logo , ADP A more human resource , The Bridge Sponsored by ADP & Bubble Talk Design , and . Welcome to Essential Time & Attendance Supervisor Timecard Basics for ADP Workforce Now. This training includes U.S. spellings and the date construct of month/day/year. You will
(which are both a form of approximate dynamic programming) used by each approach. The methods are then subjected to rigorous testing using the context of optimizing grid level storage. Key words: multistage stochastic optimization, approximate dynamic programming, energy storage, stochastic dual dynamic programming, Benders decomposition .
which other optimizing methods, such as dynamic programming (DP) and Markov decision process (MDP), are facing due to the high dimensions of HOCP. Keywords: Approximate dynamic programming (ADP), hybrid systems, optimal control 1. INTRODUCTION Recently, developments in the unmanned system
For details, see Auto Scaling User Guide. 1.2 API Calling AS supports Representational State Transfer (REST) APIs, allowing you to call APIs using HTTPS. For details about API calling, see Calling APIs. 1.3 Endpoints Region and endpoint are the request address for calling an API. Endpoints vary depending on services and regions. For the endpoints of the AS service, see Regions and Endpoints. 1 .
