Artificial Intelligence (AI) In Healthcare And Research

potentially reducing demands on human careworkers and family carers.35and prevent hospital admissions.37PUBLIC HEALTHAI apps that monitor and support patientadherence to prescribed medication andtreatment have been trialled with promisingresults, for example, in patients withtuberculosis.36 Other tools, such as Sentrian, useAI to analyse information collected by sensorsworn by patients at home. The aim is to detectsigns of deterioration to enable early interventionAI has the potential to be used to aid earlydetection of infectious disease outbreaksand sources of epidemics, such as watercontamination.38 AI has also been used to predictadverse drug reactions, which are estimated tocause up to 6.5 per cent of hospital admissionsin the UK.39LIMITS OF AIAI depends on digital data, so inconsistenciesin the availability and quality of data restrict thepotential of AI. Also, significant computing poweris required for the analysis of large and complexdata sets. While many are enthusiastic aboutthe possible uses of AI in the NHS, others pointto the practical challenges, such as the fact thatmedical records are not consistently digitisedacross the NHS, and the lack of interoperabilityand standardisation in NHS IT systems, digitalrecord keeping, and data labelling.5 There arequestions about the extent to which patients anddoctors are comfortable with digital sharing ofpersonal health data.40Humans have attributes that AI systems mightnot be able to authentically possess, such ascompassion.41 Clinical practice often involvescomplex judgments and abilities that AI currentlyis unable to replicate, such as contexualknowledge and the ability to read social cues.16There is also debate about whether some humanknowledge is tacit and cannot be taught.42Claims that AI will be able to display autonomyhave been questioned on grounds that this isa property essential to being human and bydefinition cannot be held by a machine.17ETHICAL AND SOCIAL ISSUESMany ethical and social issues raised byAI overlap with those raised by data use;automation; the reliance on technologies morebroadly; and issues that arise with the use ofassistive technologies and ‘telehealth’.The performance of symptom checker appsusing AI, has been questioned. For example, ithas been found that recommendations from appsmight be overly cautious, potentially increasingdemand for uneccessary tests and treatments.16RELIABILITY AND SAFETYTRANSPARENCY AND ACCOUNTABILITYReliability and safety are key issues where AI isused to control equipment, deliver treatment,or make decisions in healthcare. AI could makeerrors and, if an error is difficult to detect orhas knock-on effects, this could have seriousimplications.43 For example, in a 2015 clinical trial,an AI app was used to predict which patientswere likely to develop complications followingpneumonia, and therefore should be hospitalised.This app erroneously instructed doctors to sendhome patients with asthma due to its inability totake contextual information into account.44It can be difficult or impossible to determinethe underlying logic that generates the outputsproduced by AI.45 Some AI is proprietary anddeliberately kept secret, but some are simply toocomplex for a human to understand.46 Machinelearning technologies can be particularly opaquebecause of the way they continuously tweak theirown parameters and rules as they learn.47 Thiscreates problems for validating the outputs of AIsystems, and identifying errors or biases in thedata.Nuffield Council on Bioethics 4

The new EU General Data Protection Regulation(GDPR) states that data subjects have the rightnot to be subject to a decision based solelyon automated processing that produces legalor similarly significant effects. It further statesthat information provided to individuals whendata about them are used should include “theexistence of automated decision-making, (.)meaningful information about the logic involved,as well as the significance and the envisagedconsequences of such processing for the datasubject”.48 However, the scope and content ofthese restrictions - for example, whether and howAI can be intelligible - and how they will applyin the UK, remain uncertain and contested.49Related questions include who is accountable fordecisions made by AI and how anyone harmedby the use of AI can seek redress.3DATA BIAS, FAIRNESS, AND EQUITYAlthough AI applications have the potential toreduce human bias and error, they can alsoreflect and reinforce biases in the data used totrain them.50 Concerns have been raised aboutthe potential of AI to lead to discrimination inways that may be hidden or which may not alignwith legally protected characteristics, such asgender, ethnicity, disability, and age.51 The Houseof Lords Select Committee on AI has cautionedthat datasets used to train AI systems are oftenpoorly representative of the wider populationand, as a result, could make unfair decisions thatreflect wider prejudices in society. The Committeealso found that biases can be embedded in thealgorithms themselves, reflecting the beliefsand prejudices of AI developers.52 Severalcommentators have called for increased diversityamong developers to help address this issue.53The benefits of AI in healthcare might not beevenly distributed. AI might work less well wheredata are scarce or more difficult to collect orrender digitally.54 This could affect people withrare medical conditions, or others who areunderrepresented in clinical trials and researchdata, such as Black, Asian, and minority ethnicpopulations.55TRUSTThe collaboration between DeepMind and theRoyal Free Hospital in London led to publicdebate about commercial companies being givenaccess to patient data.56 Commentators havewarned that there could be a public backlashagainst AI if people feel unable to trust that thetechnologies are being developed in the publicinterest.57At a practical level, both patients and healthcareprofessionals will need to be able to trustAI systems if they are to be implementedsuccessfully in healthcare.58 Clinical trials ofIBM’s Watson Oncology, a tool used in cancerdiagnosis, was reportedly halted in someclinics as doctors outside the US did not haveconfidence in its recommendations, and feltthat the model reflected an American-specificapproach to cancer treatment.59EFFECTS ON PATIENTSAI health apps have the potential to empowerpeople to evaluate their own symptoms andcare for themselves when possible. AI systemsthat aim to support people with chronic healthconditions or disabilities could increase people’ssense of dignity, independence, and quality oflife; and enable people who may otherwise havebeen admitted to care institutions to stay at homefor longer.60 However, concerns have been raisedabout a loss of human contact and increasedsocial isolation if AI technologies are used toreplace staff or family time with patients.61AI systems could have a negative impact onindividual autonomy: for example, if they restrictchoices based on calculations about risk orwhat is in the best interests of the user.62 If AIsystems are used to make a diagnosis or devisea treatment plan, but the healthcare professionalis unable to explain how these were arrived at,this could be seen as restricting the patient’sright to make free, informed decisions abouttheir health.63 Applications that aim to imitate ahuman companion or carer raise the possibilitythat the user will be unable to judge whether theyare communicating with a real person or withtechnology. This could be experienced as a formof deception or fraud.64EFFECTS ON HEALTHCARE PROFESSIONALSHealthcare professionals may feel that theirautonomy and authority is threatened if theirexpertise is challenged by AI.65 The ethicalobligations of healthcare professionals towardsindividual patients might be affected by the use ofAI decision support systems, given these mightBioethics briefing note: Artificial intelligence (AI) in healthcare and research 5

be guided by other priorities or interests, such ascost efficiency or wider public health concerns.66As with many new technologies, the introductionof AI is likely to mean the skills and expertiserequired of healthcare professionals will change.In some areas, AI could enable automation oftasks that have previously been carried out byhumans.2 This could free up health professionalsto spend more time engaging directly withpatients. However, there are concerns that theintroduction of AI systems might be used tojustify the employment of less skilled staff.67 Thiscould be problematic if the technology fails andstaff are not able to recognise errors or carry outnecessary tasks without computer guidance. Arelated concern is that AI could make healthcareprofessionals complacent, and less likely tocheck results and challenge errors.68DATA PRIVACY AND SECURITYAI applications in healthcare make use of datathat many would consider to be sensitive andprivate. These are subject to legal controls.69However, other kinds of data that are notobviously about health status, such as socialmedia activity and internet search history, couldbe used to reveal information about the healthstatus of the user and those around them. TheNuffield Council on Bioethics has suggested thatinitiatives using data that raise privacy concernsshould go beyond compliance with the law totake account of people’s expectations about howtheir data will be used.70AI could be used to detect cyber-attacks andprotect healthcare computer systems. However,there is the potential for AI systems to be hackedto gain access to sensitive data, or spammedwith fake or biased data in ways that might noteasily be detectable.71MALICIOUS USE OF AIWhile AI has the potential to be used for good, itcould also be used for malicious purposes. Forexample, there are fears that AI could be used forcovert surveillance or screening. AI technologiesthat analyse motor behaviour, (such as the waysomeone types on a keyboard), and mobilitypatterns detected by tracking smartphones,could reveal information about a person’s healthwithout their knowledge.72 AI could be used tocarry out cyber-attacks at a lower financial costand on a greater scale.73 This has led to callsfor governments, researchers, and engineers toreflect on the dual use nature of AI and preparefor possible malicious uses of AI technologies.73CHALLENGES FOR GOVERNANCEAI has applications in fields that are subject toregulation, such as data protection, research,and healthcare. However, AI is developing in afast-moving and entrepreneurial manner thatmight challenge these established frameworks.A key question is whether AI should be regulatedas a distinct area, or whether different areas ofregulation should be reviewed with the possibleimpact of AI in mind.5Further challenges include the need to ensurethat the way AI is developed and used istransparent, accountable, and compatible withpublic interest, and balanced with the desire tod

Artificial intelligence (AI) in healthcare and research. RECENT INTEREST IN AI AI is not new, but there have been rapid advances in the field in recent years.This has in part been enabled by developments in computing power and the huge volumes of digital data that are now generated.5 A wide range of applications of AI are now being explored with considerable public and private investment and .