Best Practices In A Digital Age: Artificial Intelligence .

2y ago
58 Views
5 Downloads
1.61 MB
61 Pages
Last View : Today
Last Download : 2m ago
Upload by : Jewel Payne
Transcription

Best Practices in a Digital Age:Artificial Intelligence and LanguageAssessmentArum Perwitasari, Ph.D.Educational Testing Service (ETS) GlobalInternational Association of Teachers of English as a Foreign Language (IATEFL)IATEFL Webinar, 25 March 2021

What we will cover today Artificial Intelligence (AI) AI in Language Assessment- AI-enabled remote proctoring- AI-driven automated scoring Resources for teachers Q&ACopyright (c) 2021 by ETS. All rights reserved. ETS, the ETS logo, E-RATER, GRE, HISET, PRAXIS, PPAT, PROETHICA, PROPELL, SPEECHRATER, TOEFL, TOEFL iBT, TOEFL ITP andWORKFORCE are registered trademarks of ETS. MYBEST is a trademark of ETS. All other trademarks are property of their respective owners. 6323070052

What is AI? The simulation of human intelligencein machines that are programmed tothink like humans and mimic theiractions. These machines are able to learn withexperience and perform human-liketasks.- An intelligent entity created by humans.- Capable of thinking and acting rationallyand humanely.- Capable of performing tasks intelligentlywithout being explicitly instructed.3

AI in Language Assessment AI has made it possible for languagetest providers to conduct onlineassessments with the help of theinternet and computer networks.- AI-enabled remote proctoring- AI-driven automated scoring4

AI-enabledRemote Proctoring

AI-enabled Remote Proctoring A technology to conduct onlineassessment that prevents studentsfrom possible unfair, fraudulentactivities. Combines integrated web camera andAI-assisted facial recognitionalgorithm, and monitoring system.6

AI-enabled Remote Proctoring Monitors every single testadministration from beginning to end,records via video and captures thedesktop screen, images and chat logs. Tracks nonstandard activities such as astudent leaving the room, talking tosomeone else during the test orleaning away from the web camera.7

TOEFL iBT Home Edition Test The TOEFL iBT Home Edition test isconducted through artificialintelligence technology and the use oflive human remote proctoring servicedby ProctorU , the leading proctoringsolution for online testing.8

TOEFL iBT Home Edition Test:The role of the proctor The at home testing solutions by ETSinvolve live, remote human proctors tokeep an eye on students throughoutthe test session, in addition to best-inclass AI technology.9

TOEFL iBT Home Edition Test:The role of the proctor The proctors confirm the test taker’sidentity and scan their homeenvironment before testing begins,flag any suspicious activity andintervene if needed. Proctors’ goal is to detect anywrongdoings during the test sessionand can cancel the test immediately ifthere is any attempt to cheat.10

TOEFL iBT Home Edition vs.TOEFL iBT in test centers TOEFL iBT Home Edition test is thesame TOEFL iBT test taken at a testcenter:- Same content and format- Same on-screen experience- Same features, like MyBest scores, andinstant scoring of Reading and Listening- Same price and payment options- Same score scales and score reports11

It’s the same preparation for thetest If your students are ready for the TOEFL iBTtest, they are ready for the TOEFL iBT HomeEdition test too.Use official TOEFL iBT test prep resources to prepare, including:TOEFL PracticeOnlineThe Official Guide tothe TOEFL iBT TestTOEFL Test Preparation: TheInsider’s Guide online courseTOEFL iBT Free Practice Testets.org/toefl/test-takers/ibt/prepare12

The ONLY differences are .Students takethe test fromhomeStudents usetheir ownequipmentStudents aremonitored onlineby a humanproctor13

Equipment Requirements –ComputerTo access the test online, students need:OSDesktop orlaptop, not atablet ormobile devicePC: Windows operatingsystem, versions 10, 8 or 7Mac : OS X 10.5 or higher(10.13 High Sierra isrecommended)Chrome orFirefox browser14

Equipment Requirements –Computer & SpeakerStudents Need:Internal orexternalmicrophoneNot Allowed:Internal orexternalspeakerStudents cannot usea headset orearphones15

Equipment Requirements –CameraStudents Need:A built-in camera in the computer,or a separate webcam.Students will have to show a 360degree view of the room, includingtheir tabletop surface, before the test16

To get their computer ready,students need to .Download and install the ETSTest BrowserRun the ProctorU equipmentcheckTo fully install, students must runthe file after downloading itThis checks students’ computer,camera, microphone and speaker17

AI-drivenAutomated Scoring

AI-driven AutomatedScoring AI offers performance-specificfeedback, which is not feasible underoperational human scoring. AI can help score the responsesefficiently and reliably, especially fortest programs with large test takervolumes.19

AI-driven AutomatedScoring AI cannot fully replace human scoringof spoken and written responses.Human scores can do better sometasks (e.g., evaluate appropriateness ofa response), which is why the TOEFLiBT test uses the strength of both.20

Scoring constructed-responsesin the TOEFL iBT test:A hybrid approach Scoring for constructed responses inthe TOEFL iBT test combines thestrengths of machine and humanscoring- Human raters score all responses- Machine scoring is gradually added asautomated capabilities mature21

Rater training and scoring22

Scoring constructed-responsesin the TOEFL iBT test:A hybrid approach The e-rater Scoring Engine has beenused in the TOEFL iBT test for almost adecade. The SpeechRater Service has started tocomplement human rating as of August2019. No other major English proficiency testof its kind combines the benefits of AIand human scoring for speaking andwriting.23

Use of automated scoringtechnology for constructedresponse test sections24

The SpeechRater Service is used forassessing speaking

Assessing speaking in theTOEFL iBT test 1 independent task and 3 integratedtasks- speak about familiar topics drawing onpersonal experience and backgroundknowledge- speak about a topic based on inputmaterial on academic course content orcampus life Total time: 17 minutes (45–60seconds/response)26

Assessing speaking in theTOEFL iBT test Test takers’ responses are recordedand rated through a secure onlinenetwork on:- Topic development- Delivery- Language use The final Speaking test score (0–30) isbased on a combination of humanand SpeechRater scores27

Speaking Scoring Independent andintegrated speakingrubric (holistic)- Delivery- Language use- Topic development Ratings are 0–4 fromrubrics- Converted to a scaledscore of 0–3028

Independent speaking task29

Integrated speaking task(reading input)30

Integrated speaking task(listening input)31

Integrated speaking task(prompt)32

The SpeechRater servicescoring engineAdding AI technology to provide thebest in measurement ETS’s SpeechRater service uses artificialintelligence (AI) technology to assessand provide feedback onpronunciation, fluency, vocabulary andgrammar.34

The SpeechRater servicescoring engine The combination of AI and humanraters’ evaluation of content, meaningand language use providesunmatched accuracy and reliability.- Speaking section score now based onmore ratings (8) than before (6).- 4 human ratings 4 SpeechRater ratings.35

The SpeechRater servicescoring engineSpeechRater36

Studies supporting claimsabout the use of the speakingscores Test design and contentrepresentativeness- test development was informed by reviews ofthe English-language skills needed for studyat English-medium institutions of highereducation (Taylor & Angelis, 2008)- groups of experts laid out frameworks for anew test design (Butler et al., 2000; Jamiesonet al., 2008)- teachers’ interviews provided support for thecontent relevance, authenticity andeducational appropriateness of integratedtest tasks (Cumming et al., 2005)37

Studies supporting claimsabout the use of the speakingscores Meaningfulness of test scores- responses to the speaking tasks variedpredictably, according to proficiencylevel (Biber & Gray, 2013; Brown et al.,2005) Predicting performance in real world- scores on speaking tasks have a clearrelationship with real-world criterionmeasures (Brooks & Swain, 2014; Ockeyet al., 2015)38

Studies supporting claimsabout the use of the speakingscores Usefulness and consequences of testscores- speaking scores are useful for the initialscreening of ITAs (Xi, 2007)- reporting a separate speaking score hasprompted greater attention to thedevelopment of speaking skills (Wall &Horak, 2005, 2006, 2011)39

The e-rater Automated Scoringand Feedback Engineis used for assessingwriting

Writing Tasks 2 Writing Tasks 1 Integrated Task- Reading/listening/writing- Short academic reading and listeningmaterial- Time: 20 minutes 1 Independent Task- Response based on personal experienceor opinion- Time: 30 minutes41

Writing Scoring Independent writingrubric (holistic)- Development of ideas- Organization- Quality and accuracy oflanguage used Integrated writing rubric(holistic)- Quality of writing- Completeness andaccuracy of response Ratings are 0–5 fromrubrics- Converted to a scaledscore of 0–3042

Integrated Writing Task Test taker sees a reading (approximately300 words) for 3 minutes Then test taker hears a 2-minute lectureabout the same topic from a differentperspective or with additionalinformation Test taker sees the reading again, then aprompt Test taker has 20 minutes to respond tothe prompt43

Integrated Writing ScoringGuide Integrated writing rubric(holistic); descriptors referto:- Quality of writing- Completeness andaccuracy of response Ratings are 0–5 fromrubrics Raters also work with:- Benchmark responses- Annotations- Key points44

45

46

47

48

49

The e-rater scoring engine The TOEFL iBT test uses the e-raterautomated scoring engine in a limitedand responsible way for the Writingsection- For each writing task: 1 human rater 1automated rating Combines the judgment of humansfor content and meaning with theconsistency of automated scoring forlinguistic features50

The e-rater scoring engine In the e-rater engine, test takerresponses are evaluated in a systemwhere human raters have awardedscores. Identifies features associated withwriting proficiency in academic Englishin test taker essays. The e-rater engine offers a holisticscore for a given response to anintegrated and independent taskbesides real-time diagnostic feedback.51

The features in the e-rater scoring engine Content analysis based on vocabularymeasures Lexical complexity/diction Proportion of grammar errors Proportion of usage errors Proportion of mechanics errors Proportion of style comments Organization and development scores Features rewarding idiomatic phraseology52

Combining AI andHuman Raters

Important takeaways:Combining AI and raters Human raters evaluate content, meaningand language in a holistic manner, whileautomated scoring by the SpeechRater service and the e-rater engine evaluateslinguistic features in an analytic manner.54

Important takeaways:Combining AI and raters The TOEFL iBT test is the only large-scaletest of academic language proficiency testcombining the strengths of automatedscoring machines and human scoring, asall spoken and written responses arerated by both multiple human raters andthe aforementioned automated scoringengines.55

ETS’s commitment to validity,reliability and security ETS is taking every precaution toensure that the test students take fromhome meets the highest standards forvalidity, reliability and security. ETS is and will be committed todeveloping automated scoring systemsto meet these conditions andevaluating test taker responses with thecombination of expertise from humanraters.56

Resources for Teachers

Resources For YouTeachers and Advisors SectionOne location for all your resource needs Propell Workshops Teacher Webinars Advisor Toolkitwww.ets.org/toefl/teachers advisorsVideos Available TOEFL Resource Series for Teachers Research Behind the TOEFL Program How ETS Scores the TOEFL iBT Testwww.ets.org/toefl/teachers advisors/video library58

Free TOEFL test prep foryour students TOEFL Test Preparation: The Insider’sGuide: The TOEFL MOOC is a free selfpaced course designed by the expertswho created the TOEFL test. TOEFL iBT Free Practice Test: A full testwith all 4 sections and real past testquestions, to help them become familiarwith the test format and question types. TOEFL iBT Practice Sets: Free sets ofTOEFL iBT test questions, grouped by testsection, in PDF format. TOEFL iBT Test Prep Planner: An 8-weekpreparation plan with tips and activities tobuild each of the 4 skills.59

Inside the TOEFL Test Video series gives an indepth look at theReading, Listening,Speaking and Writingquestions, including:- Question structure- Scoring criteria- Sample responses- Skill-building tips60

Sign up to receive updates:wwww.ets.org/toefl/communicationsCopyright (c) 2021 by ETS. All rights reserved. ETS, the ETS logo, E-RATER, GRE, HISET, PRAXIS, PPAT, PROETHICA, PROPELL, SPEECHRATER, TOEFL, TOEFL iBT, TOEFL ITP andWORKFORCE are registered trademarks of ETS. MYBEST is a trademark of ETS. All other trademarks are property of their respective owners. 63230700561

Thank you andstay connected!aperwitasari@etsglobal.org

Free TOEFL test prep for your students TOEFL Test Preparation: The Insider’s Guide: The TOEFL MOOC is a free self-paced course designed by the experts who created the TOEFL test. TOEFL iBT Free Practice Test: A full test with all 4 sections and real past test questions,

Related Documents:

Switch and Zoning Best Practices 28-30 2. IP SAN Best Practices 30-32 3. RAID Group Best Practices 32-34 4. HBA Tuning 34-38 5. Hot Sparing Best Practices 38-39 6. Optimizing Cache 39 7. Vault Drive Best Practices 40 8. Virtual Provisioning Best Practices 40-43 9. Drive

The Global Association for Contact Center Best Practices & Networking www.ContactCenterWorld.com THE BEST PRACTICE SERIES Nov 11-15, 2013 Benchmarking, Networking & Best Practices IN THE CONTACT CENTER WORLD TOP RANKING PERFORMERS BEST PRACTICES CONFERENCE & AWARDS WORLD'S BEST LAS VEGAS . Kansas City Call Center

Within each category, the Election Security Best Practices Guide separates the recommendations into two levels according to their criticality to help Election Authorities prioritize the implementation of the practices: (1) Priority Best Practices and (2) Standard Best Practices. Priority Best Practices are urgently critical and form the .

Digital inclusion is defined in various ways and is often used interchangeably with terms such as digital skills, digital participation, digital competence, digital capability, digital engagement and digital literacy (Gann, 2019a). In their guide to digital inclusion for health and social care, NHS Digital (2019) describe digital

VMware ESX Host Best Practices for Citrix XenApp –Provides proven VMware best practices for vSphere hosts running XenApp workloads. Includes guidance in the areas of CPU, memory, storage, and networking. Citrix XenApp on vSphere Best Practices – Deploying Citrix XenApp on vSphere requires that proven best practices for the XenApp application continue to be followed. The focus in this section is on

8 BEST PRACTICES FOR SMALL BUSINESS GIFT CARDS BEST PRACTICES FOR SMALL BUSINESS GIFT CARDS 9. BEST PRACTICES FOR SMALL BUSINESS GIFT CARDS 11 61% . nicer, more elegant carrier, like a gift box or tin that reflects your brand well. 14 BEST PRACTICES FOR SMALL BUSINESS GIFT CARDS BEST PRACTICES FOR SMALL

BEST PRACTICES FoR CRAFT, MEdIA & VISuAL ARTISTS In ALBERTA ALBERTA BEST PRACTICES - InTRoduCTIon 1 oF 2 2 InTRoduCTIon Best Practices are industry standards, or professional guidelines, for specific fields of work. Best Practices for Craft, Media, and Visual Artists facilitate fair, ethical interactions and equitable dealings between artists, and individuals or organizations that engage the .

9 of these Best Practices were rated as no longer effective and are recommended for deletion In addition, Focus Group 1C is proposing 2 new Best Practices to address gaps identified by the Focus Group Recommended modifications to the Best Practices and new Best Practices are included in Section 8.4 of this report. 3 Background