Examination Evaluation Of The ACTFL OPI In French, Korean .
Examination Evaluation of the ACTFL OPI in French, Korean, Mandarin for theACE ReviewPrepared for:American Council on the Teaching of Foreign Languages (ACTFL)White Plains, NYPrinciple Investigator: Stephen Cubbellotti, Ph.D.,Independent Psychometric ConsultantACTFL Consultant: Troy Cox, Ph.D.,Brigham Young University
EXECUTIVE SUMMARYThis report documents the American Council on the Teaching of Foreign Languages (ACTFL) OralProficiency Interview (OPI ) from 2012 to 2014 to satisfy a review requirement of the American Councilof Education College Credit Recommendation Service (CREDIT) program. The ACTFL OPI is a liveinterview conducted telephonically or face-to-face between an ACTFL Certified OPI Tester and theindividual whose language proficiency is being assessed.ACTFL and LTI have an extensive collection of resources available publically that document the rigor ofdefining language competency as well as the precision in their assessments. All documentation cited ispublically available and citations for these resources are given in the bibliography at the end of thisdocument. The reliability information section is the only section which contains uniquely generatedstatistics for the purposes of this study. An outline of the results can be found below.Given the ordinal nature of the ACTFL proficiency scale and ACTFL OPI scores, inter-rater reliabilitywas measured by the Spearman’s R correlation, which is a coefficient of reliability appropriate for ordinaldata. Inter-rater agreement was measured by the extent to which ratings exhibited absolute (i.e., exact)and/or adjacent (i.e., /- one level) agreement. The combination of Spearman’s R and absolute/adjacentagreement results provides sufficient information about reliability.Comparisons of ACTFL OPI inter-rater reliability and agreement were made across three languages:French, Korean, and Mandarin. Comparisons were also made across language categories (i.e., languagedifficulty) and interview years (i.e., 2012 to 2014 in this sample). For inter-rater agreement, raterconcordance was further investigated by major proficiency level and sub-level.The ACTFL OPI exceeded the minimum inter-rater reliability and agreement standards. Further, thefindings are fairly consistent with results from Surface, Dierdorff, and Poncheri (2006), indicating theACTFL OPI process yields relatively stable reliability results over time.Overall, the findings support the reliability of the ACTFL OPI as an assessment of speaking proficiency.Areas for continued improvement include increasing rater agreement within the Advanced level and theNovice High-Intermediate Low border. Findings are presented in more detail in the report.The structure of this document is outlined to address several areas including: general test information,item/test content development, reliability information, validity information, scaling and item responsetheory procedures, validity of computer administration, and cut-score information.ACTFL Confidential/Proprietary Page 2
Table of ContentsEXECUTIVE SUMMARY . 2Table of Contents . 3General Test Information . 5Rationale and Purpose of the test . 5Name(s) and institutional affiliations of the principle author(s) or consultant(s) . 6Types of scores reported to examinees . 6Directions for scoring and procedures and keys . 6Item/Test Content Development . 7Specifications that define the domain(s) of content, skills, and abilities that the test samples . 7Statement of test's emphasis on each of the content, skills, and ability areas . 8Rationale for the kinds of tasks (items) that make up the test . 9Information about the Adequacy of the items on the test as a sample from the domain(s) . 9Information on the currency and representativeness of the test's items . 9Description of the item sensitivity panel review . 9Whether and/or how the items pre-tested (field tested) before inclusion in the final form . 10Item analysis results (e.g. item difficulty, discrimination, item fit statistics, correlation withexternal criteria . 10Reliability Information. 10Table 1 Concordance Table for French OPI from 2012 to 2014 . 11Table 2 Concordance Table for Korean OPI from 2012 to 2014 . 11Table 3 Concordance Table for Mandarin OPI from 2012 to 2014 . 11Internal consistency reliability . 12Table 4 Spearman’s Correlations by Language from 2012-2014. 12Table 5 Spearman’s Correlations by Year . 12Evidence for equivalence of forms of the test . 13Scorer reliability for extended response items . 13Table 6 Absolute/Adjacent Agreement by Language from 2012-2014 . 14Table 7 Absolute/Adjacent Agreement by Language and Year . 14Table 8 Absolute/Adjacent Agreement by Language and Sublevel Proficiency from 20122014 . 14Errors of classification percentage for the minimum score for granting college credit (cutscore) . 16Validity Information . 16Content-related validity . 16Criterion-related validity . 16Construct validity (if appropriate) . 17Possible test bias of the total test score . 17Evidence that time limits are appropriate and that the exam is not unduly speeded . 18Provisions for standardizing administration of the examination . 18Provisions for exam security . 18Scaling and Item Response Theory Procedures . 19Types of IRT scaling model(s) used . 19Evidence of the fit of the model(s) used . 19Evidence that new items/tests fit the current scale used . 19ACTFL Confidential/Proprietary Page 3
Validity of Computer Administration . 19Size of the operational test item pool for test . 19This is not applicable as this is not a computer-delivered test. . 19Cut-score information . 19Rationale for the particular cut-score recommended . 19Evidence for the reasonableness and appropriateness of the cut-score recommended . 20Procedures recommended to users for establishing their own cut scores (e.g. granting collegecredit) . 21Bibliography . 22ACTFL Confidential/Proprietary Page 4
General Test InformationRationale and Purpose of the testThe ACTFL OPI is a live interview conducted telephonically or face-to-face between an ACTFLCertified OPI Tester and the individual whose language proficiency is being assessed. The interviewlasts between 20 and 30 minutes depending on the proficiency level of the test taker.The primary goal of the OPI is the efficient elicitation of a ratable sample. To be ratable, a speechsample must clearly demonstrate the highest sustained level of performance of the speaker (known as the“floor”) and the level at which the speaker can no longer sustain the performance (known as the“ceiling”), over a variety of topics. A ratable sample is elicited through a series of personalized questionsthat adhere strictly to a standardized elicitation protocol. While the OPI may resemble a conversationbetween the tester and the test taker, in fact, the tester follows strict elicitation protocol and structures theinterview to respect that protocol.The “floor” and “ceiling” are determined by the test taker’s ability to meet the assessment criteria of themajor levels of the ACTFL Proficiency Guidelines and ACTFL Rating Scale. Each major level is definedas a confluence of function (task), text type, content and accuracy expectations that grow exponentially asthe scale progresses. Instead of defining oral language proficiency as a single, unitary construct, eachmajor level is defined as its own construct with the subsumption principle that as the scale progresses, therelationship between the levels is hierarchal. A ratable sample is one in which the speaker demonstratesa base level or “floor” of sustained performance of the tasks (functions) for that level and a “ceiling” levelat which the examinee cannot sustain performance of the tasks of the next higher level. Hence, eachOPI Interview contains the performance of tasks from at least two contiguous major levels.The difference between the sublevels, then, is based on the quality and quantity of the examinee’slanguage when s/he is engaged in at-level tasks. The low sublevel is indicative of a speaker who justbarely demonstrates competence when performing the tasks for the major level. The mid sublevelindicates that the speaker fulfills all the requirements of the major level with quantity and quality oflanguage across the assessment criteria. There is no doubt that the examinee can perform the functions ofthat major level; the quality and quantity of language is much more substantial than that of speakers at thelow sublevel. The high sublevel rating is different in that, not only does it indicate a speaker’s robustability to meet the criteria for the major level, but provides information related to what happens to thespeaker’s language when s/he is attempting to meet the expectations for the next higher (adjacent) majorlevel. A rating at the high sublevel is indicative of performance at the next higher level most of the time,that is to say, the speaker is unable to sustain all the criteria all the time.The OPI assesses language proficiency in terms of the ability to use spoken language effectively andappropriately in real-life situations. It does not address when, where, why, or the way in which a speakerhas acquired his/her language. The OPI is not an achievement test assessing a speaker’s acquisition ofspecific aspects of course and curriculum content, nor is it tied to any specific method of instruction. TheOPI does not compare one individual’s performance to others, but each individual performance to theassessment criteria.ACTFL Confidential/Proprietary Page 5
Name(s) and institutional affiliations of the principle author(s) or consultant(s)No authorship has ever been ascribed to the ACTFL OPI . The OPI was originally based on theSpeaking Test of the Foreign Service Institute (FSI) created in the mid-1950s by Claudia Wilds of FSI inconsultation with John B. Carroll of Harvard University. The FSI Speaking test was then adopted by theInteragency Language Roundtable (ILR) for use at the Central Intelligence Agency (CIA) and theDefense Language Institute (DLI). These speaking tests were designed to elicit speech samples that wouldalign with the government’s Interagency Language Roundtable (ILR) proficiency scale. Instead ofcreating a set of detailed test specifications with specific item specifications, a stringent interviewprotocol was created that would elicit the speech that would demonstrate what was being assessed.In 1982, the government’s OPI was adapted for use outside of the government context. The authors ofthe first ACTFL Manual with its subsequent descriptions of the ACTFL proficiency scale were Dr. PardeeLowe, Jr. and Dr. Judith E. Liskin-Gasparro. Pardee Lowe, Jr. (Ph.D., University of California, Berkeley)was former chair of the ILR Testing Committee and Chief of Testing at the Central Intelligence Agency'sLanguage School and consulted widely on language testing. As a member of the ACTFL GuidelinesProject, he focused on the guidelines' commensurability with the government’s scales so that a nationalstandard might evolve. Since the original training, experienced OPI testers have been recruited toconduct training and to ensure the protocols practiced by current testers retain their alignment with theACTFL Proficiency Guidelines.Types of scores reported to examineesExaminee scores are reported as the major level and sublevel according to the ACTFL ProficiencyGuidelines 2012 - Speaking. While the ACTFL Proficiency Guidelines are comprised of five major levelsof proficiency – Novice, Intermediate, Advanced, Superior, and Distinguished – the current exam onlytests through Superior. Together these levels form a hierarchy in which each level subsumes all lowerlevels. The major levels of Advanced, Intermediate, and Novice are divided into High, Mid, and Lowsublevels. The description of each major level is representative of a specific range of abilities.Directions for scoring and procedures and keysA 1Certified ACTFL tester elicits a sample of speech by asking questions that target the functions of thetest-taker’s floor level (the level at which the speaker is able to sustain all of the criteria for the level) andceiling level (the level at which the speaker is no longer able to sustain the criteria for the level) across avariety of topics. The speech sample is digitally recorded and stored on a secure Internet-based archive.Patterns of strength and weakness in accomplishing the assigned tasks are established by the tester. Thespeech is first placed within a major range and then matched to the sub-level description in the ACTFLProficiency Guidelines 2012 – Speaking. A first rating is assigned by the tester after which the sample isthen independently second rated by a second certified tester who is able to access the data base. The tworatings must agree exactly. Any rating discrepancy is blindly arbitrated by a third rater and an officialACTFL rating is assigned when two ratings agree exactly.For a more in depth discussion of the rating process, please refer to the section on ScorerReliability for Extended Response Items1ACTFL Confidential/Proprietary Page 6
ACTFL tests are integrative tests, i.e., they address a number of abilities simultaneously and look at themfrom a global perspective rather than from the point of view of the presence or absence of any givenlinguistic feature.Linguistic components are viewed from the wider perspective of how they contribute to the overallsample. The test taker is evaluated through the lens of proficiency. Though holistically rated, there arefour major categories of assessment criteria on which ratings are focused: the global tasks/functionsperformed with the language, the social contexts and content areas in which the language can be used, theaccuracy features which define how well the speaker performs the task pertinent to those contexts andcontent areas, and the oral text type (from individual words to extended discourse) produced. Theassessment criteria used to evaluate speaking are summarized in the chart below:Item/Test Content DevelopmentSpecifications that define the domain(s) of content, skills, and abilities that the testsamplesThe ACTFL OPI is an interactive and adaptive interview protocol that results in a unique and ratablesample of speech. No two interviews are exactly the same. The interview is interactive in that thequestions elicited by the tester are based on the responses that the test taker provides. The interview isadaptive in that content areas are based on the interests and experiences of the examinee, and the majorlevels that are targeted are based on the linguistic range demonstrated by the examinee. There are requireditem types used to target each level of proficiency. For example, the request for the telling of a story frombeginning to end is an effective item “type” or request “type” when eliciting a past narration at theAdvanced level. Testers are trained in the types of requests that are most effective for elicitation at allACTFL Confidential/Proprietary Page 7
levels. Each certified tester is, in effect, an item writer, creating the prompts that elicit demonstration ofthe major functions and other assessment criteria according to the ACTFL Scale.Following the “Warm-Up” phase, during which a preliminary range of ability of the test taker isdetermined, the tester seeks evidence of the “floor” through a series of “level checks” (questions thattarget the functions of the floor level). Level checks are followed by “probes” (questions that target thefunctions of the next higher level). The interview continues in an iterative process of alternating “levelchecks” and “probes”, thereby clearly demonstrating the test taker’s strengths and weaknesses across twocontiguous major levels. Once the tester is satisfied that a ratable sample has been produced, a “WindDown” phase brings closure to the interview. The standardized elicitation protocol of the OPI can beseen in the chart below:Statement of test's emphasis on each of the content, skills, and ability areasThe tested content, skills and ability areas are based on the Assessment Criteria for Speaking andthe descriptions contained in the ACTFL Proficiency Guidelines - Speaking. The ACTFL OPI measures how well a person spontaneously speaks language during in live interpersonalcommunication dealing with practical, social, and professional topics
Examination Evaluation of the ACTFL OPI in French, Korean, Mandarin for the ACE Review Prepared for: American Council on the Teaching of Foreign Languages (ACTFL) . The major levels of Advanced, Intermediate, and Novice are divided into High, Mid, and Low sublevels. The description of e
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
Chính Văn.- Còn đức Thế tôn thì tuệ giác cực kỳ trong sạch 8: hiện hành bất nhị 9, đạt đến vô tướng 10, đứng vào chỗ đứng của các đức Thế tôn 11, thể hiện tính bình đẳng của các Ngài, đến chỗ không còn chướng ngại 12, giáo pháp không thể khuynh đảo, tâm thức không bị cản trở, cái được
Proficiency Guidelines 2012 - Writing. The ACTFL WPT consists of four tasks dealing with practical, social, and professional topics encountered in informal and formal contexts. The writer is presented with tasks representing a limited range of proficiency levels from Novice through Superior, as described in the . ACTFL
DEPARTMENT OF BOTANY Telangana University Dichpally, Nizamabad -503322 (A State University Established under the Act No. 28 of 2006, A.P. Recognized by UGC under 2(f) and 12 (B) of UGC Act 1956)
