Advantages And Constraints In The Implementation Of .

Advances in Social Science, Education and Humanities Research, volume 257about a contemporary phenomenon or a case set within the realworld context especially when the boundaries betweenphenomenon and context are not clearly evident [13]. Dornyeialso adds that a case study is the study of the particularity andcomplexity of a single case [14]. A case or a phenomenon hereis elaborated by Miles and Huberman might be as a program,an institution, an organization, or a community [15].By employing a case study approach, this research hasseveral advantages: First, as case studies are qualitative innature, thus they have the advantage of qualitative research.Johnson claims that a case study is primarily naturalistic whichrelies on the collection of naturally occurring data [16]. In otherwords, the data of a case study is high in reliability for thenaturalness in terms of behavior, environment and all relatedaspects of the events investigated. According to Yin, adescriptive case study is “allowing an investigation to retain theholistic and meaningful characteristics of real-life events” [13].Zonabend cited in Tellis states that “case study is done bygiving special attention to completeness in observation,reconstruction, and analysis of the cases under study” [17]. Acase study researcher focuses on a single entity as it exists in itsnatural environment [16]. McMillan and Schumacher also statethat a descriptive research using a descriptive mode of inquirysimply describes an existing phenomenon by using numbers tocharacterize individuals or a group [18]. It assesses the natureof existing conditions.Qualitative research methods are used to examine questionsthat can best be answered by verbally describing howparticipants in a study perceive and interpret various aspect oftheir environment [19]. Qualitative research providesopportunities for researchers to study social phenomena inrelation to people’s everyday lives. Through a process of datainterpretation, qualitative research provides information aboutwhat, why and how a phenomenon in a society happens. This isline with what Denzin & Lincoln say that qualitative researchalso involves an interpretive, naturalistic approach to the world[20]. Yin also elaborates that employing qualitative methodprovides researchers with opportunities to represent the viewsand perspectives of the people/participants in a study [13].B. Research SiteThis research was conducted in a vocational school inBandung. Since 2016 the school has become one of adopterschools for Cambridge. In the process of teaching and learningEnglish, the school combines the 2013 curriculum incombination with several Cambridge books, like Interchange,etc. The school has already implemented Computer BasedEnglish Summative Test (CBEST). This summative test forEnglish is delivered via computers conducted in languagelaboratory. The result of the test is gained immediately after thetest since the scoring system is computerized.C. ParticipantsThere are several participants involved in this study: anEnglish teacher and the headmaster. The first participant isEnglish teacher. One English teacher (36 years old) is involvedin this study. She possesses 12 years of teaching experience.The headmaster was also involved to be interviewed to gainmore comprehensive data. The more detailed data about theparticipants are displayed in the table 2 as follows.TABLE II.PROFILES OF PARTICIPANTSParticipantTeacherN1Age36 yearsoldHead master135 2Management ofEducationExperience12 years ofteachingEnglish6yearsofmanagingschoolD. InstrumentThe data for this study were collected by doing observationand interview. Best and Kahn say that interview can be used togather information regarding an individual’s experiences andknowledge; his or her opinions, beliefs, and feelings; anddemographic data [21]. It means that interview is one ofsuitable methods to elicit data related to perceptions, beliefs,knowledge or experiences because interviews allow researchersto explore issues as they emerge in conversation. Interview alsoprovides an opportunity for the participants to express theirunderstanding or ideas in their own words. Burns also adds thatin interview, the participants can use their own words ratherthan using the words or language that has been set out by theresearch [22].The type interview done with teacher and head master issemi structured interview. The topic was about the advantagesand constrains found during the implementation of ComputerBased English Summative Tests (CBEST) in the school. Theaim of using semi structured interview is to give moreopportunities for both the researcher and the participants to dotwo-way communication. The conversation allows theresearcher to explore a wide a range of issues as they emerge inthe participants talk and to identify new themes and ideas [23].According to Burns, a semi-structured interview permitsgreater flexibility and permits a more valid response from theinformant’s perception of reality [22]. Dornyei also says thatin using semi-structured interview, although there is a set ofpre-prepared guiding questions and prompts, the format is openended and the interviewee is encouraged to elaborate on theissues raised in an exploratory manner [14]. The semistructured interview was based on a common set of questionsbut each interview followed up issues as they were raised bythe teacher and head master so that their answers aboutadvantages and constrains during the implementation ofComputer-Based Summative English Tests (CBSET) could beexplored in detailed. The interview was audio-taped andtranscribed for further analysis.E. ProcedureThe research was conducted in three steps. The first stepwas observation. The second step was interview with theteacher. The interview was conducted in the teacher office. Thesecond step was interview with the head master which wasconducted in the head master of the schoolThe next process after collecting the data is to analyze it.Burns says that the purpose of analyzing the data is to find351

Advances in Social Science, Education and Humanities Research, volume 257meaning in the data and this is one by systematically arrangingand presenting the information [22]. It has to be organized sothat comparisons, contrasts, and insights can be made anddemonstrated. Specifically, thematic analysis was used inanalyzing the data. Boyatzis elaborates thematic analysis as astrategy in qualitative research to analyze information in asystematic way in order to make the data understandable. Itorganizes and describes the data in detail according toemergent themes [24]. In doing thematic analysis, theresearcher used the phases by Braun & Clarke as the guidance.Table 3 below displays the phases of thematic analysis process[25].TABLE III.PHASES OF THEMATIC ANALYSIS [25]NO1PHASESFamiliarizing withthe data2Generatingcodes3Searchingthemes4Reviewing themes5Definingandnaming themes6Producing the reportinitialforDESCRIPTIONTranscribing data, reading and rereading the data, noting down initialideasCoding interesting features of the datain a systematic fashion across the entiredata set, collating data relevant to eachcodeCollating codes into potential themes,gathering all data relevant to eachpotential themeChecking the themes work in relationto the coded extracts (level 1) and theentire data set (level 2), generating athematic ‘map’ of the analysisOngoing analysis to refine the specificsof each theme, and overall story theanalysistells,generatingcleardefinitions and names for each themeThe final opportunity for analysis.Selection of vivid, compelling extractexamples, final analysis of selectedextracts, relating back of the analysis tothe research questions and literature,producing a scholarly report of theanalysis.The first step in the process of data analysis was organizingthe data. This step involved transcribing the data gained frominterview with teacher & head master and students’questionnaire. The data then were analyzed and interpreted toidentify the links between the data in the interviews andquestionnaire. After transcribing and translating the data, thenext procedure was labeling the data based on the data sources.The next analytic procedure was repeatedly reading of the textof the transcription of the interview and questionnaire. Thenext step was coding. Creswell states that coding process is tomake sense out of data, divide it into text or image segment,label the segments with codes, examine codes for overlap andredundancy and collapse these codes into broad themes [23]. Inthis study, coding was intended to identify certain ideas in thedata that represented the same meanings. Finally, the data werecategorized into the aspects related to advantages andconstrains in the implementation of Computer Based EnglishSummative Tests (CBEST).III. FINDINGS AND DISCUSSIONThe elaboration about the result of this study will be basedon the two research questions. The first research question isabout the advantages of implementing Computer-BasedEnglish Summative Test (CBEST). And the second researchquestion is about the constraints found during theimplementation of CBEST. The elaboration of the researchresult will be explained according to the nine attributes of CBTby Suvorov and Hegelheimer [2]: directionality, deliveryformat, media density, target skill, scoring mechanism, stakes,purpose, response type, and task type.A. CBEST ImplementationThe type of test direction in this school is linear direction. Itmeans that the test administers the same number of test itemsin the same order to all test takers. The format of the test isComputer-Based Tests (CBT). This offline delivery formatusually uses CD, DVD and Cambridge software applicationinstalled on each computer in the language laboratory. ThisCBT format only uses a single medium, which is audio forlistening test and test based reading test. The result of the test isusually evaluated by computers. The scoring mechanism isusually done by matching test takers’ responses with thecorrect preset responses. Since this CBT is done for summativetests, it usually gives medium impact for students. As Roeversays that medium stakes test can give medium impact to thetest takers, for example: progress test, placement test, etc [10].Even though summative tests are important, however studentscan still take remedial tests if the scores of summative test isbelow the passing grade. Since this is CBT is done assummative tests, so the purpose of the test is related tocurriculum. As Carr says that curriculum-related tests can beused for the purposes of admission to a program, placementinto a specific level of the program, diagnosis of test takers’strengths and weaknesses, assessment of their progress in theprogram, and their achievement of the program’s objectives[11]. There are two types of test responses which can be doneby test takers in the CBT, namely: 1) selected responses (likemultiple choice test) and 2) constructed responses (studentsdevelop their answers). Both of the responses were usedaccording to the skills which will be evaluated. This CBT canbe categorized as selective task type. The examples of selectivetask types are multiple choice questions, yes/no questions. Allelaboration above can be summarized in the table 4 as follows.TABLE IV.NO12345678ATTRIBUTESDirectionalityDelivery FormatMedia DensityTarget SkillScoring MechanismStakesPurposeResponse Type9Task TypeATTRIBUTES OF CBTCATEGORIESLinear testComputer Based Test (CBT)Single mediumSingle language skillcomputer based scoringmedium stakesCurriculum relatedSelectedresponseandconstructed responseSelective task type352

Advances in Social Science, Education and Humanities Research, volume 257B. Advantages and Constraints in Implementing ComputerBased English Summative Tests (CBEST)Based on the result of the interview with English teacherand head master of the school, it was found that theimplementation of Computer-Based English Summative Tests(CBEST) has given several advantages as well as itsconstraints. The elaboration about CBEST and its advantagesand constraints will be based on four educational aspects ofCBT: aspect of economy, aspect of system implementation,and aspect of test administration and design. Each aspect willbe explained in separated sections.C. Aspect of EconomyViewed from the economy aspects, the implementation ofCBT in the research site is more efficient that than Paper PencilBased Test (PBT). In economic perspectives, the advantagesmight include several factors: cost-effective of test in longterm, reduce paper, scoring mechanism and preparing studentsfor a more global economy. In PBT, the school usually neededa huge amount of paper to print the test booklets, while in CBTless paper is needed. So it can be said that CBT can promoteeco-friendly environment. The other advantage is about thescoring system. In CBT, the mechanism of scoring iscomputerized, so the school does not need to ask and payteachers for scoring students’ tests. Therefore, CBT can reducethe cost of scoring mechanism and teachers can be morefocused on preparing teaching materials rather than spendingtime for checking students’ tests. Moreover, as Cater, Rose,Thille and Shaffer claim that students who participate incomputer-based instruction and testing may be more preparedto complete in the global economy [26]. They argue thatstudents in the 21st century need to know how to usetechnology to obtain good jobs, and teaching students tonavigate and successfully complete online tests might helpprepare them for the future. However, in the short run, theimplementation of CALT often cost more than PBT becausethey are costly to develop and implement. Many schools mightnot currently have enough computers and some other relatedfacilities, so the cost for providing these items might need bigbudget. Moreover, the head master says, that the price forbuying a set of Cambridge assessment test is expensive,approximately IDR 800,000 per-student.D. Aspect of System ImplementationViewed from aspect of system implementation, theimplementation of CBT might give several advantages. Thefirst advantages are efficient administration. In Paper BasedTest (PBT) the school needs to print the test booklets, store it inone room before distributing them to every classroom. Thesecond advantage is accurate data collection. In CBT, data willbe more likely accurately collected and easier to store. Also,results and other data can be stored in much less space and it iseasier to retrieve. Moreover, responses generally are accuratelycaptures and scored. In PBT however, students often makesome marks on the answer sheets which can result in inaccuratescoring. On the other hand, constraints were found during theimplementation of CBT. The school actually has threecomputer laboratories to conduct CBT. Each laboratoryconsists of 120 computer sets. However, due to the number ofnew students, so the computer availability is not enough.Therefore, the school needs to manage the CBT into shiftsduring the exam weeks.E. Aspect of Test Administration and DesignViewed from the aspect of test administration and design,the implementation of CBT might give several advantages. Thefirst advantage is immediate results. In PBT, the result ofsummative tests usually takes several days to finish. Teachersas test raters often need time to assess the test and to makedecision about the tests. However, in CBT immediate viewingof scores on screen is provided in CBT to give test takers theinstant feedback. This is in line with Mojarrad et al. thatimmediate feedback, accurate test result reports and thepossibility of printing the basic testing statistics are otheradvantages of using computer in assessment field that enabletest takers take the test at any [27]. The other advantage ofCBT is shorter test duration. In PBT, one subject test usuallytakes 90-120 minutes. However, in CBT test duration can bemade shorter to approximately 60 minutes, because studentsjust need to click the answers without blackening the answersheets like in PBT. The other advantage of CBT is that it ispreferred by students. Some students have used computers toplay games and some of them might receive the instructionthrough computers. Some students might prefer CBT sincethey can customize the assessment based on their personalpreference, like colors on the screen, font types, font sizes, andso on. Due to the possibility of customizing the assessmentbased on personal preferences, some people prefer to take CBTversion of the test. For instance, all students have the option toselect their own background color and font size preference oncomputer screen. Although some students may prefer CBT,others may prefer paper and pencil-based test [26,28]. Sometest takers prefer paper-based testing process because they areaccustomed to taking notes and circling questions and/oranswers for later review. However, constraints during theimplementation of CBT are also found. One of them isstudents’ anxiety in using computers. Some students who arelack of computer ability might find it threatening in doing a testusing computer. Therefore, training for students before theCBT is needed to be conducted so students will be familiarwith the computer and will feel comfortable in doing the test.IV. CONCLUSIONThis present research investigates the advantages andconstraints found during the implementation of ComputerBased English Summative Tests (CBEST) in one vocationalschool in Bandung. All findings which have elaborated abovelead to the conclusion that the two research questions havebeen answered. For the first research question, it can beconcluded that the implementation of CBEST has been inaccordance with the theories, especially regarding to theattributes of CBT. For the second research question, it can beconcluded that the implementation of CBEST has its ownadvantages and constraints, especially in terms of its aspect ofeconomy, aspects of implementation and aspect of testadministration and design.353

Advances in Social Science, Education and Humanities Research, volume 2][13]J. Noijons, “Testing computer assisted language testing: Towards achecklist for CALT,” CALICO Journal, volume 12, no.1, 2012.R. Suvorov and V. Hegelheimer, Computer Assisted Language Testing.In Antony John Kunnan (Eds.) The Companion to LanguageAssessment, 1st Edition,. EdsJohn Wiley & Sons, Inc, 2014.P. Winke and F. Fei, Computer‐assisted language assessment.Encyclopedia of language and education, 2008, pp. 1442-1453.G.J. Ockey, “Developments and challenges in the use of computer-basedtesting for assessing second language ability,” The Modern LanguageJournal, vol. 93, pp. 836–47, 2009.D. Douglas and V. Hegelheimer, “Assessing language using computertechnology,” Annual Review of Applied Linguistics, vol. 27, pp. 115–32, 2007.C. Chapelle, W. Grabe, and M. Berns, Communicative languageproficiency: Definition and implications for TOEFL 2000. TOEFLmonograph series 10. Princeton, NJ: Educational Testing Service, 2000.L. Plakans, “Integrated assessment”, 2009. [Online]. Retrieved . Sawaki, L.J. Stricker, and A.H. Oranje, “Factor structure of theTOEFL Internet-based test,” Language Testing, vol. 26, no. 1, pp. 5–30,2009.J.C.Alderson,Diagnosing foreign language proficiency: The interfacebetween learning and assessment. London, England: ContinuumInternational Publishing, 2005.C. Roever, “Web-based Language Testing,” Language Learning &Technology, vol. 5, no. 2, pp. 84-94, 2001.N. Carr, The Shallows: What the Internet is Doing to

during the implementation of CBEST. The data were collected through observation during the implementation of CBEST and interview with teacher and headmaster. The result of this study reveals that the implementation of CBEST has its own benefits and limitations in relation to aspect of economy, implementation and test administration and test design.