Discourse Characteristics Of Writing And Speaking Task .

paraphrasing. Brown, Iwashita, and McNamara (2005) focused on spoken responses butsimilarly considered differences across score levels and independent versus integrated tasks. Thatstudy found weaker patterns of linguistic variation associated with fluency, vocabulary,grammatical accuracy, and complexity.The present project complements these previous studies by focusing on the lexicogrammatical characteristics of examinee responses on the TOEFL iBT, considering a muchlarger inventory of linguistic features than in previous research, and analyzing a larger corpus ofexam responses. Similar to the two studies cited above, though, this study focuses on the primaryconsiderations relevant to the explanation proposition of the TOEFL validity argument: analysisof the discourse characteristics of responses produced across task types, by examinees fromdifferent score levels. Thus, the study investigates three major research questions:1. Do test takers systematically vary the linguistic characteristics of discourse producedin the spoken versus written modes across different task types? If so, how?2. In what ways do exam scores correspond to systematic linguistic differences in thediscourse produced by test takers?3. How does the relationship between linguistic discourse characteristics and score levelvary across the spoken/written modes and/or task types?The first question adopts a register perspective, disregarding proficiency level. The issuehere is the extent to which the texts produced by test takers reflect awareness of the linguisticdifferences across the spoken and written modes and between integrated versus independent tasktypes; that is, have test takers developed proficiency in the appropriate use of linguistic features(e.g., vocabulary and grammar) associated with spoken versus written language, and withintegrated versus independent tasks?The second question concerns the ways in which TOEFL iBT score levels correspond tosystematic linguistic differences in the language produced by test takers. As noted above, theanalytical focus of this study is on the lexical and grammatical characteristics of the discourseproduced by the test taker groups.Finally, the third question brings the first two perspectives together, considering theinteractions of score levels, mode, and task differences as predictors of the patterns of lexicogrammatical variation.2

To address these research questions, this study presents an empirical linguistic analysis ofa corpus of TOEFL iBT exam responses, providing a comprehensive lexico-grammaticaldescription of the discourse of exam responses. As set out in the TOEFL validity argument, thelinguistic characteristics of examinees’ discourse are predicted to vary in systematic ways withtask type, mode, and score level. The investigations reported below are a first step towarddescribing those relationships.In Section 2, we briefly summarize previous research that has described the use of avariety of lexico-grammatical features in the spoken and/or written production of Englishlanguage learners. In Section 3, we introduce the TOEFL iBT context and corpus, followed by adescription of our research design and methods in Section 4. We then present and discuss theresults of our investigations into the lexico-grammatical characteristics of spoken and writtenTOEFL iBT discourse in Section 5, and conclude with a brief summary and discussion ofimplications for the TOEFL iBT in Section 6.2. A Brief Survey of Previous ResearchSeveral previous studies have described linguistic characteristics of the discourseproduced by different learner groups in attempts to document the linguistic changes associatedwith language development and different levels of proficiency. Table 1 surveys many of the mostimportant of these studies. Rather than undertaking an exhaustive survey of previous research,the purposes here are to illustrate the wide range of discourse characteristics that have beeninvestigated in these studies.3

Table 1Features Investigated in Spoken and Written Language Production, as Related to Proficiency and/or L1 (Language 1)CategoryLexicalfeaturesStudyLinguistic featuresGrant & Ginther Lexical specificity (i.e.,(2000)type/token ratios, wordlength), conjuncts, hedges,amplifiers, emphatics,demonstratives,downtonersFindingsAs proficiency increased, lexical specificity increased (i.e., longer andmore varied words were used).Ferris (1994)Higher proficiency writers used more specific lexical classes (e.g.,emphatics, hedges).Word length, speciallexical classesUses of conjuncts, amplifiers, emphatics, demonstratives, anddowntoners increased.Word length was one of the most significant predictors of holisticscores assigned to essays.Lexical variation (i.e.,Lexical variation and holistic scores assigned to compositions weretype/token variation), error- highly correlated.free variation, percentageError-free variation and holistic scores were also highly correlated.of lexical error, lexicaldensityJarvis, Grant,Bikowski, &Ferris (2003)Mean word length,type/token ratio, conjuncts,hedges, amplifiers,emphatics, downtonersCluster analysis revealed that clusters of highly-rated texts varied littlein terms of lexical diversity and use of conjuncts.Jarvis (2002)Lexical diversity(type/token ratios)Results indicated that lexical diversity did contribute to writingquality, but this relationship was dependent on the writer’s L1.4Engber (1995)Laufer & Nation Lexical frequency profiles(1995)based on proportions ofUWL, GSL 1K, GSL 2K,and offlist wordsLexical frequency profiles discriminate between proficiency levelsand correlate well with other measures of vocabulary size with lowerproficiency learners using higher proportion of high frequency wordsand higher proficiency learners using more words from the lessfrequent or offlist words.

CategoryGrammaticaland syntacticfeaturesStudyCumming et al.(2005)Linguistic featuresLexical sophistication(word length, type/tokenratios)FindingsAll proficiency levels tended to use longer words in integrated tasks.Higher proficiency learners had higher type/token ratios.The frequency of several features increased with proficiency:nominalizations, modals, first and third-person pronouns, more variedverb tense uses, passives, subordination, and complementation.Ferris (1994)Verb tenses, pronouns,adverbials, modals,negation, coordination,prepositional phrases,definite article reference,passives, relative clauses,stative forms, coordination,participials, coherencefeaturesHigher proficiency writers produced more of the more difficultsyntactic constructions such as stative forms, participial constructions,relative clauses, and adverbial clauses.Nouns and nominalizations,pronouns, adverbials,prepositions, definitearticles, present tenseverbs, stative verb be,passives, adverbialsubordination, relativeclauses, complementationUsing cluster analysis, Jarvis et al. found that judgments of essayquality depended on how linguistic features were used together ratherthan on the use of individual features.5Grant & Ginther Nouns, nominalizations,(2000)personal pronouns, verbs,modals, adjectives,adverbs, prepositions,articles, subordination,complementation, relativeclauses, adverbialsubordination, passivesJarvis et al.(2003)Higher proficiency writers used more passives, existential there,preposed adverbials, clefts, topicalizations to show “pragmaticsensitivity” and “promote textual coherence” (p. 418).Clusters of highly rated texts could differ in terms of mean wordlength, nouns and nominalizations, prepositions, and present tenseverbs.Highly rated texts varied less in terms of text length and lexicaldiversity.

CategoryRhetoricalstructureStudyCumming et al.(2005)Linguistic featuresSyntactic complexity(clauses per T-unit, wordsper T-unit)FindingsMore proficient learners produced more words per T-unit.The mean number of clauses per T-unit differed across task types, butno difference was found across proficiency level.Wolfe-Quintero, Linguistic complexityInagaki, & Kim (clauses per T-unit,(1998)dependent clause ratio)Surveyed previous empirical research on complexity and languagedevelopment, identifying the most promising lexico-grammaticalcomplexity features.Ortega (2003)Syntactic complexity(especially T-unitmeasures)Surveyed 25 previous studies of syntactic complexity in L2 writing.Hirose (2003)Deductive vs. inductiveorganizational patternsL2 organization scores did not significantly correlate with L1organization scores.6Choice of organizational pattern (deductive or inductive) did notcontribute alone to the evaluation of organization; rather, factors suchas coherence between/within paragraphs also influenced howorganization was evaluated.Kubota (1998)Location of main idea,rhetorical pattern/organizationAbout half of the participants used similar rhetorical patterns in L1and L2 essays.A positive correlation was found between L1 and L2 organization,indicating that writing proficiency in the L2 may be related to writingproficiency in the L1.Little evidence for transfer of rhetorical patterns from L1 to L2.Coffin (2004)Argument structureLower-level learners tend to use arguments composed usingexposition structures rather than a discussion-based argument.Cumming et al.(2005)Quality of argumentstructure, orientations tosource evidenceIn integrated tasks, highly proficient learners often summarized andsynthesized information from source materials, while learners in themidproficiency ranges used more phrases directly from the prompts.

CategoryFormulaiclanguageStudyCortes (2004)Linguistic featuresLexical bundlesFindingsStudent writers rarely used lexical bundles used by professionalwriters.When student writers did use the target bundles, they did not use themin the same way as professional writers.Hyland (2008)Lexical bundlesStudent writers employed a higher proportion of lexical bundles thatoutline research procedures as compared to published writers, whichmay be related to the nature of the student genres as a way ofdisplaying knowledge.Student writers tended to avoid participant-oriented bundles, perhapsdue to influences from the L1 culture and educational experience.7Howarth (1998)Collocational densityAdvanced learners are able to internalize restricted collocation orsemi-idioms, but there are too many less restricted combinations tolearn as unitary items.Altenberg &Granger (2001)Grammatical patterns,meanings, collocations ofmakeWhen compared to native English-speaking student writers, advancedlevel learners underused delexical make and used inappropriatecollocations.Note. L2 Language 2; UWL University Word List, 808 common word families in academic writing; GSL 1K 1,000 mostfrequent words in the General Service List; GSL 2K second 1,000 most frequent words in the General Service List.

As Table 1 shows, previous research has investigated the use of linguistic features at allgrammatical levels associated with English language development. Thus, the features consideredin previous studies include the following: Lexical features (e.g., type/token ratio, average word length, use of academic andgeneral service words) Word classes and general grammatical features (e.g., nouns, nominalizations,adjectives) Grammatical features that specifically relate to linguistic complexity (e.g., relativeclauses, adverbial clauses, average T-unit length, depth of embedding) Rhetorical organization (e.g., move structure of written essays) Formulaic language (e.g., collocational patterns, lexical bundles)It is worth noting that (almost) all lexico-grammatical characteristics of English areuseful indicators of register and communicative task differences (see Biber & Conrad, 2009,especially Chapter 3). By extension, it is likely that these same linguistic features are associatedwith language development and differences in language proficiency. These relationships existbecause lexico-grammatical features are functional and are used to differing extents inassociation with the communicative purposes and production circumstances of differentregisters. For example, writing development entails the productive use of lexico-grammaticalfeatures that are not naturally acquired in speech, including an increased range of vocabulary,increased range of grammatical structures (e.g., nonfinite relative clauses), and increasedcomplexity in noun phrase constructions (especially with phrasal modifiers). Languagedevelopment in speech follows a different progression and is focused more on clausal (ratherthan phrasal) modification and vocabulary diversification. As a result, the linguistic featureslisted in Table 1 represent a relatively comprehensive subset of the possible lexico-grammaticalcharacteristics of English discourse.Beginning in the 1970s, numerous researchers have focused on L2 (Language 2) writingdevelopment with an overt focus on the linguistic structures used in student texts (see, e.g.,Cooper, 1976; Ferris & Politzer, 1981; Flahive & Snow, 1980; Gipps & Ewen, 1974). This trendhas continued to the present time, so that it is common now to find second language researcherswho focus on “measures of fluency, accuracy, and complexity” in second language writing (as inthe title of the 1998 book by Wolfe-Quintero, Inagaki, & Kim). More recent studies include8

Brown et al. (2005), Ellis and Yuan (2004), Larsen-Freeman (2006), and Nelson and Van Meter(2007).Across these decades, when writing development research has focused on the linguisticdescription of student texts, one of the key concerns has been the analysis of grammaticalcomplexity. Most of these studies have adopted a deductive approach, beginning with an a prioridefinition of grammatical complexity as elaborated structures added on to simple phrases andclauses (see, e.g., Purpura, 2004, p. 91; Willis, 2003, p. 192). Specifically, most studies of L2writing development have relied on T-unit-based measures, based on the average length ofstructural units and/or the extent of clausal subordination, assuming that longer units and moresubordination reflect greater complexity. The early reliance on clausal subordination (and T-unitbased measures) is documented by Wolfe-Quintero et al. (1998), and subsequent studies havecontinued this practice (e.g., Ellis & Yuan, 2004; Larsen-Freeman, 2006; Li, 2000; Nelson &Van Meter, 2007; Norrby & Håkansson, 2007). The two previous studies of TOEFL iBT spokenand written responses (Brown et al., 2005; Cumming et al., 2006) have similarly relied heavilyon T-unit based measures for their analyses of syntactic complexity. Ortega (2003) providedstrong confirmation that current research continues to employ these same two measures, basedon a meta-analysis of empirical research on grammatical complexity in college level ESL/EFLwriting. Of the 27 studies included in her survey, 25 studies relied on the mean length of T-unit(MLTU) to measure grammatical complexity, while 11 studies used the related measure ofdependent clauses per T-unit (C/TU). No other measure was used widely across these studies.Biber and Gray (2010) and Biber, Gray, and Poonpon (2011) challenged this pervasivepractice, arguing instead that phrasal embedding is a much more important indicator of advancedwriting development than clausal embedding; these structures function mostly as noun phrasemodifiers, such as attributive adjectives, premodifying nouns, prepositional phrasepostmodifiers, and appositive noun phrase postmodifiers. Based on corpus analysis, these twostudies show that there is no empirical basis for treating all dependent clauses as a singleconstruct reflecting complexity. Rather, different types of dependent clauses are distributed inquite different ways across spoken and written registers, indicating that they represent quitedifferent types of structural complexity. Thus, for the purposes of the present research project,the full range of linguistic features associated with both clausal emb

a continuing program of research related to the TOEFL test. From 1977 to 2005, nearly 100 research and technical reports on the early versions of TOEFL were published. In 1997, a monograph series that laid the groundwork for the development of TOEFL iBT was launched. With the release of TOEFL iBT, a