Questionnaires In Second Language Research: Construction .

Preface to the Second EditionIn the Introduction to the first edition of Questionnaires in second languageresearch, I argued that in spite of the wide application of questionnairesin applied linguistics, there did not seem to be sufficient awareness inthe profession about the theory of questionnaire design. The positivereception of the book confirmed that it successfully catered to the need fora relatively non-technical and accessible text that describes systematicallyhow to construct, administer, and process questionnaires. So why write asecond edition?The reason for this new edition is not that the material in the firstedition has become outdated—research methods texts usually have amuch longer shelf life than books describing research results. Rather,over the past seven years I have thought of several ways of how the bookcould be improved by adding extra material and by filling certain gaps. Themost significant change in this revised edition involves adding a wholenew chapter to the book—Chapter 5, “Illustration: Developing a motivation questionnaire”—to provide a detailed, illustrative analysis of how anactual scientific research instrument was developed from scratch followingthe theoretical guidelines. This chapter was co-authored by my Ph.D.student, Tatsuya Taguchi, who has been directing a major test designprogram at the University of Nottingham under my guidance. Tatsuya hasalso helped me to update the references and improve several other parts ofthe material, particularly the addition of a section on translating questionnaire items in Chapter 2 and the discussion of how to collect survey dataon the Internet in Chapter 3. We also added the full form of the Japaneseinstrument, along with Chinese and Iranian versions, in the Appendices.xi

xii Preface to the Second EditionBesides these more substantial changes, I have made numerous smalleradditions, and improved the style and accuracy of the text in many places.All in all, the first edition has been given a thorough facelift! One thing hasnot changed, though: I still believe that conducting questionnaire surveyscan be an exciting and rewarding activity, and I do hope that readerswill find in this book all the technical information that they need to be ableto obtain valid and reliable results. Have fun!Zoltán Dörnyei

IntroductionOne of the most common methods of data collection in second language(L2) research is to use questionnaires of various kinds. The popularity ofquestionnaires is due to the fact that they are easy to construct, extremelyversatile, and uniquely capable of gathering a large amount of informationquickly in a form that is readily processable. Indeed, the frequency ofuse of self-completed questionnaires as a research tool in the L2 field issurpassed only by that of language proficiency tests.In spite of the wide application of questionnaires in L2 research, theredoes not seem to be sufficient awareness in the profession about thetheory of questionnaire design and processing. The usual—and in mostcases false—perception is that anybody with a bit of common sense canconstruct a good questionnaire. This situation resembles somewhat the“pre-scientific” phase of language testing (i.e., the period before the 1950s)when language tests were used without paying enough attention to theirpsychometric qualities, and every language teacher was, by definition,assumed to be capable of devising and grading tests and exams withoutany special training. It is my impression that many questionnaire usersare unaware of the fact that there is considerable relevant knowledgeand experience accumulated in various branches of the social sciences(e.g., psychometrics, social psychology, sociology). This is why it is alltoo common to find studies which start out with exciting research questions but are flawed by a badly designed or inadequately processedquestionnaire.xiii

xiv IntroductionIn one sentence . . .“The essential point is that good research cannot be built on poorlycollected data . . .”(Gillham, 2008, p. 1)This book is intended to be practical in nature. During the past 20 years Ihave found questionnaire theory to be very helpful in my own research. Idesigned my first questionnaire in the mid-1980s for my Ph.D. work and,because my specialization area—the study of L2 motivation—is veryclosely linked to the use of questionnaires, I have since then taken part as aprincipal researcher, participant, or supervisor in numerous studies surveying over 20,000 language learners. The idea to share my experience in theuse of questionnaires with a broader audience occurred to me when I wasworking on the research section of a book on motivation (Dörnyei, 2001),and, thanks to the encouragement I received from the Series Editor, SusanGass, right from the beginning, the initial idea eventually led to this book.Although questionnaire design and, more generally, survey research,have a substantial literature in the social sciences, this has not been sufficiently reflected in L2 methodology texts. With the emphasis typicallyplaced on research methods and statistical procedures in them, there wassimply not enough room for discussing specific research instruments(with the sole exception of language tests), and the issue of “questionnaires” has usually been summarized in a maximum of three to four pages.It was therefore a real surprise that, while already working on this book,I learned about another book in the making on a related topic: J. D.Brown’s (2001) Using surveys in language programs. As it happened, thetwo books are largely complementary, with few overlaps. I was fortunate tohave J. D.’s manuscript in my hands when preparing the final draft of thisbook (thanks once again, J. D.!) and I will refer it at times for a moredetailed discussion of certain topics.The structure of the book is straightforward. After an initial chapterthat discusses the nature, the merits, and the shortcomings of questionnaires, separate chapters cover the construction and the administration ofthe questionnaire, as well as the processing of questionnaire data. Chapter5 offers a detailed illustration of the theories by describing how we havedeveloped a research instrument for the purpose of surveying languagelearning motivation in Japan. This questionnaire, which can be found inAppendix B, has been used successfully since then, and our research teamhas also produced Chinese and Iranian versions of it (see Taguchi, Magid,& Papi, 2009); all the items of the three instruments are listed in AppendixA. The book is concluded by a detailed checklist that summarizes the mainprinciples and recommendations.

CHAPTER1Questionnaires in SecondLanguage ResearchAsking questions is one of the most natural ways of gathering informationand, indeed, as soon as babies have mastered the basics of their mothertongue they launch into a continuous flow of questions, and keep goingthroughout the rest of their lives. Some people such as reporters actuallymake a living of this activity and survey/polling organizations can basehighly successful businesses on it.Because the essence of scientific research is trying to find answers toquestions in a systematic manner, it is no wonder that the questionnaire hasbecome one of the most popular research instruments applied in the socialsciences. Questionnaires are certainly the most often employed data collection devices in statistical work, with the most well-known questionnairetype—the census—being the flagship of every national statistical office.The main strength of questionnaires is the ease of their construction. Inan age of computers and sophisticated word processing software it is possible to draw up something that looks respectable in a few hours. After all,as Gillham (2008) reminds us, we all know what questionnaires looklike: Hardly a week goes by without some coming our way. Ironically, thestrength of questionnaires is at the same time also their main weakness.People appear to take it for granted that everybody with reasonable intelligence can put together a questionnaire that works. Unfortunately, this isnot true: Just like in everyday life, where not every question elicits the rightanswer, it is all too common in scientific investigations to come acrossquestionnaires that fail. In fact, I believe that most questionnaires appliedin L2 research are somewhat ad hoc instruments, and questionnaires with1

2 Questionnaires in Second Language Researchsufficient (and well-documented) psychometric reliability and validity arenot that easy to come by in our field. This is of course no accident: In spiteof the growing methodological awareness that has characterized appliedlinguistics over the past two decades, the practice of questionnaire design/use has remained largely uninformed by the principles of survey researchaccumulated in the social sciences. I sometimes wonder what proportion of questionnaire constructors are actually aware that such principlesexist . . .Not indeed . . .“The world is full of well-meaning people who believe that everyonewho can write plain English and has a modicum of common sense canproduce a good questionnaire. This book is not for them.”(Oppenheim, 1992, p. 1)As already mentioned in the Introduction, my interest in questionnairesis pragmatic and practice-driven. I use them all the time and I would likethe measures obtained by them to meet high research standards. Havingfallen into many of the existing pitfalls several times, I intend for this bookto offer concrete suggestions on how to use questionnaires to best effectand how to save ourselves a lot of trouble. Drawing on my own experienceand a review of the literature, I will summarize the main principles ofconstructing and administering questionnaires, and outline the key issuesin processing and reporting questionnaire data.I would like to emphasize right at the onset that this is a “questionnairebook,” which means that I will not go into much detail about issues thatgo beyond the immediate scope of the subject; for example, I will notelaborate on topics such as overall survey design, statistical procedures,or qualitative data analysis. Readers who are interested in these areas mayrefer to my recent overview of Research methods in applied linguistics(Dörnyei, 2007), in which these topics are extensively discussed. In the“Further reading” section below I have also listed a number of good summaries of questionnaire theory that I have found particularly useful inthe past.Further ReadingThere is no shortage of books on questionnaires; many relevant and usefulworks have been written on the topic in such diverse disciplines as psychology, measurement theory, statistics, sociology, educational studies, andmarket research. In the L2 field, J. D. Brown (2001) provides a comprehensive account of survey research (which uses questionnaires as one of the

Questionnaires in Second Language Research 3main data-gathering instruments), offering a detailed account of how toprocess questionnaire data either statistically or qualitatively. In the fieldof psychological measurement, two companion volumes by Aiken (1996,1997) provide up-to-date overviews of questionnaires, inventories, ratingscales, and checklists. A new edition of DeVellis’s work (2003) conciselyexplains the theoretical and technical aspects of scale development. Of themany books specifically focusing on questionnaire design I would liketo highlight three: Oppenheim’s (1992) summary is the revised version ofhis classic work from 1966, and Sudman and Bradburn’s (1983) monograph is also a seminal volume in the area. Finally, Gillham’s (2008) slimmonograph is refreshing, with its readable and entertaining style.1.1 What Are “Questionnaires” and What Do They Measure?Although the term “questionnaire” is one that most of us are familiarwith, it is not a straightforward task to provide a precise definition for it.To start with, the term is partly a misnomer because many questionnairesdo not contain any, or many, real questions that end with a question mark.Indeed, questionnaires are often referred to under different names, such as“inventories,” “forms,” “opinionnaires,” “tests,” “batteries,” “checklists,”“scales,” “surveys,” “schedules,” “studies,” “profiles,” “indexes/indicators,”or even simply “sheets” (Aiken, 1997).Second, the general rubric of “questionnaire” has been used byresearchers in at least two broad senses:(a) Interview schedules, like the ones used in opinion polls, when someone actually conducts a live interview with the respondent, readingout a set of fixed questions and marking the respondent’s answerson an answer sheet.(b) Self-administered pencil-and-paper questionnaires, like the “consumer surveys” that we often find in our mailbox or the short formswe are asked to fill in when, for example, checking out of a hotel toevaluate the services.In this book—in accordance with Brown’s (2001) definition below—Iwill concentrate on the second type only; that is, on the self-completed,written questionnaire that respondents fill in by themselves. More specifically, the focus will be on questionnaires employed as research instrumentsfor measurement purposes to collect reliable and valid data.A Definition for “Questionnaires”“Questionnaires are any written instruments that present respondentswith a series of questions or statements to which they are to react

4 Questionnaires in Second Language Researcheither by writing out their answers or selecting from among existinganswers.”(Brown, 2001, p. 6)1.1.1 What a Questionnaire Is NotTests Are Not QuestionnairesMost scholars know that tests and questionnaires are assessment tools of acompletely different nature, but because some novice researchers mightfind written, self-completed (or self-report) questionnaires and writtentests rather similar, let me highlight the main difference between them. A“test” takes a sample of the respondent’s behavior/knowledge and, on thebasis of this sample, inferences are made about the degree of the development of the individual’s more general underlying competence/abilities/skills (e.g., overall L2 proficiency). Thus, a test measures how well someone can do something. In contrast, questionnaires do not have good orbad answers; they ask for information about the respondents (or “informants”) in a non-evaluative manner, without gauging their performanceagainst a set of criteria or against the performance of a norm group. Thus,although some commercially available questionnaires are actually called“tests,” these are not tests in the same sense as achievement or aptitudetests.“Production Questionnaires” (DCTs) Are Not QuestionnairesThe term “production questionnaire” is a relatively new name for a popularinstrument—traditionally referred to as a DCT or “discourse completiontask”—that has been the most commonly used elicitation technique in thefield of interlanguage pragmatics (see Bardovi-Harlig, 1999; Johnston,Kasper, & Ross, 1998). Although several versions exist, the common featureof production questionnaires is that they require the informant to producesome sort of authentic language data as a response to situational prompts.For example:Rushing to get to class on time, you run round the corner and bump intoone of your fellow students who was waiting there, almost knockinghim down.You:The student: Never mind, no damage done.(Johnston et al., 1998, p. 176)It is clear that these “questionnaires” are not questionnaires in the samepsychometric sense as the instruments discussed in this book. They arewritten, structured language elicitation instruments and, as such, they

Questionnaires in Second Language Research 5sample the respondent’s competence in performing certain tasks, whichmakes them similar to language tests.1.1.2 What Do Questionnaires Measure?Broadly speaking, questionnaires can yield three types of data about therespondent: factual, behavioral, and attitudinal.1. Factual questions (also called “classification” questions or “subjectdescriptors”) are used to find out about who the respondents are.They typically cover demographic characteristics (e.g., age, gender,and race), residential location, marital and socioeconomic status,level of education, religion, occupation, as well as any other background information that may be relevant to interpreting the findingsof the survey. Such additional data in L2 studies often include factsabout the learners’ language learning history, amount of time spentin an L2 environment, level of parents’ L2 proficiency, or the L2coursebook used.2. Behavioral questions are used to find out what the respondents aredoing or have done in the past. They typically ask about people’sactions, lifestyles, habits, and personal history. Perhaps the mostwell-known questions of this type in L2 studies are the items inlanguage learning strategy inventories that ask about the frequencyof the use of a particular strategy in the past.3. Attitudinal questions are used to find out what people think. This is abroad category that concerns attitudes, opinions, beliefs, interests, andvalues. These five interrelated terms are not always distinguished ordefined very clearly in the literature. Attitudes concern evaluative responses to a particular target (e.g.,people, institution, situation). They are deeply embedded in thehuman mind, and are very often not the product of rationaldeliberation of facts—they can be rooted back in our past ormodeled by certain significant people around us. For this reason,they are rather pervasive and resistant to change. Opinions are just as subjective as attitudes, but they are perceivedas being more factually based and more changeable. Peopleare always aware of their opinions but they may not be fullyconscious of their attitudes (Aiken, 1996). Beliefs have a stronger factual support than opinions and oftenconcern the question as to whether something is true, false, or“right.” Interests are preferences for particular activities. Values, on the one hand, concern preferences for “life goals” and

6 Questionnaires in Second Language Research“ways of life” (e.g., Christian values); on the other hand, they arealso used to describe the utility, importance, or worth attachedto particular activities, concepts, or objects (e.g., instrumental/utilitarian value of L2 proficiency).1.2 Using Questionnaires: Pros and Cons1.2.1 AdvantagesThe main attraction of questionnaires is their unprecedented efficiency interms of (a) researcher time, (b) researcher effort, and (c) financialresources. By administering a questionnaire to a group of people, one cancollect a huge amount of information in less than an hour, and the personal investment required will be a fraction of what would have beenneeded for, say, interviewing the same number of people. Furthermore, ifthe questionnaire is well constructed, processing the data can also be fastand relatively straightforward, especially by using some modern computersoftware. These cost–benefit considerations are very important, particularly for all those who are doing research in addition to having a full-timejob (Gillham, 2008).Cost-effectiveness is not the only advantage of questionnaires. They arealso very versatile, which means that they can be used successfully with avariety of people in a variety of situations targeting a variety of topics. Infact, as Bryman (2008

Larson-Hall A Guide to Doing Statistics in Second Language Research Using SPSS (2009) Dörnyei/Taguchi Questionnaires in Second Language Research: Con- struction, Administration, and Processing, Second Edition (2010) Of Related Interest: Gass Input, Interaction, and the Second Language Learner (1997) Gass/Sorace/Selinker Second Language Learning Data Analysis, Second