Statistical Methods 2. Questionnaire Design

community projectencouraging academics to share statistics support resourcesAll stcp resources are released under a Creative Commons licenceStatistical Methods2. Questionnaire nMarshallUniversityofSheffield

SummaryWe will consider:q What is a questionnaire?q Response ratesq Data types and question typesq Question design principlesq Questionnaire layoutq Rating scalesq Questionnaire design ffield

What is a questionnaire?q A collection of questions given to different people inthe same formq Observational or experimental (e.g. observing aclass before and after an intervention, or measuringa variable in experimental and control groups)q The focus may be opinions or factsq Can be administered by paper, phone, email or viaa websiteq Completed by the researcher, subject or a thirdpartyq Questions can be organised into groups (as often ina standard fSheffield

Comparison with other datacollection techniquesPros:q Cost-effective: Large samples can be gathered Greater potential of statistically significant resultsq Can be carried out by different peopleCons:q Honesty of responses may be an issueq Low response rates, especially for email/onlineq How to deal with ‘missing’ / ‘no opinion’ answers?q Free response answers tend to be very �eld

Mode of delivery may affectwhat is possibleq Self-completed or interviewer-led?q Is help/explanation available?q Paper, phone, online or an electronic form viaemail?q If online, questions can be delivered one at atime, with a progress bar to show level ofSheffield

Response ratesq The proportion of questionnairesgiven to your sample that arecompletedq Should be as high as possible inorder to minimise possibility ofbias (you also need to makesure your sample isrepresentative of the population)q Notorious problem with onlineand postal inghamCityUniversityPopulationSam- ‐pleResp- ‐ond- ‐entsReviewer:EllenMarshallUniversityofSheffield

How to maximise your responserateq Try to engage with the respondent as a person, e.g. giveout paper-based questionnaires on a clipboard face-to-facewhile you are waiting or email people who already know you(but watch out for possible bias)q Give your questionnaire a short and meaningful titleq Begin by introducing yourself and the purpose of yourresearchq Keep it as short and succinct as possibleq Offer incentives for responding, if appropriateq Make it look attractive, e.g. use colour and imagesq Make it convenient, e.g. enclose a stamped addressenvelope for a postal yofSheffield

Question types and data typesQuestion typeMultiple choiceData typeAnalysisNominal or Quantitative:ordinalnonparametric onlyMultiple choice scaleScalewith words at the endsQuantitative: parametric and nonparametricMultiple answermultiple choiceMultipleQuantitative: treat eachbinary (e.g. value separately – onlytrue/false) have a fewNumber fieldScaleQuantitativeRankingOrdinalQuantitative: hard toanalyse – avoidText iversityofSheffield

Activity:Question designIn pairs or small groups:q Refer to the Sample Questions handout - thiscontains a series of poorly written questions –some are real!q Look at each question in turnq State one potential problem with each questionq Spend at most 1 minute on each effield

Questions 1-41. Smoking in public places should not be abolished.Yes q No q Not sure q 2. What percentage of your annual salary do you spend onpetrol?3. If students were to receive more discount, where wouldbe most appropriate?Restaurants q Supermarkets q Bars q 4. How much money do you spend per week on alcohol, ifany? 0 q 10-20 q 20-30 q 30-40 q 40-50 q 50 q ersityReviewer:EllenMarshallUniversityofSheffield

Question design principlesq Question wording should be clear andunambiguous: avoid double negatives – Q1q Questions should not be too difficult toanswer – Q2q Responses should exhaust all possibleoptions or have a residual option such as'Other' – Q3q Responses should be mutually exclusive ifonly one option can be chosen – Q4 (e.g. eld

Questions 5-95. Are you thinking of booking a family holiday for this summer?Yesq Noq Not sureq 6. Wouldn’t you agree that the government’s decision to re-classifyCannabis as a Class B drug is the right one?Yes q No q Not sure q 7. How would you rate our service?123Excellent Very good Good4Undecided5Poor8. Have you ever stolen stationery from work for home use?Yes q No q Not sure q 9. Do you have or have you ever had a health condition which haslasted for over 6 months and has limited the kind of work you can do?Yes q No q Not sure q ersityReviewer:EllenMarshallUniversityofSheffield

Question design principles (2)Avoid:q Possible incorrect responses (boxes next towrong response) – Q5q Leading questions – Q6q Biased scaling – Q7q “Prestige bias” – may not be answeredtruthfully – Q8q Overly complicated questions – d

Questions 10-1110. Rank the following televisionprogrammes in the order thatyou watch them, indicating themost watched channel with a 1and the least watched channelwith a 10.11. Indicate if you agree ordisagree: A chemical used infood production that has anegligible cancer risk should beprohibited even though it delaysspoilage, prevents rancidity, orprolongs storage ITV4Channel 4FiveSky OneVirgin 1Sky Sports 1Sky Movies PremierReviewer:EllenMarshallUniversityofSheffield

Question design principles (3)q Avoid asking to rank too many items (andnumbers don’t make sense!) – Q10q In fact, it is better to avoid ranking questionsaltogether as they are difficult to analyse – askabout each item individually then rank themean response valuesq Ensure a questionnaire is the best vehicle forcollecting your data – ld

Question design principles (4)q Make the first few questions easy, e.g.demographicsq Then have your most important questionsq Leave open/complex questions to the endq Avoid very similar questions, unless you areconstructing scalesq Try to minimise bias through:Ø Socially acceptable responsesØ Respondent pleasing the questionnaire setter mixpositively and negatively worded questionsØ Too many or too few answer categories use 6 or 7(see ��eld

Demographic questionsq Good to start with theseq Examples are:Ø Ø Ø Ø Ø Ø AgeGenderLevel of educationSocioeconomic backgroundEmploymentRegionq Useful for comparing groupsq Only ask what you eld

Questionnaire layoutGood layout can increase response rate andaccuracy:q Make the design eye catchingq Be clear and succinctq Group questions logicallyq Sequence the questions sensiblyq Use consistent wording and answer formatsq Be ��eld

Activity: Questionnaireevaluationq 10 minutesq In pairs or small groups, evaluate one of thequestionnaires provided according to theprinciples given on the checksheet:Ø Ø Ø Ø National Student SurveyParent questionnaireCoffee consumption questionnaireDiabetes questionnaireq Report back your findings to the main �eld

Rating scalesq A rating scale is an ordered scale of responsesq They are commonly used in questionnairesq They typically have 5 or 7 values:Ø Less values reduce data qualityØ More values are hard to process mentally (people end up justchoosing bottom, middle or top values)q There is an argument to have an even number ofvalues (e.g. 6) to force respondents to make a choice,although most people tend to choose more positiveanswers anywayq There is also an argument to word some of thequestions negatively to stop respondents from givingresponses without effield

Example 1Indicate the extent to which you agree or disagreewith the statement:“Staff on this workshop are good at explaining things”q q q q q ields ordinal data:limited analysis ffield

Example 2Indicate the extent to which you agree or disagreewith the statement:“Staff on this workshop are good at explaining s richer scale data:more analysis ffield

Likert scalesq Many people refer to rating scales like the previousexamples as Likert scalesq Strictly speaking, a Likert scale is a series of numberedrating scale questions which are used to measureresponses to a series of carefully developed 'attitudestatements’, e.g. via a focus groupq The Likert scale value is then the overall sum of thenumbered ratings (once negatively worded items havebeen reversed). However this should not be donewithout first testing the scale for reliability (notrecommended for Level 4 or 5 students)q For more information eld

Questionnaire design process:Don't let the tail wag the dog!q Do not start designing your questionnaire bysimply writing questionsq Start by first considering what your researchquestions areq Use this to determine what data you need andthe analysis you will doq Test your questionnaire in a pilot ��eld

Carry out a pilot surveyq Always pilot your questionnaire with a fewfriends before carrying out your full surveyq This will allow you to:Ø Ø Ø Ø Ø Check the integrity of your questionsIdentify missing response categoriesCheck for spelling and grammatical mistakesAssess any difficultiesCheck the analysis you plan to d

RecapWe have looked at:q What is a questionnaireq Maximising response ratesq Data types and question typesq Question design principlesq Questionnaire layoutq Rating scalesq Questionnaire design ffield

BibliogrpahyFink, A. (2013) How to Conduct Surveys: A step-by-step guide, 5th ed.,London: SAGE.Frazer, L. and Lawley, M. (2000) Questionnaire Design andAdministration: A practical guide, New York: Wiley.Loughborough University (n. d.) Questionnaire Design. [pdf] Available f [Accessed 6/01/14].Oppenheim, A. N. (2000) Questionnaire Design, Interviewing andAttitude Measurement, New ed., London: Continuum.Rea, L. M. and Parker, R. A. (2005) Designing and Conducting SurveyResearch, 3rd ed., San Francisco, CA: Jossey-Bass.Trochim, W. M. K. (2006) Likert Scaling. Available .php ersityofSheffield

What is a questionnaire ! Maximising response rates ! Data types and question types ! Question design principles ! Questionnaire layout ! Rating scales ! Questionnaire design process Peter&Samuels& Birmingham&City&University& Reviewer:&Ellen&Marshall& www.statstutor.ac.uk& University&of&Sheffield&