Session Overview

• Designing Effective Questionnaires

  • From ideas to items

  • Question & response types

  • Example items

• Designing Effective Interviews

  • From ideas to interviews

  • Interview schedules

  • Successful probes & follow-ups

  • Piloting

Introduction

• In psychology, we often want to measure things we cannot directly observe

• Within quantitative methods, measurement refers to the way we record behaviours, feelings, attitudes, & characteristics in a quantifiable sense

• We can measure variables like height & weight, but we can also measure 'unquantifiable' constructs like 'happiness' or 'lecture-induced boredom'

• We can make attempts to measure these feelings through observation...

  • I could count how many times you smile during a conversation, or how many times you yawn in the next hour

• But often we rely on our participants' abilities to self-report their behaviours, feelings, attitudes, & characteristics instead

From Ideas to Items

• We most often ask participants to self-report through questionnaires, BUT someone has to create them at some point... (your task for the lab report)

• When designing any questionnaire, the first task is identifying and defining what topic(s) we want to be covered in the survey

• The aim here is to break down big ideas into smaller concepts that can be measured with specific items in a questionnaire

• For our session today, let's say that our topic is 'lecture-induced boredom' 😴

  • With the rationale that in the future I can administer this questionnaire after every class to see what teaching techniques and subjects are the least boring...

• The next step is to brainstorm what thoughts, feelings, behaviours, or characteristics define our construct

From Ideas to Items: Define

So, what even is 'lecture-induced boredom'?

Question Types

• So now we have an idea of what 'lecture-induced boredom' is, we can design some questions to measure it...

• For our next step, we need to decide what type of questions we're going to ask:

• Open Questions

  • Empty text boxes

• Closed Questions

  • Yes/no
  • Multiple choice
  • Ranking scales
  • Likert scales

Question Types

• So now we have an idea of what 'lecture-induced boredom' is, we can design some questions to measure it...

• For our next step, we need to decide what type of questions we're going to ask:

• Open Questions

  • Empty text boxes

• Closed Questions

  • Yes/no
  • Multiple choice
  • Ranking scales
  • Likert scales

Likert Scales

• Usually 5 or 7 points covering the two extremes with a neutral option in the middle e.g.,

  • Not very often to very often

  • Very untrue of me to very true of me

  • Not at all important to extremely important

  • Extremely unlikely to extremely likely

  • Strongly disagree to strongly agree

  • Never true to always true

From Ideas to Items

• Now we've decided on a Likert scale we can come up with our items!

  • Items in this context, refers to the different statements we want to include in our questionnaire

• As a starting point, it can be helpful to look at how our construct has been measured previously, or how it's been measured in similar contexts

• We can use those examples as inspiration for coming up with our own items (but don't steal!!)

• Looking at existing measures can also help us identify what gaps there are in previous work that our questionnaire can fill

My Attempts

• I had a go at coming up with my own items in the next few slides...

• I used the Boredom Proneness Scale by Farmer & Sunberg (1986) as inspiration...

• Using what you already know about questionnaire design, I want you to have a go at telling me what's right or wrong with them...

My Attempts

My Attempts

My Attempts

My Attempts

My Attempts

My Attempts

My Attempts

Some Final Q-Tips

• Use statements not questions

• Make sure your response options fit the wording of the item

• Be specific, ask one thing per question

• Avoid double-negatives & double-barrelled questions

• Use neutral language - don't present one response as being more desirable

• Avoid jargon

• Consider your sample, are there any factors that might impact their understanding of your items?

• Pilot your questionnaire to ensure it makes sense to others

Qualitative Methods

• With qualitative methods we are usually trying to understand someone's personal experiences

• We're more concerned with meaning and understanding subjective experiences

• There are many different types of qualitative research, and a lot of them involve using interviews as a method of data collection

• 3 different types of interview:

  • unstructured
  • semi-structured
  • structured

• Could involve individuals OR groups

From Ideas to Interviews

• The process of coming up with questions for interviews is quite similar to questionnaires...

• You firstly need to identify the research topic or area of interest

• For unstructured interviews the research topic can be relatively general, for structured interviews it needs to be more focused

• For more structured interviews, you would then create an interview schedule (or an interview guide) that contains the questions you plan to ask all of your participant(s)

  • For more sensitive topics you might also create a distress protocol based on the questions in your interview schedule (we'll talk more about ethics in interviewing next week!)

From Ideas to Interviews

• To help you come up with questions, think about what type of information you would like to obtain: opinions, attitudes, knowledge, beliefs, behaviours, sensory, experiences, feelings, demographics...

• Create a bank of questions that cover each of these domains for your research topic, focusing on the past, the present, and the future

  • e.g. "Thinking back to when you were a child, how did you feel about transitioning to high school? (feelings in the past)

• Aim for open questions that elicit story-telling - remember the goal is to understand your participants' experiences in depth

• Test them out - think about how you might answer them yourself/go through the questions with a friend

Let's give it a go!

If we were interested in R anxiety what interview questions could we ask?

Swaying from the Schedule

• Sometimes you might need to ask questions that are not in your interview schedule

• Participants could offer up interesting points that you hadn't originally considered when creating your interview schedule, but now want to explore further

• Or it could be that your participant isn't very forthcoming, or something they've said is inconsistent/unclear

• In these cases, you can sway from the interview schedule to probe or follow-up on prior responses

Successful Probes & Follow-ups

• Avoid directly asking 'why?' - it can come off as confrontational, instead, try using 'how'

• Pay attention to your participant and actively listen

  • Otherwise you might miss important opportunities to probe!

• When you create your interview schedule, include some answer-based paths to take to help you prepare follow-ups and probes in advance

  • I.e., if the participant says x, then I'll ask them y

• Be flexible when creating your interview schedule, sometimes things you don't anticipate might come up

Piloting

• It's easier to clarify what you mean by a question during an interview because you're present and involved in the process (vs a questionnaire...)

• But it's much better to avoid any misunderstanding from the start - participants may feel uncomfortable asking for clarity

• You should pilot your interview questions (and questionnaires too) before recruiting participants for the main study

• Some questions might turn out to be easily misunderstood or actually irrelevant to the research question

• After piloting, you can revise any problematic questions to be more effective

Best Practice

• Ask one question at a time, avoiding questions that elicit short answers

• It's important to build rapport so start with simple questions to ease your participant into the interview

  • Choose questions that your participant won't find too difficult to answer, they're likely to feel a little nervous, & might need some time to warm up!

• When moving between topics, tell your participant and give them the opportunity to add anything before you go onto the next question

• Keep difficult or potentially sensitive questions towards the end of the interview (when you both feel more comfortable with the interview process)

• Close the interview with an opportunity for participants to add any more information ("is there anything else you’d like to add before we end?")

• Tailor the language and pace of the interview to your sample

  • Consider any factors (such as age, culture, language etc.,) that might affect how participants understand and respond to your questions

Summary

• We heavily rely on self-report measures in psychology, and so we need to design these measures to be valid and reliable

  • To get good data, we need to ask good questions!!

• The process of creating questions (for surveys OR interviews) generally starts with defining our research topic or area of interest

• We then want to design questions that look at the different thoughts, feelings, behaviours, characteristics, opinions, attitudes, & experiences related to our research topic

• We should pilot any questionnaires or interview schedules before administering them to ensure there aren't any errors in understanding, and to make sure our questions are actually relevant to our topic, making any adjustments as necessary

• We can then evaluate how 'good' our questions are with statistics, which we'll be covering in the next few weeks! Woohoo! 😁

That's all - happy questioning!