Analysis Plans:
NeveR gonna let you down
Dr Danielle Evans
18 Oct 2024
Overview
Why plan my analyses?
- As best practice in research and data analysis
- To dodge bad decisions & help your future self 🥰
- To avoid HorroR stoRies!
How to plan my analyses?
- Defining your research question & formulating hypotheses
- Operationalising variables
- Mapping research questions to data & identifying appropriate analyses
Workshop activity
Top Tip! Hover over keywords for definitions! Go on, try it now, I dare you...
So what is an ✨Analysis Plan✨?
An outline
A recipe
A plan of action
A detailed and structured document that outlines the methods, procedures, and steps you will follow to analyse data to answer your research question(s)
- AKA a roadmap for how you intend to approach your data analysis in a systematic and organised manner
Essentially, your best friend when it comes to conducting your analysis!
But, why???
It's best practice within science and research to always plan your analyses before you start data collection
- But even if you're using secondary data sources, you should still plan your analyses!
Planning in advance helps you to avoid making bad design, analytical, or statistical decisions:
A lot of the (rash) decisions you make will seem sensible at the time and can come back to haunt you later! 👻
& the beauty of R is that if you plan your analysis in advance, you could even write all the code you need to prep and analyse your data before you even have any!
But, why??? (pt 2)
Your ethics application will include various questions that require decision-making - completing an analysis plan will often mean you've already made those decisions in advance!
Helps you avoid analysis procrastination
When you then go to analyse your data in R, it'll be 100x easier if you've made a plan that you can follow like a recipe!
Your analysis plan is never gonna give you up, its never gonna let you down, its never gonna run around and desert you:
"The analysis plan was really helpful as it made me realise I didn’t have clear research questions and hypotheses (lol) this made me arrange a meeting with my supervisor and we could go through the sheet together which was really helpful. Even more detail would have been great. I referred to it a lot during analysis"
-- Former Student
HorroR StoRies!
Careful planning now can help you avoid any disasters later on!
- Just trust me on this one OK...
Planning now will help you catch issues with your design that would otherwise lead to a new ethics application, or losing very valuable data
- Things like counterbalancing, randomisation, how to operationalise your variables, deciding what to measure, setting up the conditions of repeated/mixed designs correctly
HorroR stoRies are very rare, but the ones I've seen are always preventable design issues and not R
And if that's not enough incentive, an analysis plan must be completed to access the R Help Desk next term!
Fine, I'm convinced, now what?
Download & open the analysis plan document!
Now, the first thing we need to do is identify our research question
Fine, I'm convinced, now what?
Download & open the analysis plan document!
Now, the first thing we need to do is identify our research question
With the people around you, take turns to:
Briefly summarise the background motivation of your study
Clearly state the research question your study will answer
Briefly discuss and ask/answer questions about the study
Write down your research question in 1-2 crystal clear sentences in the first box in the doc!
07:00
Hypotheses
Once we have defined our research question(s), we can start to form hypotheses!
Remember, we want to avoid HARKing - write your plan and stick to it!
Hypotheses
Once we have defined our research question(s), we can start to form hypotheses!
Remember, we want to avoid HARKing - write your plan and stick to it!
With the people around you, take turns to:
Clearly state the hypotheses your study will test
Discuss what your results will look like for your hypotheses to be supported
Write down specific, testable hypotheses in your analysis plan!
05:00
Design
Is your design correlational or is it experimental?
How will you collect your data? (i.e., surveys, secondary data sources, lab experiments)
Is it an independent, repeated measures, or mixed design?
How many groups or conditions will you have?
How will participants be assigned to conditions?
Will you control for any variables? Are there any potential confounds?
Design
Is your design correlational or is it experimental?
How will you collect your data? (i.e., surveys, secondary data sources, lab experiments)
Is it an independent, repeated measures, or mixed design?
How many groups or conditions will you have?
How will participants be assigned to conditions?
Will you control for any variables? Are there any potential confounds?
Complete the design & confound Qs on the plan!
06:00
Measurement
What concepts are you actually measuring? & How? Are they categorical or continuous?
RTs, test scores, questionnaire responses, scales, conditions/groups/levels etc
How are the conditions/groups/levels defined?
For continuous variables, what does a high and low score represent? What's the possible range? What units are they measured in?
Measurement
What concepts are you actually measuring? & How? Are they categorical or continuous?
RTs, test scores, questionnaire responses, scales, conditions/groups/levels etc
How are the conditions/groups/levels defined?
For continuous variables, what does a high and low score represent? What's the possible range? What units are they measured in?
With the people around you, take turns to:
State what variables you'll be measuring as your primary interest, discuss exactly how you'll measure each one
Write it on your plan - the more detail the better!!
10:00
tip: categorical data can be tricky to work with sometimes, if you can measure it continuously, do so! then you can transform it to be categorical later if you need to, but you cant do it the other way round!
Data Manipulation
Will you need to wrangle or transform your data in any way?
Do you need to categorise people into groups depending on their responses/scores?
Do you need to create composite scores for your measures? Will these be means or totals?
Have you looked at the scoring guidelines for any pre-existing measures you're using to find out?
Data Manipulation
Will you need to wrangle or transform your data in any way?
Do you need to categorise people into groups depending on their responses/scores?
Do you need to create composite scores for your measures? Will these be means or totals?
Have you looked at the scoring guidelines for any pre-existing measures you're using to find out?
With the people around you, take turns to:
- Discuss it & write it down!
08:00
Participants
Who will they be? Will you have any specific inclusion/exclusion criteria?
Will you adjust for any participant characteristics? How will you identify real vs inattentive responses, or mistakes?
What information will you need to collect to produce informative participant summaries in your method section?
Participants
Who will they be? Will you have any specific inclusion/exclusion criteria?
Will you adjust for any participant characteristics? How will you identify real vs inattentive responses, or mistakes?
What information will you need to collect to produce informative participant summaries in your method section?
With the people around you, take turns to:
State your participants and inclusion/exclusion criteria
Discuss what an informative participant summary would look like for your study
Write it on your plan!
07:00
Thinking ahead to analyses
It's important to think about how your data map onto your research questions, otherwise you'll end up in a muddle!
So, to answer your research question(s), what do you need to investigate? What relationships or effects are of interest? Will you make any comparisons between groups?
What analyses are appropriate?
Thinking ahead to analyses
It's important to think about how your data map onto your research questions, otherwise you'll end up in a muddle!
So, to answer your research question(s), what do you need to investigate? What relationships or effects are of interest? Will you make any comparisons between groups?
What analyses are appropriate?
Take some time to think, & then write it down:
07:00
What's next?
Fill out the rest of the analysis plan!
- Make as many decisions as possible before you've seen the data
Keep thinking about how your data map onto your research questions
- The process of going from random, messy numbers to answering your research question depends on you!
There are some questions in the plan that you might not immediately know the answer to - and that's OK!!!
Your supervisor can help you to complete it (but I can't.. 😥)
A lot of the decisions you make will be specific to your project, so discuss your analysis options with your supervisor and get their help!
They can help you avoid horroR stoRies!
What's next?
When it comes to using R, we want to pick our battles and make it easier for ourselves
Putting in the work now will hugely benefit your future self!
You could even make your own codebook to make it easier when working with your data!
But as always...
What's next?
When it comes to using R, we want to pick our battles and make it easier for ourselves
Putting in the work now will hugely benefit your future self!
You could even make your own codebook to make it easier when working with your data!
But as always...
Don't panic, you got this!!
