Which statistical test should I use?

What are you trying to find out?

Pick the option that best describes your objective. You do not need to know the name of a statistical test.

• Compare a measurement between groups or time points — next question
• See whether two things are related — next question
• Compare counts or proportions between categories — next question
• Check how accurate a diagnostic test is — next question
• Check agreement or consistency (raters or scale items) — next question
• Predict an outcome from one or more factors — next question
• Compare groups while allowing for more (two factors, a covariate, several outcomes, or repeated visits in groups) — next question
• How long until an event happens (survival, relapse, discharge) — next question
• Follow one measurement over time (a series of readings by date, month or year) — next question
• Find the underlying factors behind a set of questionnaire items — next question
• Weigh the evidence for and against an effect (Bayesian) — next question

What kind of measurement are you comparing?

A number is something measured on a scale, like a score, weight or blood pressure. A category is a label, like improved / not improved.

• A number (score, measurement, level) — next question
• A category (yes/no, improved/not improved) — next question
• A ranked category (mild / moderate / severe, a Likert score) — next question

What kind of values are the two things you are relating?

Ranked values are ordered categories, such as mild / moderate / severe, where the gaps between them are not equal.

• Both are numbers on a scale — run Pearson correlation
• At least one is a ranked category — run Spearman correlation
• One is a yes/no and the other is a number — run Independent t-test

Are these counts from the same participants, measured twice?

Choose 'the same participants' if each person appears twice - classified before and after, or judged by two methods. Choose 'different participants' if each person is counted once, in one group or the other.

• The same participants, classified twice — run McNemar
• Different participants in each group — run Chi-square

What does the test being evaluated report?

Choose 'positive or negative' when the test gives a verdict. Choose 'a number' when it gives a value you would have to pick a cut-off for, like a biomarker concentration or a score.

• Positive or negative — run Diagnostic accuracy
• A number you would set a cut-off on — run ROC analysis

What are you checking the consistency of?

Choose 'items' for a questionnaire where several questions measure the same thing. Choose 'raters' when different people (or repeat measurements) rated the same subjects.

• Items within a questionnaire or scale — run Cronbach's alpha
• Different raters or repeated measurements — next question

What kind of outcome are you predicting?

A number is measured on a scale; a yes/no outcome has exactly two possibilities.

• A number (score, measurement) — run Linear regression
• A yes/no outcome — run Logistic regression
• An ordered category (mild / moderate / severe, a Likert grade) — run Ordinal logistic regression
• A category with three or more possibilities in no order (blood group, type of anaemia) — run Multinomial logistic regression

Which of these describes your comparison?

Choose the one that matches how the study was designed. If none does, go back and choose 'Compare a measurement between groups or time points'.

• Two or more grouping factors at once (for example treatment and sex) — run Factorial ANOVA
• One grouping, allowing for a measured covariate (for example the baseline score or age) — run ANCOVA
• Several related outcomes compared across the groups together — run MANOVA
• The same participants measured at several visits, in one group or several — run Linear mixed model

What do you want to know about the time to the event?

Participants who had not had the event when they were last seen are 'censored'; both methods use them for as long as they were followed.

• How long it takes, and whether it differs between groups — run Kaplan-Meier with log-rank test
• How several factors together change the risk of the event — run Cox regression

What do you want to know about the series?

A series is one measurement taken again and again over time, in order: monthly admissions, daily temperature, yearly incidence. The order matters, so these methods are not for comparing groups. Most of them need every step recorded with no gaps.

• Is it rising or falling over time, and by how much? — run Mann-Kendall trend test with Sen's slope
• Is it steady, or does it wander (is it stationary)? — run Stationarity (ADF and KPSS)
• Is each reading related to the ones before it? — run Autocorrelation (ACF, PACF and Ljung-Box)
• Does it repeat in a yearly, quarterly or weekly cycle, and how strongly? — run Seasonal decomposition
• Did it change when something began (a programme, a policy, a treatment)? — run Interrupted time series (segmented regression)
• Forecast the next values — next question
• Does one series help predict another? — run Granger causality

What do you want to find out about the items?

Choose 'test' when you already know which items belong together (from the scale's manual or an earlier study); choose 'discover' when you do not.

• Reduce many items to a few summary scores — run Principal component analysis
• Discover which underlying factors the items measure — run Exploratory factor analysis
• Test a factor structure you already expect — run Confirmatory factor analysis

What are you weighing the evidence for?

A Bayes factor says how much more likely the data are with an effect than without one, so it can also show evidence that there is no difference, which a p-value cannot.

• A difference between two separate groups — run Bayesian independent t-test
• A change in the same participants, measured twice — run Bayesian paired t-test
• A relationship between two measurements — run Bayesian Pearson correlation

How many groups or time points are you comparing?

Count the sets of measurements you want to compare. Comparing a before and an after measurement counts as two.

• Just one (against a known or expected value) — run One-sample t-test
• Two — next question
• Three or more — next question

How many groups or time points are you comparing?

Count the groups you are comparing the ranking across - two treatment arms, or three visits.

• Two — next question
• Three or more — next question

What did each rater record?

Choose 'a measurement' for anything on a scale - a score, a length, a concentration. Choose 'a category' when each rater picked a label, like present / absent or mild / moderate / severe.

• A measurement on a scale — run Intraclass correlation
• A category or label — run Cohen's kappa

How should the forecast be built?

Both give a forecast with a 95% prediction interval, and both are checked on the data they were fitted to. They differ in what they assume about the pattern.

• From how each reading depends on the ones before it (ARIMA, with differencing and cycles) — run ARIMA forecast
• From a smoothed level, trend and cycle that update as new readings arrive (exponential smoothing) — run Exponential smoothing forecast

Are these measurements from the same participants?

Choose 'the same participants' if each person was measured twice, for example before and after treatment. Choose 'different participants' if you have two separate groups of people.

• The same participants, measured twice — run Paired t-test
• Two different groups of participants — run Independent t-test

Are these measurements from the same participants?

Choose 'the same participants' if everyone was measured at every time point, for example baseline, 3 months and 6 months.

• The same participants, measured repeatedly — run Repeated-measures ANOVA
• Separate groups of participants — run One-way ANOVA

Are these ratings from the same participants?

Choose 'the same participants' if each person was graded twice, for example before and after treatment.

• The same participants, graded twice — run Wilcoxon signed-rank
• Two different groups of participants — run Mann-Whitney U

Are these ratings from the same participants?

Choose 'the same participants' if everyone was graded at every time point, for example baseline, 3 months and 6 months.

• The same participants, graded repeatedly — run Friedman test
• Different groups of participants — run Kruskal-Wallis

Pearson correlation

Both variables are measured on a scale, so Pearson's correlation measures how strongly they move together in a straight line.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Spearman correlation.

Spearman correlation

At least one variable is a ranked category, so Spearman's correlation, which works on ranks rather than raw values, is appropriate.

If it does not: Pearson correlation.

Independent t-test

You are comparing a numerical measurement between two separate groups of participants, which is what an independent samples t-test is designed for.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.
• Equal variance — The groups being compared are about equally spread out. A group that varies far more than the others makes the comparison of their averages unreliable. Checked with Levene's test.

If it does not: Welch's t-test, Mann-Whitney U.

McNemar

Each participant was classified twice, so the two counts are linked and a chi-square test would treat them as though they came from different people. McNemar's test looks only at the participants who changed, which is where the evidence of a change actually is.

Chi-square

Each participant is counted once and the groups are separate, so you are comparing how counts are distributed across categories, which is what a chi-square test of independence assesses.

What has to hold:

• Expected cell counts — Every cell of the table would be expected to hold at least five cases. Chi-square is an approximation that stops holding when the counts get small. Checked with Expected cell counts.

If it does not: Fisher's exact.

Diagnostic accuracy

Comparing a positive/negative test against a gold standard gives sensitivity, specificity, predictive values and accuracy, each reported with a confidence interval.

ROC analysis

The test reports a number, so there is no single sensitivity and specificity to report - there is one pair for every cut-off. An ROC curve shows them all, and the area under it summarises how well the test separates the two groups whatever cut-off you eventually choose. Once you have chosen one, the accuracy panel reports that cut-off.

If it does not: Diagnostic accuracy.

Cronbach's alpha

Cronbach's alpha measures how consistently the items of a scale measure the same underlying construct.

Linear regression

Linear regression estimates how much the outcome changes for each unit change in the predictor, and can adjust for several predictors at once.

Logistic regression

With a yes/no outcome, logistic regression estimates the odds of the outcome occurring, reported as an odds ratio.

Ordinal logistic regression

With an ordered outcome, ordinal logistic regression estimates the odds of being in a higher category for each predictor, using the order without treating the grades as measured numbers.

If it does not: Multinomial logistic regression.

Multinomial logistic regression

With three or more unordered outcome categories, multinomial logistic regression compares each category with a reference category, giving a relative risk ratio for each predictor.

Factorial ANOVA

With two grouping factors, the question a factorial design exists to ask is whether the effect of one depends on the other - the interaction. A factorial ANOVA tests each factor and the interaction together; separate one-way ANOVAs never test the interaction at all.

ANCOVA

ANCOVA compares the groups after allowing for a covariate that also affects the outcome, such as the baseline score. It removes a baseline imbalance and makes the comparison more precise, provided the covariate relates to the outcome the same way in every group - which it checks.

MANOVA

Several related outcomes tested one at a time find a difference somewhere far more often than 5% of the time. MANOVA tests them together, with follow-up ANOVAs for each outcome.

Linear mixed model

The same participants were measured repeatedly, so their measurements are linked. A linear mixed model gives each participant their own baseline (a random intercept), compares groups over time, and keeps the visits a participant did attend when they missed one.

If it does not: Repeated-measures ANOVA.

Kaplan-Meier with log-rank test

Kaplan-Meier curves estimate the proportion still free of the event over time, using each participant for as long as they were followed, and the log-rank test compares the curves between groups.

If it does not: Cox regression.

Cox regression

Cox regression estimates how each factor changes the rate of the event - a hazard ratio - with the others held fixed, using censored participants for as long as they were followed.

If it does not: Kaplan-Meier with log-rank test.

Mann-Kendall trend test with Sen's slope

The Mann-Kendall test asks whether later readings tend to be above earlier ones, without assuming a straight line or normal errors, and Sen's slope gives the size of the trend. It leaves blanks out and does not need equal spacing.

If it does not: Interrupted time series (segmented regression), Seasonal decomposition.

Stationarity (ADF and KPSS)

A series that is stationary has the same mean and spread whenever it is looked at; many methods assume it. The augmented Dickey-Fuller and KPSS tests start from opposite hypotheses and are read together, and the same pair on the differenced series says how many differences are needed.

If it does not: Autocorrelation (ACF, PACF and Ljung-Box), Mann-Kendall trend test with Sen's slope.

Autocorrelation (ACF, PACF and Ljung-Box)

The autocorrelation and partial autocorrelation functions show how each reading relates to the ones before it, and the Ljung-Box test asks whether any of that is more than chance. It shows whether readings can be treated as independent and where a cycle repeats.

If it does not: Stationarity (ADF and KPSS), ARIMA forecast.

Seasonal decomposition

Classical decomposition splits the series into a trend, a repeating seasonal pattern and what is left, and gives the strength of the trend and of the seasonality. It needs the cycle length and two complete cycles.

If it does not: Autocorrelation (ACF, PACF and Ljung-Box), Exponential smoothing forecast.

Interrupted time series (segmented regression)

Segmented regression fits the trend before the intervention and asks whether the level, the slope or both changed when it began, allowing for autocorrelated errors. A before-and-after t-test of the same readings would ignore the trend and the order.

If it does not: Mann-Kendall trend test with Sen's slope.

Granger causality

Granger causality asks whether past values of one series improve the prediction of another beyond its own past, in both directions. It shows predictive precedence, not causation, and needs both series stationary.

If it does not: Autocorrelation (ACF, PACF and Ljung-Box).

Principal component analysis

Principal component analysis replaces many correlated items with a few weighted totals that keep most of their variation. It summarises the items; it does not claim a construct lies underneath them.

If it does not: Exploratory factor analysis.

Exploratory factor analysis

Exploratory factor analysis finds the underlying factors that would explain why the items correlate, without saying in advance which item belongs where, and tests whether that many factors is enough.

If it does not: Principal component analysis.

Confirmatory factor analysis

When you already know which items should measure which factor, confirmatory factor analysis tests that structure against your data and reports how well it fits.

Bayesian independent t-test

The Bayesian independent t-test reports a Bayes factor: how much more likely the data are with a difference between the groups than without one, so it can show evidence of no difference as well as of one.

If it does not: Independent t-test.

Bayesian paired t-test

The Bayesian paired t-test weighs the evidence for and against a change in the same participants, reported as a Bayes factor.

If it does not: Paired t-test.

Bayesian Pearson correlation

The Bayesian correlation weighs the evidence for and against a linear relationship between two measurements, reported as a Bayes factor.

If it does not: Pearson correlation.

One-sample t-test

You are comparing one set of measurements against a known value, so a one-sample t-test is appropriate provided the values are roughly normally distributed.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Wilcoxon signed-rank.

Intraclass correlation

The intraclass correlation coefficient measures how closely repeated measurements or different raters agree on the same subjects when what they record is a number on a scale.

Cohen's kappa

Each rater chose a label rather than measuring a quantity, so agreement is counted rather than correlated. Cohen's kappa measures how much the raters agreed beyond what chance alone would produce, which raw percentage agreement does not.

ARIMA forecast

ARIMA models each reading from the ones before it after differencing, with a seasonal part when there is a cycle, and forecasts with a prediction interval. The residuals are tested for any pattern the model missed.

If it does not: Exponential smoothing forecast, Autocorrelation (ACF, PACF and Ljung-Box).

Exponential smoothing forecast

Exponential smoothing (the ETS family, including Holt-Winters) weights recent readings most and carries a level, an optional trend and an optional seasonal pattern forward. The form is chosen by corrected AIC from those the series allows.

If it does not: ARIMA forecast, Seasonal decomposition.

Paired t-test

The same participants were measured twice, so the two measurements are linked. A paired t-test uses that pairing, which makes it more sensitive than treating the measurements as separate groups.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Wilcoxon signed-rank.

Repeated-measures ANOVA

Each participant was measured at three or more time points, so a repeated-measures ANOVA accounts for the fact that the measurements come from the same people.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Friedman test.

One-way ANOVA

You are comparing a numerical measurement across three or more separate groups. One-way ANOVA tests them together, avoiding the inflated error rate you would get from running several t-tests.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.
• Equal variance — The groups being compared are about equally spread out. A group that varies far more than the others makes the comparison of their averages unreliable. Checked with Levene's test.

If it does not: Kruskal-Wallis.

Wilcoxon signed-rank

Each participant was graded twice on a ranked scale. The Wilcoxon signed-rank test works on the direction and size of each person's change in rank, which is what an ordinal grade can support - the distance between 'mild' and 'moderate' is not a measured quantity.

Mann-Whitney U

You are comparing a ranked grade between two separate groups. Mann-Whitney U compares the ranks themselves rather than the means, which is the only thing an ordinal scale supports.

Friedman test

The same participants were graded on three or more occasions. The Friedman test ranks each participant's own grades across the occasions, so it uses the pairing without assuming the grades are measurements.

Kruskal-Wallis

You are comparing a ranked grade across three or more separate groups. Kruskal-Wallis compares the ranks across all of them at once, avoiding the inflated error rate of several pairwise tests.