Which statistical test should I use?

What are you trying to find out?

Pick the option that best describes your objective. You do not need to know the name of a statistical test.

• Compare a measurement between groups or time points — next question
• See whether two things are related — next question
• Compare counts or proportions between categories — next question
• Check how accurate a diagnostic test is — next question
• Check agreement or consistency (raters or scale items) — next question
• Predict an outcome from one or more factors — next question

What kind of measurement are you comparing?

A number is something measured on a scale, like a score, weight or blood pressure. A category is a label, like improved / not improved.

• A number (score, measurement, level) — next question
• A category (yes/no, improved/not improved) — next question
• A ranked category (mild / moderate / severe, a Likert score) — next question

What kind of values are the two things you are relating?

Ranked values are ordered categories, such as mild / moderate / severe, where the gaps between them are not equal.

• Both are numbers on a scale — run Pearson correlation
• At least one is a ranked category — run Spearman correlation

Are these counts from the same participants, measured twice?

Choose 'the same participants' if each person appears twice - classified before and after, or judged by two methods. Choose 'different participants' if each person is counted once, in one group or the other.

• The same participants, classified twice — run McNemar
• Different participants in each group — run Chi-square

What does the test being evaluated report?

Choose 'positive or negative' when the test gives a verdict. Choose 'a number' when it gives a value you would have to pick a cut-off for, like a biomarker concentration or a score.

• Positive or negative — run Diagnostic accuracy
• A number you would set a cut-off on — run ROC analysis

What are you checking the consistency of?

Choose 'items' for a questionnaire where several questions measure the same thing. Choose 'raters' when different people (or repeat measurements) rated the same subjects.

• Items within a questionnaire or scale — run Cronbach's alpha
• Different raters or repeated measurements — next question

What kind of outcome are you predicting?

A number is measured on a scale; a yes/no outcome has exactly two possibilities.

• A number (score, measurement) — run Linear regression
• A yes/no outcome — run Logistic regression

How many groups or time points are you comparing?

Count the sets of measurements you want to compare. Comparing a before and an after measurement counts as two.

• Just one (against a known or expected value) — run One-sample t-test
• Two — next question
• Three or more — next question

How many groups or time points are you comparing?

Count the groups you are comparing the ranking across - two treatment arms, or three visits.

• Two — next question
• Three or more — next question

What did each rater record?

Choose 'a measurement' for anything on a scale - a score, a length, a concentration. Choose 'a category' when each rater picked a label, like present / absent or mild / moderate / severe.

• A measurement on a scale — run Intraclass correlation
• A category or label — run Cohen's kappa

Are these measurements from the same participants?

Choose 'the same participants' if each person was measured twice, for example before and after treatment. Choose 'different participants' if you have two separate groups of people.

• The same participants, measured twice — run Paired t-test
• Two different groups of participants — run Independent t-test

Are these measurements from the same participants?

Choose 'the same participants' if everyone was measured at every time point, for example baseline, 3 months and 6 months.

• The same participants, measured repeatedly — run Repeated-measures ANOVA
• Separate groups of participants — run One-way ANOVA

Are these ratings from the same participants?

Choose 'the same participants' if each person was graded twice, for example before and after treatment.

• The same participants, graded twice — run Wilcoxon signed-rank
• Two different groups of participants — run Mann-Whitney U

Are these ratings from the same participants?

Choose 'the same participants' if everyone was graded at every time point, for example baseline, 3 months and 6 months.

• The same participants, graded repeatedly — run Friedman test
• Different groups of participants — run Kruskal-Wallis

Pearson correlation

Both variables are measured on a scale, so Pearson's correlation measures how strongly they move together in a straight line.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Spearman correlation.

Spearman correlation

At least one variable is a ranked category, so Spearman's correlation, which works on ranks rather than raw values, is appropriate.

If it does not: Pearson correlation.

McNemar

Each participant was classified twice, so the two counts are linked and a chi-square test would treat them as though they came from different people. McNemar's test looks only at the participants who changed, which is where the evidence of a change actually is.

Chi-square

Each participant is counted once and the groups are separate, so you are comparing how counts are distributed across categories, which is what a chi-square test of independence assesses.

What has to hold:

• Expected cell counts — Every cell of the table would be expected to hold at least five cases. Chi-square is an approximation that stops holding when the counts get small. Checked with Expected cell counts.

If it does not: Fisher's exact.

Diagnostic accuracy

Comparing a positive/negative test against a gold standard gives sensitivity, specificity, predictive values and accuracy, each reported with a confidence interval.

ROC analysis

The test reports a number, so there is no single sensitivity and specificity to report - there is one pair for every cut-off. An ROC curve shows them all, and the area under it summarises how well the test separates the two groups whatever cut-off you eventually choose. Once you have chosen one, the accuracy panel reports that cut-off.

If it does not: Diagnostic accuracy.

Cronbach's alpha

Cronbach's alpha measures how consistently the items of a scale measure the same underlying construct.

Linear regression

Linear regression estimates how much the outcome changes for each unit change in the predictor, and can adjust for several predictors at once.

Logistic regression

With a yes/no outcome, logistic regression estimates the odds of the outcome occurring, reported as an odds ratio.

One-sample t-test

You are comparing one set of measurements against a known value, so a one-sample t-test is appropriate provided the values are roughly normally distributed.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Wilcoxon signed-rank.

Intraclass correlation

The intraclass correlation coefficient measures how closely repeated measurements or different raters agree on the same subjects when what they record is a number on a scale.

Cohen's kappa

Each rater chose a label rather than measuring a quantity, so agreement is counted rather than correlated. Cohen's kappa measures how much the raters agreed beyond what chance alone would produce, which raw percentage agreement does not.

Paired t-test

The same participants were measured twice, so the two measurements are linked. A paired t-test uses that pairing, which makes it more sensitive than treating the measurements as separate groups.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Wilcoxon signed-rank.

Independent t-test

You are comparing a numerical measurement between two separate groups of participants, which is what an independent samples t-test is designed for.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.
• Equal variance — The groups being compared are about equally spread out. A group that varies far more than the others makes the comparison of their averages unreliable. Checked with Levene's test.

If it does not: Welch's t-test, Mann-Whitney U.

Repeated-measures ANOVA

Each participant was measured at three or more time points, so a repeated-measures ANOVA accounts for the fact that the measurements come from the same people.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.

If it does not: Friedman test.

One-way ANOVA

You are comparing a numerical measurement across three or more separate groups. One-way ANOVA tests them together, avoiding the inflated error rate you would get from running several t-tests.

What has to hold:

• Normality — The values are spread roughly symmetrically around their average, rather than bunched at one end or dominated by a few extreme readings. Checked with Shapiro-Wilk.
• Equal variance — The groups being compared are about equally spread out. A group that varies far more than the others makes the comparison of their averages unreliable. Checked with Levene's test.

If it does not: Kruskal-Wallis.

Wilcoxon signed-rank

Each participant was graded twice on a ranked scale. The Wilcoxon signed-rank test works on the direction and size of each person's change in rank, which is what an ordinal grade can support - the distance between 'mild' and 'moderate' is not a measured quantity.

Mann-Whitney U

You are comparing a ranked grade between two separate groups. Mann-Whitney U compares the ranks themselves rather than the means, which is the only thing an ordinal scale supports.

Friedman test

The same participants were graded on three or more occasions. The Friedman test ranks each participant's own grades across the occasions, so it uses the pairing without assuming the grades are measurements.

Kruskal-Wallis

You are comparing a ranked grade across three or more separate groups. Kruskal-Wallis compares the ranks across all of them at once, avoiding the inflated error rate of several pairwise tests.