How to Analyse a Pretest-Posttest Design with ANCOVA

Four ways to analyse pretest-posttest data — which one is right?

The same dataset can be analysed four different ways; they ask different questions and can give different answers. The most common mistake in thesis drafts is picking one of the two weakest options:

ANCOVA, step by step

Order matters: the assumption checks come before the main analysis, because a violated assumption changes the method rather than just adding a footnote.

1. 01

   Choose the covariate on theoretical grounds

   In a pretest-posttest design the covariate is naturally the pretest score. If you add further covariates (age, baseline motivation, disease duration), each must meet two conditions: it has to be meaningfully related to the outcome, and it must be UNAFFECTED by the intervention — that is, measured before it. Using a variable measured after the intervention as a covariate removes part of the treatment effect from the model.

2. 02

   Verify the covariate-outcome relationship is linear

   ANCOVA assumes a linear relationship between the covariate and the posttest. A scatterplot drawn separately for each group is the fastest way to see this. If the relationship is curved or absent, the covariate adds nothing — rather than forcing ANCOVA, the design needs rethinking.

3. 03

   Test the homogeneity of regression slopes

   Temporarily add a group × covariate interaction term to the model. This term should be NON-significant (p > .05). If it is, drop the interaction and run the actual ANCOVA. If it is significant, the effect of the intervention varies by pretest level; reporting a single "group difference" would then be misleading and the analysis moves into a moderation framework.

4. 04

   Check the remaining assumptions on the residuals

   Normality is assessed on the model residuals, not on the raw scores; homogeneity of variance is tested with Levene's test; outliers are judged individually in the covariate-outcome plane. In small groups a single outlying observation can visibly shift the adjusted means.

5. 05

   Run the model and read the group effect

   The model is posttest ~ group + pretest. The F value on the group row tells you whether the groups differ significantly once the pretest is controlled. A significant covariate row is expected and welcome: it shows the pretest predicts the posttest, confirming the covariate was well chosen.

6. 06

   Report the adjusted means

   The real output of ANCOVA is the adjusted (estimated marginal) means — the posttest values expected if all groups had the same pretest average. Best practice is to table the raw means with standard deviations alongside the adjusted means with standard errors; reviewers want to see the gap between the two.

7. 07

   Add effect size and pairwise comparisons

   Partial eta squared (ηp²) is the standard effect size for ANCOVA. With three or more groups, which pairs differ is determined from the adjusted means using pairwise comparisons with a correction such as Bonferroni; uncorrected multiple comparisons inflate the Type I error rate.

ANCOVA assumptions: how to check them, what to do when they fail

Listing the assumptions isn't enough; every row needs a "what if it doesn't hold?" answer. That is exactly what gets asked in a defence:

How to interpret partial eta squared (ηp²)

"It came out significant" is not a finding on its own; the size of the difference belongs in the report too. The commonly used reference bands are:

Common mistakes

• Comparing posttest scores with an independent t-test when pretest data exists — if baselines differ, the finding is biased outright

• Running separate paired t-tests in each group and concluding "it rose significantly in the treatment group but not in the control, so the method works"; one test being significant does not show that the DIFFERENCE between groups is significant

• Never testing the homogeneity of regression slopes — it is ANCOVA's most critical precondition and the question most often asked in a defence

• Running ANCOVA but tabling the raw posttest means; what belongs in the table are the adjusted means with their standard errors

• Using a variable measured after the intervention as a covariate — part of the treatment effect leaks into it, systematically shrinking the estimate

• Adding five or six covariates on a "more control is better" logic; each covariate costs a degree of freedom, and highly intercorrelated covariates destabilise the model

• Checking normality on the raw scores; in ANCOVA normality is assessed on the residuals

• Presenting ANCOVA on non-randomised groups with markedly different baselines as "we equated the groups" — statistical adjustment does not replace random assignment

• Running uncorrected multiple pairwise comparisons with three or more groups (Bonferroni or Šidák is needed)

• Never reporting the effect size (ηp²), which APA and most journals require

• Forcing a three-measurement design (pretest, posttest, follow-up) into a single ANCOVA; a mixed model fits that design better

How to write it up in the results section

An APA-formatted ANCOVA result carries three pieces of information at once: the assumption check, the test statistic for the group effect, and the adjusted means. A usable template: "The homogeneity of regression slopes assumption was satisfied, F(1, 76) = 0.84, p = .362. With pretest scores controlled as a covariate, there was a significant difference between groups on posttest scores, F(1, 77) = 12.45, p = .001, ηp² = .139. Adjusted means were 74.26 (SE = 1.12) for the treatment group and 67.84 (SE = 1.11) for the control group." Note that the degrees of freedom drop by one because of the covariate — the detail most often missed in ANCOVA write-ups.

Putting both the raw means with standard deviations and the adjusted means with standard errors in the table lets the reader see how much the adjustment moved things; a good results section shows those columns side by side. For figures, either a line plot of pretest and posttest means by group or a bar chart of adjusted means with confidence intervals will do. We prepare these tables and figures ready to paste into your thesis or manuscript, formatted to your journal's or institute's template.

If you're working with intact groups: an honest framing

In most education, nursing and field research, groups cannot be randomly assigned; existing classes, wards or clinics serve as treatment and control groups. ANCOVA is still the right tool in that design, but it pays to be clear about what it does: it applies a statistical adjustment for pretest scores, it does not actually make the groups equivalent. The possibility that the groups differ systematically on something unmeasured — teacher experience, school resources, patient profile — is nowhere in the model. Indeed, the fact that gain-score analysis and ANCOVA can point in different directions in quasi-experimental designs is a known and still-debated issue in the methodology literature, known as Lord's paradox; both are considered legitimate because they answer different questions, and the choice depends on the research question.

In practice this means writing the finding not as a firm causal claim ("the method produced this much effect") but as "with baseline level controlled, a difference of this size was observed between groups", and stating the absence of randomisation openly in the limitations. That is precisely the point a reviewer will press, and having it written in advance strengthens the paper rather than weakening it. In our reports we keep the strength of the claim within what your data can carry, and hand you the defence questions already answered.