Statistics Error Log Template (Mistake Journal and 25 Misconceptions)

Statistics Error Log: [Your course, exam or thesis]

A mistake journal for learning statistics: a log table for every error you make in homework, practice exams or analysis, a six-type error taxonomy to classify each one, a catalogue of 25 classic statistics misconceptions with the correct idea and the template that fixes it, fix-it drills, and a weekly review routine. It is for students, thesis writers and analysts who want each mistake to happen only once. It speeds up learning because errors you classify, explain and re-test on a schedule turn into rules you remember, while errors you only correct tend to come back. The worked examples use fictional data; replace them with your own. Source basis: retrieval and spacing research (Roediger and Karpicke 2006; Cepeda et al. 2006) and the misinterpretation catalogue of Greenland et al. (2016), written in our own words.

Document control

Why a mistake journal works

Every learner makes a small number of mistakes again and again. A mistake that you correct and forget will usually return, because the wrong idea is still the first thing memory offers. Writing the mistake down, naming its type and its root cause, and turning the fix into a short rule that you later test yourself on, replaces the wrong idea with the right one through retrieval practice and spacing.

The log also shows patterns. If most of your errors are "reading the question", more statistics study will not help: a slower first read will. If most are "concept" errors about p-values and intervals, the concept cards (academic/stats-hypothesis-testing, academic/stats-confidence-intervals) are the best use of your next week.

Error-type taxonomy

Classify in this order, stopping at the first "no":

Error log

Re-test record:

Error profile

Counting errors by type tells you where your next study hour pays most. In the fictional month below, concept and procedure errors dominate: the learner should spend the next week on the concept cards and the test selector, not on more arithmetic practice.

Catalogue of 25 classic misconceptions

Fix-it drills

Each drill takes one wrong statement, names its error type, explains why it is wrong, and rewrites it correctly.

Drill A (fully worked). Statement in a lab report: "The Welch t-test gave p = .27, so the two Brightwell Bakery (fictional) flour suppliers produce loaves of the same weight."

1. Error type: interpretation (the number is right, the conclusion is not).
2. Why it is wrong: a non-significant result means the data are compatible with no difference, but they may also be compatible with a sizeable difference. The size of plausible differences is in the CI, not in the p-value.
3. Check the CI: suppose the 95% CI for the difference is [-6, 20] grams. Differences up to 20 grams remain plausible.
4. Rewrite: "The mean weights did not differ significantly (p = .27); the 95% CI for the difference, [-6, 20] grams, is too wide to rule out a practically relevant difference."
5. Fix rule for flashcards: "Non-significant: report the CI and say what it cannot rule out."

Drill B (half worked). Statement: "Students who used the Lakemoor University (fictional) tutoring service had grades 0.4 points higher, r = .41, p < .001, so tutoring improves grades."

1. Error type: [ ].
2. Why it is wrong: the data are [ ], so students chose whether to use tutoring; [ ] such as motivation could explain both.
3. Rewrite: "Use of the tutoring service was [ ] with higher grades (r = .41, p < .001)."
4. Fix rule: [ ].

Drill C (unsolved). Statement in a thesis draft: "With n = 2,000 the Shapiro-Wilk test rejected normality (p = .002), so we could not use regression and switched to a Spearman correlation." Name the error type(s), explain what is wrong and rewrite the decision.

Mixed set (interleaved). Mark each statement right or wrong; for wrong ones, give the misconception number.

1. "The SE of a mean falls as the sample size grows."
2. "Our two 95% CIs overlap slightly, so the difference cannot be significant."
3. "An odds ratio of 2.0 means the outcome is twice as likely."
4. "Because $r = 0.05$, study hours and grades are unrelated."
5. "The paired t-test is a one-sample t-test on the within-pair differences."
6. "The ANOVA was significant, F(2, 12) = 12.80, so all three sections differ."

Weekly review routine

Where to find new problems for re-tests: the Exercise C sections of the method templates, the scenarios in academic/stats-practice-which-test, and past exam papers. Use a different data set from the original error, so that you test the rule rather than your memory of one answer.

Self-test questions

1. Name the six error types in the order you should check them.
2. You used scipy.stats.ttest_ind(a, b) and reported a pooled p-value when the course requires Welch. Which type is this, and what is the fix rule?
3. Why is a fix rule written as a short sentence, and where should it go next?
4. When is an error "closed" in this log?
5. State the correct idea for misconception 7 in one sentence.

My notes and next steps

Most frequent error type this month: [ ]. Focus template for next week: [academic/...]. Errors still open: [ ]. Confidence that I will not repeat my top error (1-5): [ ]. Questions for the tutor: [ ].

Sources and further reading

• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350.
• Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.
• Roediger, H. L., and Karpicke, J. D. (2006). Test-enhanced learning: taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255.
• Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., and Rohrer, D. (2006). Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380.
• Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., and Willingham, D. T. (2013). Improving students' learning with effective learning techniques. Psychological Science in the Public Interest, 14(1), 4–58.
• Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook, 2024 edition. StatsRef, https://www.statsref.com/; OpenIntro Statistics; NIST/SEMATECH e-Handbook of Statistical Methods.

Answer key

Drill B. 1: interpretation. 2: observational; a confounder (for example motivation or prior grades). 3: "...was associated with higher grades..." 4: "No randomisation, no causal verbs: say associated with."

Drill C. Error types: concept (normality of the outcome is not the regression assumption; the assumption concerns residuals, misconception 14) and procedure (with n = 2,000, a normality test flags trivial departures, misconception 15; and switching from regression to a correlation changes the question). Rewrite: "We fitted the planned regression and checked the residuals with a Q-Q plot and a residuals-versus-fitted plot; with n = 2,000 the inference on coefficients is robust to mild non-normality. As a sensitivity analysis we report robust (HC3) standard errors."

Mixed set. 1 right. 2 wrong (misconception 6). 3 wrong (21: it doubles the odds, not the probability, unless the outcome is rare). 4 wrong (11: only the linear association is near zero; also check a scatter plot). 5 right. 6 wrong (18: at least one mean differs; use post hoc tests).

Self-test.

1. Reading the question, concept, procedure, calculation, software, interpretation (calculation and software are checked at the same step: by hand or by the tool).
2. Software. Fix rule: "In scipy write equal_var=False for two independent groups."
3. A short rule can be recalled in one step and tested as a flashcard; it goes into academic/stats-flashcards-spaced-review with a re-test date.
4. When you solve a new problem of the same kind correctly on two re-tests at least a week apart.
5. The SD estimates the spread of the population and does not shrink with n; the SE of the mean, $\sigma/\sqrt{n}$ (estimated by $s/\sqrt{n}$), does.

Revision history