# Statistics Error Log: [Your course, exam or thesis]

> A mistake journal for learning statistics: a log table for every error you make in homework, practice exams or
> analysis, a six-type error taxonomy to classify each one, a catalogue of 25 classic statistics misconceptions with
> the correct idea and the template that fixes it, fix-it drills, and a weekly review routine. It is for students,
> thesis writers and analysts who want each mistake to happen only once. It speeds up learning because errors you
> classify, explain and re-test on a schedule turn into rules you remember, while errors you only correct tend to come
> back. The worked examples use fictional data; replace them with your own.
> Source basis: retrieval and spacing research (Roediger and Karpicke 2006; Cepeda et al. 2006) and the
> misinterpretation catalogue of Greenland et al. (2016), written in our own words.

<!--
Guidance: How to use this template
- Log an error within 24 hours of finding it: the problem, your answer, the correct answer, the error type, the root cause and a fix rule.
- Classify each error with the taxonomy. The type decides the fix: a concept error needs re-learning, a calculation error needs a checking habit.
- Write the fix rule as one sentence you could put on a flashcard, then add it to academic/stats-flashcards-spaced-review.
- Set a re-test date (3 and 10 days later). An error is closed only when you solve a new problem of the same kind correctly on the re-test date.
- Once a week, count errors by type and read the misconception catalogue entries that match.
- Related templates: academic/stats-flashcards-spaced-review, academic/stats-worked-problem, academic/stats-formula-sheet, academic/stats-learning-roadmap, academic/stats-practice-which-test, academic/stats-hypothesis-testing, academic/stats-confidence-intervals, academic/study-guide
-->

## Document control

<!-- Guidance: One log per course or project. Start a new version each term so you can compare error profiles. -->

| Field | Value |
|---|---|
| Course or project | [STAT 201, Lakemoor University (fictional)] |
| Learner or analyst | [Your name] |
| Period covered | YYYY-MM-DD to YYYY-MM-DD |
| Software and versions | [R 4.5.1 / Python 3.14 with scipy 1.18.1, statsmodels 0.15.0] |
| Date | YYYY-MM-DD |
| Version | [0.1] |

## Why a mistake journal works

<!-- Guidance: Read once. The log is not a list of failures; it is the most personalised study material you will ever have. -->

Every learner makes a small number of mistakes again and again. A mistake that you correct and forget will usually
return, because the wrong idea is still the first thing memory offers. Writing the mistake down, naming its type and
its root cause, and turning the fix into a short rule that you later test yourself on, replaces the wrong idea with
the right one through retrieval practice and spacing.

The log also shows patterns. If most of your errors are "reading the question", more statistics study will not help:
a slower first read will. If most are "concept" errors about p-values and intervals, the concept cards
(academic/stats-hypothesis-testing, academic/stats-confidence-intervals) are the best use of your next week.

## Error-type taxonomy

<!-- Guidance: Pick the earliest step in the chain where it went wrong. A wrong method chosen because the question was misread is a "reading" error, not a "procedure" error. -->

| Type | What went wrong | Typical example | Usual fix |
|---|---|---|---|
| Reading the question | misread what was asked or given | used the SD when the question gave the SE | underline what is given and what is asked |
| Concept | wrong idea about what a quantity means | "p = .03 means a 3% chance $H_0$ is true" | re-learn the concept card, explain it in plain words |
| Procedure | wrong method or wrong steps for the design | analysed paired data with an independent-samples test | test selector, worked-problem steps 2 and 3 |
| Calculation | right method, arithmetic or formula slip | divided by $n$ instead of $n-1$ | hand check against software, estimate size first |
| Software | wrong function, option or default | scipy `ttest_ind` without `equal_var=False` | read the output heading, set options explicitly |
| Interpretation | right number, wrong conclusion | causal claim from an observational study | report template, Feynman check |

Classify in this order, stopping at the first "no":

```mermaid
flowchart TD
  A["Did I read the question correctly?"] -- "No" --> R["Reading the question"]
  A -- "Yes" --> B["Did I understand the idea behind it?"]
  B -- "No" --> C["Concept"]
  B -- "Yes" --> D["Did I choose the right method and steps?"]
  D -- "No" --> P["Procedure"]
  D -- "Yes" --> E["Were the numbers computed correctly?"]
  E -- "No" --> F{"By hand or in software?"}
  F -- "By hand" --> K["Calculation"]
  F -- "Software" --> S["Software"]
  E -- "Yes" --> I["Interpretation"]
```

## Error log

<!-- Guidance: One row per error. Keep "my answer" exactly as you wrote it: the exact wording reveals the misconception. -->

| Date | Problem (source, short) | My answer | Correct answer | Error type | Root cause | Fix rule | Re-test date |
|---|---|---|---|---|---|---|---|
| *Example:* YYYY-MM-DD | PS3 Q2: 95% CI [4.1, 7.9] for mean sleep hours | "95% of students sleep 4.1 to 7.9 h" | the interval estimates the mean, not individual values | Concept | confused a CI with a reference range | "A CI is about a parameter, not about individual data values" | YYYY-MM-DD |
| *Example:* YYYY-MM-DD | Practice exam Q5: before/after scores, 12 students | Welch t, p = .21 | paired t, p = .004 | Procedure | did not check whether the same students appear twice | "Same units measured twice: compute differences first" | YYYY-MM-DD |
| *Example:* YYYY-MM-DD | Lab 4: two-group comparison in Python | pooled p = .031 | Welch p = .048 | Software | scipy default is pooled | "In scipy write `equal_var=False` every time" | YYYY-MM-DD |
| *Example:* YYYY-MM-DD | PS4 Q1: SE of mean, s = 12, n = 36 | 0.33 | 2 | Calculation | divided by $n$, not $\sqrt{n}$ | "SE uses the square root of n" | YYYY-MM-DD |
| *Example:* YYYY-MM-DD | Thesis survey: tutoring use and grades, r = .41 | "tutoring raises grades" | "tutoring use is associated with higher grades" | Interpretation | causal wording for observational data | "No randomisation, no causal verbs" | YYYY-MM-DD |
| *Example:* YYYY-MM-DD | Quiz 2 Q3: "at least 2 of 5" | P(X = 2) | P(X ≥ 2) | Reading the question | skipped "at least" | "Circle every inequality word before computing" | YYYY-MM-DD |
| YYYY-MM-DD | [Problem] | [My answer] | [Correct answer] | [Type] | [Root cause] | [Fix rule] | YYYY-MM-DD |
| YYYY-MM-DD | [Problem] | [My answer] | [Correct answer] | [Type] | [Root cause] | [Fix rule] | YYYY-MM-DD |

Re-test record:

| Error (date) | Re-test 1 (date, new problem, right/wrong) | Re-test 2 (date, right/wrong) | Closed |
|---|---|---|---|
| *Example:* paired vs independent (YYYY-MM-DD) | YYYY-MM-DD, branch visits before/after, right | YYYY-MM-DD, right | Y |
| [Error] | YYYY-MM-DD, [ ] | YYYY-MM-DD, [ ] | [ ] |

## Error profile

<!-- Guidance: Count your open and closed errors by type each week. The chart below shows a fictional month; replace the counts with yours. -->

Counting errors by type tells you where your next study hour pays most. In the fictional month below, concept and
procedure errors dominate: the learner should spend the next week on the concept cards and the test selector, not on
more arithmetic practice.

```vega-lite
{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "description": "Errors by type in one month for a fictional learner.",
  "width": 360,
  "height": 200,
  "data": {"values": [
    {"type": "Reading", "count": 3}, {"type": "Concept", "count": 7}, {"type": "Procedure", "count": 6},
    {"type": "Calculation", "count": 2}, {"type": "Software", "count": 4}, {"type": "Interpretation", "count": 5}
  ]},
  "mark": "bar",
  "encoding": {
    "x": {"field": "type", "type": "nominal", "sort": "-y", "title": "Error type"},
    "y": {"field": "count", "type": "quantitative", "title": "Errors logged"}
  }
}
```

## Catalogue of 25 classic misconceptions

<!-- Guidance: When you log a concept or interpretation error, find the matching entry here and open the template in the last column. Read the correct idea aloud and explain it in your own example. -->

| # | Misconception | Correct idea | Template that fixes it |
|---|---|---|---|
| 1 | The p-value is the probability that $H_0$ is true | it is the probability of data at least this extreme, assuming $H_0$ is true | academic/stats-hypothesis-testing |
| 2 | p > .05 shows there is no effect | absence of evidence is not evidence of absence; look at the CI, which may include large effects | academic/stats-confidence-intervals |
| 3 | A significant result is an important result | significance depends on n; importance depends on the effect size and context | academic/stats-effect-sizes |
| 4 | A computed 95% CI has a 95% probability of containing $\mu$ | 95% describes the long-run procedure; a Bayesian credible interval carries the probability statement | academic/stats-confidence-intervals |
| 5 | A 95% CI contains 95% of the data | it is a range for a parameter; individual values spread much wider | academic/stats-confidence-intervals |
| 6 | Overlapping CIs mean no significant difference | two 95% CIs can overlap while the difference is significant; build a CI for the difference | academic/stats-confidence-intervals |
| 7 | A bigger sample makes the SD smaller | the SD estimates the population spread and does not shrink; the SE shrinks with $\sqrt{n}$ | academic/stats-sampling-distributions |
| 8 | The central limit theorem makes the data normal | it makes the sampling distribution of the mean approximately normal | academic/stats-sampling-distributions |
| 9 | $P(A \mid B) = P(B \mid A)$ | they differ unless $P(A) = P(B)$ (or the events never occur together); use Bayes' theorem with the base rate | academic/stats-probability-basics |
| 10 | Correlation shows causation | confounding and reverse causation are possible without randomisation | academic/stats-correlation |
| 11 | r = 0 means no relationship | r measures only linear association; a curved relation can have r near 0 | academic/stats-correlation |
| 12 | A regression slope in observational data is a causal effect | it is an adjusted association; causal claims need design or strong assumptions | academic/stats-linear-regression |
| 13 | A high $R^2$ means the model is correct | $R^2$ can be high with wrong functional form or violated assumptions; check residuals | academic/stats-regression-model-report |
| 14 | Regression requires the outcome to be normal | the assumption is about the residuals (errors), and mainly matters for small-sample inference | academic/stats-assumption-checklist |
| 15 | Always run a normality test and obey its p-value | small samples: low power; large samples: trivial departures flagged; use plots and design | academic/stats-assumption-checklist |
| 16 | Before-and-after data are two independent groups | the same units measured twice are paired; analyse the differences | academic/stats-t-tests |
| 17 | Many pairwise t-tests are fine instead of ANOVA | the family-wise Type I error grows; use ANOVA with Tukey or Games-Howell | academic/stats-one-way-anova |
| 18 | A significant ANOVA shows which groups differ | it shows at least one mean differs; post hoc tests show which | academic/stats-one-way-anova |
| 19 | Nonparametric tests have no assumptions and always compare medians | they still assume independent observations; Mann-Whitney tests whether one group tends to have larger values, which equals a median comparison only if the two shapes are the same | academic/stats-nonparametric-tests |
| 20 | Chi-square works on percentages | it needs counts; expected counts should mostly be at least 5 | academic/stats-contingency-tables |
| 21 | An odds ratio is the same as a relative risk | they are close only when the outcome is rare | academic/stats-contingency-tables |
| 22 | A logistic coefficient is a change in probability | it is a change in log odds; $e^{b}$ is an odds ratio | academic/stats-logistic-regression |
| 23 | Observed power explains a non-significant result | power computed from the observed effect is a function of the p-value and adds nothing | academic/stats-power-sample-size |
| 24 | Deleting incomplete cases is always safe | in general it is guaranteed unbiased only under MCAR (with some exceptions, such as missingness that depends only on regression predictors) and it loses power; document and consider multiple imputation | academic/stats-data-cleaning-log |
| 25 | Correcting for multiple tests is only needed for the tests you report | every test you ran (or would have run) counts; pre-specify in an analysis plan | academic/stats-analysis-plan |

## Fix-it drills

<!-- Guidance: Drills turn a fix rule into a reflex. Study Drill A, complete Drill B, then do Drill C and the mixed set without notes. -->

Each drill takes one wrong statement, names its error type, explains why it is wrong, and rewrites it correctly.

**Drill A (fully worked).** Statement in a lab report: "The Welch t-test gave p = .27, so the two Brightwell Bakery
(fictional) flour suppliers produce loaves of the same weight."

1. Error type: interpretation (the number is right, the conclusion is not).
2. Why it is wrong: a non-significant result means the data are compatible with no difference, but they may also be
   compatible with a sizeable difference. The size of plausible differences is in the CI, not in the p-value.
3. Check the CI: suppose the 95% CI for the difference is [-6, 20] grams. Differences up to 20 grams remain plausible.
4. Rewrite: "The mean weights did not differ significantly (p = .27); the 95% CI for the difference, [-6, 20] grams,
   is too wide to rule out a practically relevant difference."
5. Fix rule for flashcards: "Non-significant: report the CI and say what it cannot rule out."

**Drill B (half worked).** Statement: "Students who used the Lakemoor University (fictional) tutoring service had
grades 0.4 points higher, r = .41, p < .001, so tutoring improves grades."

1. Error type: [ ].
2. Why it is wrong: the data are [ ], so students chose whether to use tutoring; [ ] such as motivation could
   explain both.
3. Rewrite: "Use of the tutoring service was [ ] with higher grades (r = .41, p < .001)."
4. Fix rule: [ ].

**Drill C (unsolved).** Statement in a thesis draft: "With n = 2,000 the Shapiro-Wilk test rejected normality
(p = .002), so we could not use regression and switched to a Spearman correlation." Name the error type(s), explain
what is wrong and rewrite the decision.

**Mixed set (interleaved).** Mark each statement right or wrong; for wrong ones, give the misconception number.

1. "The SE of a mean falls as the sample size grows."
2. "Our two 95% CIs overlap slightly, so the difference cannot be significant."
3. "An odds ratio of 2.0 means the outcome is twice as likely."
4. "Because $r = 0.05$, study hours and grades are unrelated."
5. "The paired t-test is a one-sample t-test on the within-pair differences."
6. "The ANOVA was significant, F(2, 12) = 12.80, so all three sections differ."

## Weekly review routine

<!-- Guidance: 30 minutes, same day each week. Do it even when the week had few errors: closing old errors is half the value. -->

| Step | Time | What to do |
|---|---|---|
| 1 | 5 min | Count new errors by type; update the error profile chart values |
| 2 | 10 min | For each error due for re-test, solve a new problem of the same kind without notes; record right or wrong |
| 3 | 5 min | Close errors with two correct re-tests at least a week apart; reschedule the rest (+3 days if wrong) |
| 4 | 5 min | Read the catalogue entries that match this week's concept and interpretation errors |
| 5 | 5 min | Turn every new fix rule into a flashcard; pick next week's focus template from the most frequent type |

Where to find new problems for re-tests: the Exercise C sections of the method templates, the scenarios in
academic/stats-practice-which-test, and past exam papers. Use a different data set from the original error, so that
you test the rule rather than your memory of one answer.

## Self-test questions

<!-- Guidance: Answer from memory; then check the Answer key. -->

1. Name the six error types in the order you should check them.
2. You used `scipy.stats.ttest_ind(a, b)` and reported a pooled p-value when the course requires Welch. Which type is
   this, and what is the fix rule?
3. Why is a fix rule written as a short sentence, and where should it go next?
4. When is an error "closed" in this log?
5. State the correct idea for misconception 7 in one sentence.

## My notes and next steps

<!-- Guidance: Name next week's focus from your error profile and one concrete action. -->

Most frequent error type this month: [ ]. Focus template for next week: [academic/...]. Errors still open: [ ].
Confidence that I will not repeat my top error (1-5): [ ]. Questions for the tutor: [ ].

## Sources and further reading

<!-- Guidance: The research behind the routine and the misconception list. -->

- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016).
  Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. *European Journal of
  Epidemiology*, 31(4), 337–350.
- Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. *The
  American Statistician*, 70(2), 129–133.
- Roediger, H. L., and Karpicke, J. D. (2006). Test-enhanced learning: taking memory tests improves long-term
  retention. *Psychological Science*, 17(3), 249–255.
- Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., and Rohrer, D. (2006). Distributed practice in verbal recall
  tasks: a review and quantitative synthesis. *Psychological Bulletin*, 132(3), 354–380.
- Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., and Willingham, D. T. (2013). Improving students' learning
  with effective learning techniques. *Psychological Science in the Public Interest*, 14(1), 4–58.
- Further reading (by name only): de Smith, M. J. (2024). *Statistical Analysis Handbook*, 2024 edition. StatsRef,
  https://www.statsref.com/; *OpenIntro Statistics*; NIST/SEMATECH *e-Handbook of Statistical Methods*.

## Answer key

<!-- Guidance: Check after attempting. Log any drill you got wrong in the log above: errors about errors count too. -->

**Drill B.** 1: interpretation. 2: observational; a confounder (for example motivation or prior grades). 3: "...was
associated with higher grades..." 4: "No randomisation, no causal verbs: say associated with."

**Drill C.** Error types: concept (normality of the outcome is not the regression assumption; the assumption concerns
residuals, misconception 14) and procedure (with n = 2,000, a normality test flags trivial departures,
misconception 15; and switching from regression to a correlation changes the question). Rewrite: "We fitted the
planned regression and checked the residuals with a Q-Q plot and a residuals-versus-fitted plot; with n = 2,000 the
inference on coefficients is robust to mild non-normality. As a sensitivity analysis we report robust (HC3) standard
errors."

**Mixed set.** 1 right. 2 wrong (misconception 6). 3 wrong (21: it doubles the odds, not the probability, unless the
outcome is rare). 4 wrong (11: only the linear association is near zero; also check a scatter plot). 5 right. 6 wrong
(18: at least one mean differs; use post hoc tests).

**Self-test.**

1. Reading the question, concept, procedure, calculation, software, interpretation (calculation and software are
   checked at the same step: by hand or by the tool).
2. Software. Fix rule: "In scipy write `equal_var=False` for two independent groups."
3. A short rule can be recalled in one step and tested as a flashcard; it goes into
   academic/stats-flashcards-spaced-review with a re-test date.
4. When you solve a new problem of the same kind correctly on two re-tests at least a week apart.
5. The SD estimates the spread of the population and does not shrink with n; the SE of the mean, $\sigma/\sqrt{n}$
   (estimated by $s/\sqrt{n}$), does.

## Revision history

<!-- Guidance: New version each term or after a major re-organisation of the log. -->

| Version | Date | Author | Change |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [Name] | First version from template |
| [0.2] | YYYY-MM-DD | [Name] | [Started term 2 log] |