Data Transformations Method Card Template (Statistics, R and Python)
Data transformations method card – log, sqrt, Box-Cox, Yeo-Johnson, logit, z-scores, centring and ranks, percent-change interpretation and smearing back-transformation, with R and Python
Anleitung: Use it when a variable is skewed, its variance grows with the mean, or predictors need rescaling. Work through the decision table, estimate Box-Cox lambda, fit on the transformed scale, convert coefficients to percent changes and back-transform predictions with smearing. The main pitfall is trying transformations until a test becomes significant; decide the scale in advance and report it.
Vorschau
Data Transformations Method Card: [Your variable or research question]
A method card for transforming variables before analysis: log, log(x + c), square root, reciprocal, Box-Cox, Yeo-Johnson, logit and arcsine for proportions, z-scores and min-max scaling, centring for interactions, and ranks. It shows when a transformation helps, how to estimate the Box-Cox power, how to read regression coefficients after a log transform (as percent changes), and how to back-transform predictions without bias (geometric mean, Duan's smearing). It is for students preparing for an exam, master's and PhD students whose thesis data are skewed, and analysts who need interpretable models. It speeds up learning by putting the decisions in order, showing one example fully solved, one half solved and one left for you, and ending with self-test questions whose answers sit in a separate key. The worked examples use fictional data; replace them with your own. Source basis: Box and Cox (1964), Yeo and Johnson (2000), Duan (1983), Tukey (1977), Warton and Hui (2011), the NIST/SEMATECH e-Handbook of Statistical Methods, and the R and SciPy documentation, all written up in original words and examples.
Document control
| Field | Value |
|---|---|
| Course or project | [STAT 201 assignment / thesis chapter 4 / client model] |
| Learner or analyst | [Your name] |
| Data set | [Lakemoor University student help desk wait times, 20 visits (fictional) / your data set and source] |
| Software and versions | R 4.5.1 (stats, MASS 7.3.65, car 3.1.5 optional); Python 3.14 with numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0 |
| Date | YYYY-MM-DD |
| Version | [0.1] |
When to use it
One-sentence rule: transform a variable when the transformed scale is more natural for the science (growth, ratios, concentrations) or makes a model's assumptions hold (linearity, constant variance, roughly normal residuals), and when you can still explain the result on the original scale.
Why transform at all? Three good reasons:
- The process is multiplicative. Waiting times, incomes, reaction times and concentrations often grow by percentages, not by fixed amounts. On a log scale, multiplicative effects become additive, and a linear model fits.
- The variance grows with the mean. Right-skewed positive data usually have more spread where values are large. A log or square root stabilises the variance (counts: square root; positive skewed data: log).
- The model needs a common scale. z-scores and min-max scaling put predictors on comparable scales; centring makes main effects interpretable in a model with an interaction.
| Use when | Do not use when |
|---|---|
| Positive, right-skewed outcome whose residuals fan out (log, Box-Cox) | You only want a smaller p-value (that is a forking path; see Common mistakes) |
| The effect is naturally a percentage or ratio (log) | The research question is about the arithmetic mean on the original scale (use a GLM with a log link instead) |
| Predictors on wildly different scales (z-scores) | The outcome is a count of events (use Poisson or negative binomial regression) |
| An interaction makes main effects hard to read (centring) | The outcome is successes out of trials (use logistic regression, not arcsine or logit of proportions) |
| Heavy outliers and a rank-based answer is acceptable (ranks) | The data contain zeros or negatives and you would add an arbitrary constant without a sensitivity check |
Alternatives: generalised linear models with a log link (Gamma, Poisson, negative binomial) model the mean on the
original scale directly (academic/stats-count-regression); robust standard errors handle non-constant variance
without changing the scale; rank-based tests (academic/stats-nonparametric-tests); bootstrap confidence intervals
(academic/stats-resampling-methods); logistic regression for binary and binomial outcomes
(academic/stats-logistic-regression).
The question and the data
Question (example). At the Lakemoor University student help desk (fictional), how much shorter is the wait when more staff are on duty? Outcome: wait time in minutes (ratio scale, strictly positive, right-skewed). Predictors: staff on duty (1, 2 or 3) and arrival hour (9 to 16). Design: 20 visits observed over two weeks.
| Visit | Wait (min) | Staff | Hour | Visit | Wait (min) | Staff | Hour | |
|---|---|---|---|---|---|---|---|---|
| 1 | 2.1 | 3 | 9 | 11 | 3.0 | 3 | 11 | |
| 2 | 3.4 | 3 | 10 | 12 | 1.8 | 3 | 9 | |
| 3 | 1.2 | 3 | 11 | 13 | 9.7 | 2 | 14 | |
| 4 | 5.6 | 2 | 9 | 14 | 4.8 | 2 | 12 | |
| 5 | 8.9 | 2 | 12 | 15 | 2.5 | 3 | 10 | |
| 6 | 2.8 | 3 | 14 | 16 | 11.2 | 1 | 15 | |
| 7 | 14.5 | 1 | 13 | 17 | 7.4 | 2 | 13 | |
| 8 | 4.1 | 2 | 15 | 18 | 3.9 | 3 | 11 | |
| 9 | 6.3 | 2 | 10 | 19 | 30.5 | 1 | 16 | |
| 10 | 22.0 | 1 | 16 | 20 | 5.1 | 2 | 12 |
Summary on the original scale: mean 7.54 minutes, median 4.95, geometric mean 5.31. The mean is well above the median, the classic sign of right skew.
The histograms below show the same 20 waits on the original scale and on the natural log scale. The long right tail on the left panel disappears on the right panel.
Assumptions and how to check each
| Requirement | Check | R | Python | If it fails |
|---|---|---|---|---|
| Log, Box-Cox and reciprocal need | min(x) > 0 | min(x) | x.min() | Yeo-Johnson, log(x + c) with a sensitivity check, or a GLM |
| Proportions for logit lie strictly inside (0, 1) | Any 0 or 1? | range(p) | p.min(), p.max() | Logistic regression on counts; empirical logit |
| Model residuals roughly symmetric after transform | Histogram, Q-Q plot, skewness | qqnorm(resid(m)) | sm.qqplot(m.resid) | Try another power; use robust or rank methods |
| Variance roughly constant after transform | Residuals vs fitted plot | plot(m, which = 1) | plt.scatter(m.fittedvalues, m.resid) | Try another power; robust (sandwich) SEs; GLM |
| The relationship is linear on the transformed scale | Scatter of transformed y vs x | plot(x, log(y)) | plt.scatter(x, np.log(y)) | Transform x too, or add a spline |
| The result can be explained on the original scale | Write the interpretation sentence | Prefer log (percent change) over an odd Box-Cox power |
How it works
A transformation replaces each value by for a fixed, monotone function , so the order of the values is kept but the distances between them change. Concave functions (log, square root, negative powers) pull in the right tail and stretch the left; they reduce right skew. Convex functions (squares) do the opposite. Tukey's ladder of powers orders the common choices by strength: , , , , , . The further down the ladder, the stronger the pull on the right tail. Box and Cox turned the ladder into one formula with a power estimated from the data.
Power and log transforms.
Box-Cox (for ):
is chosen by maximum likelihood: the value that makes the transformed data (or the residuals of a linear model) look most like a normal sample with constant variance. A 95% confidence interval for contains every value whose profile log-likelihood is within of the maximum. In practice you round to a nearby interpretable power inside the interval: 1 (none), 0.5 (square root), 0 (log), -1 (reciprocal).
Yeo-Johnson extends Box-Cox to zero and negative values. For it applies the Box-Cox formula to ; for it applies the mirror image with power to :
Proportions. For a proportion strictly between 0 and 1:
The logit maps (0, 1) onto the whole real line and is the link of logistic regression. The arcsine square root was once the standard variance-stabiliser for binomial proportions. It is now discouraged: when you have counts of successes out of trials, logistic regression models the data directly and gives interpretable odds ratios, while the arcsine scale has no natural interpretation (Warton and Hui 2011).
Scaling, centring and ranks.
These are linear (z, min-max, centring) or order-only (ranks). Linear rescaling never changes the shape of a distribution, the correlation between variables or the p-value of a slope; it only changes units. Ranks throw away the distances and keep the order, which is what rank-based tests and Spearman's correlation use.
Interpreting a log-outcome coefficient. In the model , a one-unit increase in multiplies the (geometric) mean of by :
For small (say ), is a good approximation; for larger values it is not ( means a 63% fall, not 99%).
Back-transformation bias. Because the log is concave, the mean of the logs is smaller than the log of the mean (Jensen's inequality). So is the geometric mean, which for skewed data is below the arithmetic mean, and estimates the median of at (if the log-scale errors are symmetric), not its mean. Duan's smearing estimator corrects the prediction of the mean without assuming normality:
Under normal log-scale errors with variance the factor is ; smearing replaces that assumption with the average of the exponentiated residuals.
| Symbol | Meaning |
|---|---|
| Original-scale predictor and outcome | |
| Box-Cox or Yeo-Johnson power parameter | |
| Constant added before a log when zeros occur | |
| Proportion between 0 and 1 | |
| Sample mean and standard deviation (divisor ) | |
| Intercept and slope of a regression on the transformed scale | |
| Residual of observation on the transformed scale | |
| Sample size | |
| Percent change in per one-unit change in |
Which transformation, in what order of questions:
Decision table
| Situation | Transformation | Interpretation after transform | Back-transform | Caution |
|---|---|---|---|---|
| Positive, right-skewed, multiplicative (times, incomes, concentrations) | Percent change; ratio of geometric means | gives geometric mean or median; smear for the mean | Not for zeros | |
| Contains zeros, otherwise as above | or Yeo-Johnson | Approximate percent change for large only | Results depend on ; report a sensitivity check; consider a GLM | |
| Counts with variance about equal to the mean | Hard to interpret | Square, with bias | Prefer Poisson regression | |
| Strong right skew, rates per unit (time per task) | Rate (tasks per minute) | ; order reverses | Use to keep the order | |
| Unknown best power, positive data | Box-Cox, by ML | Only if is rounded to 0, 0.5, 1 or -1 | Inverse Box-Cox | Odd powers like 0.37 are hard to explain |
| Any sign, need symmetry | Yeo-Johnson | Mostly for prediction or screening | Inverse Yeo-Johnson | Coefficients lose a simple meaning |
| Proportion from a continuous measure (share of time) | logit | Log-odds; odds ratio per unit | Needs | |
| Proportion from counts (passed out of enrolled) | None: logistic regression | Odds ratio | Predicted probability | Arcsine square root is discouraged |
| Predictors on different scales | z-score | Change per 1 SD | Units become sample-specific | |
| Inputs for distance-based methods | Min-max to [0, 1] | Relative position in observed range | One outlier squashes the rest | |
| Model with an interaction | Centre at mean or meaningful value | Main effect = effect at the centre of the other variable | Add the centre back | Does not change the interaction slope or fit |
| Heavy outliers, ordinal question | Ranks | Order only | Not possible | Distances are lost |
Worked example A (fully solved): log-transforming wait times
Hypotheses. : staffing level is not related to wait time ( on the log scale). : it is (), two-sided, .
Step 1. Shape before and after. Raw waits: skewness = 1.93, Shapiro-Wilk p = .0002. Log waits: skewness 0.34, Shapiro-Wilk p = .986. The square root (skewness 1.19, p = .033) and the reciprocal (1.37, p = .017) are worse choices here. Shapiro-Wilk is used as a descriptive aid only: with large samples it rejects trivial departures, with small ones it misses real ones.
Step 2. Box-Cox. The maximum-likelihood power is , 95% CI [-0.688, 0.318]. The interval contains 0 (log) and excludes 0.5 (square root) and 1 (no transformation), so the log is supported by the data and gives the most interpretable scale. When Box-Cox is estimated for the regression residuals (given staff), R gives , even closer to 0. Box-Cox targets the distribution of the model's errors, so estimate it with the predictors you will use.
Step 3. Fit the model on the log scale. ; 95% CI for the slope [-1.2168, -0.7710].
Step 4. Convert to percent change (by hand). , so each extra staff member multiplies the typical wait by 0.37: a change of . CI: and .
Step 5. Back-transform predictions. For one staff member: minutes. That is the predicted median (and geometric mean) wait. The smearing factor is , so the predicted mean wait is minutes. The smeared mean is 5.7% above the naive back-transform, so the naive value understates the mean wait.
| Staff | Naive (median) | Smeared (mean) | Observed mean | Observed geometric mean |
|---|---|---|---|---|
| 1 | 17.48 | 18.48 | 19.55 | 18.17 |
| 2 | 6.47 | 6.84 | 6.49 | 6.23 |
| 3 | 2.40 | 2.53 | 2.59 | 2.44 |
The naive predictions track the observed geometric means, and the smeared predictions track the observed arithmetic means for 1 and 3 staff. For 2 staff the observed mean is lower than the smeared prediction for two reasons: the fitted line sits slightly above the 2-staff geometric mean (6.47 vs 6.23), and the 8 visits with 2 staff happen to have less log-scale spread than the others, while smearing uses one pooled correction for all groups (it assumes constant variance on the log scale).
R code (base R, stats and MASS; car is optional):
library(MASS) # ships with R; boxcox()
desk <- data.frame(
wait = c(2.1,3.4,1.2,5.6,8.9,2.8,14.5,4.1,6.3,22.0,3.0,1.8,9.7,4.8,2.5,11.2,7.4,3.9,30.5,5.1),
staff = c(3,3,3,2,2,3,1,2,2,1,3,3,2,2,3,1,2,3,1,2),
hour = c(9,10,11,9,12,14,13,15,10,16,11,9,14,12,10,15,13,11,16,12)
)
# Box-Cox: profile log-likelihood over a fine grid of lambda
bc0 <- boxcox(lm(wait ~ 1, data = desk), lambda = seq(-2, 2, 0.001), plotit = FALSE)
lam0 <- bc0$x[which.max(bc0$y)]
ci0 <- range(bc0$x[bc0$y > max(bc0$y) - qchisq(0.95, 1) / 2])
cat("Box-Cox lambda (no predictors):", lam0, " 95% CI:", ci0, "\n")
bc1 <- boxcox(lm(wait ~ staff, data = desk), lambda = seq(-2, 2, 0.001), plotit = FALSE)
cat("Box-Cox lambda (given staff): ", bc1$x[which.max(bc1$y)], "\n")
# Log-outcome model and percent change
m <- lm(log(wait) ~ staff, data = desk)
round(coef(m), 4)
round(100 * (exp(coef(m)["staff"]) - 1), 1)
round(100 * (exp(confint(m)["staff", ]) - 1), 1)
smear <- mean(exp(resid(m)))
round(smear, 4)
nd <- data.frame(staff = 1:3)
round(cbind(nd, naive = exp(predict(m, nd)), smeared = exp(predict(m, nd)) * smear), 3)
# Yeo-Johnson and Box-Cox via car (optional): install.packages("car")
if (requireNamespace("car", quietly = TRUE)) {
print(round(car::powerTransform(desk$wait, family = "yjPower")$lambda, 4))
print(round(car::powerTransform(desk$wait, family = "bcPower")$lambda, 4))
}
head(round(as.numeric(scale(desk$wait)), 3), 5) # z-scores
head(rank(desk$wait), 5) # ranks
Box-Cox lambda (no predictors): -0.179 95% CI: -0.688 0.317
Box-Cox lambda (given staff): -0.024
(Intercept) staff
3.8552 -0.9939
staff
-63
2.5 % 97.5 %
-70.4 -53.7
[1] 1.0567
staff naive smeared
1 1 17.484 18.476
2 2 6.471 6.839
3 3 2.395 2.531
desk$wait
-0.3954
desk$wait
-0.1786
[1] -0.736 -0.560 -0.858 -0.263 0.184
[1] 3 7 1 12 15
A practical R note: name the data frame something other than df. MASS::boxcox() refits the model internally, and
with a data frame called df it picks up R's df() function (the F density) instead, failing with "'data' must be a
data.frame, environment, or list". The same refit fails inside your own function on a local data frame. Fitting the
model with lm(..., y = TRUE, qr = TRUE) avoids the refit; the example simply uses the name desk.
Python code (numpy, pandas, scipy, statsmodels):
import numpy as np, pandas as pd
from scipy import stats
import statsmodels.formula.api as smf
desk = pd.DataFrame({
"wait": [2.1,3.4,1.2,5.6,8.9,2.8,14.5,4.1,6.3,22.0,3.0,1.8,9.7,4.8,2.5,11.2,7.4,3.9,30.5,5.1],
"staff": [3,3,3,2,2,3,1,2,2,1,3,3,2,2,3,1,2,3,1,2],
"hour": [9,10,11,9,12,14,13,15,10,16,11,9,14,12,10,15,13,11,16,12],
})
w = desk["wait"]
# 1. Shape before and after
for name, x in [("raw", w), ("log", np.log(w)), ("sqrt", np.sqrt(w)), ("recip", 1/w)]:
print(f"{name:6s} skew = {stats.skew(x):6.3f} Shapiro p = {stats.shapiro(x).pvalue:.4f}")
# 2. Box-Cox and Yeo-Johnson (marginal, no predictors)
bc, lam, ci = stats.boxcox(w, alpha=0.05)
print("Box-Cox lambda =", round(lam, 4), " 95% CI =", np.round(ci, 3))
yj, lam_yj = stats.yeojohnson(w)
print("Yeo-Johnson lambda =", round(lam_yj, 4))
# 3. Means on the original and log scale
print("mean =", round(w.mean(), 3), " median =", round(w.median(), 3),
" geometric mean =", round(np.exp(np.log(w).mean()), 3))
# 4. Log-outcome regression and percent change
m = smf.ols("np.log(wait) ~ staff", data=desk).fit()
b, (lo, hi) = m.params["staff"], m.conf_int().loc["staff"]
print("b0 =", round(m.params["Intercept"], 4), " b1 =", round(b, 4), " 95% CI", round(lo, 4), round(hi, 4))
print("percent change per extra staff =", round(100*np.expm1(b), 1),
" 95% CI", round(100*np.expm1(lo), 1), round(100*np.expm1(hi), 1))
# 5. Back-transformation: naive vs Duan smearing
smear = np.mean(np.exp(m.resid))
print("smearing factor =", round(smear, 4))
for s in (1, 2, 3):
naive = np.exp(m.params["Intercept"] + b*s)
obs = desk.loc[desk.staff == s, "wait"]
print(f"staff={s}: naive = {naive:.3f} smeared = {naive*smear:.3f} "
f"observed mean = {obs.mean():.3f} observed geo mean = {np.exp(np.log(obs).mean()):.3f}")
# 6. Scaling and ranks (first five rows)
print("z :", np.round(np.asarray(stats.zscore(w, ddof=1))[:5], 3))
print("min-max :", np.round(((w - w.min())/(w.max() - w.min()))[:5].to_numpy(), 3))
print("rank :", stats.rankdata(w)[:5])
# 7. Centring before an interaction
x, h = desk["staff"], desk["hour"]
print("corr(staff, staff*hour) raw =", round(np.corrcoef(x, x*h)[0, 1], 3),
" centred =", round(np.corrcoef(x - x.mean(), (x - x.mean())*(h - h.mean()))[0, 1], 3))
# 8. Proportions
p = np.array([0.62, 0.95, 0.80, 0.50, 0.98])
print("logit :", np.round(np.log(p/(1 - p)), 3))
print("arcsine :", np.round(np.arcsin(np.sqrt(p)), 3))
raw skew = 1.927 Shapiro p = 0.0002
log skew = 0.341 Shapiro p = 0.9855
sqrt skew = 1.191 Shapiro p = 0.0326
recip skew = 1.368 Shapiro p = 0.0171
Box-Cox lambda = -0.1786 95% CI = [-0.688 0.318]
Yeo-Johnson lambda = -0.3954
mean = 7.54 median = 4.95 geometric mean = 5.305
b0 = 3.8552 b1 = -0.9939 95% CI -1.2168 -0.771
percent change per extra staff = -63.0 95% CI -70.4 -53.7
smearing factor = 1.0567
staff=1: naive = 17.484 smeared = 18.476 observed mean = 19.550 observed geo mean = 18.169
staff=2: naive = 6.471 smeared = 6.839 observed mean = 6.487 observed geo mean = 6.228
staff=3: naive = 2.395 smeared = 2.531 observed mean = 2.587 observed geo mean = 2.442
z : [-0.736 -0.56 -0.858 -0.263 0.184]
min-max : [0.031 0.075 0. 0.15 0.263]
rank : [ 3. 7. 1. 12. 15.]
corr(staff, staff*hour) raw = 0.858 centred = 0.375
logit : [0.49 2.944 1.386 0. 3.892]
arcsine : [0.907 1.345 1.107 0.785 1.429]
R and Python compared. The Box-Cox agrees (-0.1786 from car::powerTransform and SciPy; -0.179 from
the MASS::boxcox grid). The upper CI limit is 0.317 in R and 0.318 in SciPy only because R searched a grid with step
0.001. scipy.stats.boxcox estimates for the variable alone; MASS::boxcox works on a fitted model, so it can
estimate for the residuals given predictors (-0.024 here). Yeo-Johnson agrees (-0.3954). The z-scores agree
because scale() in R and zscore(ddof=1) both divide by with ; SciPy's default ddof=0 would give slightly
larger values.
Interpretation. Waits are right-skewed and multiplicative, and the Box-Cox interval supports a log scale. On that scale, each extra staff member is associated with a 63% shorter typical wait. Predictions back-transformed with smearing estimate the mean wait; without smearing they estimate the median.
APA-style report sentence. "Wait times were right-skewed (M = 7.54 min, Mdn = 4.95) and were log-transformed (Box-Cox = -0.18, 95% CI [-0.69, 0.32]). Each additional staff member on duty was associated with a 63.0% shorter wait, = -0.99, 95% CI [-1.22, -0.77] on the log scale, corresponding to a 53.7% to 70.4% reduction, n = 20. Predicted mean waits (Duan smearing) were 18.5, 6.8 and 2.5 minutes for one, two and three staff."
Interpreting coefficients after log transforms
| Model | Coefficient means | Exact conversion | Example |
|---|---|---|---|
| (level-level) | Change in units per 1 unit of | none | : 2 more points per hour |
| (log-level) | Multiplicative change in per 1 unit of | % | : 63.0% shorter wait per extra staff |
| (level-log) | Change in units per 1% increase in | : 0.05 points per 1% more hours; 0.48 points per 10% more | |
| (log-log) | Elasticity: % change in per 1% change in | % | : 0.4% per 1%; 3.9% per 10% |
| with a binary | Ratio of geometric means between groups | : group 1 is 28.4% higher |
Use base-10 logs only if the audience thinks in powers of ten (concentrations, decibels); the conversion is then
, not . State which log you used. In R and NumPy, log() is the natural log.
Worked example B (half solved): prediction and two-step change
Using and the smearing factor 1.0567:
- Predicted log wait with 2 staff: [ ].
- Predicted median wait with 2 staff: [ ] minutes.
- Predicted mean wait with 2 staff (smearing): [ ] [ ] minutes.
- Percent change in typical wait from 1 to 3 staff (two extra): [ ]%.
- Why is the answer to step 4 not simply ? [ ]
- The observed mean with 2 staff is 6.49 minutes. Which prediction, median or smeared mean, should you compare it with, and why do they differ? [ ]
Exercise C (unsolved): interpretation drill
- Harbour City Library (fictional) records four daily e-book download counts: 2, 8, 4, 16. Compute the arithmetic mean and the geometric mean. Which is a better "typical day" for this multiplicative series, and why?
- A log-level model of Brightwell Bakery (fictional) batch waste gives for a new mixer (binary). State the effect as a percent change.
- A log-log model gives for hours of oven time on energy use. Interpret it for a 10% increase in oven time.
- In a log-level model a binary predictor has . By what percent is the outcome higher in group 1?
- A course pass rate is 0.80. Compute its logit and explain why logistic regression on the pass counts is preferred to an arcsine transform of the rate.
- You centre "hour" at its mean before fitting
log(wait) ~ staff * hour. Which coefficients change meaning, and which do not change at all?
Reading software output
| Output line | Meaning | Report it? |
|---|---|---|
Box-Cox lambda ... 95% CI | ML power and its likelihood interval | Yes, when you justify the transformation |
(Intercept) 3.8552 | Predicted log wait at staff = 0 (outside the data) | Rarely; centre staff for a meaningful intercept |
staff -0.9939 | Change in log wait per extra staff | Yes, with the percent conversion |
2.5 % 97.5 % after exp() | CI on the percent-change scale | Yes |
smearing factor 1.0567 | Correction from median to mean on the original scale | Yes, when you report predicted means |
naive vs smeared | Predicted median vs predicted mean | Say which one you report |
Yeo-Johnson lambda | Power for data with zeros or negatives | Only if you use it |
Shapiro p | Descriptive check of shape | Optional; prefer plots |
Reporting template
Fill-in sentence. "[Outcome] was [right-skewed / heteroscedastic] (M = [ ], Mdn = [ ]) and was [log / square-root / Box-Cox] transformed [because the process is multiplicative / Box-Cox = [ ], 95% CI [ , ]]. Each one-unit increase in [predictor] was associated with a [ ]% [higher / lower] [geometric mean / median] [outcome], = [ ], 95% CI [ , ] on the log scale. Predicted means on the original scale used [Duan smearing / the normal-theory correction]."
| Variable | Transformation | Reason | Decided before seeing results? | Back-transform method |
|---|---|---|---|---|
| Example: wait | natural log | multiplicative process; Box-Cox CI includes 0 | Yes (analysis plan) | Duan smearing |
| Example: hour | centred at 12.1 | interaction with staff | Yes | add 12.1 |
| [variable] | [ ] | [ ] | [ ] | [ ] |
Results checklist.
- Reason for the transformation stated (science or diagnostics), and decided before looking at p-values.
- The base of the log stated (natural log unless said otherwise).
- Constant reported, with a sensitivity check, if was used.
- Coefficients converted to percent changes or ratios, with CIs converted the same way.
- Back-transformed predictions labelled as medians (geometric means) or means (smearing).
- Residual plots checked on the transformed scale.
- Original-scale descriptive statistics also reported.
Common mistakes and misconceptions
| Mistake or misconception | Why it is wrong | Do this instead |
|---|---|---|
| Trying transformations until the test is significant | Choosing among many analyses after seeing results inflates the false positive rate (a "garden of forking paths") | Decide the scale from the science or from diagnostics on the residuals, write it in the analysis plan, and report it |
| "The outcome must be normal, so I transform it" | Linear models assume normal errors, not a normal outcome; a skewed outcome can have normal residuals once predictors are in | Check residuals of the fitted model, not the raw histogram |
| Reading on a log outcome as "99% lower" | The exact change is ; only works for small | Convert with |
| Back-transforming a mean of logs and calling it the mean | is the geometric mean (5.31 vs 7.54 here) | Call it the geometric mean or median, or apply smearing |
| Adding 1 before a log without thinking | The result depends on ; with values near 0, behaves almost linearly | Report , check = 0.5 and 1, or use a GLM or Yeo-Johnson |
| Using arcsine square root on proportions from counts | Discouraged: no interpretable scale and ignores the binomial structure | Logistic regression on successes and failures |
| Expecting z-scores to "normalise" the data | z-scores are linear; the shape is unchanged | Use z-scores for scale, a power transform for shape |
| Believing centring removes "real" multicollinearity | Centring lowers the correlation between and (0.86 to 0.38 here) but does not change the fit or the interaction slope | Centre for interpretation of main effects, not to fix inference |
| Transforming a Likert item or a bounded score with a log | Bounded scales are not multiplicative | Use ordinal models or treat the summed scale as interval |
| Reporting only transformed-scale numbers | Readers cannot judge practical size in log minutes | Give original-scale summaries and back-transformed effects |
Self-test questions
- Box-Cox gives = 0.42 with 95% CI [0.18, 0.69]. Which simple transformation would you choose, and why not the exact 0.42?
- In a log-level regression, = 0.08. Give the exact and the approximate percent change. When does the approximation break down?
- You back-transform the mean of log incomes and get 31,200 USD; the arithmetic mean is 38,900 USD. Which is "wrong"?
- Why does
scipy.stats.boxcoxandMASS::boxcox(lm(y ~ x))sometimes give different for the same ? - A colleague runs a t-test on raw data (p = .09), then on log data (p = .04), then reports the log version. What is wrong, and what should have happened?
- Does converting predictors to z-scores change the p-value of their slopes in a linear regression? Does it change the slopes?
My notes and next steps
| Item | Entry |
|---|---|
| Confidence choosing a transformation (1-5) | [ ] |
| Confidence interpreting log coefficients (1-5) | [ ] |
| Confidence back-transforming (1-5) | [ ] |
| Open questions | [What I am still unsure about] |
| Mistakes to log | [Link to academic/stats-error-log entry] |
| Next review dates | YYYY-MM-DD (2 days), YYYY-MM-DD (1 week), YYYY-MM-DD (1 month) |
| Next template | [academic/stats-linear-regression / academic/stats-count-regression] |
Sources and further reading
Based on the original method papers listed below and the NIST/SEMATECH e-Handbook of Statistical Methods; all text, data and examples are original.
- Box, G. E. P., and Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society, Series B, 26(2), 211–243.
- Yeo, I.-K., and Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87(4), 954–959.
- Duan, N. (1983). Smearing estimate: a nonparametric retransformation method. Journal of the American Statistical Association, 78(383), 605–610.
- Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
- Warton, D. I., and Hui, F. K. C. (2011). The arcsine is asinine: the analysis of proportions in ecology. Ecology, 92(1), 3–10.
- Gelman, A., and Loken, E. (2013). The garden of forking paths: why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Working paper, Department of Statistics, Columbia University.
- NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.3.5 Box-Cox Linearity Plot, and section 1.3.3.6 Box-Cox Normality Plot, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain.
- Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook (2024 edition, StatsRef); Diez, D. M., Çetinkaya-Rundel, M., and Barr, C. D., OpenIntro Statistics (CC BY-SA).
- Software documentation: R
MASS::boxcox(),car::powerTransform(),scale(); SciPystats.boxcox(),stats.yeojohnson(),stats.zscore(); statsmodelsols.
Answer key
Worked example B. (1) . (2) minutes (median and geometric mean). (3) minutes. (4) , so . (5) Percent changes multiply rather than add: two steps of give , an 86.3% fall. A change of -126% is impossible for a positive quantity. (6) Compare the observed mean (6.49) with the smeared mean (6.84), and the observed geometric mean (6.23) with the median prediction (6.47). The smeared mean is above the observed mean because the fitted median is already a little above the observed geometric mean, and smearing uses one pooled correction while the 2-staff visits have less log-scale spread than the others.
Exercise C. (1) Arithmetic mean ; geometric mean . The geometric mean, because the values differ by factors of 2 and the arithmetic mean is pulled up by 16. (2) : about one third less waste with the new mixer. (3) more energy for 10% more oven time (elasticity 0.25). (4) higher. (5) . Logistic regression uses the number passed and the number enrolled, weights courses by their size, keeps predictions inside 0 to 1 and gives odds ratios; the arcsine scale has no interpretation and ignores the binomial variance. (6) The coefficient of staff now means the effect of staff at the mean hour (not at hour 0, which is outside the data); the intercept changes too. The interaction coefficient, its p-value, the residuals and do not change.
Self-test.
- Square root (): it is inside the CI and has a familiar meaning; 0.42 is barely better in likelihood and impossible to explain. The log (0) and no transform (1) are both outside the CI.
- Exact: ; approximate: 8%. The approximation drifts once exceeds about 0.1 (for : 64.9% exact vs 50% approximate).
- Neither: 31,200 USD is the geometric mean (close to the median for log-symmetric data), 38,900 USD is the arithmetic mean. They answer different questions; label them.
- SciPy estimates for the marginal distribution of ;
MASS::boxcoxon a model estimates it for the residuals given . Group or predictor differences make the marginal distribution more skewed than the residuals (here -0.18 vs -0.02). - The transformation was chosen because it gave a significant result, so the reported p-value does not have its stated meaning. The scale should have been chosen in advance (science or residual diagnostics) and both analyses reported as a sensitivity check.
- The p-values (and t statistics) do not change. The slopes do change: they become effects per one SD of the predictor instead of per original unit.
Revision history
| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [Your name] | First draft from the template |
| 0.2 | YYYY-MM-DD | [Your name] | Example: Replaced Lakemoor example with thesis reaction times; log chosen in analysis plan |
Ähnliche Vorlagen
Data Cleaning Log Template (Statistics, Missing Data, R and Python)
Reproducible data cleaning with an audit log: duplicates, range and consistency checks, pre-set outlier policy, missing data mechanisms (MCAR, MAR, MNAR), imputation choices, R and Python code
AkademischStatistical Analysis Plan Template (Statistics, Thesis or Project)
Statistical analysis plan for a thesis or project: estimands, hypotheses, sample size, primary model, pre-set assumption fallbacks, missing data, multiplicity, sensitivity, deviations log
AkademischStatistical Analysis Report Template (Statistics, Thesis or Client)
Complete statistical analysis report for a thesis results chapter or client: plain-language summary, methods, descriptives, main analyses, assumption checks, sensitivity, limitations, reproducibility appendix, reviewer checklist