Markdown Editor

Data Transformations Method Card Template (Statistics, R and Python)

Data transformations method card – log, sqrt, Box-Cox, Yeo-Johnson, logit, z-scores, centring and ranks, percent-change interpretation and smearing back-transformation, with R and Python

AkademischVollständigBox and Cox (1964); Yeo and Johnson (2000); Duan (1983); Warton and Hui (2011); NIST/SEMATECH e-Handbook (Box-Cox)statsdatatransformationsmath

Anleitung: Use it when a variable is skewed, its variance grows with the mean, or predictors need rescaling. Work through the decision table, estimate Box-Cox lambda, fit on the transformed scale, convert coefficients to percent changes and back-transform predictions with smearing. The main pitfall is trying transformations until a test becomes significant; decide the scale in advance and report it.

Vorschau

Data Transformations Method Card: [Your variable or research question]

A method card for transforming variables before analysis: log, log(x + c), square root, reciprocal, Box-Cox, Yeo-Johnson, logit and arcsine for proportions, z-scores and min-max scaling, centring for interactions, and ranks. It shows when a transformation helps, how to estimate the Box-Cox power, how to read regression coefficients after a log transform (as percent changes), and how to back-transform predictions without bias (geometric mean, Duan's smearing). It is for students preparing for an exam, master's and PhD students whose thesis data are skewed, and analysts who need interpretable models. It speeds up learning by putting the decisions in order, showing one example fully solved, one half solved and one left for you, and ending with self-test questions whose answers sit in a separate key. The worked examples use fictional data; replace them with your own. Source basis: Box and Cox (1964), Yeo and Johnson (2000), Duan (1983), Tukey (1977), Warton and Hui (2011), the NIST/SEMATECH e-Handbook of Statistical Methods, and the R and SciPy documentation, all written up in original words and examples.

Document control

FieldValue
Course or project[STAT 201 assignment / thesis chapter 4 / client model]
Learner or analyst[Your name]
Data set[Lakemoor University student help desk wait times, 20 visits (fictional) / your data set and source]
Software and versionsR 4.5.1 (stats, MASS 7.3.65, car 3.1.5 optional); Python 3.14 with numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0
DateYYYY-MM-DD
Version[0.1]

When to use it

One-sentence rule: transform a variable when the transformed scale is more natural for the science (growth, ratios, concentrations) or makes a model's assumptions hold (linearity, constant variance, roughly normal residuals), and when you can still explain the result on the original scale.

Why transform at all? Three good reasons:

  1. The process is multiplicative. Waiting times, incomes, reaction times and concentrations often grow by percentages, not by fixed amounts. On a log scale, multiplicative effects become additive, and a linear model fits.
  2. The variance grows with the mean. Right-skewed positive data usually have more spread where values are large. A log or square root stabilises the variance (counts: square root; positive skewed data: log).
  3. The model needs a common scale. z-scores and min-max scaling put predictors on comparable scales; centring makes main effects interpretable in a model with an interaction.
Use whenDo not use when
Positive, right-skewed outcome whose residuals fan out (log, Box-Cox)You only want a smaller p-value (that is a forking path; see Common mistakes)
The effect is naturally a percentage or ratio (log)The research question is about the arithmetic mean on the original scale (use a GLM with a log link instead)
Predictors on wildly different scales (z-scores)The outcome is a count of events (use Poisson or negative binomial regression)
An interaction makes main effects hard to read (centring)The outcome is successes out of trials (use logistic regression, not arcsine or logit of proportions)
Heavy outliers and a rank-based answer is acceptable (ranks)The data contain zeros or negatives and you would add an arbitrary constant without a sensitivity check

Alternatives: generalised linear models with a log link (Gamma, Poisson, negative binomial) model the mean on the original scale directly (academic/stats-count-regression); robust standard errors handle non-constant variance without changing the scale; rank-based tests (academic/stats-nonparametric-tests); bootstrap confidence intervals (academic/stats-resampling-methods); logistic regression for binary and binomial outcomes (academic/stats-logistic-regression).

The question and the data

Question (example). At the Lakemoor University student help desk (fictional), how much shorter is the wait when more staff are on duty? Outcome: wait time in minutes (ratio scale, strictly positive, right-skewed). Predictors: staff on duty (1, 2 or 3) and arrival hour (9 to 16). Design: 20 visits observed over two weeks.

VisitWait (min)StaffHourVisitWait (min)StaffHour
12.139113.0311
23.4310121.839
31.2311139.7214
45.629144.8212
58.9212152.5310
62.83141611.2115
714.5113177.4213
84.1215183.9311
96.32101930.5116
1022.0116205.1212

Summary on the original scale: mean 7.54 minutes, median 4.95, geometric mean 5.31. The mean is well above the median, the classic sign of right skew.

The histograms below show the same 20 waits on the original scale and on the natural log scale. The long right tail on the left panel disappears on the right panel.

Assumptions and how to check each

RequirementCheckRPythonIf it fails
Log, Box-Cox and reciprocal need x>0x > 0min(x) > 0min(x)x.min()Yeo-Johnson, log(x + c) with a sensitivity check, or a GLM
Proportions for logit lie strictly inside (0, 1)Any 0 or 1?range(p)p.min(), p.max()Logistic regression on counts; empirical logit ln⁡y+0.5n−y+0.5\ln\frac{y + 0.5}{n - y + 0.5}
Model residuals roughly symmetric after transformHistogram, Q-Q plot, skewnessqqnorm(resid(m))sm.qqplot(m.resid)Try another power; use robust or rank methods
Variance roughly constant after transformResiduals vs fitted plotplot(m, which = 1)plt.scatter(m.fittedvalues, m.resid)Try another power; robust (sandwich) SEs; GLM
The relationship is linear on the transformed scaleScatter of transformed y vs xplot(x, log(y))plt.scatter(x, np.log(y))Transform x too, or add a spline
The result can be explained on the original scaleWrite the interpretation sentencePrefer log (percent change) over an odd Box-Cox power

How it works

A transformation replaces each value xx by f(x)f(x) for a fixed, monotone function ff, so the order of the values is kept but the distances between them change. Concave functions (log, square root, negative powers) pull in the right tail and stretch the left; they reduce right skew. Convex functions (squares) do the opposite. Tukey's ladder of powers orders the common choices by strength: x2x^2, xx, x\sqrt{x}, ln⁡x\ln x, −1/x-1/\sqrt{x}, −1/x-1/x. The further down the ladder, the stronger the pull on the right tail. Box and Cox turned the ladder into one formula with a power λ\lambda estimated from the data.

Power and log transforms.

ln⁡(x),ln⁡(x+c)  (c>0 when zeros occur),x,1x\ln(x), \qquad \ln(x + c)\ \ (c > 0 \text{ when zeros occur}), \qquad \sqrt{x}, \qquad \frac{1}{x}

Box-Cox (for x>0x > 0):

x(λ)={xλ−1λ,λ≠0ln⁡x,λ=0x^{(\lambda)} = \begin{cases} \dfrac{x^{\lambda} - 1}{\lambda}, & \lambda \neq 0 \\[2mm] \ln x, & \lambda = 0 \end{cases}

λ\lambda is chosen by maximum likelihood: the value that makes the transformed data (or the residuals of a linear model) look most like a normal sample with constant variance. A 95% confidence interval for λ\lambda contains every value whose profile log-likelihood is within χ1,0.952/2=1.92\chi^2_{1,0.95}/2 = 1.92 of the maximum. In practice you round to a nearby interpretable power inside the interval: 1 (none), 0.5 (square root), 0 (log), -1 (reciprocal).

Yeo-Johnson extends Box-Cox to zero and negative values. For x≥0x \ge 0 it applies the Box-Cox formula to x+1x + 1; for x<0x < 0 it applies the mirror image with power 2−λ2 - \lambda to 1−x1 - x:

ψ(x,λ)={(x+1)λ−1λ,x≥0, λ≠0ln⁡(x+1),x≥0, λ=0−(1−x)2−λ−12−λ,x<0, λ≠2−ln⁡(1−x),x<0, λ=2\psi(x, \lambda) = \begin{cases} \dfrac{(x + 1)^{\lambda} - 1}{\lambda}, & x \ge 0,\ \lambda \neq 0 \\[2mm] \ln(x + 1), & x \ge 0,\ \lambda = 0 \\[2mm] -\dfrac{(1 - x)^{2 - \lambda} - 1}{2 - \lambda}, & x < 0,\ \lambda \neq 2 \\[2mm] -\ln(1 - x), & x < 0,\ \lambda = 2 \end{cases}

Proportions. For a proportion pp strictly between 0 and 1:

logit⁡(p)=ln⁡p1−p,arcsine square root: arcsin⁡p\operatorname{logit}(p) = \ln\frac{p}{1 - p}, \qquad \text{arcsine square root: } \arcsin\sqrt{p}

The logit maps (0, 1) onto the whole real line and is the link of logistic regression. The arcsine square root was once the standard variance-stabiliser for binomial proportions. It is now discouraged: when you have counts of successes out of trials, logistic regression models the data directly and gives interpretable odds ratios, while the arcsine scale has no natural interpretation (Warton and Hui 2011).

Scaling, centring and ranks.

zi=xi−xˉs,mi=xi−min⁡(x)max⁡(x)−min⁡(x),ci=xi−xˉ,ri=rank⁡(xi)z_i = \frac{x_i - \bar{x}}{s}, \qquad m_i = \frac{x_i - \min(x)}{\max(x) - \min(x)}, \qquad c_i = x_i - \bar{x}, \qquad r_i = \operatorname{rank}(x_i)

These are linear (z, min-max, centring) or order-only (ranks). Linear rescaling never changes the shape of a distribution, the correlation between variables or the p-value of a slope; it only changes units. Ranks throw away the distances and keep the order, which is what rank-based tests and Spearman's correlation use.

Interpreting a log-outcome coefficient. In the model ln⁡y=β0+β1x+ε\ln y = \beta_0 + \beta_1 x + \varepsilon, a one-unit increase in xx multiplies the (geometric) mean of yy by eβ1e^{\beta_1}:

%Δy=100(eβ1−1)\%\Delta y = 100\left(e^{\beta_1} - 1\right)

For small β1\beta_1 (say ∣β1∣<0.1|\beta_1| < 0.1), 100β1100\beta_1 is a good approximation; for larger values it is not (β1=−0.99\beta_1 = -0.99 means a 63% fall, not 99%).

Back-transformation bias. Because the log is concave, the mean of the logs is smaller than the log of the mean (Jensen's inequality). So exp⁡(ln⁡y‾)\exp(\overline{\ln y}) is the geometric mean, which for skewed data is below the arithmetic mean, and exp⁡(β^0+β^1x)\exp(\hat\beta_0 + \hat\beta_1 x) estimates the median of yy at xx (if the log-scale errors are symmetric), not its mean. Duan's smearing estimator corrects the prediction of the mean without assuming normality:

E^(y∣x)=exp⁡(β^0+β^1x)⋅1n∑i=1neε^i\widehat{E}(y \mid x) = \exp\left(\hat\beta_0 + \hat\beta_1 x\right) \cdot \frac{1}{n}\sum_{i=1}^{n} e^{\hat\varepsilon_i}

Under normal log-scale errors with variance σ2\sigma^2 the factor is eσ2/2e^{\sigma^2/2}; smearing replaces that assumption with the average of the exponentiated residuals.

SymbolMeaning
x,yx, yOriginal-scale predictor and outcome
λ\lambdaBox-Cox or Yeo-Johnson power parameter
ccConstant added before a log when zeros occur
ppProportion between 0 and 1
xˉ,s\bar{x}, sSample mean and standard deviation (divisor n−1n-1)
β0,β1\beta_0, \beta_1Intercept and slope of a regression on the transformed scale
ε^i\hat\varepsilon_iResidual of observation ii on the transformed scale
nnSample size
%Δy\%\Delta yPercent change in yy per one-unit change in xx

Which transformation, in what order of questions:

Decision table

SituationTransformationInterpretation after transformBack-transformCaution
Positive, right-skewed, multiplicative (times, incomes, concentrations)ln⁡x\ln xPercent change; ratio of geometric meansevaluee^{\text{value}} gives geometric mean or median; smear for the meanNot for zeros
Contains zeros, otherwise as aboveln⁡(x+c)\ln(x + c) or Yeo-JohnsonApproximate percent change for large xx onlyevalue−ce^{\text{value}} - cResults depend on cc; report a sensitivity check; consider a GLM
Counts with variance about equal to the meanx\sqrt{x}Hard to interpretSquare, with biasPrefer Poisson regression
Strong right skew, rates per unit (time per task)1/x1/xRate (tasks per minute)1/value1/\text{value}; order reversesUse −1/x-1/x to keep the order
Unknown best power, positive dataBox-Cox, λ\lambda by MLOnly if λ\lambda is rounded to 0, 0.5, 1 or -1Inverse Box-CoxOdd powers like 0.37 are hard to explain
Any sign, need symmetryYeo-JohnsonMostly for prediction or screeningInverse Yeo-JohnsonCoefficients lose a simple meaning
Proportion from a continuous measure (share of time)logitLog-odds; odds ratio per unit1/(1+e−value)1/(1 + e^{-\text{value}})Needs 0<p<10 < p < 1
Proportion from counts (passed out of enrolled)None: logistic regressionOdds ratioPredicted probabilityArcsine square root is discouraged
Predictors on different scalesz-scoreChange per 1 SDx=xˉ+szx = \bar{x} + s zUnits become sample-specific
Inputs for distance-based methodsMin-max to [0, 1]Relative position in observed rangex=min⁡+m(max⁡−min⁡)x = \min + m(\max - \min)One outlier squashes the rest
Model with an interactionCentre at mean or meaningful valueMain effect = effect at the centre of the other variableAdd the centre backDoes not change the interaction slope or fit
Heavy outliers, ordinal questionRanksOrder onlyNot possibleDistances are lost

Worked example A (fully solved): log-transforming wait times

Hypotheses. H0H_0: staffing level is not related to wait time (β1=0\beta_1 = 0 on the log scale). H1H_1: it is (β1≠0\beta_1 \neq 0), two-sided, α=0.05\alpha = 0.05.

Step 1. Shape before and after. Raw waits: skewness g1g_1 = 1.93, Shapiro-Wilk p = .0002. Log waits: skewness 0.34, Shapiro-Wilk p = .986. The square root (skewness 1.19, p = .033) and the reciprocal (1.37, p = .017) are worse choices here. Shapiro-Wilk is used as a descriptive aid only: with large samples it rejects trivial departures, with small ones it misses real ones.

Step 2. Box-Cox. The maximum-likelihood power is λ^=−0.179\hat\lambda = -0.179, 95% CI [-0.688, 0.318]. The interval contains 0 (log) and excludes 0.5 (square root) and 1 (no transformation), so the log is supported by the data and gives the most interpretable scale. When Box-Cox is estimated for the regression residuals (given staff), R gives λ^=−0.024\hat\lambda = -0.024, even closer to 0. Box-Cox targets the distribution of the model's errors, so estimate it with the predictors you will use.

Step 3. Fit the model on the log scale. ln⁡y^=3.8552−0.9939×staff\widehat{\ln y} = 3.8552 - 0.9939 \times \text{staff}; 95% CI for the slope [-1.2168, -0.7710].

Step 4. Convert to percent change (by hand). e−0.9939=0.370e^{-0.9939} = 0.370, so each extra staff member multiplies the typical wait by 0.37: a change of 100(0.370−1)=−63.0%100(0.370 - 1) = -63.0\%. CI: 100(e−1.2168−1)=−70.4%100(e^{-1.2168} - 1) = -70.4\% and 100(e−0.7710−1)=−53.7%100(e^{-0.7710} - 1) = -53.7\%.

Step 5. Back-transform predictions. For one staff member: e3.8552−0.9939=17.48e^{3.8552 - 0.9939} = 17.48 minutes. That is the predicted median (and geometric mean) wait. The smearing factor is 120∑eε^i=1.0567\frac{1}{20}\sum e^{\hat\varepsilon_i} = 1.0567, so the predicted mean wait is 17.48×1.0567=18.4817.48 \times 1.0567 = 18.48 minutes. The smeared mean is 5.7% above the naive back-transform, so the naive value understates the mean wait.

StaffNaive eμ^e^{\hat\mu} (median)Smeared (mean)Observed meanObserved geometric mean
117.4818.4819.5518.17
26.476.846.496.23
32.402.532.592.44

The naive predictions track the observed geometric means, and the smeared predictions track the observed arithmetic means for 1 and 3 staff. For 2 staff the observed mean is lower than the smeared prediction for two reasons: the fitted line sits slightly above the 2-staff geometric mean (6.47 vs 6.23), and the 8 visits with 2 staff happen to have less log-scale spread than the others, while smearing uses one pooled correction for all groups (it assumes constant variance on the log scale).

R code (base R, stats and MASS; car is optional):

library(MASS)   # ships with R; boxcox()
desk <- data.frame(
  wait  = c(2.1,3.4,1.2,5.6,8.9,2.8,14.5,4.1,6.3,22.0,3.0,1.8,9.7,4.8,2.5,11.2,7.4,3.9,30.5,5.1),
  staff = c(3,3,3,2,2,3,1,2,2,1,3,3,2,2,3,1,2,3,1,2),
  hour  = c(9,10,11,9,12,14,13,15,10,16,11,9,14,12,10,15,13,11,16,12)
)
# Box-Cox: profile log-likelihood over a fine grid of lambda
bc0 <- boxcox(lm(wait ~ 1, data = desk), lambda = seq(-2, 2, 0.001), plotit = FALSE)
lam0 <- bc0$x[which.max(bc0$y)]
ci0 <- range(bc0$x[bc0$y > max(bc0$y) - qchisq(0.95, 1) / 2])
cat("Box-Cox lambda (no predictors):", lam0, " 95% CI:", ci0, "\n")
bc1 <- boxcox(lm(wait ~ staff, data = desk), lambda = seq(-2, 2, 0.001), plotit = FALSE)
cat("Box-Cox lambda (given staff):  ", bc1$x[which.max(bc1$y)], "\n")
# Log-outcome model and percent change
m <- lm(log(wait) ~ staff, data = desk)
round(coef(m), 4)
round(100 * (exp(coef(m)["staff"]) - 1), 1)
round(100 * (exp(confint(m)["staff", ]) - 1), 1)
smear <- mean(exp(resid(m)))
round(smear, 4)
nd <- data.frame(staff = 1:3)
round(cbind(nd, naive = exp(predict(m, nd)), smeared = exp(predict(m, nd)) * smear), 3)
# Yeo-Johnson and Box-Cox via car (optional): install.packages("car")
if (requireNamespace("car", quietly = TRUE)) {
  print(round(car::powerTransform(desk$wait, family = "yjPower")$lambda, 4))
  print(round(car::powerTransform(desk$wait, family = "bcPower")$lambda, 4))
}
head(round(as.numeric(scale(desk$wait)), 3), 5)   # z-scores
head(rank(desk$wait), 5)                          # ranks
Box-Cox lambda (no predictors): -0.179  95% CI: -0.688 0.317
Box-Cox lambda (given staff):   -0.024
(Intercept)       staff
     3.8552     -0.9939
staff
  -63
 2.5 % 97.5 %
 -70.4  -53.7
[1] 1.0567
  staff  naive smeared
1     1 17.484  18.476
2     2  6.471   6.839
3     3  2.395   2.531
desk$wait
  -0.3954
desk$wait
  -0.1786
[1] -0.736 -0.560 -0.858 -0.263  0.184
[1]  3  7  1 12 15

A practical R note: name the data frame something other than df. MASS::boxcox() refits the model internally, and with a data frame called df it picks up R's df() function (the F density) instead, failing with "'data' must be a data.frame, environment, or list". The same refit fails inside your own function on a local data frame. Fitting the model with lm(..., y = TRUE, qr = TRUE) avoids the refit; the example simply uses the name desk.

Python code (numpy, pandas, scipy, statsmodels):

import numpy as np, pandas as pd
from scipy import stats
import statsmodels.formula.api as smf

desk = pd.DataFrame({
    "wait":  [2.1,3.4,1.2,5.6,8.9,2.8,14.5,4.1,6.3,22.0,3.0,1.8,9.7,4.8,2.5,11.2,7.4,3.9,30.5,5.1],
    "staff": [3,3,3,2,2,3,1,2,2,1,3,3,2,2,3,1,2,3,1,2],
    "hour":  [9,10,11,9,12,14,13,15,10,16,11,9,14,12,10,15,13,11,16,12],
})
w = desk["wait"]

# 1. Shape before and after
for name, x in [("raw", w), ("log", np.log(w)), ("sqrt", np.sqrt(w)), ("recip", 1/w)]:
    print(f"{name:6s} skew = {stats.skew(x):6.3f}  Shapiro p = {stats.shapiro(x).pvalue:.4f}")

# 2. Box-Cox and Yeo-Johnson (marginal, no predictors)
bc, lam, ci = stats.boxcox(w, alpha=0.05)
print("Box-Cox lambda =", round(lam, 4), " 95% CI =", np.round(ci, 3))
yj, lam_yj = stats.yeojohnson(w)
print("Yeo-Johnson lambda =", round(lam_yj, 4))

# 3. Means on the original and log scale
print("mean =", round(w.mean(), 3), " median =", round(w.median(), 3),
      " geometric mean =", round(np.exp(np.log(w).mean()), 3))

# 4. Log-outcome regression and percent change
m = smf.ols("np.log(wait) ~ staff", data=desk).fit()
b, (lo, hi) = m.params["staff"], m.conf_int().loc["staff"]
print("b0 =", round(m.params["Intercept"], 4), " b1 =", round(b, 4), " 95% CI", round(lo, 4), round(hi, 4))
print("percent change per extra staff =", round(100*np.expm1(b), 1),
      " 95% CI", round(100*np.expm1(lo), 1), round(100*np.expm1(hi), 1))

# 5. Back-transformation: naive vs Duan smearing
smear = np.mean(np.exp(m.resid))
print("smearing factor =", round(smear, 4))
for s in (1, 2, 3):
    naive = np.exp(m.params["Intercept"] + b*s)
    obs = desk.loc[desk.staff == s, "wait"]
    print(f"staff={s}: naive = {naive:.3f}  smeared = {naive*smear:.3f}  "
          f"observed mean = {obs.mean():.3f}  observed geo mean = {np.exp(np.log(obs).mean()):.3f}")

# 6. Scaling and ranks (first five rows)
print("z       :", np.round(np.asarray(stats.zscore(w, ddof=1))[:5], 3))
print("min-max :", np.round(((w - w.min())/(w.max() - w.min()))[:5].to_numpy(), 3))
print("rank    :", stats.rankdata(w)[:5])

# 7. Centring before an interaction
x, h = desk["staff"], desk["hour"]
print("corr(staff, staff*hour) raw =", round(np.corrcoef(x, x*h)[0, 1], 3),
      " centred =", round(np.corrcoef(x - x.mean(), (x - x.mean())*(h - h.mean()))[0, 1], 3))

# 8. Proportions
p = np.array([0.62, 0.95, 0.80, 0.50, 0.98])
print("logit   :", np.round(np.log(p/(1 - p)), 3))
print("arcsine :", np.round(np.arcsin(np.sqrt(p)), 3))
raw    skew =  1.927  Shapiro p = 0.0002
log    skew =  0.341  Shapiro p = 0.9855
sqrt   skew =  1.191  Shapiro p = 0.0326
recip  skew =  1.368  Shapiro p = 0.0171
Box-Cox lambda = -0.1786  95% CI = [-0.688  0.318]
Yeo-Johnson lambda = -0.3954
mean = 7.54  median = 4.95  geometric mean = 5.305
b0 = 3.8552  b1 = -0.9939  95% CI -1.2168 -0.771
percent change per extra staff = -63.0  95% CI -70.4 -53.7
smearing factor = 1.0567
staff=1: naive = 17.484  smeared = 18.476  observed mean = 19.550  observed geo mean = 18.169
staff=2: naive = 6.471  smeared = 6.839  observed mean = 6.487  observed geo mean = 6.228
staff=3: naive = 2.395  smeared = 2.531  observed mean = 2.587  observed geo mean = 2.442
z       : [-0.736 -0.56  -0.858 -0.263  0.184]
min-max : [0.031 0.075 0.    0.15  0.263]
rank    : [ 3.  7.  1. 12. 15.]
corr(staff, staff*hour) raw = 0.858  centred = 0.375
logit   : [0.49  2.944 1.386 0.    3.892]
arcsine : [0.907 1.345 1.107 0.785 1.429]

R and Python compared. The Box-Cox λ^\hat\lambda agrees (-0.1786 from car::powerTransform and SciPy; -0.179 from the MASS::boxcox grid). The upper CI limit is 0.317 in R and 0.318 in SciPy only because R searched a grid with step 0.001. scipy.stats.boxcox estimates λ\lambda for the variable alone; MASS::boxcox works on a fitted model, so it can estimate λ\lambda for the residuals given predictors (-0.024 here). Yeo-Johnson agrees (-0.3954). The z-scores agree because scale() in R and zscore(ddof=1) both divide by ss with n−1n-1; SciPy's default ddof=0 would give slightly larger values.

Interpretation. Waits are right-skewed and multiplicative, and the Box-Cox interval supports a log scale. On that scale, each extra staff member is associated with a 63% shorter typical wait. Predictions back-transformed with smearing estimate the mean wait; without smearing they estimate the median.

APA-style report sentence. "Wait times were right-skewed (M = 7.54 min, Mdn = 4.95) and were log-transformed (Box-Cox λ^\hat\lambda = -0.18, 95% CI [-0.69, 0.32]). Each additional staff member on duty was associated with a 63.0% shorter wait, bb = -0.99, 95% CI [-1.22, -0.77] on the log scale, corresponding to a 53.7% to 70.4% reduction, n = 20. Predicted mean waits (Duan smearing) were 18.5, 6.8 and 2.5 minutes for one, two and three staff."

Interpreting coefficients after log transforms

ModelCoefficient bb meansExact conversionExample
y=β0+β1xy = \beta_0 + \beta_1 x (level-level)Change in yy units per 1 unit of xxnoneb=2b = 2: 2 more points per hour
ln⁡y=β0+β1x\ln y = \beta_0 + \beta_1 x (log-level)Multiplicative change in yy per 1 unit of xx100(eb−1)100(e^{b} - 1)%b=−0.99b = -0.99: 63.0% shorter wait per extra staff
y=β0+β1ln⁡xy = \beta_0 + \beta_1 \ln x (level-log)Change in yy units per 1% increase in xxbln⁡(1.01)≈b/100b \ln(1.01) \approx b/100b=5b = 5: 0.05 points per 1% more hours; 0.48 points per 10% more
ln⁡y=β0+β1ln⁡x\ln y = \beta_0 + \beta_1 \ln x (log-log)Elasticity: % change in yy per 1% change in xx100(1.01b−1)100(1.01^{b} - 1)%b=0.4b = 0.4: 0.4% per 1%; 3.9% per 10%
ln⁡y\ln y with a binary xxRatio of geometric means between groupsebe^{b}b=0.25b = 0.25: group 1 is 28.4% higher

Use base-10 logs only if the audience thinks in powers of ten (concentrations, decibels); the conversion is then 10b10^{b}, not ebe^{b}. State which log you used. In R and NumPy, log() is the natural log.

Worked example B (half solved): prediction and two-step change

Using ln⁡y^=3.8552−0.9939×staff\widehat{\ln y} = 3.8552 - 0.9939 \times \text{staff} and the smearing factor 1.0567:

  1. Predicted log wait with 2 staff: 3.8552−0.9939×2=3.8552 - 0.9939 \times 2 = [ ].
  2. Predicted median wait with 2 staff: e[ ]=e^{[\ ]} = [ ] minutes.
  3. Predicted mean wait with 2 staff (smearing): [ ] ×1.0567=\times 1.0567 = [ ] minutes.
  4. Percent change in typical wait from 1 to 3 staff (two extra): 100(e2×(−0.9939)−1)=100(e^{2 \times (-0.9939)} - 1) = [ ]%.
  5. Why is the answer to step 4 not simply 2×(−63.0%)=−126%2 \times (-63.0\%) = -126\%? [ ]
  6. The observed mean with 2 staff is 6.49 minutes. Which prediction, median or smeared mean, should you compare it with, and why do they differ? [ ]

Exercise C (unsolved): interpretation drill

  1. Harbour City Library (fictional) records four daily e-book download counts: 2, 8, 4, 16. Compute the arithmetic mean and the geometric mean. Which is a better "typical day" for this multiplicative series, and why?
  2. A log-level model of Brightwell Bakery (fictional) batch waste gives b=−0.40b = -0.40 for a new mixer (binary). State the effect as a percent change.
  3. A log-log model gives b=0.25b = 0.25 for hours of oven time on energy use. Interpret it for a 10% increase in oven time.
  4. In a log-level model a binary predictor has b=0.25b = 0.25. By what percent is the outcome higher in group 1?
  5. A course pass rate is 0.80. Compute its logit and explain why logistic regression on the pass counts is preferred to an arcsine transform of the rate.
  6. You centre "hour" at its mean before fitting log(wait) ~ staff * hour. Which coefficients change meaning, and which do not change at all?

Reading software output

Output lineMeaningReport it?
Box-Cox lambda ... 95% CIML power and its likelihood intervalYes, when you justify the transformation
(Intercept) 3.8552Predicted log wait at staff = 0 (outside the data)Rarely; centre staff for a meaningful intercept
staff -0.9939Change in log wait per extra staffYes, with the percent conversion
2.5 % 97.5 % after exp()CI on the percent-change scaleYes
smearing factor 1.0567Correction from median to mean on the original scaleYes, when you report predicted means
naive vs smearedPredicted median vs predicted meanSay which one you report
Yeo-Johnson lambdaPower for data with zeros or negativesOnly if you use it
Shapiro pDescriptive check of shapeOptional; prefer plots

Reporting template

Fill-in sentence. "[Outcome] was [right-skewed / heteroscedastic] (M = [ ], Mdn = [ ]) and was [log / square-root / Box-Cox] transformed [because the process is multiplicative / Box-Cox λ^\hat\lambda = [ ], 95% CI [ , ]]. Each one-unit increase in [predictor] was associated with a [ ]% [higher / lower] [geometric mean / median] [outcome], bb = [ ], 95% CI [ , ] on the log scale. Predicted means on the original scale used [Duan smearing / the normal-theory correction]."

VariableTransformationReasonDecided before seeing results?Back-transform method
Example: waitnatural logmultiplicative process; Box-Cox CI includes 0Yes (analysis plan)Duan smearing
Example: hourcentred at 12.1interaction with staffYesadd 12.1
[variable][ ][ ][ ][ ]

Results checklist.

  • Reason for the transformation stated (science or diagnostics), and decided before looking at p-values.
  • The base of the log stated (natural log unless said otherwise).
  • Constant cc reported, with a sensitivity check, if ln⁡(x+c)\ln(x + c) was used.
  • Coefficients converted to percent changes or ratios, with CIs converted the same way.
  • Back-transformed predictions labelled as medians (geometric means) or means (smearing).
  • Residual plots checked on the transformed scale.
  • Original-scale descriptive statistics also reported.

Common mistakes and misconceptions

Mistake or misconceptionWhy it is wrongDo this instead
Trying transformations until the test is significantChoosing among many analyses after seeing results inflates the false positive rate (a "garden of forking paths")Decide the scale from the science or from diagnostics on the residuals, write it in the analysis plan, and report it
"The outcome must be normal, so I transform it"Linear models assume normal errors, not a normal outcome; a skewed outcome can have normal residuals once predictors are inCheck residuals of the fitted model, not the raw histogram
Reading b=−0.99b = -0.99 on a log outcome as "99% lower"The exact change is 100(eb−1)=−63.0%100(e^{b} - 1) = -63.0\%; 100b100b only works for small bbConvert with ebe^{b}
Back-transforming a mean of logs and calling it the meanexp⁡(ln⁡y‾)\exp(\overline{\ln y}) is the geometric mean (5.31 vs 7.54 here)Call it the geometric mean or median, or apply smearing
Adding 1 before a log without thinkingThe result depends on cc; with values near 0, ln⁡(x+1)\ln(x + 1) behaves almost linearlyReport cc, check cc = 0.5 and 1, or use a GLM or Yeo-Johnson
Using arcsine square root on proportions from countsDiscouraged: no interpretable scale and ignores the binomial structureLogistic regression on successes and failures
Expecting z-scores to "normalise" the dataz-scores are linear; the shape is unchangedUse z-scores for scale, a power transform for shape
Believing centring removes "real" multicollinearityCentring lowers the correlation between xx and x⋅zx \cdot z (0.86 to 0.38 here) but does not change the fit or the interaction slopeCentre for interpretation of main effects, not to fix inference
Transforming a Likert item or a bounded score with a logBounded scales are not multiplicativeUse ordinal models or treat the summed scale as interval
Reporting only transformed-scale numbersReaders cannot judge practical size in log minutesGive original-scale summaries and back-transformed effects

Self-test questions

  1. Box-Cox gives λ^\hat\lambda = 0.42 with 95% CI [0.18, 0.69]. Which simple transformation would you choose, and why not the exact 0.42?
  2. In a log-level regression, bb = 0.08. Give the exact and the approximate percent change. When does the approximation break down?
  3. You back-transform the mean of log incomes and get 31,200 USD; the arithmetic mean is 38,900 USD. Which is "wrong"?
  4. Why does scipy.stats.boxcox and MASS::boxcox(lm(y ~ x)) sometimes give different λ\lambda for the same yy?
  5. A colleague runs a t-test on raw data (p = .09), then on log data (p = .04), then reports the log version. What is wrong, and what should have happened?
  6. Does converting predictors to z-scores change the p-value of their slopes in a linear regression? Does it change the slopes?

My notes and next steps

ItemEntry
Confidence choosing a transformation (1-5)[ ]
Confidence interpreting log coefficients (1-5)[ ]
Confidence back-transforming (1-5)[ ]
Open questions[What I am still unsure about]
Mistakes to log[Link to academic/stats-error-log entry]
Next review datesYYYY-MM-DD (2 days), YYYY-MM-DD (1 week), YYYY-MM-DD (1 month)
Next template[academic/stats-linear-regression / academic/stats-count-regression]

Sources and further reading

Based on the original method papers listed below and the NIST/SEMATECH e-Handbook of Statistical Methods; all text, data and examples are original.

  • Box, G. E. P., and Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society, Series B, 26(2), 211–243.
  • Yeo, I.-K., and Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87(4), 954–959.
  • Duan, N. (1983). Smearing estimate: a nonparametric retransformation method. Journal of the American Statistical Association, 78(383), 605–610.
  • Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley.
  • Warton, D. I., and Hui, F. K. C. (2011). The arcsine is asinine: the analysis of proportions in ecology. Ecology, 92(1), 3–10.
  • Gelman, A., and Loken, E. (2013). The garden of forking paths: why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Working paper, Department of Statistics, Columbia University.
  • NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.3.5 Box-Cox Linearity Plot, and section 1.3.3.6 Box-Cox Normality Plot, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain.
  • Further reading (by name only): de Smith, M. J. (2024). Statistical Analysis Handbook (2024 edition, StatsRef); Diez, D. M., Çetinkaya-Rundel, M., and Barr, C. D., OpenIntro Statistics (CC BY-SA).
  • Software documentation: R MASS::boxcox(), car::powerTransform(), scale(); SciPy stats.boxcox(), stats.yeojohnson(), stats.zscore(); statsmodels ols.

Answer key

Worked example B. (1) 3.8552−1.9878=1.86743.8552 - 1.9878 = 1.8674. (2) e1.8674=6.47e^{1.8674} = 6.47 minutes (median and geometric mean). (3) 6.47×1.0567=6.846.47 \times 1.0567 = 6.84 minutes. (4) e−1.9878=0.137e^{-1.9878} = 0.137, so 100(0.137−1)=−86.3%100(0.137 - 1) = -86.3\%. (5) Percent changes multiply rather than add: two steps of ×0.370\times 0.370 give ×0.3702=×0.137\times 0.370^2 = \times 0.137, an 86.3% fall. A change of -126% is impossible for a positive quantity. (6) Compare the observed mean (6.49) with the smeared mean (6.84), and the observed geometric mean (6.23) with the median prediction (6.47). The smeared mean is above the observed mean because the fitted median is already a little above the observed geometric mean, and smearing uses one pooled correction while the 2-staff visits have less log-scale spread than the others.

Exercise C. (1) Arithmetic mean =30/4=7.5= 30/4 = 7.5; geometric mean =(2×8×4×16)1/4=10241/4=5.66= (2 \times 8 \times 4 \times 16)^{1/4} = 1024^{1/4} = 5.66. The geometric mean, because the values differ by factors of 2 and the arithmetic mean is pulled up by 16. (2) 100(e−0.40−1)=−33.0%100(e^{-0.40} - 1) = -33.0\%: about one third less waste with the new mixer. (3) 100(1.10.25−1)=2.4%100(1.1^{0.25} - 1) = 2.4\% more energy for 10% more oven time (elasticity 0.25). (4) 100(e0.25−1)=28.4%100(e^{0.25} - 1) = 28.4\% higher. (5) ln⁡(0.8/0.2)=ln⁡4=1.386\ln(0.8/0.2) = \ln 4 = 1.386. Logistic regression uses the number passed and the number enrolled, weights courses by their size, keeps predictions inside 0 to 1 and gives odds ratios; the arcsine scale has no interpretation and ignores the binomial variance. (6) The coefficient of staff now means the effect of staff at the mean hour (not at hour 0, which is outside the data); the intercept changes too. The interaction coefficient, its p-value, the residuals and R2R^2 do not change.

Self-test.

  1. Square root (λ=0.5\lambda = 0.5): it is inside the CI and has a familiar meaning; 0.42 is barely better in likelihood and impossible to explain. The log (0) and no transform (1) are both outside the CI.
  2. Exact: 100(e0.08−1)=8.3%100(e^{0.08} - 1) = 8.3\%; approximate: 8%. The approximation drifts once ∣b∣|b| exceeds about 0.1 (for b=0.5b = 0.5: 64.9% exact vs 50% approximate).
  3. Neither: 31,200 USD is the geometric mean (close to the median for log-symmetric data), 38,900 USD is the arithmetic mean. They answer different questions; label them.
  4. SciPy estimates λ\lambda for the marginal distribution of yy; MASS::boxcox on a model estimates it for the residuals given xx. Group or predictor differences make the marginal distribution more skewed than the residuals (here -0.18 vs -0.02).
  5. The transformation was chosen because it gave a significant result, so the reported p-value does not have its stated meaning. The scale should have been chosen in advance (science or residual diagnostics) and both analyses reported as a sensitivity check.
  6. The p-values (and t statistics) do not change. The slopes do change: they become effects per one SD of the predictor instead of per original unit.

Revision history

VersionDateAuthorChange summary
0.1YYYY-MM-DD[Your name]First draft from the template
0.2YYYY-MM-DD[Your name]Example: Replaced Lakemoor example with thesis reaction times; log chosen in analysis plan

Ähnliche Vorlagen