# Data Transformations Method Card: [Your variable or research question] > A method card for transforming variables before analysis: log, log(x + c), square root, reciprocal, Box-Cox, > Yeo-Johnson, logit and arcsine for proportions, z-scores and min-max scaling, centring for interactions, and ranks. > It shows when a transformation helps, how to estimate the Box-Cox power, how to read regression coefficients after a > log transform (as percent changes), and how to back-transform predictions without bias (geometric mean, Duan's > smearing). It is for students preparing for an exam, master's and PhD students whose thesis data are skewed, and > analysts who need interpretable models. It speeds up learning by putting the decisions in order, showing one example > fully solved, one half solved and one left for you, and ending with self-test questions whose answers sit in a > separate key. The worked examples use fictional data; replace them with your own. > Source basis: Box and Cox (1964), Yeo and Johnson (2000), Duan (1983), Tukey (1977), Warton and Hui (2011), the > NIST/SEMATECH e-Handbook of Statistical Methods, and the R and SciPy documentation, all written up in original words > and examples. ## Document control | Field | Value | |---|---| | Course or project | [STAT 201 assignment / thesis chapter 4 / client model] | | Learner or analyst | [Your name] | | Data set | [Lakemoor University student help desk wait times, 20 visits (fictional) / your data set and source] | | Software and versions | R 4.5.1 (stats, MASS 7.3.65, car 3.1.5 optional); Python 3.14 with numpy 2.5.4, pandas 3.1.0, scipy 1.18.1, statsmodels 0.15.0 | | Date | YYYY-MM-DD | | Version | [0.1] | ## When to use it **One-sentence rule:** transform a variable when the transformed scale is more natural for the science (growth, ratios, concentrations) or makes a model's assumptions hold (linearity, constant variance, roughly normal residuals), and when you can still explain the result on the original scale. Why transform at all? Three good reasons: 1. **The process is multiplicative.** Waiting times, incomes, reaction times and concentrations often grow by percentages, not by fixed amounts. On a log scale, multiplicative effects become additive, and a linear model fits. 2. **The variance grows with the mean.** Right-skewed positive data usually have more spread where values are large. A log or square root stabilises the variance (counts: square root; positive skewed data: log). 3. **The model needs a common scale.** z-scores and min-max scaling put predictors on comparable scales; centring makes main effects interpretable in a model with an interaction. | Use when | Do not use when | |---|---| | Positive, right-skewed outcome whose residuals fan out (log, Box-Cox) | You only want a smaller p-value (that is a forking path; see Common mistakes) | | The effect is naturally a percentage or ratio (log) | The research question is about the arithmetic mean on the original scale (use a GLM with a log link instead) | | Predictors on wildly different scales (z-scores) | The outcome is a count of events (use Poisson or negative binomial regression) | | An interaction makes main effects hard to read (centring) | The outcome is successes out of trials (use logistic regression, not arcsine or logit of proportions) | | Heavy outliers and a rank-based answer is acceptable (ranks) | The data contain zeros or negatives and you would add an arbitrary constant without a sensitivity check | **Alternatives:** generalised linear models with a log link (Gamma, Poisson, negative binomial) model the mean on the original scale directly (`academic/stats-count-regression`); robust standard errors handle non-constant variance without changing the scale; rank-based tests (`academic/stats-nonparametric-tests`); bootstrap confidence intervals (`academic/stats-resampling-methods`); logistic regression for binary and binomial outcomes (`academic/stats-logistic-regression`). ## The question and the data **Question (example).** At the Lakemoor University student help desk (fictional), how much shorter is the wait when more staff are on duty? **Outcome:** wait time in minutes (ratio scale, strictly positive, right-skewed). **Predictors:** staff on duty (1, 2 or 3) and arrival hour (9 to 16). **Design:** 20 visits observed over two weeks. | Visit | Wait (min) | Staff | Hour | | Visit | Wait (min) | Staff | Hour | |---|---|---|---|---|---|---|---|---| | 1 | 2.1 | 3 | 9 | | 11 | 3.0 | 3 | 11 | | 2 | 3.4 | 3 | 10 | | 12 | 1.8 | 3 | 9 | | 3 | 1.2 | 3 | 11 | | 13 | 9.7 | 2 | 14 | | 4 | 5.6 | 2 | 9 | | 14 | 4.8 | 2 | 12 | | 5 | 8.9 | 2 | 12 | | 15 | 2.5 | 3 | 10 | | 6 | 2.8 | 3 | 14 | | 16 | 11.2 | 1 | 15 | | 7 | 14.5 | 1 | 13 | | 17 | 7.4 | 2 | 13 | | 8 | 4.1 | 2 | 15 | | 18 | 3.9 | 3 | 11 | | 9 | 6.3 | 2 | 10 | | 19 | 30.5 | 1 | 16 | | 10 | 22.0 | 1 | 16 | | 20 | 5.1 | 2 | 12 | Summary on the original scale: mean 7.54 minutes, median 4.95, geometric mean 5.31. The mean is well above the median, the classic sign of right skew. The histograms below show the same 20 waits on the original scale and on the natural log scale. The long right tail on the left panel disappears on the right panel. ```vega-lite { "$schema": "https://vega.github.io/schema/vega-lite/v5.json", "title": "Help desk wait: original scale and log scale (fictional)", "data": {"values": [ {"wait": 2.1}, {"wait": 3.4}, {"wait": 1.2}, {"wait": 5.6}, {"wait": 8.9}, {"wait": 2.8}, {"wait": 14.5}, {"wait": 4.1}, {"wait": 6.3}, {"wait": 22.0}, {"wait": 3.0}, {"wait": 1.8}, {"wait": 9.7}, {"wait": 4.8}, {"wait": 2.5}, {"wait": 11.2}, {"wait": 7.4}, {"wait": 3.9}, {"wait": 30.5}, {"wait": 5.1} ]}, "transform": [{"calculate": "log(datum.wait)", "as": "log_wait"}], "hconcat": [ { "width": 180, "height": 180, "mark": "bar", "encoding": { "x": {"field": "wait", "bin": {"step": 4, "extent": [0, 32]}, "title": "Wait (minutes)"}, "y": {"aggregate": "count", "title": "Visits"} } }, { "width": 180, "height": 180, "mark": "bar", "encoding": { "x": {"field": "log_wait", "bin": {"step": 0.5, "extent": [0, 3.5]}, "title": "ln(wait)"}, "y": {"aggregate": "count", "title": "Visits"} } } ] } ``` ## Assumptions and how to check each | Requirement | Check | R | Python | If it fails | |---|---|---|---|---| | Log, Box-Cox and reciprocal need $x > 0$ | `min(x) > 0` | `min(x)` | `x.min()` | Yeo-Johnson, log(x + c) with a sensitivity check, or a GLM | | Proportions for logit lie strictly inside (0, 1) | Any 0 or 1? | `range(p)` | `p.min(), p.max()` | Logistic regression on counts; empirical logit $\ln\frac{y + 0.5}{n - y + 0.5}$ | | Model residuals roughly symmetric after transform | Histogram, Q-Q plot, skewness | `qqnorm(resid(m))` | `sm.qqplot(m.resid)` | Try another power; use robust or rank methods | | Variance roughly constant after transform | Residuals vs fitted plot | `plot(m, which = 1)` | `plt.scatter(m.fittedvalues, m.resid)` | Try another power; robust (sandwich) SEs; GLM | | The relationship is linear on the transformed scale | Scatter of transformed y vs x | `plot(x, log(y))` | `plt.scatter(x, np.log(y))` | Transform x too, or add a spline | | The result can be explained on the original scale | Write the interpretation sentence | | | Prefer log (percent change) over an odd Box-Cox power | ## How it works A transformation replaces each value $x$ by $f(x)$ for a fixed, monotone function $f$, so the order of the values is kept but the distances between them change. **Concave** functions (log, square root, negative powers) pull in the right tail and stretch the left; they reduce right skew. **Convex** functions (squares) do the opposite. Tukey's **ladder of powers** orders the common choices by strength: $x^2$, $x$, $\sqrt{x}$, $\ln x$, $-1/\sqrt{x}$, $-1/x$. The further down the ladder, the stronger the pull on the right tail. Box and Cox turned the ladder into one formula with a power $\lambda$ estimated from the data. **Power and log transforms.** $$ \ln(x), \qquad \ln(x + c)\ \ (c > 0 \text{ when zeros occur}), \qquad \sqrt{x}, \qquad \frac{1}{x} $$ **Box-Cox** (for $x > 0$): $$ x^{(\lambda)} = \begin{cases} \dfrac{x^{\lambda} - 1}{\lambda}, & \lambda \neq 0 \\[2mm] \ln x, & \lambda = 0 \end{cases} $$ $\lambda$ is chosen by maximum likelihood: the value that makes the transformed data (or the residuals of a linear model) look most like a normal sample with constant variance. A 95% confidence interval for $\lambda$ contains every value whose profile log-likelihood is within $\chi^2_{1,0.95}/2 = 1.92$ of the maximum. In practice you round to a nearby interpretable power inside the interval: 1 (none), 0.5 (square root), 0 (log), -1 (reciprocal). **Yeo-Johnson** extends Box-Cox to zero and negative values. For $x \ge 0$ it applies the Box-Cox formula to $x + 1$; for $x < 0$ it applies the mirror image with power $2 - \lambda$ to $1 - x$: $$ \psi(x, \lambda) = \begin{cases} \dfrac{(x + 1)^{\lambda} - 1}{\lambda}, & x \ge 0,\ \lambda \neq 0 \\[2mm] \ln(x + 1), & x \ge 0,\ \lambda = 0 \\[2mm] -\dfrac{(1 - x)^{2 - \lambda} - 1}{2 - \lambda}, & x < 0,\ \lambda \neq 2 \\[2mm] -\ln(1 - x), & x < 0,\ \lambda = 2 \end{cases} $$ **Proportions.** For a proportion $p$ strictly between 0 and 1: $$ \operatorname{logit}(p) = \ln\frac{p}{1 - p}, \qquad \text{arcsine square root: } \arcsin\sqrt{p} $$ The logit maps (0, 1) onto the whole real line and is the link of logistic regression. The arcsine square root was once the standard variance-stabiliser for binomial proportions. It is now discouraged: when you have counts of successes out of trials, logistic regression models the data directly and gives interpretable odds ratios, while the arcsine scale has no natural interpretation (Warton and Hui 2011). **Scaling, centring and ranks.** $$ z_i = \frac{x_i - \bar{x}}{s}, \qquad m_i = \frac{x_i - \min(x)}{\max(x) - \min(x)}, \qquad c_i = x_i - \bar{x}, \qquad r_i = \operatorname{rank}(x_i) $$ These are **linear** (z, min-max, centring) or **order-only** (ranks). Linear rescaling never changes the shape of a distribution, the correlation between variables or the p-value of a slope; it only changes units. Ranks throw away the distances and keep the order, which is what rank-based tests and Spearman's correlation use. **Interpreting a log-outcome coefficient.** In the model $\ln y = \beta_0 + \beta_1 x + \varepsilon$, a one-unit increase in $x$ multiplies the (geometric) mean of $y$ by $e^{\beta_1}$: $$ \%\Delta y = 100\left(e^{\beta_1} - 1\right) $$ For small $\beta_1$ (say $|\beta_1| < 0.1$), $100\beta_1$ is a good approximation; for larger values it is not ($\beta_1 = -0.99$ means a 63% fall, not 99%). **Back-transformation bias.** Because the log is concave, the mean of the logs is smaller than the log of the mean (Jensen's inequality). So $\exp(\overline{\ln y})$ is the **geometric mean**, which for skewed data is below the arithmetic mean, and $\exp(\hat\beta_0 + \hat\beta_1 x)$ estimates the **median** of $y$ at $x$ (if the log-scale errors are symmetric), not its mean. Duan's **smearing estimator** corrects the prediction of the mean without assuming normality: $$ \widehat{E}(y \mid x) = \exp\left(\hat\beta_0 + \hat\beta_1 x\right) \cdot \frac{1}{n}\sum_{i=1}^{n} e^{\hat\varepsilon_i} $$ Under normal log-scale errors with variance $\sigma^2$ the factor is $e^{\sigma^2/2}$; smearing replaces that assumption with the average of the exponentiated residuals. | Symbol | Meaning | |---|---| | $x, y$ | Original-scale predictor and outcome | | $\lambda$ | Box-Cox or Yeo-Johnson power parameter | | $c$ | Constant added before a log when zeros occur | | $p$ | Proportion between 0 and 1 | | $\bar{x}, s$ | Sample mean and standard deviation (divisor $n-1$) | | $\beta_0, \beta_1$ | Intercept and slope of a regression on the transformed scale | | $\hat\varepsilon_i$ | Residual of observation $i$ on the transformed scale | | $n$ | Sample size | | $\%\Delta y$ | Percent change in $y$ per one-unit change in $x$ | Which transformation, in what order of questions: ```mermaid flowchart TD A["What is the problem?"] --> B{"Skew or fanning residuals?"} A --> C{"Predictors on different scales?"} A --> D{"Interaction term in the model?"} B -- "Outcome is a count" --> E["Poisson or negative binomial GLM"] B -- "Outcome is successes out of trials" --> F["Logistic regression"] B -- "Positive continuous" --> G["Estimate Box-Cox lambda"] G --> H{"Lambda CI includes 0?"} H -- "Yes" --> I["Use log: percent-change interpretation"] H -- "No" --> J["Round lambda to 1, 0.5 or -1 if inside CI"] B -- "Zeros or negatives" --> K["Yeo-Johnson or GLM, not arbitrary log of x plus c"] C -- "Yes" --> L["z-scores, report units"] D -- "Yes" --> M["Centre the predictors at the mean or a meaningful value"] I --> N["Check residuals again"] J --> N K --> N ``` ### Decision table | Situation | Transformation | Interpretation after transform | Back-transform | Caution | |---|---|---|---|---| | Positive, right-skewed, multiplicative (times, incomes, concentrations) | $\ln x$ | Percent change; ratio of geometric means | $e^{\text{value}}$ gives geometric mean or median; smear for the mean | Not for zeros | | Contains zeros, otherwise as above | $\ln(x + c)$ or Yeo-Johnson | Approximate percent change for large $x$ only | $e^{\text{value}} - c$ | Results depend on $c$; report a sensitivity check; consider a GLM | | Counts with variance about equal to the mean | $\sqrt{x}$ | Hard to interpret | Square, with bias | Prefer Poisson regression | | Strong right skew, rates per unit (time per task) | $1/x$ | Rate (tasks per minute) | $1/\text{value}$; order reverses | Use $-1/x$ to keep the order | | Unknown best power, positive data | Box-Cox, $\lambda$ by ML | Only if $\lambda$ is rounded to 0, 0.5, 1 or -1 | Inverse Box-Cox | Odd powers like 0.37 are hard to explain | | Any sign, need symmetry | Yeo-Johnson | Mostly for prediction or screening | Inverse Yeo-Johnson | Coefficients lose a simple meaning | | Proportion from a continuous measure (share of time) | logit | Log-odds; odds ratio per unit | $1/(1 + e^{-\text{value}})$ | Needs $0 < p < 1$ | | Proportion from counts (passed out of enrolled) | None: logistic regression | Odds ratio | Predicted probability | Arcsine square root is discouraged | | Predictors on different scales | z-score | Change per 1 SD | $x = \bar{x} + s z$ | Units become sample-specific | | Inputs for distance-based methods | Min-max to [0, 1] | Relative position in observed range | $x = \min + m(\max - \min)$ | One outlier squashes the rest | | Model with an interaction | Centre at mean or meaningful value | Main effect = effect at the centre of the other variable | Add the centre back | Does not change the interaction slope or fit | | Heavy outliers, ordinal question | Ranks | Order only | Not possible | Distances are lost | ## Worked example A (fully solved): log-transforming wait times **Hypotheses.** $H_0$: staffing level is not related to wait time ($\beta_1 = 0$ on the log scale). $H_1$: it is ($\beta_1 \neq 0$), two-sided, $\alpha = 0.05$. **Step 1. Shape before and after.** Raw waits: skewness $g_1$ = 1.93, Shapiro-Wilk p = .0002. Log waits: skewness 0.34, Shapiro-Wilk p = .986. The square root (skewness 1.19, p = .033) and the reciprocal (1.37, p = .017) are worse choices here. Shapiro-Wilk is used as a descriptive aid only: with large samples it rejects trivial departures, with small ones it misses real ones. **Step 2. Box-Cox.** The maximum-likelihood power is $\hat\lambda = -0.179$, 95% CI [-0.688, 0.318]. The interval contains 0 (log) and excludes 0.5 (square root) and 1 (no transformation), so the log is supported by the data and gives the most interpretable scale. When Box-Cox is estimated for the regression residuals (given staff), R gives $\hat\lambda = -0.024$, even closer to 0. Box-Cox targets the distribution of the model's errors, so estimate it with the predictors you will use. **Step 3. Fit the model on the log scale.** $\widehat{\ln y} = 3.8552 - 0.9939 \times \text{staff}$; 95% CI for the slope [-1.2168, -0.7710]. **Step 4. Convert to percent change (by hand).** $e^{-0.9939} = 0.370$, so each extra staff member multiplies the typical wait by 0.37: a change of $100(0.370 - 1) = -63.0\%$. CI: $100(e^{-1.2168} - 1) = -70.4\%$ and $100(e^{-0.7710} - 1) = -53.7\%$. **Step 5. Back-transform predictions.** For one staff member: $e^{3.8552 - 0.9939} = 17.48$ minutes. That is the predicted **median** (and geometric mean) wait. The smearing factor is $\frac{1}{20}\sum e^{\hat\varepsilon_i} = 1.0567$, so the predicted **mean** wait is $17.48 \times 1.0567 = 18.48$ minutes. The smeared mean is 5.7% above the naive back-transform, so the naive value understates the mean wait. | Staff | Naive $e^{\hat\mu}$ (median) | Smeared (mean) | Observed mean | Observed geometric mean | |---|---|---|---|---| | 1 | 17.48 | 18.48 | 19.55 | 18.17 | | 2 | 6.47 | 6.84 | 6.49 | 6.23 | | 3 | 2.40 | 2.53 | 2.59 | 2.44 | The naive predictions track the observed geometric means, and the smeared predictions track the observed arithmetic means for 1 and 3 staff. For 2 staff the observed mean is lower than the smeared prediction for two reasons: the fitted line sits slightly above the 2-staff geometric mean (6.47 vs 6.23), and the 8 visits with 2 staff happen to have less log-scale spread than the others, while smearing uses one pooled correction for all groups (it assumes constant variance on the log scale). **R code** (base R, stats and MASS; car is optional): ```r library(MASS) # ships with R; boxcox() desk <- data.frame( wait = c(2.1,3.4,1.2,5.6,8.9,2.8,14.5,4.1,6.3,22.0,3.0,1.8,9.7,4.8,2.5,11.2,7.4,3.9,30.5,5.1), staff = c(3,3,3,2,2,3,1,2,2,1,3,3,2,2,3,1,2,3,1,2), hour = c(9,10,11,9,12,14,13,15,10,16,11,9,14,12,10,15,13,11,16,12) ) # Box-Cox: profile log-likelihood over a fine grid of lambda bc0 <- boxcox(lm(wait ~ 1, data = desk), lambda = seq(-2, 2, 0.001), plotit = FALSE) lam0 <- bc0$x[which.max(bc0$y)] ci0 <- range(bc0$x[bc0$y > max(bc0$y) - qchisq(0.95, 1) / 2]) cat("Box-Cox lambda (no predictors):", lam0, " 95% CI:", ci0, "\n") bc1 <- boxcox(lm(wait ~ staff, data = desk), lambda = seq(-2, 2, 0.001), plotit = FALSE) cat("Box-Cox lambda (given staff): ", bc1$x[which.max(bc1$y)], "\n") # Log-outcome model and percent change m <- lm(log(wait) ~ staff, data = desk) round(coef(m), 4) round(100 * (exp(coef(m)["staff"]) - 1), 1) round(100 * (exp(confint(m)["staff", ]) - 1), 1) smear <- mean(exp(resid(m))) round(smear, 4) nd <- data.frame(staff = 1:3) round(cbind(nd, naive = exp(predict(m, nd)), smeared = exp(predict(m, nd)) * smear), 3) # Yeo-Johnson and Box-Cox via car (optional): install.packages("car") if (requireNamespace("car", quietly = TRUE)) { print(round(car::powerTransform(desk$wait, family = "yjPower")$lambda, 4)) print(round(car::powerTransform(desk$wait, family = "bcPower")$lambda, 4)) } head(round(as.numeric(scale(desk$wait)), 3), 5) # z-scores head(rank(desk$wait), 5) # ranks ``` ```text Box-Cox lambda (no predictors): -0.179 95% CI: -0.688 0.317 Box-Cox lambda (given staff): -0.024 (Intercept) staff 3.8552 -0.9939 staff -63 2.5 % 97.5 % -70.4 -53.7 [1] 1.0567 staff naive smeared 1 1 17.484 18.476 2 2 6.471 6.839 3 3 2.395 2.531 desk$wait -0.3954 desk$wait -0.1786 [1] -0.736 -0.560 -0.858 -0.263 0.184 [1] 3 7 1 12 15 ``` A practical R note: name the data frame something other than `df`. `MASS::boxcox()` refits the model internally, and with a data frame called `df` it picks up R's `df()` function (the F density) instead, failing with "'data' must be a data.frame, environment, or list". The same refit fails inside your own function on a local data frame. Fitting the model with `lm(..., y = TRUE, qr = TRUE)` avoids the refit; the example simply uses the name `desk`. **Python code** (numpy, pandas, scipy, statsmodels): ```python import numpy as np, pandas as pd from scipy import stats import statsmodels.formula.api as smf desk = pd.DataFrame({ "wait": [2.1,3.4,1.2,5.6,8.9,2.8,14.5,4.1,6.3,22.0,3.0,1.8,9.7,4.8,2.5,11.2,7.4,3.9,30.5,5.1], "staff": [3,3,3,2,2,3,1,2,2,1,3,3,2,2,3,1,2,3,1,2], "hour": [9,10,11,9,12,14,13,15,10,16,11,9,14,12,10,15,13,11,16,12], }) w = desk["wait"] # 1. Shape before and after for name, x in [("raw", w), ("log", np.log(w)), ("sqrt", np.sqrt(w)), ("recip", 1/w)]: print(f"{name:6s} skew = {stats.skew(x):6.3f} Shapiro p = {stats.shapiro(x).pvalue:.4f}") # 2. Box-Cox and Yeo-Johnson (marginal, no predictors) bc, lam, ci = stats.boxcox(w, alpha=0.05) print("Box-Cox lambda =", round(lam, 4), " 95% CI =", np.round(ci, 3)) yj, lam_yj = stats.yeojohnson(w) print("Yeo-Johnson lambda =", round(lam_yj, 4)) # 3. Means on the original and log scale print("mean =", round(w.mean(), 3), " median =", round(w.median(), 3), " geometric mean =", round(np.exp(np.log(w).mean()), 3)) # 4. Log-outcome regression and percent change m = smf.ols("np.log(wait) ~ staff", data=desk).fit() b, (lo, hi) = m.params["staff"], m.conf_int().loc["staff"] print("b0 =", round(m.params["Intercept"], 4), " b1 =", round(b, 4), " 95% CI", round(lo, 4), round(hi, 4)) print("percent change per extra staff =", round(100*np.expm1(b), 1), " 95% CI", round(100*np.expm1(lo), 1), round(100*np.expm1(hi), 1)) # 5. Back-transformation: naive vs Duan smearing smear = np.mean(np.exp(m.resid)) print("smearing factor =", round(smear, 4)) for s in (1, 2, 3): naive = np.exp(m.params["Intercept"] + b*s) obs = desk.loc[desk.staff == s, "wait"] print(f"staff={s}: naive = {naive:.3f} smeared = {naive*smear:.3f} " f"observed mean = {obs.mean():.3f} observed geo mean = {np.exp(np.log(obs).mean()):.3f}") # 6. Scaling and ranks (first five rows) print("z :", np.round(np.asarray(stats.zscore(w, ddof=1))[:5], 3)) print("min-max :", np.round(((w - w.min())/(w.max() - w.min()))[:5].to_numpy(), 3)) print("rank :", stats.rankdata(w)[:5]) # 7. Centring before an interaction x, h = desk["staff"], desk["hour"] print("corr(staff, staff*hour) raw =", round(np.corrcoef(x, x*h)[0, 1], 3), " centred =", round(np.corrcoef(x - x.mean(), (x - x.mean())*(h - h.mean()))[0, 1], 3)) # 8. Proportions p = np.array([0.62, 0.95, 0.80, 0.50, 0.98]) print("logit :", np.round(np.log(p/(1 - p)), 3)) print("arcsine :", np.round(np.arcsin(np.sqrt(p)), 3)) ``` ```text raw skew = 1.927 Shapiro p = 0.0002 log skew = 0.341 Shapiro p = 0.9855 sqrt skew = 1.191 Shapiro p = 0.0326 recip skew = 1.368 Shapiro p = 0.0171 Box-Cox lambda = -0.1786 95% CI = [-0.688 0.318] Yeo-Johnson lambda = -0.3954 mean = 7.54 median = 4.95 geometric mean = 5.305 b0 = 3.8552 b1 = -0.9939 95% CI -1.2168 -0.771 percent change per extra staff = -63.0 95% CI -70.4 -53.7 smearing factor = 1.0567 staff=1: naive = 17.484 smeared = 18.476 observed mean = 19.550 observed geo mean = 18.169 staff=2: naive = 6.471 smeared = 6.839 observed mean = 6.487 observed geo mean = 6.228 staff=3: naive = 2.395 smeared = 2.531 observed mean = 2.587 observed geo mean = 2.442 z : [-0.736 -0.56 -0.858 -0.263 0.184] min-max : [0.031 0.075 0. 0.15 0.263] rank : [ 3. 7. 1. 12. 15.] corr(staff, staff*hour) raw = 0.858 centred = 0.375 logit : [0.49 2.944 1.386 0. 3.892] arcsine : [0.907 1.345 1.107 0.785 1.429] ``` **R and Python compared.** The Box-Cox $\hat\lambda$ agrees (-0.1786 from `car::powerTransform` and SciPy; -0.179 from the `MASS::boxcox` grid). The upper CI limit is 0.317 in R and 0.318 in SciPy only because R searched a grid with step 0.001. `scipy.stats.boxcox` estimates $\lambda$ for the variable alone; `MASS::boxcox` works on a fitted model, so it can estimate $\lambda$ for the residuals given predictors (-0.024 here). Yeo-Johnson agrees (-0.3954). The z-scores agree because `scale()` in R and `zscore(ddof=1)` both divide by $s$ with $n-1$; SciPy's default `ddof=0` would give slightly larger values. **Interpretation.** Waits are right-skewed and multiplicative, and the Box-Cox interval supports a log scale. On that scale, each extra staff member is associated with a 63% shorter typical wait. Predictions back-transformed with smearing estimate the mean wait; without smearing they estimate the median. **APA-style report sentence.** "Wait times were right-skewed (*M* = 7.54 min, *Mdn* = 4.95) and were log-transformed (Box-Cox $\hat\lambda$ = -0.18, 95% CI [-0.69, 0.32]). Each additional staff member on duty was associated with a 63.0% shorter wait, $b$ = -0.99, 95% CI [-1.22, -0.77] on the log scale, corresponding to a 53.7% to 70.4% reduction, *n* = 20. Predicted mean waits (Duan smearing) were 18.5, 6.8 and 2.5 minutes for one, two and three staff." ## Interpreting coefficients after log transforms | Model | Coefficient $b$ means | Exact conversion | Example | |---|---|---|---| | $y = \beta_0 + \beta_1 x$ (level-level) | Change in $y$ units per 1 unit of $x$ | none | $b = 2$: 2 more points per hour | | $\ln y = \beta_0 + \beta_1 x$ (log-level) | Multiplicative change in $y$ per 1 unit of $x$ | $100(e^{b} - 1)$% | $b = -0.99$: 63.0% shorter wait per extra staff | | $y = \beta_0 + \beta_1 \ln x$ (level-log) | Change in $y$ units per 1% increase in $x$ | $b \ln(1.01) \approx b/100$ | $b = 5$: 0.05 points per 1% more hours; 0.48 points per 10% more | | $\ln y = \beta_0 + \beta_1 \ln x$ (log-log) | Elasticity: % change in $y$ per 1% change in $x$ | $100(1.01^{b} - 1)$% | $b = 0.4$: 0.4% per 1%; 3.9% per 10% | | $\ln y$ with a binary $x$ | Ratio of geometric means between groups | $e^{b}$ | $b = 0.25$: group 1 is 28.4% higher | Use base-10 logs only if the audience thinks in powers of ten (concentrations, decibels); the conversion is then $10^{b}$, not $e^{b}$. State which log you used. In R and NumPy, `log()` is the natural log. ## Worked example B (half solved): prediction and two-step change Using $\widehat{\ln y} = 3.8552 - 0.9939 \times \text{staff}$ and the smearing factor 1.0567: 1. Predicted log wait with 2 staff: $3.8552 - 0.9939 \times 2 =$ [ ]. 2. Predicted median wait with 2 staff: $e^{[\ ]} =$ [ ] minutes. 3. Predicted mean wait with 2 staff (smearing): [ ] $\times 1.0567 =$ [ ] minutes. 4. Percent change in typical wait from 1 to 3 staff (two extra): $100(e^{2 \times (-0.9939)} - 1) =$ [ ]%. 5. Why is the answer to step 4 not simply $2 \times (-63.0\%) = -126\%$? [ ] 6. The observed mean with 2 staff is 6.49 minutes. Which prediction, median or smeared mean, should you compare it with, and why do they differ? [ ] ## Exercise C (unsolved): interpretation drill 1. Harbour City Library (fictional) records four daily e-book download counts: 2, 8, 4, 16. Compute the arithmetic mean and the geometric mean. Which is a better "typical day" for this multiplicative series, and why? 2. A log-level model of Brightwell Bakery (fictional) batch waste gives $b = -0.40$ for a new mixer (binary). State the effect as a percent change. 3. A log-log model gives $b = 0.25$ for hours of oven time on energy use. Interpret it for a 10% increase in oven time. 4. In a log-level model a binary predictor has $b = 0.25$. By what percent is the outcome higher in group 1? 5. A course pass rate is 0.80. Compute its logit and explain why logistic regression on the pass counts is preferred to an arcsine transform of the rate. 6. You centre "hour" at its mean before fitting `log(wait) ~ staff * hour`. Which coefficients change meaning, and which do not change at all? ## Reading software output | Output line | Meaning | Report it? | |---|---|---| | `Box-Cox lambda ... 95% CI` | ML power and its likelihood interval | Yes, when you justify the transformation | | `(Intercept) 3.8552` | Predicted log wait at staff = 0 (outside the data) | Rarely; centre staff for a meaningful intercept | | `staff -0.9939` | Change in log wait per extra staff | Yes, with the percent conversion | | `2.5 % 97.5 %` after `exp()` | CI on the percent-change scale | Yes | | `smearing factor 1.0567` | Correction from median to mean on the original scale | Yes, when you report predicted means | | `naive` vs `smeared` | Predicted median vs predicted mean | Say which one you report | | `Yeo-Johnson lambda` | Power for data with zeros or negatives | Only if you use it | | `Shapiro p` | Descriptive check of shape | Optional; prefer plots | ## Reporting template **Fill-in sentence.** "[Outcome] was [right-skewed / heteroscedastic] (*M* = [ ], *Mdn* = [ ]) and was [log / square-root / Box-Cox] transformed [because the process is multiplicative / Box-Cox $\hat\lambda$ = [ ], 95% CI [ , ]]. Each one-unit increase in [predictor] was associated with a [ ]% [higher / lower] [geometric mean / median] [outcome], $b$ = [ ], 95% CI [ , ] on the log scale. Predicted means on the original scale used [Duan smearing / the normal-theory correction]." | Variable | Transformation | Reason | Decided before seeing results? | Back-transform method | |---|---|---|---|---| | *Example:* wait | natural log | multiplicative process; Box-Cox CI includes 0 | Yes (analysis plan) | Duan smearing | | *Example:* hour | centred at 12.1 | interaction with staff | Yes | add 12.1 | | [variable] | [ ] | [ ] | [ ] | [ ] | **Results checklist.** - [ ] Reason for the transformation stated (science or diagnostics), and decided before looking at p-values. - [ ] The base of the log stated (natural log unless said otherwise). - [ ] Constant $c$ reported, with a sensitivity check, if $\ln(x + c)$ was used. - [ ] Coefficients converted to percent changes or ratios, with CIs converted the same way. - [ ] Back-transformed predictions labelled as medians (geometric means) or means (smearing). - [ ] Residual plots checked on the transformed scale. - [ ] Original-scale descriptive statistics also reported. ## Common mistakes and misconceptions | Mistake or misconception | Why it is wrong | Do this instead | |---|---|---| | Trying transformations until the test is significant | Choosing among many analyses after seeing results inflates the false positive rate (a "garden of forking paths") | Decide the scale from the science or from diagnostics on the residuals, write it in the analysis plan, and report it | | "The outcome must be normal, so I transform it" | Linear models assume normal **errors**, not a normal outcome; a skewed outcome can have normal residuals once predictors are in | Check residuals of the fitted model, not the raw histogram | | Reading $b = -0.99$ on a log outcome as "99% lower" | The exact change is $100(e^{b} - 1) = -63.0\%$; $100b$ only works for small $b$ | Convert with $e^{b}$ | | Back-transforming a mean of logs and calling it the mean | $\exp(\overline{\ln y})$ is the geometric mean (5.31 vs 7.54 here) | Call it the geometric mean or median, or apply smearing | | Adding 1 before a log without thinking | The result depends on $c$; with values near 0, $\ln(x + 1)$ behaves almost linearly | Report $c$, check $c$ = 0.5 and 1, or use a GLM or Yeo-Johnson | | Using arcsine square root on proportions from counts | Discouraged: no interpretable scale and ignores the binomial structure | Logistic regression on successes and failures | | Expecting z-scores to "normalise" the data | z-scores are linear; the shape is unchanged | Use z-scores for scale, a power transform for shape | | Believing centring removes "real" multicollinearity | Centring lowers the correlation between $x$ and $x \cdot z$ (0.86 to 0.38 here) but does not change the fit or the interaction slope | Centre for interpretation of main effects, not to fix inference | | Transforming a Likert item or a bounded score with a log | Bounded scales are not multiplicative | Use ordinal models or treat the summed scale as interval | | Reporting only transformed-scale numbers | Readers cannot judge practical size in log minutes | Give original-scale summaries and back-transformed effects | ## Self-test questions 1. Box-Cox gives $\hat\lambda$ = 0.42 with 95% CI [0.18, 0.69]. Which simple transformation would you choose, and why not the exact 0.42? 2. In a log-level regression, $b$ = 0.08. Give the exact and the approximate percent change. When does the approximation break down? 3. You back-transform the mean of log incomes and get 31,200 USD; the arithmetic mean is 38,900 USD. Which is "wrong"? 4. Why does `scipy.stats.boxcox` and `MASS::boxcox(lm(y ~ x))` sometimes give different $\lambda$ for the same $y$? 5. A colleague runs a t-test on raw data (p = .09), then on log data (p = .04), then reports the log version. What is wrong, and what should have happened? 6. Does converting predictors to z-scores change the p-value of their slopes in a linear regression? Does it change the slopes? ## My notes and next steps | Item | Entry | |---|---| | Confidence choosing a transformation (1-5) | [ ] | | Confidence interpreting log coefficients (1-5) | [ ] | | Confidence back-transforming (1-5) | [ ] | | Open questions | [What I am still unsure about] | | Mistakes to log | [Link to `academic/stats-error-log` entry] | | Next review dates | YYYY-MM-DD (2 days), YYYY-MM-DD (1 week), YYYY-MM-DD (1 month) | | Next template | [`academic/stats-linear-regression` / `academic/stats-count-regression`] | ## Sources and further reading Based on the original method papers listed below and the NIST/SEMATECH e-Handbook of Statistical Methods; all text, data and examples are original. - Box, G. E. P., and Cox, D. R. (1964). An analysis of transformations. *Journal of the Royal Statistical Society, Series B*, 26(2), 211–243. - Yeo, I.-K., and Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. *Biometrika*, 87(4), 954–959. - Duan, N. (1983). Smearing estimate: a nonparametric retransformation method. *Journal of the American Statistical Association*, 78(383), 605–610. - Tukey, J. W. (1977). *Exploratory Data Analysis*. Addison-Wesley. - Warton, D. I., and Hui, F. K. C. (2011). The arcsine is asinine: the analysis of proportions in ecology. *Ecology*, 92(1), 3–10. - Gelman, A., and Loken, E. (2013). *The garden of forking paths: why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time*. Working paper, Department of Statistics, Columbia University. - NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.3.5 Box-Cox Linearity Plot, and section 1.3.3.6 Box-Cox Normality Plot, https://www.itl.nist.gov/div898/handbook/ (accessed 2026-10-25). Public domain. - Further reading (by name only): de Smith, M. J. (2024). *Statistical Analysis Handbook* (2024 edition, StatsRef); Diez, D. M., Çetinkaya-Rundel, M., and Barr, C. D., *OpenIntro Statistics* (CC BY-SA). - Software documentation: R `MASS::boxcox()`, `car::powerTransform()`, `scale()`; SciPy `stats.boxcox()`, `stats.yeojohnson()`, `stats.zscore()`; statsmodels `ols`. ## Answer key **Worked example B.** (1) $3.8552 - 1.9878 = 1.8674$. (2) $e^{1.8674} = 6.47$ minutes (median and geometric mean). (3) $6.47 \times 1.0567 = 6.84$ minutes. (4) $e^{-1.9878} = 0.137$, so $100(0.137 - 1) = -86.3\%$. (5) Percent changes multiply rather than add: two steps of $\times 0.370$ give $\times 0.370^2 = \times 0.137$, an 86.3% fall. A change of -126% is impossible for a positive quantity. (6) Compare the observed mean (6.49) with the smeared mean (6.84), and the observed geometric mean (6.23) with the median prediction (6.47). The smeared mean is above the observed mean because the fitted median is already a little above the observed geometric mean, and smearing uses one pooled correction while the 2-staff visits have less log-scale spread than the others. **Exercise C.** (1) Arithmetic mean $= 30/4 = 7.5$; geometric mean $= (2 \times 8 \times 4 \times 16)^{1/4} = 1024^{1/4} = 5.66$. The geometric mean, because the values differ by factors of 2 and the arithmetic mean is pulled up by 16. (2) $100(e^{-0.40} - 1) = -33.0\%$: about one third less waste with the new mixer. (3) $100(1.1^{0.25} - 1) = 2.4\%$ more energy for 10% more oven time (elasticity 0.25). (4) $100(e^{0.25} - 1) = 28.4\%$ higher. (5) $\ln(0.8/0.2) = \ln 4 = 1.386$. Logistic regression uses the number passed and the number enrolled, weights courses by their size, keeps predictions inside 0 to 1 and gives odds ratios; the arcsine scale has no interpretation and ignores the binomial variance. (6) The coefficient of staff now means the effect of staff at the mean hour (not at hour 0, which is outside the data); the intercept changes too. The interaction coefficient, its p-value, the residuals and $R^2$ do not change. **Self-test.** 1. Square root ($\lambda = 0.5$): it is inside the CI and has a familiar meaning; 0.42 is barely better in likelihood and impossible to explain. The log (0) and no transform (1) are both outside the CI. 2. Exact: $100(e^{0.08} - 1) = 8.3\%$; approximate: 8%. The approximation drifts once $|b|$ exceeds about 0.1 (for $b = 0.5$: 64.9% exact vs 50% approximate). 3. Neither: 31,200 USD is the geometric mean (close to the median for log-symmetric data), 38,900 USD is the arithmetic mean. They answer different questions; label them. 4. SciPy estimates $\lambda$ for the marginal distribution of $y$; `MASS::boxcox` on a model estimates it for the residuals given $x$. Group or predictor differences make the marginal distribution more skewed than the residuals (here -0.18 vs -0.02). 5. The transformation was chosen because it gave a significant result, so the reported p-value does not have its stated meaning. The scale should have been chosen in advance (science or residual diagnostics) and both analyses reported as a sensitivity check. 6. The p-values (and t statistics) do not change. The slopes do change: they become effects per one SD of the predictor instead of per original unit. ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [Your name] | First draft from the template | | 0.2 | YYYY-MM-DD | [Your name] | *Example:* Replaced Lakemoor example with thesis reaction times; log chosen in analysis plan |