Bài 5: Linear Regression (Simple & Multiple)
Series: Business Analytics Cheatsheet Series
Topics:
linear-regressionleast-squaresresidual-analysisinferencer-squaredSource: Regression deck — Simple Linear Regression and Multivariable Linear Regression. Running example: house price ($1000s) vs size (square feet), n = 10. CRISP-DM position: Modeling + Evaluation (and diagnostic analytics).
Core claim: the coefficient is the easy part. The assumptions, the residual diagnostics, and the range of validity are what make the estimate usable.
1. Correlation vs Regression
| Term | Type | Definition / Usage |
|---|---|---|
| Scatter diagram | Chart | First step, always. Shows whether a linear model is even plausible before any fitting. |
| Correlation analysis | Method | Measures the strength of linear association between two variables. Concerned only with strength — no causal effect is implied. |
| Regression analysis | Method | Two purposes: (1) predict the dependent variable from at least one independent variable; (2) explain the impact of a change in an independent variable on the dependent variable. |
| Dependent variable (Y) | Definition | The variable to be predicted or explained. Also: response, criterion, outcome. |
| Independent variable (X) | Definition | The variable used to explain Y. Also: predictor, explanatory variable, regressor. |
| Types of relationship | Concept | Linear (positive/negative) · curvilinear · no relationship. Curvilinear data fitted with a straight line yields a valid-looking but wrong coefficient — check the scatter. |
2. The Simple Linear Regression Model
| Term | Type | Definition / Usage |
|---|---|---|
| Population model | Formula | Yᵢ = β₀ + β₁Xᵢ + εᵢ. Linear component: β₀ + β₁Xᵢ. Random error component: εᵢ. Only one independent variable; changes in Y are assumed to be caused by changes in X. |
| β₀ — population intercept | Parameter | Expected Y when X = 0. |
| β₁ — population slope | Parameter | Expected change in Y per one-unit change in X. |
| εᵢ — error term | Parameter | Unexplained deviation of observation i from the population line. |
| Prediction line (estimated equation) | Formula | Ŷᵢ = b₀ + b₁Xᵢ. b₀, b₁ are sample estimates of β₀, β₁. Individual residuals eᵢ have mean zero by construction. |
| Least Squares Method | Estimation method | Choose b₀, b₁ minimizing Σ(Yᵢ − Ŷᵢ)² = Σ(Yᵢ − (b₀ + b₁Xᵢ))². Minimizes squared vertical distances — which is why it is outlier-sensitive. |
| Interpreting b₀ | Interpretation | Estimated average Y when X = 0 — meaningful only if X = 0 is inside the observed range. Example: no house has 0 sq ft, so b₀ = 98.24833 merely represents the portion of price not explained by size, for houses within the observed size range. |
| Interpreting b₁ | Interpretation | Estimated change in average Y per one-unit increase in X. Example: b₁ = 0.10977 ⇒ average house value rises by 0.10977 × $1000 = $109.77 per additional square foot. |
| Worked model | Example | house price = 98.24833 + 0.10977 × (square feet). Predicting a 2,000 sq ft house: 98.25 + 0.1098(2000) = 317.85 ⇒ $317,850. |
| Interpolation | Practice | Prediction inside the observed range of X. Valid. |
| Extrapolation | Anti-pattern | Prediction outside the observed range of X. Do not do it — the linear form was never tested there. |
3. Measures of Variation
| Term | Type | Definition / Usage |
|---|---|---|
| Decomposition | Formula | SST = SSR + SSE. Total variation splits into explained and unexplained. |
| SST — Total Sum of Squares | Metric | Σ(Yᵢ − Ȳ)². Variation of observed Y around its own mean. The benchmark any model must beat. |
| SSR — Regression Sum of Squares | Metric | Σ(Ŷᵢ − Ȳ)². Explained variation attributable to the X–Y relationship. |
| SSE — Error Sum of Squares | Metric | Σ(Yᵢ − Ŷᵢ)². Variation attributable to factors other than X. |
| r² — Coefficient of Determination | Metric | r² = SSR / SST, with 0 ≤ r² ≤ 1. Proportion of total variation in Y explained by variation in X. Example: 18934.9348 / 32600.5 = 0.58082 ⇒ 58.08% of house price variation explained by square feet. |
| Reading r² | Interpretation | r² = 1: perfect linear relationship, 100% explained. 0 < r² < 1: partial explanation. r² = 0: no linear relationship — Y does not depend linearly on X. A high r² does not validate the model — check residuals. |
| Sᵧₓ — Standard Error of Estimate | Metric | Sᵧₓ = √(SSE/(n−2)) = √(Σ(Yᵢ − Ŷᵢ)²/(n−2)). Standard deviation of observations around the regression line, in the units of Y. |
| Judging Sᵧₓ | Interpretation | Always relative to the size of Y. Example: Sᵧₓ = $41.33K is moderately small against house prices in the $200–300K range. Small Sᵧₓ ⇒ points cluster tightly on the line. |
4. LINE Assumptions
| Letter | Assumption | Definition / Usage |
|---|---|---|
| L | Linearity | The underlying X–Y relationship is linear. Check: residuals vs X should show no curvature. |
| I | Independence of Errors | Error values are statistically independent. Check: residuals vs time/order should show no pattern; use Durbin-Watson for time-ordered data. |
| N | Normality of Error | Errors ε are normally distributed for any given X. Check: normal probability plot of residuals, histogram, box-and-whisker. Matters for inference (t/F tests), not for the point estimate. |
| E | Equal Variance (Homoscedasticity) | Error variance is constant across all levels of X. Check: residuals vs X should show a constant band, not a funnel. |
5. Residual Analysis
| Term | Type | Definition / Usage |
|---|---|---|
| Residual | Definition | eᵢ = Yᵢ − Ŷᵢ. Difference between observed and predicted value for observation i. The residuals — not r² — are where model failure is visible. |
| Residuals vs X (linearity) | Diagnostic | Random scatter around zero ⇒ linear. Systematic curve ⇒ not linear; add a term or transform. |
| Residuals vs order (independence) | Diagnostic | Random ⇒ independent. Runs or cycles ⇒ autocorrelation. |
| Normal probability plot | Diagnostic | Residuals plotted against theoretical normal quantiles. Points near the straight line ⇒ normality acceptable. |
| Residuals vs X (equal variance) | Diagnostic | Constant vertical spread ⇒ homoscedastic. Widening/narrowing fan ⇒ heteroscedastic; consider transforming Y or using robust/weighted least squares. |
| Worked check | Example | House-price residual plot spans roughly −49 to +64 with no visible pattern against square feet ⇒ does not appear to violate the regression assumptions. |
Autocorrelation
| Term | Type | Definition / Usage |
|---|---|---|
| Autocorrelation | Concept | Correlation of residuals over time — residuals in one period related to another period. Violates the independence assumption. Cyclical residual patterns signal positive autocorrelation. |
| Durbin-Watson statistic | Statistical test | D = Σᵢ₌₂ⁿ(eᵢ − eᵢ₋₁)² / Σᵢ₌₁ⁿ eᵢ². Range 0 ≤ D ≤ 4. H₀: residuals not correlated. H₁: positive autocorrelation present. |
| Reading D | Interpretation | D ≈ 2 ⇒ H₀ plausible. D < 2 ⇒ possible positive autocorrelation. D > 2 ⇒ possible negative autocorrelation. Decision rule using table values d_L, d_U (for n and k): reject H₀ if D < d_L; inconclusive if d_L ≤ D ≤ d_U; do not reject if D > d_U. |
6. Inference About the Slope
| Term | Type | Definition / Usage |
|---|---|---|
| S_b₁ — standard error of the slope | Metric | S_b₁ = Sᵧₓ / √SSX = Sᵧₓ / √Σ(Xᵢ − X̄)². Measures variation in the estimated slope across possible samples. Larger spread in X ⇒ smaller S_b₁ ⇒ more precise slope. |
| t-test for the slope | Statistical test | H₀: β₁ = 0 (no linear relationship). H₁: β₁ ≠ 0. Statistic: t = (b₁ − β₁)/S_b₁, with df = n − 2. |
| Worked t-test | Example | t = (0.10977 − 0)/0.03297 = 3.32938. df = 8, critical t = ±2.3060 at α = .05 ⇒ reject H₀. p-value = 0.01039 < .05 ⇒ same conclusion. Sufficient evidence that square footage affects house price. |
| F-test for significance | Statistical test | F = MSR/MSE where MSR = SSR/k and MSE = SSE/(n − k − 1); F(k, n−k−1). Example: F = 18934.9348/1708.1957 = 11.0848; critical F.05(1,8) = 5.32 ⇒ reject H₀. In simple regression the F-test and the slope t-test are equivalent (t² = F, 3.329² ≈ 11.08). |
| Confidence interval for the slope | Interval estimate | b₁ ± t_(n−2) · S_b₁. Example: (0.0337, 0.1858) ⇒ 95% confident the average price impact is between $33.70 and $185.80 per square foot. The interval excludes 0 ⇒ significant relationship at α = .05. Report the interval, not just the p-value — it carries the effect size. |
| t-test for a correlation coefficient | Statistical test | H₀: ρ = 0; H₁: ρ ≠ 0. Statistic t = (r − ρ)/√((1 − r²)/(n − 2)), df = n − 2, where r = +√r² if b₁ > 0 and r = −√r² if b₁ < 0. Example: r = .762, t = 3.329 ⇒ reject H₀, linear association at the 5% level. Same value as the slope t-test — they test the same hypothesis. |
7. Interval Estimates for Y
| Term | Type | Definition / Usage |
|---|---|---|
| CI for the mean of Y given X | Interval estimate | Ŷ ± t_(n−2) · Sᵧₓ · √hᵢ, where hᵢ = 1/n + (Xᵢ − X̄)²/Σ(Xᵢ − X̄)². Uncertainty about the average Y at that X. Width grows with distance from X̄. |
| Prediction Interval for an individual Y | Interval estimate | Ŷ ± t_(n−2) · Sᵧₓ · √(1 + hᵢ). The extra 1 + reflects the added uncertainty of a single case. Always wider than the CI. |
| Worked prediction interval | Example | 95% PI for an individual 2,000 sq ft house: 317.85 ± 102.28 ⇒ $215,500 to $420,070. Note how wide it is relative to the point estimate — this is the honest uncertainty, and the number stakeholders should see. |
| CI vs PI | Distinction | CI answers “where is the average?”; PI answers “where will the next one land?”. Quoting a CI when the decision concerns an individual case understates risk substantially. |
8. Pitfalls & Strategy
Pitfalls of regression analysis
| Pitfall | Type | Consequence |
|---|---|---|
| Unaware of the least-squares assumptions | Anti-pattern | Invalid inference; confident wrong conclusions. |
| Not knowing how to evaluate the assumptions | Anti-pattern | Violations pass unnoticed. |
| Not knowing the alternatives when an assumption fails | Anti-pattern | Model shipped despite known violation. |
| Using a model without subject-matter knowledge | Anti-pattern | Nonsense coefficients, omitted confounders. |
| Extrapolating outside the relevant range | Anti-pattern | Predictions where the model was never tested. |
Strategy for avoiding them
1. Start with a scatter diagram of X vs Y.
2. Perform residual analysis:
- residuals vs X → linearity, homoscedasticity
- histogram / stem-and-leaf / boxplot / normal probability plot → normality
- residuals vs order + Durbin-Watson → independence
3. If an assumption is violated → alternative method or model
(transform, weighted least squares, robust regression, add terms).
4. If no violation → test coefficient significance; build CIs and PIs.
5. Never predict outside the relevant range.
9. Multiple Regression Analysis (MRA)
| Term | Type | Definition / Usage |
|---|---|---|
| MRA | Method | Studies the relationship between one dependent variable and two or more independent variables. Purposes: prediction · explanation · theory building. |
| Design requirements | Constraint | One dependent variable; two or more independent variables; sample size ≥ 50, and at least 10 cases per independent variable. Violating this produces unstable coefficients. |
| Assumptions | Assumption set | Independence: each subject’s scores independent of others’. Normality: Y normally distributed for each combination of X levels; each variable normally distributed. Homoscedasticity: equal variance of Y across all combinations of X levels. Linearity: Y–X relation linear when all other independent variables are held constant. |
| Model | Formula | Y' = a + b₁X₁ + b₂X₂ + … + b_kX_k. |
| Partial regression coefficient (b) | Parameter | Effect of one X on Y holding all other X’s constant. This conditioning is what distinguishes it from a simple regression slope — the two rarely match. |
| R² | Metric | Proportion of variation in Y predictable from the set of independent variables. In simple regression, R² = r². |
| Adjusted R² | Metric | R² penalized for the number of predictors. Use it when comparing models of different size — plain R² never decreases when you add a variable. |
| Worked example | Example | Predicting academic achievement (N = 103): Y' = 36.83 + 3.52·X_ASC + (−0.44)·X_GSC. For GSC = 4, ASC = 6: 36.83 + 3.52(6) − 0.44(4) = 56.23. |
Significance testing in MRA
| Test | Type | Definition / Usage |
|---|---|---|
| Testing R² | F test | Is R² significantly different from 0? Example: R² = .16 (R = .41) ⇒ 16% of AA variance explained by GSC + ASC. F_obs = 9.52 > F_crit(.05/2,100) = 3.09 ⇒ reject H₀; a significant relationship exists between AA and the linear composite. |
| Comparing nested models | F test of ΔR² | Tests whether added predictors improve fit. Example: Model 1 Y' = 35.37 + 3.38·X_ASC (R² = .160) vs Model 2 with GSC added (R² = .161). F_obs = .119 < F_crit(.05/1,100) = 3.94 ⇒ GSC adds nothing once ASC is in the model. A textbook illustration of why R² alone is not evidence. |
| Testing individual b’s | t test | One t-test per partial regression coefficient. |
| Comparing predictors to each other | t test on β | Compare standardized partial regression coefficients (β). Raw b’s are in different units and cannot be ranked directly. |
Simple vs Multiple regression
| Aspect | Simple | Multiple |
|---|---|---|
| Predictors | One X | Set (X₁ … X_k) |
| Coefficients | One | One per independent variable (partial) |
| Variance explained | r² from X | R² from the set of X’s |
| Interpretation | Slope = effect of X | Partial b = effect of X holding others constant |
10. Reading a regression output
Using the house-price Excel output as the template:
| Field | Value | What it tells you |
|---|---|---|
| Multiple R | 0.76211 | Correlation between observed and predicted Y. |
| R Square | 0.58082 | 58.08% of Y variation explained. |
| Adjusted R Square | 0.52842 | Penalized for predictor count; the honest number for model comparison. |
| Standard Error | 41.33032 | Sᵧₓ — typical prediction error, in Y units ($1000s). |
| Observations | 10 | n. Far below the ≥50 guidance — treat all inference as illustrative. |
| F / Significance F | 11.0848 / 0.01039 | Overall model significance. |
| Coefficient (Square Feet) | 0.10977 | +$109.77 per square foot. |
| Standard Error (slope) | 0.03297 | S_b₁ — precision of that estimate. |
| t Stat / P-value | 3.32938 / 0.01039 | Slope significantly different from 0. |
| Lower/Upper 95% | 0.03374 / 0.18580 | Effect size range; excludes 0 ⇒ significant. |
Definition of Done
- Scatter diagram inspected before fitting.
- All four LINE assumptions checked with named diagnostics; Durbin-Watson run for time-ordered data.
- Coefficients reported with confidence intervals, not just p-values.
- Intercept interpreted only if X = 0 lies within the observed range.
- Prediction intervals (not confidence intervals) quoted whenever the decision concerns an individual case.
- No prediction outside the relevant range of X.
- For MRA: n ≥ 50 and ≥ 10 cases per predictor; adjusted R² used for model comparison; nested models compared with an F-test on ΔR²; predictors ranked using standardized β only.
🔗 Cùng series
Bài 4: EDA & Visualization for Decision-Making
Bài 6: Classification Evaluation: ROC/PR, Thresholding, Cost-Sensitive, Calibration
Nếu bài này hữu ích, hãy chia sẻ cho người đang học Data Science / Business Analytics. Mọi góp ý về lỗi kỹ thuật rất được hoan nghênh.