Bài 3: Data Quality & Feature Engineering

Series: Business Analytics Cheatsheet Series

Topics: data-quality feature-engineering data-leakage encoding pipelines

Source: BA3 — Data Quality & Feature Engineering. Running case: telco churn, 10k customers, 35 variables, predict churn in the next 30 days. CRISP-DM position: Data Understanding + Data Preparation.

Core claim: data quality sets the ceiling on model performance. Leakage-safe pipelines are not optional.

1. Data Quality

Term	Type	Definition / Usage
Data quality dimensions	Framework	Accuracy · Completeness · Consistency · Validity · Timeliness · Provenance. Direct impact on model performance and operating cost. Score each dimension per column, not per table.
Data Quality Scorecard	Artifact	Concrete thresholds, e.g. % missing < 2% per feature; duplicate rate < 0.5%; out-of-range < 0.1%. Without numeric thresholds “quality” is unenforceable.
Data SLA	Governance	Delivery guarantee, e.g. daily H+1 before 08:00; stable schema outside maintenance windows. Breach = alert, not a Slack message.
Data Quality Lifecycle	Process	Ingest → Profile → Clean → Validate → Monitor → Feedback. Combine rule-based checks with descriptive statistics; neither alone catches everything.
Data profiling	Technique	Distributions, min/max, outliers, cardinality, missing map. Visuals: histogram, boxplot, correlation heatmap, category bars. First 30 minutes of any new dataset.
Duplicates	Diagnostic	Exact-key dedup vs fuzzy dedup (edit distance). Choose based on whether the key is system-generated or user-entered.
Orphan records	Diagnostic	Broken foreign keys. Detect via referential integrity checks; silently dropped joins are a common source of biased samples.
Units & value domains	Technique	Normalize currencies, percentages, units (km ↔ m). Encode validity rules explicitly, e.g. `age ∈ [0, 120]`.

2. Missing Data

Mechanisms

Term	Type	Definition / Usage
MCAR — Missing Completely At Random	Concept	Missingness independent of both observed and unobserved values. Safe to drop rows; estimates stay unbiased, only power is lost. Rare in practice.
MAR — Missing At Random	Concept	Missingness depends on observed variables. Correctable by conditioning: group-wise imputation, MICE. Most common realistic assumption.
MNAR — Missing Not At Random	Concept	Missingness depends on the missing value itself (e.g. high earners not reporting income). No imputation fixes this; requires modeling the missingness or collecting better data.
Impact of ignoring mechanism	Anti-pattern	Biased estimates, lower statistical power, model bias. The mechanism determines the remedy — diagnose before imputing.

Diagnosis

Term	Type	Definition / Usage
Missingness map	Diagnostic	Column/row heatmap (1 = missing). Reveals block structure from join failures or pipeline outages.
Missingness vs label	Diagnostic	Test whether missingness correlates with the target. If it does, missingness itself is a feature.
Little’s MCAR test (idea)	Statistical test	Tests whether missingness patterns are consistent with MCAR. Rejection ⇒ not MCAR; non-rejection is weak evidence.
Distribution comparison	Diagnostic	Compare feature distributions in missing vs non-missing groups. Divergence ⇒ MAR or MNAR.
Worked example	Example	Income missing more in Prepaid; tenure missing for new customers → likely MAR by `plan_type`; income possibly MNAR. Different columns, different mechanisms, different fixes.

Strategies

Term	Type	Definition / Usage
Row/column drop	Technique	Only for excessive missingness or low-value features. Document the threshold; dropping rows silently changes your population.
`is_missing` indicator	Technique	Binary flag added alongside imputation. Cheap, preserves the signal that the value was absent. Forgetting this is a listed pitfall.
Simple imputation	Technique	Mean / median / mode. Median and mode are robust to outliers; mean is not.
Group-wise imputation	Technique	Median within a segment, e.g. median income by `plan_type`. Correct choice when missingness is MAR conditional on that segment.
kNN imputation	Algorithm	Find k nearest neighbors → average/majority. Pro: preserves multivariate structure. Con: compute-heavy, scale-sensitive. Must scale inside the pipeline to avoid leakage.
MICE — Multiple Imputation by Chained Equations	Algorithm	Model each incomplete feature iteratively until convergence; produce multiple datasets and pool estimates. Best for MAR; preserves correlation structure.
Time-aware imputation	Technique	Forward/backward fill, interpolation, seasonal Kalman. Never fill forward across a train/test boundary.
Business-rule imputation	Technique	Domain formulas, e.g. `income = base_salary × factor`, with caps applied. Auditable and explainable — preferable in regulated contexts.
Assessing imputation impact	Practice	Compare AUC/RMSE across strategies; check distribution drift post-imputation; confirm predictive signal survives.

Pitfalls

Pitfall	Type	Fix
Global-mean imputation on the full dataset	Anti-pattern	Fit the imputer on training folds only.
No `is_missing` flag	Anti-pattern	Add it by default; drop only if proven useless.
Imputing before the train/test split	Anti-pattern	Split first, always. This is leakage.

3. Data Leakage

Term	Type	Definition / Usage
Data leakage	Anti-pattern (definition)	Information from the future or from the test set entering training. Signature: over-optimistic training/validation scores, sharp drop in production.
Target leakage	Leakage type	Features contain the label or its consequences. Example: `refund_in_30d` in a churn model — the refund happens after churn. Mitigation: drop, or time-shift to before the prediction point.
Train–test contamination	Leakage type	Preprocessing (scalers, encoders, imputers, feature selection) fitted on the full dataset. Mitigation: fit inside CV folds via `Pipeline`.
Temporal leakage	Leakage type	Using post-hoc features for forecasting. Mitigation: freeze feature windows to pre-T0 data only; use `TimeSeriesSplit` / rolling origin.
Anti-leakage pipeline principles	Practice	1) Fit preprocessing only on training folds. 2) Use `Pipeline` / `ColumnTransformer`. 3) Cross-validate the full pipeline, not the model alone.

Splitting strategies

Term	Type	Definition / Usage
Holdout	Split	Single train/test partition. Fast; high variance on small data.
KFold / StratifiedKFold	Split	k partitions; stratified preserves class balance. Default for i.i.d. tabular data with imbalance.
TimeSeriesSplit	Split	Expanding/rolling window respecting chronology. Mandatory for temporal targets.
Group-aware split	Split	Keep all rows of one entity (customer) in the same fold. Required whenever there are multiple rows per customer — otherwise the same customer appears in train and test.

4. Categorical Encoding

Term	Type	Definition / Usage
Encoding overview	Taxonomy	One-Hot · Ordinal · Target/Mean · WoE · Hashing · Frequency/Count. Recurring issues: high cardinality, rare categories, target-encoding leakage.
One-Hot Encoding (OHE)	Technique	One binary column per level. Pro: simple, no ordering assumption. Con: dimensionality explosion on high cardinality. Tip: group rare levels into `Other`; switch to hashing if unbounded.
Ordinal Encoding	Technique	Map categories to integers with a true ordering (S < M < L < XL). Do not use for unordered categories — it injects a false distance.
Target / Mean Encoding	Technique	Replace category with the target mean, with smoothing. Must be computed inside the CV loop or it leaks. Add noise/smoothing to reduce overfit.
WoE — Weight of Evidence	Technique	`WoE = ln((Good_i/Bad_i) / (Good/Bad))`. Standard in credit risk. Pro: monotonicity, interpretability, works with logistic regression scorecards. Con: requires binning and sufficient samples per bin.
Hashing trick	Technique	Map categories into a fixed-size space via a hash function. Pro: handles unbounded cardinality and streaming. Con: collisions, loss of interpretability.
Frequency / Count encoding	Technique	Replace category with its occurrence count. Cheap high-cardinality tactic; works well with tree models.
High-cardinality tactics	Playbook	Collapse rare levels → frequency/count encoding → CV-safe target encoding → hashing + embeddings. Escalate in that order.
Text features	Technique	Bag-of-Words / TF-IDF, n-grams, stopword removal. Pair with linear/logistic models or tree-based learners.
Entity embeddings	Technique	Learn dense representations of categories with a neural net. Strong for high-cardinality features with interactions; costs interpretability.

5. Scaling, Transforms & Outliers

Term	Type	Definition / Usage
Why/when to scale	Principle	Scale-sensitive algorithms: kNN, k-means, SVM, PCA, L1/L2 regression. Tree-based models are largely insensitive, but scaling helps when mixing model families or ensembling.
StandardScaler	Technique	`z = (x − μ)/σ`. Default. Assumes roughly symmetric distributions.
MinMaxScaler	Technique	Rescale to [0,1]. Useful for bounded inputs; very sensitive to outliers.
RobustScaler	Technique	Center on median, scale by IQR. Choose when outliers are present and legitimate.
Fit/transform discipline	Practice	Always fit on train; transform train and test. Non-negotiable.
Log transform	Transform	Compresses right skew; requires x > 0. First tool for revenue/usage distributions.
Box-Cox	Transform	Power family, requires x > 0. Estimates the exponent that best normalizes.
Yeo-Johnson	Transform	Power family valid on x ∈ ℝ (handles zeros and negatives). Use when Box-Cox is inapplicable.
Quantile / Rank-Gauss	Transform	Map to a uniform or normal distribution by rank. Useful for linear models with normality assumptions; destroys the original scale’s meaning.
Outlier detection	Technique	Z-scores · IQR rule `[Q1 − 1.5·IQR, Q3 + 1.5·IQR]` · model-based: Isolation Forest, LOF. Note: outliers may carry business signal (fraud, whales) — investigate before removing.
Outlier handling	Technique	Capping / Winsorization · log transform · robust loss functions · conditional removal after domain investigation. Never remove silently.

6. Feature Engineering

Term	Type	Definition / Usage
Transformations toolkit	Technique set	Binning (equal-width / equal-frequency) · interactions · polynomials · ratios and rates (e.g. `ARPU = revenue/usage`) · time-normalized rates. Ratios often outperform raw components.
Date–time features	Technique	Calendar parts (day/week/month/quarter/year), holiday flags; cyclic seasonality via `sin(2πt/T)`, `cos(2πt/T)`; recency/freshness = days since last activity.
Geo features	Technique	Haversine distance, regional clustering, point density; geohash grid encoding for regional aggregation.
Aggregation / group-by features	Technique	Counts and frequencies per user/plan; group mean/median/std; shares and ratios. Use leave-one-out aggregation to reduce leakage of the row’s own target.
Time-series features	Technique	Lags (t−1, t−7, t−30), rolling mean/std/max/min, exponential moving averages, differencing. All windows must end before T0.
Text/NLP features (extended)	Technique	Sentiment scores, keyword flags (e.g. complaints); TF-IDF + SVD (LSA) for dimension reduction.

7. Feature Selection

Term	Type	Definition / Usage
Three families	Taxonomy	Filter (model-agnostic statistics) · Wrapper (search using a model) · Embedded (selection inside model fitting). Cost and fidelity increase left to right.
Chi-square	Filter / statistical test	Categorical feature vs categorical target. Requires adequate expected cell counts.
ANOVA F-test	Filter / statistical test	Numeric feature vs class label. Assumes approximate normality and equal variances.
Mutual Information	Filter	Captures non-linear dependence; no distributional assumption. Slower, needs binning/estimation choices.
RFE — Recursive Feature Elimination	Wrapper	Fit estimator, drop weakest features, repeat. Choose k by cross-validation. Trade-off: accuracy vs compute time.
SFS / SBS	Wrapper	Sequential forward / backward selection. Same trade-off as RFE; greedy, no optimality guarantee.
Lasso (L1)	Embedded	Drives coefficients to exactly zero → sparsity and automatic selection. Sensitive to correlated features (picks one arbitrarily).
Ridge (L2)	Embedded	Shrinks coefficients, reduces variance; does not zero them out. Handles multicollinearity better than Lasso.
Elastic Net	Embedded	Convex combination of L1 and L2. Default when features are both numerous and correlated.
Tree-based importance	Embedded	Gini/impurity importance (fast, biased toward high-cardinality) vs permutation importance (slower, more trustworthy). Prefer permutation for reporting.
SHAP	Embedded / explainability	Additive attribution per prediction. Use for explanation and selection support — not as evidence of causality.

Dimensionality reduction (see file 07)

Term	Type	Definition / Usage
PCA	Technique	Preserves variance in orthogonal components. Linear; requires standardization.
t-SNE / UMAP	Technique	Visualization only. Do not feed into forecasting models; stochastic, hyperparameter-sensitive, axes uninterpretable.

8. Pipelines, Validation & Reproducibility

Term	Type	Definition / Usage
Golden rules	Practice	1) Never fit preprocessing on the full dataset. 2) Use `ColumnTransformer` for per-type pipelines. 3) Wrap hyperparameter search around the pipeline.
Canonical pipeline	Architecture	`Raw → Split → (Impute → Encode → Scale → FeatureSelect) → Model → Eval`. For temporal data, enforce pre-T0 feature windows at the split step.
Post-preprocessing evaluation	Practice	Metrics: ROC-AUC / PR-AUC (imbalance), Brier score, KS. Compare impute/encode/scale strategies under an identical CV scheme.
Schema contracts	Governance	Types, ranges, constraints agreed between producer and consumer, with SLAs. Version the schema; plan backfills for breaking changes.
Pre-deploy checks	Practice	Null %, ranges, cardinality, drift. Block deployment on failure.
Production monitoring	Practice	Alerts and thresholds on the same checks, running continuously.
Reproducibility	Practice	Random seeds, environment files, dependency locking.
Experiment tracking	Practice	Log params, metrics, artifacts, and data snapshots. Without the data snapshot the experiment is not reproducible.

Anti-patterns

Anti-pattern	Fix
Future-derived features	Freeze feature windows to pre-T0.
Preprocessing fitted on full data	Fit inside CV folds via `Pipeline`.
Unbounded one-hot encoding	Cap cardinality; group rare levels; hash.
Ignoring rare categories	Explicit `Other` bucket with a documented threshold.
No `is_missing` flags	Add by default.
Ignoring group-aware splits	Split by entity when rows repeat per customer.

9. Reference snippets

Group-wise imputation + missingness flag (pandas)

X['income'] = X.groupby('plan_type')['income'].transform(
    lambda s: s.fillna(s.median())
)
X['income_is_missing'] = X['income'].isna().astype(int)

CV-safe target encoding (scikit-learn + category_encoders)

from category_encoders.target_encoder import TargetEncoder
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression

te = TargetEncoder(cols=['plan_id'], smoothing=0.2)
model = Pipeline([
    ('pre', ColumnTransformer([('plan', te, ['plan_id'])], remainder='drop')),
    ('clf', LogisticRegression(max_iter=200)),
])

Key formulas

Standardize: z = (x − μ) / σ
IQR: IQR = Q3 − Q1; cap at [Q1 − 1.5·IQR, Q3 + 1.5·IQR]
Cyclic time encoding: sin(2πt/T), cos(2πt/T)
WoE: ln((Good_i/Bad_i) / (Good/Bad))

Definition of Done

Data quality scorecard exists with numeric thresholds and a data SLA.
Missingness mechanism diagnosed per column (MCAR/MAR/MNAR) before any imputation.
is_missing flags present for every imputed column.
No preprocessing fitted outside training folds; the whole pipeline is cross-validated.
Split strategy matches the data structure: time-aware for temporal, group-aware for repeated entities.
High-cardinality features handled with a documented escalation (rare-level grouping → target encoding → hashing).
Seeds, environment, params, metrics, and data snapshot logged.

References: Kuhn & Johnson (2019), Feature Engineering and Selection · Tan et al., Introduction to Data Mining · scikit-learn docs (Pipeline, ColumnTransformer, Imputation, Encoding, Scaling) · credit-scoring literature on monotone binning, WoE/IV.

🔗 Cùng series

Bài 2: CRISP-DM, KPI Tree & Problem Framing
Bài 4: EDA & Visualization for Decision-Making

Nếu bài này hữu ích, hãy chia sẻ cho người đang học Data Science / Business Analytics. Mọi góp ý về lỗi kỹ thuật rất được hoan nghênh.