Bài 2: CRISP-DM, KPI Tree & Problem Framing
Series: Business Analytics Cheatsheet Series
Topics:
crisp-dmkpi-treeproblem-framingab-testingguardrailsSource: BA1 — Introduction to Business Analytics: CRISP-DM · KPI Tree · Problem Framing. Organized by CRISP-DM phase. Cross-phase topics (KPI Tree, problem framing, templates) sit in their own sections at the end.
Core claim of the deck: up to 80% of analytics failures come from a misframed problem or KPI — not from a weak model. Framing is the highest-leverage step.
0. Orientation
| Term | Type | Definition / Usage |
|---|---|---|
| Business Analytics | Discipline | Process that turns data into insight and action. Spans descriptive → diagnostic → predictive → prescriptive layers. Outputs: reports/KPIs, models, A/B tests, operational recommendations. |
| BI vs DA/DS vs AI/ML | Taxonomy | BI: reporting, dashboards, KPI monitoring (past/present). DA/DS: root-cause, forecasting, segmentation, optimization. AI/ML: learn from data to automate prediction/recommendation inside the product. |
| Data Value Chain | Framework | Collect → Store → Clean/Transform → Analyze/Model → Deploy → Monitor. Treat as a product lifecycle: quality, reliability, repeatability. Principle: start from the decision to be supported, not from the data you happen to have. |
| Team roles | Org | Product/Data PM, Data Analyst, Data Scientist, Data Engineer, BI Dev, MLOps/Platform. Core skills: SQL, statistics, EDA, visualization, modeling, data storytelling. Supporting: domain knowledge, experiment design, data ethics & governance. |
| Trustworthy AI | Governance | Fairness, safety, transparency, auditability. Operationalized through KPI definitions, data quality, lineage & catalog, RACI. Privacy/security/compliance are constraints, not afterthoughts. |
| CRISP-DM | Framework | Cross-Industry Standard Process for Data Mining. Six phases, explicitly iterative — loop back when new insight or a goal change appears. Artifacts per phase: charter, data dictionary, feature list, model card, runbook. |
1. Business Understanding (BU)
Goal: convert a business objective into a decision, a measurable success definition, and a bounded scope.
| Term | Type | Definition / Usage |
|---|---|---|
| SMART objective | Technique | Specific, Measurable, Achievable, Relevant, Time-bound. Example: “+15% Q4 revenue YoY.” Vague goals (“improve engagement”) cannot be evaluated and should be rejected at charter review. |
| Decision-centric framing | Principle | Identify the decision analytics will inform before choosing a method. Test question: “What decision will this inform?” If there is no answer, do not start the work. |
| Success KPI | Metric | The primary measure the project is judged on. One primary; 2–4 supporting. More than one primary = no primary. |
| Guardrail | Metric | Hard constraint. Example: “do not reduce overall CR by more than 1pp.” Any solution violating it is not deployed — state this in the charter, not in the readout. |
| Hypothesis (H1, H2…) | Artifact | Testable statement linking a lever to a KPI. Example: H1 = bundles/upsell raise AOV without hurting CR; H2 = mobile checkout optimization raises mobile CR. Each must name the data needed to test it. |
| Project Charter | Artifact | Business problem & opportunity, objective, in/out of scope, stakeholders & RACI, hypotheses, success criteria & guardrails, timeline, risks & mitigation. The single approval gate before data work starts. |
| Scope (In / Out) | Artifact | Explicit exclusions matter more than inclusions. Example out-of-scope: paid acquisition, pricing/discount policy, backend overhaul, desktop-only work. |
| RACI | Governance | Assign per workstream: Sponsor = Accountable; Marketing/UX/DS = Responsible; DE/Finance = Consulted; Exec = Informed. |
| Risk register | Artifact | Risk → mitigation pairs. Typical: solution hurts CR → guardrail monitoring + phased A/B + rollback; poor data quality → early audit + rule-based fallback; Q4 seasonality → use historical + current-year data. |
| Data rules / canonical definitions | Governance | Agree on what a session, transaction, user, and attribution are before measuring. Disagreement here invalidates everything downstream. |
BU outputs checklist: problem statement · scope · stakeholders & RACI · analysis questions · hypotheses · KPIs & guardrails · timeline · risks · data definitions.
2. Data Understanding (DU)
Goal: verify that reality matches the charter’s assumptions before committing engineering time.
| Term | Type | Definition / Usage |
|---|---|---|
| Source inventory | Technique | For each source: schema, size, freshness, access permissions. Gate the project on access being granted, not promised. |
| EDA (at DU stage) | Technique | Distributions, outliers, missingness, temporal consistency. Purpose here is validation, not insight generation. |
| KPI reconciliation | Technique | Match KPI definitions against real fields: order date vs ship date, returns, cancellations, order edits. Expect a gap; document it. |
| Funnel analysis | Technique | session → view → add_to_cart → checkout → purchase. Quantify absolute drop-off per stage, not just rates. |
| Segmentation (quick win) | Technique | Split by channel/device (mobile vs desktop, organic vs paid) before modeling. Often reveals that the “average” problem does not exist in any segment. |
| Double-counting detection | Diagnostic | Retries and timeouts inflate purchase counts. Deduplicate on transaction key before any revenue claim. |
| Leakage detection (early) | Diagnostic | Flag fields generated after the decision point at DU, not at modeling. Cheaper to exclude than to debug later. |
3. Data Preparation (DP)
Goal: produce a leakage-safe, reproducible modeling dataset.
| Term | Type | Definition / Usage |
|---|---|---|
| Cleaning | Technique | Handle missing values and outliers; normalize units, currency, timezone. Do this inside a fitted pipeline, not in an ad-hoc notebook cell. |
| Feature engineering | Technique | Derive decision-relevant signals: RFM (recency/frequency/monetary), price_sensitivity, device_speed. Domain knowledge beats feature count. |
| Time-based split | Technique | Train on pre-T0, evaluate on post-T0. Mandatory whenever the production use is forecasting a future event. Random splits leak temporal information. |
| Data tests | Technique | Schema, ranges, freshness, uniqueness. Run as pre-deploy gates with alert thresholds. |
| Taxonomy & data dictionary | Artifact | Standardized event naming, field semantics, late-arriving data policy. |
| Feature Store | Infrastructure | Reuse features across models with lineage and versioning; guarantees train/serve parity. |
| Data leakage | Anti-pattern | Any information unavailable at decision time entering training. Symptom: excellent validation, collapse in production. See file 03 for the full taxonomy. |
4. Modeling (M)
Goal: the simplest model that supports the decision, with a defensible baseline.
| Term | Type | Definition / Usage |
|---|---|---|
| Task selection | Decision | Classification (churn) · Regression (AOV) · Clustering (segments) · Optimization (pricing). Derived from the decision, not from data availability. |
| Baseline | Practice | Simple model + business rules first. Compare AUC/MAE/lift against it. A model that does not beat rules should not ship. |
| Threshold tuning | Technique | Set the decision threshold from business goals and error costs, not from the 0.5 default. See file 06. |
| Time-aware cross-validation | Technique | Rolling-origin / TimeSeriesSplit. Standard k-fold overstates performance for temporal targets. |
| Regularization & feature selection | Technique | L1/L2 and embedded importance to control overfitting and reduce serving cost. |
| Explainability (e.g. SHAP) | Technique | Attribute predictions to features. Required for regulated decisions and for stakeholder trust; treat as an explanation, not a causal statement. |
| Bias check | Governance | Compare performance across protected/vulnerable segments before deployment. |
| Packaging | Practice | Model card, reproducible environment, dependency locking, seed fixing. |
5. Evaluation (E)
Goal: separate technical performance from business impact — they are not the same and often disagree.
| Term | Type | Definition / Usage |
|---|---|---|
| Technical evaluation | Metric set | AUC/PR-AUC, RMSE/MAE, calibration, cohort stability. Answers: does the model rank/predict well? |
| Business evaluation | Metric set | Revenue/profit uplift, KPI movement, guardrail impact. Answers: does deploying it make money without breaking anything? |
| Experimentation | Method | A/B test, holdout, careful sequential testing. The only credible bridge from model metric to business metric. |
| Cohort stability | Diagnostic | Performance consistent across cohorts and time slices. Instability predicts production degradation. |
| “Technical success, business failure” | Anti-pattern | High AUC, zero adoption. Fix by involving the decision owner from BU and by testing the intervention, not the model. |
6. Deployment (D)
Goal: operate the model as a product with defined failure modes.
| Term | Type | Definition / Usage |
|---|---|---|
| Batch vs real-time scoring | Decision | Batch for periodic campaigns; real-time when the decision happens in a user session. Latency budget belongs in the charter. |
| Canary / Blue-Green | Deployment pattern | Canary = gradual traffic ramp; Blue-Green = atomic switch between two full environments. Both require an automated rollback trigger. |
| Monitoring | Practice | Data drift, model drift, SLA/latency, alerting, auto-rollback. Silence is not health — instrument the absence of predictions too. |
| MLOps | Practice | CI/CD, model registry, feature–model consistency. |
| Handover package | Artifact | Project charter · data dictionary · EDA report & quality checklist · model card · evaluation report · rollout plan · monitoring dashboard · operations runbook · improvement plan. |
CRISP-DM common pitfalls
| Pitfall | Type | Fix |
|---|---|---|
| Starting from data, not decisions | Anti-pattern | Ask “what decision does this inform?” at charter review; kill the project if unanswered. |
| Unmeasurable KPIs | Anti-pattern | Require formula + window + unit + owner for every KPI. |
| Data leakage | Anti-pattern | Time-based splits; fit preprocessing on training folds only; exclude post-decision fields. |
| Inconsistent KPI definitions across teams | Anti-pattern | One canonical definition in the data dictionary; dashboards read from it. |
| Great model, zero business impact | Anti-pattern | Design the intervention and the experiment alongside the model, not after it. |
KPI Tree
Purpose: connect a North Star to levers a named person can pull.
| Term | Type | Definition / Usage |
|---|---|---|
| KPI Tree | Framework | Hierarchical decomposition: North Star → drivers → levers → owners. Makes trade-offs explicit and prevents disconnected metric sprawl. |
| 5-step process | Process | 1) Choose the North Star (Monthly Revenue, MRR, GMV). 2) Decompose into influenceable drivers. 3) Attach levers and owners per branch. 4) Define guardrails, trade-offs, instrumentation, update cadence. 5) Connect to dashboards + weekly review. |
| KPI selection principles | Principle | Fewer but better; measurable, available, actionable. Avoid vanity metrics unless tied to a decision. |
| E-commerce revenue tree | Example | Revenue = Sessions × CR × AOV. CR = P(view→cart) × P(cart→checkout) × P(checkout→purchase). AOV = Σ(price × qty)/#orders. Levers: bundles, upsell, free-shipping threshold. |
| Numeric illustration | Example | 1,000,000 sessions/mo · CR 2.0% · AOV 420k → revenue ≈ 8.4bn VND. Target +15% → 9.66bn. O1: CR → 2.3% (AOV fixed). O2: AOV → 483k (CR fixed). Quantifying options this way turns strategy debate into arithmetic. |
| SaaS MRR tree | Example | MRR = #Customers × ARPA × (1 − Churn); #Customers = New Signups × Activation Rate × Conversion to Paid. Levers: pricing tiers, onboarding, success motions, retention plays. |
| Measurement quality | Practice | Canonical definitions (returns/cancellations, order edit dates); fixed windows for CR/AOV by channel/device; bot traffic removed. Health metrics alongside: latency, payment errors, inventory availability. |
| Measurement plan / instrumentation | Artifact | Events: page_view, add_to_cart, begin_checkout, purchase. Schema: user_id, session_id, device, source, campaign, value. Fixed update frequency and named dashboard owner. |
| Dashboard design | Practice | Board 1: North Star + driver-tree heatmap (R/Y/G). Board 2: funnel by device/channel. Board 3: AOV by category. Features: cohort filters, event annotations (sales, UI changes). |
| KPI Tree pitfalls | Anti-pattern | Overlapping branches / double counting · missing guardrails · no clear owner or lever · weak causal validation · refresh slower than the decision cadence. |
Problem Framing
| Term | Type | Definition / Usage |
|---|---|---|
| DOC — Decision, Options, Criteria | Framework | D: the choice to be made. O: candidate options. C: the KPI set used to compare them. A problem statement without all three is not actionable. |
| Problem Statement Template | Artifact | Context (current situation + goal) → Decision → Options → Criteria & KPIs. One page maximum. |
| Business question → analytics task | Technique | “Grow revenue 15% without more ad spend” → Task 1 (diagnostic): find funnel bottlenecks & affected cohorts. Task 2 (prescriptive): rank improvements by ROI. Always translate before choosing a method. |
| Target variable definition | Technique | Make it explicit and time-bounded: “purchase within 7 days of session (binary).” Ambiguous targets produce unusable models. |
| Decision-point leakage | Anti-pattern | Excluding variables generated after the decision point (e.g. applied voucher). Pair with time-based splits and drift/cohort monitoring. |
| Constraints / Assumptions / Risks | Artifact | Constraints: engineering resources, rollout time, SLA. Assumptions: stable seasonality, data reliability. Risks: ad policy change, supply disruption, behavior shift. State assumptions explicitly so they can be invalidated. |
| Success criteria & guardrails | Artifact | Primary: +15% revenue. Secondary: CR drop <1pp, stable AOV. Technical: lift ≥ X%, acceptable SE/CI, latency <200ms. Ethical: no worsened experience for vulnerable groups; transparent pricing. |
| Impact/Effort matrix | Prioritization | Rank opportunities into quick wins · big bets · fillers · avoid. Output: a 6–8 week backlog with milestones. |
Anti-patterns & fixes
| Anti-pattern | Fix |
|---|---|
| Analytics not tied to a decision | Ask “what decision will this inform?” before scoping. |
| Vague / unmeasurable KPIs | Standardize definitions and sources in the data dictionary. |
| Technical success, business failure | Experiment; canary release; measure the intervention. |
A/B Testing basics
| Term | Type | Definition / Usage |
|---|---|---|
| Randomization unit | Design choice | User or session. Choose the unit the intervention actually affects; mismatches cause interference and inflated significance. |
| Primary metric | Design choice | One metric, declared before launch. Everything else is secondary or a guardrail. |
| Peeking | Anti-pattern | Checking results repeatedly and stopping at significance. Inflates false positives; fix the duration in advance or use a sequential test designed for it. |
| Group independence | Assumption | No spillover between arms. Violated by social features, shared inventory, marketplace supply. |
| Test duration | Design choice | Derived from traffic and minimum detectable effect (MDE), plus at least one full weekly cycle. |
| Sample size rule of thumb (CR) | Formula | n ≈ 16 × p(1−p) / Δ² per group. Example: p = 2%, Δ = 0.3pp (0.003) → n ≈ 34,844 per group. Use a proper calculator for multiple KPIs or adjusted tests. |
Templates
Problem Framing Canvas
Context & Goals:
Decision (D):
Options (O): O1 … O2 … O3 …
Criteria & KPIs (C):
Constraints:
Assumptions:
Risks:
Experiment / Measurement plan:
Owner: Timeline:
Required data preparation:
KPI Tree (blank)
North Star: ______________________________
Branch 1 → driver → lever → owner
Branch 2 → driver → lever → owner
Branch 3 → driver → lever → owner
Guardrails: ______________________________
Instrumentation & cadence: _______________
CRISP-DM checklist
BU: SMART goals · KPIs/guardrails · stakeholders & RACI
DU: inventory · EDA · data definitions · quality issues
DP: missing/outliers · feature list · time-aware splits
M : baselines · evaluation · explainability · versioning
E : business evaluation · experiment plan
D : deployment · monitoring · runbook · rollback
Definition of Done
- Charter approved with one primary KPI and at least one guardrail.
- Every hypothesis names the data required to test it.
- KPI Tree has an owner and a lever on every leaf.
- Splits are time-aware; no post-decision features in the feature list.
- Experiment plan exists before the model is built.
- Handover package complete before deployment sign-off.
References: Davenport & Harris, Competing on Analytics · Provost & Fawcett, Data Science for Business · CRISP-DM guide · Google Analytics / Amplitude event taxonomy docs.
🔗 Cùng series
Bài 1: Business Analytics Foundations & Types of Analytics
Bài 3: Data Quality & Feature Engineering
Nếu bài này hữu ích, hãy chia sẻ cho người đang học Data Science / Business Analytics. Mọi góp ý về lỗi kỹ thuật rất được hoan nghênh.