Applied Machine Learning Research Lab

Process of Data Science

Teaching / CSCI 4146 & CSCI 6409

Summer 2026 · Dalhousie University

Process of Data Science

A model is only one step in a decision system. This project-driven course follows the full lifecycle: frame the problem, choose metrics, build and audit models, diagnose AI failures, estimate causal effects, and monitor what happens after deployment.

12 weeks5 team projectsDecision-centered AI

What you will learn

01Frame the real decision

Translate a broad goal into north-star, input, and guardrail metrics using a structured data-science workflow.

02Explain model behavior

Use SHAP, LIME, counterfactual reasoning, slicing, and subgroup analysis to find how and where models fail.

03Diagnose modern AI systems

Analyze prompts, logs, embeddings, clusters, and LLM-as-a-judge pipelines to map failure patterns.

04Move from prediction to action

Apply causal inference, uplift modeling, and monitoring to support decisions and measure downstream impact.

The full lifecycle

MeasureFrame goals and metrics
→
ModelPredict and validate
→
ExplainAudit and debug
→
DecideEstimate impact
→
MonitorTrack production

Weekly syllabus

WeekThemeTopics and studio work
1Data science as decision making

CRISP-DM, problem framing, business impact, and the Python/Jupyter environment

2Metrics that reflect the goal

North-star and guardrail metrics; confusion matrices as business tools; dashboard workshop

3Evidence and predictive baselines

Statistical significance, power, actionability, regression, classification, and validation pipelines

4Evaluation and audit targets

Calibration, overfitting, transparency, XGBoost/LightGBM, and preparation for model auditing

5Explain model behavior

LIME, SHAP, counterfactuals, feature importance, partial dependence, and interaction effects

6Find hidden failures

Slicing, error attribution, bias discovery, group fairness, proxy variables, and Fairlearn

7Debug AI systems

Failure taxonomies for ML and LLM pipelines; logs, prompts, labels, and conversation analysis

8Map the failure surface

Embeddings, clustering, semantic patterns, LLM judges, root-cause dashboards

9Reason causally

Counterfactuals, propensity scores, matching, common support, and synthetic controls

10Estimate decision impact

Uplift modeling, heterogeneous treatment effects, Qini curves, ROI, and persuadables

11Monitor production reality

Drift, feedback loops, silent failure, fairness decay, governance, Evidently, and Streamlit

12Integrate and communicate

The complete lifecycle, project showcase, industry deployment patterns, and final clinic

Five projects, one connected portfolio

Project 1Data analysis and metric design
Project 2Model explanation and audit
Project 3LLM observability and debugging
Project 4Causal inference and decision support
Project 5Responsible AI monitoring

Projects are completed in teams of one to three. Each produces a technical submission and a ten-minute “impression talk” modeled after a concise stakeholder update. Assessment emphasizes technical execution, actionable insight, debugging, communication, and the ability to defend decisions.

Assessment at a glance

80% Five projects20% Participation

This course is offered as CSCI 4146 and CSCI 6409. It assumes prior machine-learning foundations and focuses on the broader system in which models operate.