Lec 3: Honest models and meaningful evaluation
DGIN 5401 · Dalhousie University
| Step | Question | Evidence you should provide |
|---|---|---|
| Define | What exactly is the prediction study? | data card, time zero, outcome window, split diagram |
| Constrain | What complexity can the information support? | event-count rationale, baseline, learning curve |
| Evaluate | Do risks mean what they say in the target setting? | discrimination, calibration, uncertainty, workload |
| Extend | What changes when time or text changes the task? | revised estimand, output contract, evaluation plan |
| Act | What action and claim can the evidence justify? | threshold scenarios, utility, limitations, audit trail |
A professional deliverable
The model is one component. It should be embedded in a defensible chain from care context all the way to possible action.
Sketch, from memory:
For the AIRO-Sim example, define:
For binary outcome prevalence \(\bar y\), the constant probability model predicts:
\[ \hat p_i=\bar y_{train}. \]
This baseline can have high accuracy when SSI is rare, yet provide no real discrimination.
This provides a worst possible case for log loss, Brier score, calibration, and expected counts.
\[ z_i=\mathbf{x}_i^\top\mathbf{w}+b, \qquad \hat p_i=\sigma(z_i) \]
Here, \(\sigma(\cdot)\) is the sigmoid function and describes a function on \([0,1]\).
Why begin here?
For continuous outcome \(y\), the prediction is:
\[ \hat y_i=\beta_0+\sum_j\beta_jx_{ij} \]
Mean squared error:
\[ \text{MSE}=\frac{1}{n}\sum_i(y_i-\hat y_i)^2. \]
The coefficients describe changes in the fitted conditional mean under the model’s assumptions. These are not necessarily causal effects.
For a binary target \(y_i\in\{0,1\}\):
\[ \ell_i=-\left[y_i\log \hat p_i+(1-y_i)\log(1-\hat p_i)\right]. \]
This loss strongly penalizes confident wrong predictions and encourages probability estimates rather than only hard labels.
Risk
Optimizing log loss does not choose a clinically appropriate threshold or guarantee subgroup calibration. By itself, it’s a very generic approach.
For one predictor, polynomial regression generalizes the linear model by adding powers of (x):
\[ \hat y^{(d)}(x) = \beta_0+\beta_1x+\beta_2x^2+\cdots+\beta_dx^d = \sum_{j=0}^{d}\beta_jx^j. \]
The degree \(d\) controls flexibility. The model is nonlinear in \(x\), but still linear in the fitted coefficients \(\beta_j\).
rng = np.random.default_rng(5401)
x = np.linspace(0, 1, 14)
y_true = np.sin(2*np.pi*x)
y = y_true + rng.normal(0, .18, size=len(x))
xx = np.linspace(0,1,400)
fig, ax = plt.subplots(figsize=(8,4))
ax.scatter(x,y,label="observed")
for deg in [1,3,11]:
coef = np.polyfit(x,y,deg)
ax.plot(xx,np.polyval(coef,xx),label=f"degree {deg}")
ax.plot(xx,np.sin(2*np.pi*xx),linestyle="--",label="data-generating curve")
ax.set_ylim(-1.8,1.8)
ax.legend(ncol=2)
plt.show()Solution: make the weights themselves smaller, to reduce big fluctuations
Ridge / \(L_2\) penalty:
\[ \mathcal{J}=\mathcal{L}+\lambda\frac{1}{2}\sum_j\beta_j^2. \]
Lasso / \(L_1\) penalty:
\[ \mathcal{J}=\mathcal{L}+\lambda\sum_j|\beta_j|. \]
Regularization can reduce variance and stabilize estimates, but it does not rescue poor labels, leakage, or non-representative data.
Risk
Repeatedly checking the test set turns it into another validation set and invalidates your results.
Each simulated patient contributes four similar records.
A random row split allows the model to encounter some visits from a patient during training and other visits during testing, whereas the patient-held-out split test (and therefore tests generalizability) with genuinely unseen people.
rng = np.random.default_rng(5401)
n_patients = 220
visits = 4
patient_signal = rng.normal(size=n_patients)
y_patient = (patient_signal + rng.normal(scale=.7,size=n_patients) > 0).astype(int)
X = np.repeat(patient_signal, visits)[:,None] + rng.normal(scale=.08,size=(n_patients*visits,1))
y = np.repeat(y_patient, visits)
patient_id = np.repeat(np.arange(n_patients), visits)
# A nearest-neighbour-style memorization score for illustration
perm = rng.permutation(len(y))
cut = int(.7*len(y))
tr, te = perm[:cut], perm[cut:]
pred_random = np.array([y[tr[np.argmin(abs(X[tr,0]-X[i,0]))]] for i in te])
acc_random = (pred_random==y[te]).mean()
train_patients = np.arange(0,150)
test_patients = np.arange(150,n_patients)
tr2 = np.where(np.isin(patient_id,train_patients))[0]
te2 = np.where(np.isin(patient_id,test_patients))[0]
pred_group = np.array([y[tr2[np.argmin(abs(X[tr2,0]-X[i,0]))]] for i in te2])
acc_group = (pred_group==y[te2]).mean()
print({"random-row accuracy": acc_random, "patient-held-out accuracy": acc_group}){'random-row accuracy': np.float64(0.7613636363636364), 'patient-held-out accuracy': np.float64(0.7071428571428572)}Repeated records from one patient can let a model recognize the patient rather than learn a portable clinical pattern.
Suppose a registry contains 3,000 surgeries and the 30-day SSI rate is 2%.
Those 60 events specifically may carry much of the information about how predictors relate to SSI than the other 2940 rows.
Now ask:
How much data do we need? Heuristics like “10 events per variable” are not universal design solutions. Use formal sample-size planning and shrinkage principles appropriate to the intended model and estimand. (Efthimiou et al. 2024; Riley, Debray, et al. 2024)
Signals of excess complexity
Defences
A (too-)flexible model can match the particular rare events in the training data very closely, while learning patterns that are too sample-specific to predict new patients reliably.
| Family | Strengths | Questions to ask |
|---|---|---|
| Regularized regression / GAM | stable baseline, additive effects, inspectable | are interactions or nonlinearity essential? |
| Trees / boosting | nonlinearities, interactions, tabular performance | calibration, missingness behaviour, shift? |
| Neural networks | representation learning, multimodal data | data scale, optimization, maintenance, explanation? |
| Survival / longitudinal models | time-to-event and repeated data | censoring, landmarks, dynamic validation? |
| Foundation / generative models | flexible text and multimodal interfaces | factuality, privacy, versioning, prompt dependence? |
\[ g(E[Y\mid X])=\beta_0+f_1(x_1)+\cdots+f_p(x_p) \]
GAMs allow nonlinear univariate effects while preserving an additive structure that can be plotted.
They can be an excellent middle ground between linear models and opaque high-capacity models, especially when decision-makers need stable, reviewable risk functions. (Caruana et al. 2015)
Decision trees partition features into rules. Trees can also be ‘ensembled’.
👍️ Advantages:
👎️ Risks:
\[ \mathbf{h}^{(1)} = \phi\!\left(XW^{(1)}+\mathbf{b}^{(1)}\right), \qquad \hat y = f\!\left(\mathbf{h}^{(1)}W^{(2)}+\mathbf{b}^{(2)}\right). \]
What each layer does
\(XW+\mathbf b\) forms weighted combinations of the current features. The activation \(\phi(\cdot)\) introduces nonlinearity. Without it, stacked affine layers collapse mathematically to a single affine transformation.
Hidden layers \(\mathbf h^{(l)}\) learn representations useful for the training objective.
Additional layers can support increasingly complex learned representations.
Depth is not clinical knowledge
A deeper network may learn a useful representation, a shortcut, or noise. Clinical meaning comes from the data, labels, constraints, validation, and intended use—not from depth itself.
Deep models are useful when patterns are distributed across high-dimensional inputs such as:
Classic successes in dermatology and retinal imaging demonstrated the capacity of convolutional networks, while later work emphasized external evaluation and workflow integration. (Esteva et al. 2017; Gulshan et al. 2016; Beede et al. 2020)
Methodological reviews repeatedly identify leakage, weak external validation, and inappropriate comparisons. (Varoquaux and Cheplygina 2022; Kapoor et al. 2024)
Oversampling, undersampling, SMOTE (Chawla et al. 2002), or class weighting may change the training distribution.
Possible benefit
The learner pays more attention to rare examples or a chosen classification loss.
Common cost
Predicted probabilities no longer correspond to the target population without correction and recalibration.
In simulations and clinical examples, common imbalance corrections produced strong overprediction without improving discrimination; similar sensitivity/specificity trade-offs could be achieved by changing the threshold instead. (Goorbergh et al. 2022)
Separate three choices
How to train, how to estimate risk, and where to place a decision threshold are not the same problem.
flowchart LR A[Original development data] --> B[Outer train fold] A --> T[Untouched validation / test fold] B --> R[Resample or reweight only here] R --> M[Fit preprocessing and model] M --> T T --> E[Evaluate at natural prevalence]
Never oversample before splitting. Synthetic minority examples derived from validation or test observations leak information across the boundary.
| Predicted high risk | Predicted low risk | |
|---|---|---|
| SSI occurs | True positive | False negative |
| SSI does not occur | False positive | True negative |
\[ \text{Sensitivity}=\frac{TP}{TP+FN},\qquad \text{Specificity}=\frac{TN}{TN+FP} \]
\[ \text{Precision}=\frac{TP}{TP+FP},\qquad \text{NPV}=\frac{TN}{TN+FN} \]
Sensitivity and specificity condition on outcome status. Precision and NPV depend strongly on prevalence and the evaluation setting.
If SSI prevalence is 3%, predicting “no SSI” for every surgery yields 97% accuracy.
Report at least:
Risk
Neither curve selects the threshold or tells you whether the resulting action is worthwhile.
A model can rank patients well but still give poor probability estimates \(\hat p\). Calibration asks, e.g.:
Among patients assigned a predicted risk of about 10%, do about 10% actually experience the outcome?
For perfect calibration,
\[ P(Y=1 \mid \hat p=p) \approx p. \]
Calibration is not discrimination
A model can distinguish higher-risk from lower-risk patients well and still assign the wrong absolute probabilities. If a predicted 20% risk actually corresponds to 35% observed risk, a threshold or clinical decision based on that number may be badly misleading.
Calibration should also be examined in the population, place, and time in which the model is intended to be used. (Van Calster et al. 2016; Collins, Dhiman, et al. 2024)
The plot groups patients with similar predicted risks. For each group, it compares the mean predicted risk with the observed event proportion.
How to read the graph
Two useful numerical summaries are:
Brier score:
\[ \text{Brier}=\frac{1}{n}\sum_i(\hat p_i-y_i)^2. \]
Log loss:
\[ -\frac{1}{n}\sum_i[y_i\log\hat p_i+(1-y_i)\log(1-\hat p_i)]. \]
Both reward accurate probability estimates and penalize overconfidence. Compare them with a baseline (e.g., prevalence-only) and report uncertainty.
rng = np.random.default_rng(7)
n = 1200
p = rng.beta(1.2, 14, size=n)
y = rng.binomial(1, np.clip(.01 + 1.6*p,0,.8))
rows=[]
for t in [.03,.05,.08,.12,.20]:
pred=p>=t
tp=((pred)&(y==1)).sum(); fp=((pred)&(y==0)).sum(); fn=((~pred)&(y==1)).sum()
rows.append({"threshold":t,"flagged":int(pred.sum()),"TP":int(tp),"FP":int(fp),"FN":int(fn)})
pd.DataFrame(rows)| threshold | flagged | TP | FP | FN | |
|---|---|---|---|---|---|
| 0 | 0.03 | 879 | 152 | 727 | 13 |
| 1 | 0.05 | 686 | 137 | 549 | 28 |
| 2 | 0.08 | 455 | 111 | 344 | 54 |
| 3 | 0.12 | 246 | 75 | 171 | 90 |
| 4 | 0.20 | 74 | 32 | 42 | 133 |
Pause and discuss
At these thresholds, who is asked to do what—and how often?
The observed outcome does not change when the threshold moves; the decision classification does.
Lowering the threshold creates more alerts: some false negatives become true positives, while some true negatives become false positives. Raising it reverses those changes.
Move the slider and watch what changes:
The decision depends on the downstream consequences:
\[ \operatorname{cost}(t) = c_{\mathrm{FN}}\operatorname{FN}(t) + c_{\mathrm{FP}}\operatorname{FP}(t). \]
Threshold selection should involve the people who bear the workload and harm.
Decision-curve analysis compares the value of model-guided action with “act on all” and “act on none,” weighting false positives according to a threshold probability. (Vickers, Van Calster, and Steyerberg 2016)
\[ \text{Net benefit}=\frac{TP}{N}-\frac{FP}{N}\frac{p_t}{1-p_t}. \]
Risk
Net benefit is only meaningful when the threshold encodes a defensible trade-off and the proposed action matches the analysis.
Report confidence intervals or resampling distributions for:
Small event counts can make apparently precise model outputs highly uncertain. (Riley et al. 2025; Riley, Debray, et al. 2024)
Pre-specify clinically and socially relevant groups, then report:
Chest-radiograph studies illustrate that underdiagnosis can be systematically concentrated in underserved populations. (Seyyed-Kalantari et al. 2021; Lotter et al. 2024)
Useful validation axes include:
Separate model transportability from local recalibration and implementation effectiveness. (Riley, Archer, et al. 2024)
Temporal methods differ in what history they retain, how they handle irregular observations, and whether they predict at a fixed horizon. E.g.,
Summarize clinically meaningful windows: use features in a conventional predictor. (Ghassemi et al. 2015)
Model temporal dynamics explicitly: autoregressive or state-space models represent temporal dependence and changing states; Gaussian processes can accommodate irregular sampling. (L.-W. H. Lehman et al. 2015; Ghassemi et al. 2015; Futoma, Hariharan, and Heller 2017)
Learn sequence representations: recurrent networks (e.g., LSTM, GRU), temporal CNNs, and transformer or attention-based models learn features from sequences. (Che et al. 2018; Catling and Wolff 2020; Li et al. 2020)
Choose the prediction schedule before the architecture
Specify when the prediction is made, what history is available at that moment, how far ahead it predicts, how often it updates, and what action follows.
“SSI within 30 days” converts several different trajectories into the same binary label.
| Patient | Follow-up | Event | Binary 30-day label | Information lost |
|---|---|---|---|---|
| A | 30 d | SSI on day 4 | 1 | early event |
| B | 30 d | SSI on day 27 | 1 | late event |
| C | 30 d | none | 0 | full event-free follow-up |
| D | 8 d | none observed | ? | incomplete follow-up |
| E | 12 d | death | competing event | SSI no longer observable in the same way |
Time-to-event question
What is the probability of remaining event-free beyond time \(t\), given what is known now?
flowchart LR O[Time origin<br/>ICU transfer] --> F[Follow-up] F --> E[SSI event] F --> C[Censoring<br/>event status later unknown] F --> K[Competing event<br/>death or another event precludes SSI]
Changing any of these changes the estimand and the analysis.
Let \(T\) denote the time from a defined origin to the event of interest, and let \(t\) be a particular follow-up time; then the survival function is:
\[ S(t)=P(T>t) \]
The hazard is an instantaneous event rate among those still at risk:
\[ h(t)=\lim_{\Delta t\to 0}\frac{P(t\leq T<t+\Delta t\mid T\geq t)}{\Delta t} \]
Survival probability
\(S(30)=0.94 \equiv\) “Probability of remaining SSI-free beyond day 30 is 94%.”
Hazard
“Current event rate around day 30 among patients still SSI-free.”
Here, each update estimates the remaining risk of SSI by postoperative day 30 using a new information set. A fixed horizon after each landmark (e.g., risk over the next 7 days) would be a different estimand and should be labelled accordingly.
At every landmark, ask:
More updates do not guarantee more value
A 2025 comparison of static and dynamic infection-risk models found that, largely, modelling choices and competing risks affected both calibration and performance. (Gao et al. 2025)
A compressed history of LMs:
Clinical embeddings are shaped by domain, task, and corpus. (Khattak et al. 2019)
For each token (or event), attention asks which other items are relevant, assigns them weights, and combines the information they carry. (Vaswani et al. 2017)
Canonical multi-head pattern
Source: Vaswani et al. (2017), via Wikimedia Commons; resized only. CC BY-SA 4.0.
Clinical example: attention across EHR timeline
In BEHRT, diagnoses run downward in chronological order. A diagnosis selected on the left is compared with events elsewhere in the history. The example includes a long-range association between rheumatoid arthritis and later enthesopathies or synovial disorders. (Li et al. 2020)
Each row is a query token and each column is a possible key token. A darker cell means that, in this attention head, the query placed more weight on that key while constructing its contextual representation.
One ClinicalBERT self-attention head applied to an illustrative clinical sentence. Red rendering from the nwams ClinicalBERT tutorial, based on Huang et al., Fig. 5. (nwams, n.d.; Huang, Altosaar, and Ranganath 2020)
Do not assume that attention is explanation
This heatmap shows one head, in one layer, for one sentence. A large weight does not by itself establish causality, or a faithful explanation of prediction. (Jain and Wallace 2019)
Discriminative: learn \(P(Y\mid X)\) or a decision boundary.
Generative: learn aspects of \(P(X,Y)\), \(P(X)\), or next-token distributions that can generate new outputs.
These categories overlap in modern systems. A generative model can be used for classification; a discriminative model can sit inside a generative workflow.
Pause and discuss
In this context, how ought we evaluate: generated text quality, downstream decision quality, or patient/operational outcome?
Generalist medical AI aims to build reusable models that can support many tasks and adapt with little task-specific labelled data. (Moor et al. 2023; Tu et al. 2024)
Potential roles and representative systems
Summarize and extract information: Clinical-T5 supports text-to-text clinical tasks; GatorTron was evaluated for concept and relation extraction from clinical records. (E. Lehman and Johnson 2023; Yang et al. 2022)
Answer questions with evidence: Med-PaLM targets medical question answering; Almanac augments an LLM with retrieval from curated medical sources. (Singhal et al. 2023; Zakka et al. 2024)
Assist documentation: ambient AI scribes generate draft notes from clinical conversations, to reduce documentation burden and increase practitioner well-being. (Seth, Carretas, and Rudzicz 2024; Afshar, Baumann, et al. 2025)
Interpret multiple modalities: Med-PaLM M combines language with information such as medical images, genomics, and structured clinical data. (Tu et al. 2024)
Conduct diagnostic dialogue: AMIE was developed for conversational history-taking, diagnostic reasoning, and communication. (Tu et al. 2025; McDuff et al. 2025)
Use tools and execute workflows: MedAgentBench evaluates LLM agents that retrieve information and perform multi-step tasks through a FHIR-compatible EHR environment. (Jiang et al. 2025)
Generation changes the evaluation contract
For generated output, “is the class correct?” is no longer enough. Evaluation may need to examine factuality, coverage, clinical reasoning, safety, bias and equity, confidence, naturalness, privacy, et c.
MEDIC, e.g., combines different tasks and metrics across five dimensions of clinical competence; others add user-centred, trustworthiness, privacy, empathy, and operational measures. No single score establishes that a generative system is clinically useful. (Kanithi et al. 2026; Abbasian et al. 2024)
A model can perform well on exams or simulated consultations yet fail to improve a clinician–AI team.
This is especially relevant for results on AIRO-Sim.
In a randomized trial of 50 physicians, access to an LLM did not significantly improve diagnostic reasoning compared with conventional resources, despite strong standalone model performance. (Goh et al. 2024)
Evidence
Human–AI performance depends on interface, human psychology, task allocation, training, time, trust, and how disagreements are resolved.
A prognostic model estimates what may happen under the patterns in its development data.
A treatment decision requires a comparison between alternative actions. Let \(Y^a\) denote the adverse outcome that would occur under action \(a\), with \(a_0\) as the comparator. For patients with characteristics \(x\),
\[ B(x) = P(Y^{a_0}=1\mid X=x) - P(Y^{a}=1\mid X=x). \]
A positive \(B(x)\) means that action \(a\) is expected to reduce absolute outcome risk relative to the comparator. This is an expected conditional benefit: both potential outcomes cannot be observed for the same patient. (Kent, Steyerberg, and Klaveren 2018; Geloven et al. 2020)
A high-risk patient may have:
A moderate-risk patient may have:
Baseline risk can still help predict absolute benefit when relative treatment effects and harms are reasonably stable, but risk and benefit are not “two sides of the same coin.”
If a predictor prompts effective treatment, the resulting prevention of outcomes can make that predictor appear less prognostic in historical data. This treatment paradox can cause an ordinary prognostic model to underestimate risk under no treatment. (Sperrin et al. 2018; Geloven et al. 2020)
A risk score is not a treatment rule
Before recommending an action, specify the intervention, comparator, expected benefit, harms and burden then ask whether model-guided decisions improve net benefit relative to usual care.
Possible questions:
Risk
An attractive explanation can be unstable, incomplete, or persuasive without being faithful. (Ghassemi, Oakden-Rayner, and Beam 2021b)
First, not every problem is linear (Rudin 2019). A suspiciously strong model might be using:
Pause and discuss
For each top feature, ask: could this exist, in this form, at the intended prediction time in a new patient?
Proposed claim: “At the end of surgery, our model can identify patients at high risk of SSI within 30 days so that enhanced wound follow-up can be targeted.”
Available data: 2,400 surgeries over 12 years; 72 recorded SSIs (3%); 30 candidate registry variables; repeated operations for some patients; and a registry redesign after year 6.
The proposed analysis
Your role: independent review panel
⏲️ 15 minutes: Would you approve this study as written?
Identify the three most consequential threats to a credible result. For each one, explain why it matters and propose the smallest repair that would make the first experiment defensible.
A strong project will ideally lean into aspects of machine learning (e.g., modelling itself or its use) but also into aspects of reproducibility so another professional can reproduce and evaluate your work.
Make the evidence auditable
Make the limits actionable
Ambition with discipline
Build with curiosity, confidence, and ambition, but keep every claim proportional to the evidence! (Collins, Moons, et al. 2024; Moons et al. 2025; Vasey et al. 2022; Lekadir et al. 2025)
DGIN 5401 · Operationalized ML in Healthcare
