Responsible AI by Design

Module 8 – Responsible AI Design & Procurement

Frank Rudzicz

2025-10-01

What we mean by
“AI procurement”

  • Working definition: Selecting, contracting, and governing AI systems so they remain safe, effective, and reversible across their lifecycle
    • Not a one-off purchase.
  • Decisions first, tools second (outcomes > features)
    • i.e., Do not be one of these people:
      “everyone is doing LLMs, ∴ we need to do LLMs”.
  • Evidence over claims (docs, tests, logs)
  • Lifecycle obligations (monitoring, updates, incidents, exit)

Tip

👉 We’re buying capability under uncertainty. Our job is to make uncertainty tractable with evidence and enforceable obligations. 👈

Today

  • Why AI procurement is different (lifecycle, opacity, systemic risk)
  • What to ask vendors up front (evidence pack & sandboxing)
  • Contract patterns that reduce risk (data/IP, audit, SLAs, exit)
  • Technical assurance you can require (security, fairness, robustness)
  • Mini‑vignette + Activity 8.1: Build your 1‑page checklist

Why AI procurement is different

You may already have a software procurement practice in place.

  • Models change over time: updates & drift mean obligations must be lifecycle, not one‑off delivery.
  • Opaque supply chains: foundation models, data vendors, labeling — need upstream documentation and change‑control.
  • Systemic + reputational risk: fairness/privacy/security and public trust → cross‑functional ownership (legal/privacy/security/data/product).
  • Evidence over claims: require artifacts you can independently verify.

What to ask vendors
(RFI/RFP “evidence pack” (1/2))

You have a Request for Information (RFI) or Request for Proposal (RFP) framework.

  • Model & system documentation
    • Model Cards (Mitchell et al. 2019) 👀 and System Cards (intended use, limitations, subgroup performance, safety mitigations, evaluation methods).
    • Datasheets for Datasets (Gebru et al. 2021) 👀 for key training/eval sets.
  • Security & robustness
    • Latest 🔗 OWASP Top 10 risks for LLM/GenAI
    • Spoilers: prompt injection, sensitive information disclosure, supply chain, data poisoning, improper output handling, excessive agency…

What to ask vendors
(RFI/RFP “evidence pack” (2/2))

  • Risk & compliance (light‑touch)
    • One‑page mapping to your policy; if needed, reference ISO 42001 / ISO 23894 / NIST AI RMF / EU AI Act at a high level.
  • Operations & transparency
    • Post‑market monitoring plan, logging scope & retention, incident thresholds & timelines; update cadence and rollback process.
    • Usage boundaries: data‑use/training restrictions, human‑in‑the‑loop, explanation support, sandbox access for evaluation.

Best practices: Contracts

  1. Data & IP: No training on your data/outputs without written approval; dataset lineage & licenses; IP/copyright indemnity.
  2. Documentation & transparency: Maintain Model/System Cards, Datasheets, user docs, and (when high‑risk) a technical dossier.
  3. Evaluation & audit rights: Independent testing pre‑award and periodically; access to logs/prompts/artifacts; cooperative evidence production.
  4. Security: Controls aligned to OWASP LLM Top 10; vulnerability disclosure; timely patching; tenant/data segregation; incident timelines.
  5. Performance & SLAs: Accuracy/quality targets (with subgroup floors), latency/uptime, hallucination/error‑rate ceilings, explainability on request.
  6. Change control: Notice & re‑certification on major model or provider changes; rollback rights.
  7. Exit & portability: Export of data, prompts, logs, and fine‑tunes; transition assistance; escrow or functional equivalence on termination.

Best practices: Contracts

  • Pattern library to adapt:
    • EU Model Contractual Clauses for AI procurement (MCC‑AI)
      • light & high‑risk versions with commentary.
      • 🔗 Europa.eu

EU MCC-AI — What you get

  • A model clause library for public buyers procuring AI, updated 5 Mar 2025 to align with the EU AI Act.
  • Two tracks:
    • Full (high-risk) — comprehensive obligations aligned to high-risk duties.
    • Light (non-high-risk) — leaner set for proportionate control.
  • Comes with an in-depth Commentary (how to customize, where to tighten/relax).
  • Translated and field-tested across EU buyers; usable as a starting point outside the EU with local law addenda.
  • Reminder: MCC-AI is a contractual add-on, not a whole contract (you still need your standard terms).

🔗 Updated EU AI model contractual clauses (MCC-AI)

MCC-AI clause patterns to adapt

Clause pattern (what to borrow) What it does Where to tune for your context
Documentation & transparency (tech dossier + Model/System Cards + Datasheets) Forces concrete artifacts (intended use, limits, subgroup metrics, safety mitigations) Specify formats you accept; require updates on each model/version change
Risk mgmt & post-market monitoring Makes risk controls ongoing (not one-and-done) and mandates monitoring plans Define review cadence; align to your governance (e.g., quarterly reviews)
Testing & audit rights Grants sandbox, pre-award testing, periodic re-tests; obliges cooperation and log access Add subgroup floors; require change-impact summaries after each update
Security & incidents Aligns with recognized risks (e.g., OWASP LLM Top-10); sets incident notice windows Name timelines (e.g., P1 ≤ 24h); require vuln disclosure & patch SLAs
Data & IP / training-use Controls data lineage/licences; bans training on your data/outputs without consent Add “no fine-tuning on our data” unless expressly approved
Performance & SLAs Sets quality targets and error ceilings (incl. hallucination) Include subgroup floors + latency/uptime; define acceptance tests
Change control & re-certification Requires notice & re-assessment for major updates or provider swaps Define what counts as “major” (e.g., base model change; Δperf > X%)
Subcontractors & supply-chain Ensures flow-down of duties to subprocessors Require list of subprocessors + approval rights
Exit & portability Guarantees export of data/prompts/logs/fine-tunes; transition help Add escrow or “functional equivalence” if escrow isn’t possible

Quick-start adaptation (5 steps)

  1. Classify the use case → start with Light; switch to Full if risk is high (safety/rights-impacting).
  2. Add a Contract Annex for: (i) evidence pack, (ii) testing & audit, (iii) incident SLAs, (iv) change control, (v) exit.
  3. Insert your evaluation pack: sandbox on your data; subgroup floors; robustness & LLM security checks.
  4. Localize: bolt on your privacy/records laws (e.g., DADM/AIA inventory for Canada, retention rules, data-location).
  5. Define update triggers (what requires re-cert) and the diff report vendors must deliver after each release.

Tip

If you only do one thing, require a post-award monitoring plan + periodic evidence refresh—this is where most contracts fail.

Example model cards

We spoke a lot about ‘model cards’ (Mitchell et al. 2019) 👀.

Model card template

Model details

Field Value
Owner Demo Vendor Inc.
Model type Multi-class classifier (routes citizen service tickets to 6 queues)
Version 0.3 (2025-09-01)
Interface REST API, batch CSV
Training data Internal labeled tickets (2019–2024) + synthetic augmentation for rare classes
Evaluation data Stratified 2024-Q4 holdout; sandbox supports customer shadow data

Note

Assurance summary. Robustness checks; security tested against OWASP LLM Top 10; subgroup audits quarterly; change-control & rollback documented.

Intended use & users

  • Intended. Route tickets (benefits, permits, waste, roads, recreation, other) and provide triage hints to agents.
  • Users. Contact-centre agents; not for automated eligibility/enforcement decisions.
  • Decision context. Low–medium risk; human-in-the-loop mandatory.

Out-of-scope

  • Automated adverse decisions without human review
  • Audio/image inputs (text only)
  • Training on customer data without written approval

Performance summary

Metric Value
Macro-F1 0.80 (±0.01)
Top-1 accuracy 0.86
Latency (P95) 180 ms (CPU), 95 ms (GPU)
Uptime (90d) 99.9%


Subgroup F1 (2024-Q4 holdout; floors ≥ 0.78)

Subgroup F1
General 0.84
Newcomers 0.78
Rural 0.80
Seniors 0.76
Accessibility 0.79


Calibration (holdout)

ECE ≈ 0.02; isotonic calibration applied post fine-tune.


Evaluation data & methods

Area Notes
Holdout split Stratified by queue & time (last quarter); dual-review labels (κ = 0.87).
Fairness eval Subgroup metrics where available: newcomer status, region (urban/rural), age band, accessibility needs.
Robustness eval Adversarial typos; injection strings in free text; out-of-domain class names.
Security tests OWASP LLM Top-10 threat model; no tool-use plugins in production.

Limitations

  • Lower performance on short, multi-intent messages (<10 words).
  • Seasonal event names can shift priors; additional monitoring during large events.
  • Subgroup labels are self-reported/incomplete → fairness estimates have uncertainty.

Ethical considerations & mitigations

Risk Potential harm Mitigation Residual
Misrouting urgent tickets Delays Urgency gate; human review Low
Subgroup performance gaps Unequal wait times Floors + quarterly audits; targeted fine-tunes Medium
Data leakage Privacy breach Pseudonymization; logging redaction; DLP rules Low
Prompt injection Tool abuse No tools; filters; canary strings Low


Monitoring & update policy

  • Drift monitors on class priors & confidence; alert if weekly Δ > 5%.
  • Quarterly re-evaluation on a customer sample; publish deltas.
  • Major updates: 14-day notice; rollback path documented.
  • P1 incident: notify <24 h; post-mortem within 5 business days.

Versioning

Version Notes
v0.3 Added calibration; improved rural F1 0.76 → 0.80
v0.2 Expanded training data; added urgency gate
v0.1 Initial internal release

Licensing & contact

Demo content © 2025 Demo Vendor Inc. Classroom use only.

Contact: ai-procurement@example.org

Procurement evidence checklist (request these)

  • Model Card (this file) & System Card
  • Datasheets for Datasets
  • Fairness & robustness results (incl. subgroup floors)
  • Security testing report (OWASP LLM Top 10)
  • Logging scope, retention, and incident runbooks
  • Change-control policy and release notes

Quick pointers to frameworks

  • 🇨🇦 Canada (GoC) — Directive on Automated Decision‑Making (applies to systems developed or procured after Apr 2020); use the AIA for risk scoping.
  • 🏈 US (Federal) — OMB M‑24‑10 sets minimum practices and procurement terms for rights/safety‑impacting AI.
  • 🇬🇧 UK — Practical buyer playbooks: Guidelines for AI Procurement and WEF AI Procurement in a Box.

WEF “AI Procurement in a Box”

Quickstart to using the WEF toolkit

  1. Define the decision & outcomes (not the tool). Use the Workbook’s problem framing prompts.
  2. Risk scoping & stakeholders (users, impacted groups, failure modes). Capture early assumptions and harm hypotheses.
  3. Market engagement + sandbox: request demo + evaluation pack (docs + access) before RFP.
  4. RFP/RFQ with evidence asks mapped from the Guidelines:
    • Model/System Cards, Datasheets; subgroup performance; red-team & OWASP LLM Top-10 checks; logging/incident plan; usage boundaries.
  5. Scoring & testing: adapt Workbook checklists into weighted criteria; require on-your-data evaluation and change-impact testing.
  6. Post-award governance: change control, monitoring cadence, incident timelines, and exit/portability.

Common pitfalls & fixes

  • Vague outcomes → add measurable acceptance tests and subgroup floors.
  • EU-specific refs copied verbatim → keep structure, swap legal anchors for your jurisdiction.
  • Provider-only duties → add buyer/deployer duties (e.g., supply eval data, run AIA/inventory, staff human oversight).
  • No update pathway → define major-change thresholds, re-test scope, and rollback rights.

🔗 EU Resource page with overview & downloads

Due diligence question bank

Tip

Translate answers into measurable obligations and acceptance criteria. Put these into your contract/agreements.

  • Purpose & users — What decisions will the AI influence? Worst‑case error? Oversight points?
  • Data — Origins, licenses/consent, sensitive attributes handling, representativeness, de‑identification.
  • Performance — Baseline + subgroup metrics; eval datasets & protocols; drift expectations & alert thresholds.
  • Security — Attack surface (prompt injection, data exfiltration), tool integrations, red‑team results, patch cadence.
  • Governance — Roles (business owner/CAIO), change control, subprocessors, audit trail, business continuity.
  • Compliance — Where data compute lives; cross‑border transfers; applicable sectoral rules.

Activity 8.1: Ethical AI Procurement

Task (20–30 min): Draft a \(\leq\) 1‑page checklist your organization (or a fictional organization you devise) will apply before and during any AI purchase.

Include:
- Purpose & risk (use case, harm analysis, oversight)
- Evidence pack (docs at RFP)
- Security & robustness (OWASP LLM checks, red‑team, patch cadence)
- Fairness & performance (metrics & cadence)
- Contract must‑haves (data/IP, audit, SLAs, change control, incidents, exit)
- Lifecycle (monitoring schedule, owners, review gates)

✍️ Deliverable: email me (or upload to Teams) your checklist

Summary

  • Treat AI procurement as evidence‑driven and lifecycle‑bound.
    • AI is Computer Science
  • Ask for documentation you can verify and security/fairness test results.
  • Put audit, change‑control, and exit in the contract.
  • Use frameworks sparingly — as anchors, not centerpieces.

Next module: Building an ethical AI culture

References

Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. “Datasheets for Datasets.” arXiv. https://doi.org/10.48550/arXiv.1803.09010.
Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. “Model Cards for Model Reporting.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–29. https://doi.org/10.1145/3287560.3287596.