Platform Blueprint · Commercial In Confidence · Version 1.0
Proofmark
AI Decision Governance & Audit Trail Platform — complete technical and operational reference for the Proofmark platform, its measurement framework, and regulatory mapping.
Contents
01Platform Overview
07Metric Catalogue
02How It Works
08Drift Calculation (PSI)
03The Decision Record
09Regulatory Mapping
04Clients & Sectors
10Governance Dashboard
05Buyer Personas
11Platform Tiers
06Indicators & Assessments
12Roadmap
Proofmark is an AI decision governance and audit trail platform for organisations operating AI models in regulated environments. It does not replace, retrain, or interfere with existing AI systems. Instead, it sits as a lightweight integration layer alongside them — capturing a structured record of every material AI decision at the moment it is made, in an immutable, queryable, and board-reportable format.
Core Premise
Regulated entities using AI are increasingly required to prove what their models decided, why, and what happened next — for every material decision. The audit trail is not optional. The question is whether it is structured, immutable, and retrievable on demand, or reconstructed under pressure after the fact. Proofmark makes it the former.
The Problem
APRA-regulated entities — banks, insurers, and superannuation funds — are deploying AI models at scale across credit decisioning, claims triage, fraud detection, customer scoring, and underwriting. These models make thousands of material decisions per day. Regulators under CPS 230 and CPG 234, and consumer regulators under ASIC RG 271 and RG 274, now require documented evidence that:
- AI decisions are logged with sufficient context to be re-examined
- Human oversight is recorded for material outcomes
- Models are monitored for performance drift over time
- Boards receive governance reporting on AI model behaviour
- Records are tamper-evident and audit-ready
Most organisations have none of this infrastructure. Their AI models produce outputs that are logged at the application layer, if at all, in formats not designed for regulatory examination. When a regulator, auditor, or plaintiff asks "show me every decision this model made for customers with characteristic X" — they cannot answer quickly, completely, or with integrity guarantees.
The Solution
Proofmark provides a single API endpoint that any AI pipeline can call at inference time. One call per decision, taking less than 100ms, captures the complete Decision Record and writes it to an immutable, multi-tenant, RLS-secured audit store. The platform then provides a governance dashboard, drift monitoring, and board-ready reporting on top of that store — giving every role from operations to board a view appropriate to their accountability.
<100ms
Ingest latency per decision
6
Fields in every Decision Record
100%
Immutability guarantee
<60s
Time to board-ready report
Design Principles
No rip-outs. No rebuilds.
Proofmark integrates with existing AI systems via a single API call. No model changes, no pipeline redesigns. Add one function call to your inference layer.
Immutability by design
Every record is assigned a SHA-256 immutability hash computed before insert. Database triggers block any UPDATE or DELETE. Hash is re-verified on every read.
Regulator-first structure
The 6-field Decision Record is designed around APRA and ASIC requirements, not convenience. Every field has a regulatory rationale.
Multi-tenant isolation
Row-level security ensures complete data isolation between tenants. Even a bug in application code cannot expose cross-tenant data — the database enforces the boundary.
01 / INTEGRATE
AI Pipeline Calls Proofmark
After your model produces a decision, your pipeline calls POST /v1/decisions with the API key for that model. One call, <100ms, fire-and-continue.
02 / VALIDATE
Record Validated & Flagged
Proofmark validates the payload, resolves the confidence flag (NONE / WARN / REVIEW_REQUIRED), assigns a UUID, and computes the immutability hash before write.
03 / STORE
Immutable Record Written
The Decision Record is written to the append-only audit store inside a transaction. The DB trigger prevents any subsequent modification or deletion.
04 / OVERSEE
Human Review Recorded
For flagged or material decisions, a reviewer logs their action via the dashboard or API. One review write is permitted per decision (confirmed, overridden, escalated).
05 / GOVERN
Dashboard & Reports
Compliance teams, auditors, and the board access real-time governance views, drift monitoring, outcome distribution analysis, and PDF board reports.
Technical Architecture
Ingest Layer
- Endpoint: POST /v1/decisions
- Auth: API key per model (Bearer token, SHA-256 hashed at rest, prefix pm_live_)
- Payload: JSON, max 256kb, validated on receipt
- Response: decision_id, confidence_flag, review_required signal, created_at
- Latency target: <100ms p99
Storage Layer
- Database: PostgreSQL with row-level security
- Multi-tenancy: Shared database, tenant isolation via SET LOCAL app.tenant_id per transaction
- Immutability: DB trigger blocks UPDATE/DELETE on decisions table; application-layer SHA-256 hash verified on every read
- Schema: decisions, models, api_keys, users, tenants, refresh_tokens
API Layer
- Runtime: Node.js / Express
- Auth: JWT (15-minute access token) + refresh token rotation (7-day, revocable)
- Routes: /auth, /admin (model & key management), /v1/decisions (ingest, review, query, export)
- Export: CSV, up to 50,000 rows per request
Dashboard Layer
- Deployment: Static SPA (single-file, no build step)
- Views: Login, Dashboard, Decisions, Decision Detail, AI Models, Model Detail, Drift Monitor, Board Reports
- Print: Board Reports view is print-optimised (window.print() with print CSS)
API Key Model
Each AI model registered in Proofmark has one or more API keys. Keys are scoped to the model — a key for CreditAI can only log decisions attributed to CreditAI. Key format is pm_live_<64hex>. The raw key is shown once at generation and never stored — only its SHA-256 hash is retained. Key revocation takes effect immediately.
Integration Code (one call per inference)
await fetch('https://api.proofmark.com.au/v1/decisions', {
method: 'POST',
headers: { 'Authorization': 'Bearer pm_live_...', 'Content-Type': 'application/json' },
body: JSON.stringify({
model_version, decision_timestamp, inputs, raw_output,
confidence, model_decision, value, customer_id,
product, regulatory_class, final_outcome
})
});
Every AI decision logged to Proofmark produces a Decision Record — a structured, immutable document covering six mandatory governance domains. The fields are designed to satisfy the information requirements of APRA supervisors, internal auditors, legal teams, and board risk committees. The six fields form a complete chain of accountability from model input to human outcome.
| # | Field | Description | Regulatory Rationale | Required |
|---|
| 01 |
Inputs |
The exact feature set provided to the model at inference time — the data the model "saw" when it made this decision. Stored as structured JSON. Enables replay, counterfactual analysis, and fairness auditing. |
CPG 234 §31RG 271 §22 |
Required |
| 02 |
Model Output |
The raw numeric output (probability score, regression value, or classification logit) and the model's decision label (APPROVE / DECLINE / FLAG / ESCALATE / REFER / HOLD) produced by the model before any human review. |
CPS 230 §43CPG 234 §34 |
Required |
| 03 |
Confidence |
The model's confidence or probability score for its decision (0.0–1.0), the resolved confidence flag (NONE / WARN / REVIEW_REQUIRED), and the thresholds configured for this model. Used to route decisions to human review and to measure model reliability over time. |
CPG 234 §35RG 274 §22 |
Conditional |
| 04 |
Human Oversight |
The identity of the human reviewer (tokenised employee ID), the timestamp of review, and the review action taken (CONFIRMED / OVERRIDDEN / ESCALATED / REFERRED), including the reason for any override. Records the human-in-the-loop moment that regulators require. |
CPS 230 §44RG 271 §29 |
Required for material |
| 05 |
Final Outcome |
The outcome that was actually applied — which may differ from the model's decision if a human overrode it. The outcome_differs flag is set to true when a human changed the model's recommendation. This field captures the effective regulatory consequence. |
CPS 230 §43RG 271 §22 |
Required |
| 06 |
Audit Metadata |
System-generated fields: UUID (generated in application before insert), tenant_id, model_id, model_version, regulatory_class (MATERIAL / NON_MATERIAL), business_unit, created_at timestamp, and the SHA-256 immutability hash of the complete record. Enables querying, filtering, integrity verification, and regulatory classification. |
CPS 230 §43ASIC RG 271 |
System-generated |
Immutability Hash
The immutability hash is a SHA-256 digest of the canonical JSON of all core Decision Record fields, computed in the application layer before the database INSERT. The UUID is included in the hash computation, meaning it must be generated in the application (not the database) so that it is known before the hash is calculated. The hash is stored alongside the record and re-computed and compared on every GET /v1/decisions/:id request. Any discrepancy between stored and recomputed hash indicates post-write tampering.
Two-Layer Immutability Guarantee
Layer 1 — Database: A PostgreSQL trigger fires BEFORE UPDATE and BEFORE DELETE on the decisions table and raises an exception, blocking any modification. A single exception is permitted: setting reviewed = true for the one human oversight write. All other fields are permanently locked after INSERT.
Layer 2 — Application: The SHA-256 hash is recomputed on every read and compared to the stored value. A mismatch surfaces immediately in the API response (integrity.hash_valid: false) and is logged as a critical integrity event, regardless of how the tampering occurred (DB-level direct edit, migration error, storage corruption).
Proofmark is designed for APRA-regulated entities operating AI models in material decision-making contexts. The primary regulatory obligations arise from APRA's CPS 230 (Operational Risk Management) and CPG 234 (Information Security), and ASIC's RG 271 (Internal Dispute Resolution) and RG 274 (Credit Licensees). The secondary driver is director liability under the Corporations Act for failures in AI governance oversight.
Target Sectors
| Sector | Regulated by | Primary AI Use Cases | Key Obligations |
|---|
| Authorised Deposit-taking Institutions (ADIs) |
APRA |
Credit decisioning, fraud detection, anti-money laundering, customer scoring |
CPS 230CPG 234RG 271 |
| General Insurers |
APRA |
Claims triage, fraud detection, underwriting automation, risk scoring |
CPS 230CPG 234RG 274 |
| Life Insurers |
APRA |
Underwriting decisioning, claims assessment, disability classification |
CPS 230CPG 234 |
| Superannuation Funds |
APRA |
Member risk classification, hardship assessment, advice routing |
CPS 230SPS 530 |
| Credit Licensees (non-ADI) |
ASIC |
Consumer credit decisioning, affordability assessment, hardship triage |
RG 271RG 274NCCP |
Use Cases
Credit Decisioning
Home loans, personal loans, business credit — AI models producing APPROVE / DECLINE outputs on material applications. Proofmark captures input features (income, LVR, credit score, employment), raw probability, confidence, and the human reviewer's confirmation or override. Required for RG 271 complaint response and APRA supervisory review.
Claims Triage
General and life insurers using AI to route claims to fast-track approval, investigation, or manual assessment. Proofmark logs the classification, the features the model used, and whether an assessor confirmed or escalated. Critical for RG 274 fairness monitoring and APRA operational risk reporting.
Fraud Detection
Real-time transaction fraud scoring. Proofmark records the flagged transaction, model confidence, and the analyst's review action. Provides the audit trail needed when a false positive triggers a customer complaint or when a missed fraud event requires post-incident examination.
Customer & Risk Scoring
Portfolio-level models assigning risk tiers, credit limits, or product eligibility. Proofmark captures scoring outputs over time, enabling longitudinal fairness analysis, outcome distribution monitoring (RG 274), and regulatory cohort comparisons.
Economic Buyer
Chief Risk Officer
Owns the APRA relationship. Needs to demonstrate to supervisors that AI model governance is in place — documented, auditable, and board-reported. Primary fear: regulatory finding or enforcement action citing inadequate AI oversight. Values: APRA compliance posture dashboard, drift alerts, board report export.
Technical Buyer
Chief Data / AI Officer
Accountable for AI model performance and responsible deployment. Needs to monitor models in production for drift and outcome anomalies without building custom tooling. Values: per-model drift monitoring, outcome distribution analysis, confidence flag trends, API-first integration that doesn't touch model code.
Governance User
Head of AI Governance / Risk
Day-to-day owner of AI risk policy and human-in-the-loop processes. Manages reviewer workflows, monitors unreviewed decisions, and prepares board packs. Values: decisions dashboard, review rate metrics, unreviewed material decision alerts, one-click board report generation.
Assurance User
Head of Internal Audit
Needs to independently verify that AI governance controls operated as described. Requires read-only access to full decision records, integrity verification, and exportable audit evidence. Values: CSV export, integrity hash verification, immutable record guarantee, scoped read-only credentials.
Operations User
Compliance / Model Risk Analyst
Runs the day-to-day review queue. Records human oversight decisions, manages reviewer workflows, monitors confidence flag rates, and escalates anomalies. Values: filtered decisions view, single-decision detail with all 6 fields, review action capture, CSV export for ad hoc analysis.
Technical User
ML Engineer / Data Engineer
Implements and maintains the API integration. Manages API keys, monitors ingest rates, and diagnoses integration issues. Values: simple REST API, clear payload schema, immediate feedback on confidence flags and review_required signals, API key management portal.
Proofmark resolves a set of categorical indicators at multiple levels — per-decision, per-model, and per-period. These indicators drive routing, alerting, dashboard display, and board reporting. They are not metrics (which are calculated, continuous measures) — they are categorical assessments that classify a decision or model into a governance state.
Decision-Level Indicators
| Indicator | Values | Description | When Resolved |
|---|
| Confidence Flag |
NONE
WARN
REVIEW_REQUIRED
|
Indicates whether the model's confidence score falls within acceptable bounds. WARN means the score is below the warn threshold. REVIEW_REQUIRED means it falls below the mandatory review threshold — human oversight must be recorded. |
On ingest (POST /v1/decisions) |
| Materiality Class |
MATERIAL
NON_MATERIAL
|
Set by the calling system at ingest (or defaulting to MATERIAL). Indicates whether this decision crosses the materiality threshold defined for this model — typically based on dollar value. MATERIAL decisions are subject to mandatory human review obligations. |
On ingest (caller-supplied or default) |
| Model Decision |
APPROVE
DECLINE
FLAG
ESCALATE
REFER
HOLD
|
The label produced by the AI model. This is what the model recommended before any human review. Immutable after logging. |
On ingest (caller-supplied) |
| Review Status |
REVIEWED
AUTO
UNREVIEWED
|
Whether a human review has been recorded. REVIEWED: review action logged. AUTO: no review required (low value, high confidence, non-material). UNREVIEWED: review expected but not yet recorded — appears in the unreviewed queue and APRA readiness report. |
Derived from reviewed field + materiality + confidence flag |
| Review Action |
CONFIRMED
OVERRIDDEN
ESCALATED
REFERRED
|
The action taken by the human reviewer. CONFIRMED: reviewer agreed with the model. OVERRIDDEN: reviewer changed the outcome (sets outcome_differs = true). ESCALATED: referred to a higher authority within the entity. REFERRED: sent to external party (e.g. dispute body). |
On review write (POST /v1/decisions/:id/review) |
| Outcome Differs |
YES
NO
|
Boolean flag set to true when a human reviewer changed the model's decision (review_action = OVERRIDDEN). Drives override rate calculation and fairness analysis. The final_outcome field reflects the human's decision, not the model's, when this is true. |
On review write when action = OVERRIDDEN |
| Integrity Status |
HASH VERIFIED
INTEGRITY FAIL
|
Result of re-computing the SHA-256 immutability hash on read and comparing to the stored value. HASH VERIFIED means the record is intact. INTEGRITY FAIL indicates post-write tampering, storage corruption, or a migration error and triggers an immediate security alert. |
On every GET /v1/decisions/:id read |
Model-Level Indicators
| Indicator | Values | Description |
|---|
| Drift Status |
OK
NEAR LIMIT
BREACH
|
Governance assessment of the model's current PSI drift score against its configured threshold. OK: PSI ≤ 85% of threshold. NEAR LIMIT: 85–100% of threshold (monitoring required). BREACH: PSI exceeds threshold — governance action required per CPG 234 §35. |
| Drift Trend |
STABLE
RISING
ACCELERATING
|
Direction and rate of change of the PSI score over the trailing 4 weeks. RISING: PSI increased in 3 of the last 4 weeks. ACCELERATING: week-over-week delta is growing. Used for early warning before a threshold breach occurs. |
| Model Status |
ACTIVE
ARCHIVED
|
Whether the model is currently active (accepting decisions) or archived (historical records only, API keys revoked). An ARCHIVED model's decisions remain in the audit trail permanently. |
Proofmark computes 16 metrics across four domains: Volume, Oversight, Confidence, and Model Health. Each metric is defined below with its full descriptor, calculation, threshold levels, and regulatory mapping. Metrics are computed over configurable periods (day, week, month, quarter) and per model, per business unit, or across all models.
Description
Total number of AI decisions logged to Proofmark within a defined period, for a given model or across all models. The primary volume indicator. Used to verify completeness of logging (all model inferences should produce a Decision Record).
Formula
COUNT(decisions) WHERE period AND [model_id = x]
Unit
Integer count (absolute)
Thresholds
No inherent threshold. Monitored for unexpected drops (>20% week-over-week decrease suggests integration failure or missed logging) or unexpected spikes.
Regulatory Mapping
CPS 230 §43 Completeness of AI decision audit trail
Dashboard Location
Dashboard KPI (Total Decisions Logged), Model Detail (per-model count), Board Report (model summary table)
Description
The proportion of all logged decisions that have a human review action recorded. The broadest measure of human oversight coverage across all AI decisions regardless of materiality or confidence flag.
Formula
review_rate = (COUNT decisions WHERE reviewed = true) / COUNT(decisions) × 100
Thresholds
OK ≥ 90% WARN 75–90% CRITICAL < 75% — Note: not all decisions require review; a review rate of 70% may be acceptable if the remaining 30% are non-material low-confidence decisions. Context is required.
Regulatory Mapping
CPS 230 §44 Human oversight of material AI decisions
Dashboard Location
Dashboard KPI (Human Review Rate), APRA Readiness panel, Board Report executive summary
Description
The proportion of MATERIAL decisions (those above the model's materiality threshold) that have a human review action recorded. This is the highest-priority oversight metric — regulators are primarily concerned with unreviewed decisions that affected customers in material ways.
Formula
hc_review_rate = (COUNT decisions WHERE reviewed = true AND regulatory_class = 'MATERIAL')
/ COUNT(decisions WHERE regulatory_class = 'MATERIAL') × 100
Thresholds
TARGET ≥ 95% WARN 85–95% CRITICAL < 85% — Any unreviewed MATERIAL decision is a governance exception requiring explanation in board reporting.
Regulatory Mapping
CPS 230 §44RG 271 §29 Human oversight specifically for material/consequential decisions
Dashboard Location
Dashboard KPI (High-Consequence Review), APRA Readiness panel, Board Report Key Findings
Description
The proportion of reviewed decisions where a human reviewer changed the model's recommended outcome (review_action = OVERRIDDEN and outcome_differs = true). A high override rate may indicate model underperformance, threshold misconfiguration, or systematic reviewer bias. A very low rate may indicate rubber-stamping of AI decisions without genuine review.
Formula
override_rate = (COUNT decisions WHERE outcome_differs = true AND reviewed = true)
/ COUNT(decisions WHERE reviewed = true) × 100
Thresholds
OK 1–10% INVESTIGATE >10% or <0.5% — Both extremes warrant investigation. Policy threshold set per model in model configuration.
Regulatory Mapping
CPG 234 §34RG 271 §22 Model performance monitoring; human override documentation
Dashboard Location
Board Report executive summary, Model Detail (per-model), Decisions table (outcome_differs column)
Description
The elapsed time between a decision being logged and a human review action being recorded. Measures the operational responsiveness of the human oversight process. High TTR for REVIEW_REQUIRED decisions indicates operational risk — decisions may be applied before review occurs. Applies only to decisions where reviewed = true.
Formula
TTR = AVG(reviewed_at - decision_timestamp)
for all decisions WHERE reviewed = true AND period
TTR_material = AVG(reviewed_at - decision_timestamp)
for all decisions WHERE reviewed = true AND regulatory_class = 'MATERIAL'
Unit
Duration (hours, reported as mean and P95)
Thresholds
Policy-defined per model. Common targets: REVIEW_REQUIRED decisions <4 hours; MATERIAL decisions <24 hours; all other reviewed <72 hours.
Regulatory Mapping
CPS 230 §44 Timeliness of human oversight for material decisions
Dashboard Location
Board Report model summary table (Phase 2)
Description
Statistical distribution of model confidence scores across all decisions in a period. Provides insight into model certainty, calibration, and whether score distributions are shifting over time. Computed as mean, median, standard deviation, and percentile bands. A model whose confidence distribution shifts materially between periods may be experiencing drift even if output labels haven't changed.
Formula
confidence_mean = AVG(confidence) WHERE period AND model_id
confidence_median = PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY confidence)
confidence_p10 = PERCENTILE_CONT(0.1) WITHIN GROUP (ORDER BY confidence)
confidence_p90 = PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY confidence)
Unit
Float 0.0–1.0 (mean, median, P10, P90)
Thresholds
No universal threshold — compared to model baseline. A >5 percentage point shift in mean confidence over 4 weeks is flagged for investigation.
Regulatory Mapping
CPG 234 §35 Model performance monitoring
Dashboard Location
Drift Monitor — confidence distribution panel (Phase 3 full implementation)
Description
The proportion of decisions receiving each confidence flag (NONE / WARN / REVIEW_REQUIRED) in a period. A rising REVIEW_REQUIRED rate indicates the model is becoming less certain — potentially drifting. A sudden spike in WARN or REVIEW_REQUIRED may indicate a data quality issue at the ingestion point.
Formula
flag_rate(flag) = COUNT(decisions WHERE confidence_flag = flag AND period)
/ COUNT(decisions WHERE period) × 100
Unit
Percentage (0–100%) per flag type
Thresholds
REVIEW_REQUIRED rate: OK <5% WARN 5–15% CRITICAL >15% — configured per model based on historical baseline.
Regulatory Mapping
CPG 234 §35 Model confidence and reliability monitoring
Dashboard Location
Decisions table (flag column), Dashboard KPI (flagged decisions count)
Description
Population Stability Index — the primary measure of model input drift. Quantifies how much the distribution of a model's input features (or output scores) has changed relative to a baseline period (typically the month the model was registered or last validated). A model whose inputs shift significantly may be operating outside the conditions under which it was validated, making its outputs unreliable. PSI is computed per model per week and tracked in the Drift Monitor view.
Formula
See Section 08 for full PSI calculation with worked example.
Unit
Float ≥ 0 (higher = more drift)
Thresholds
STABLE PSI < 0.10 — No significant change. Model operating within validated conditions.
MONITOR 0.10 ≤ PSI < 0.20 — Some shift detected. Increase monitoring frequency.
NEAR LIMIT 0.20 ≤ PSI < threshold — Significant shift. Investigate root cause.
BREACH PSI ≥ configured threshold (default 0.30) — Governance action required. Report to risk committee.
Regulatory Mapping
CPG 234 §35 Ongoing model performance monitoring and drift response
Dashboard Location
Drift Monitor (sparkline charts, alert table, KPI cards), Board Report model summary table
Description
The rate of change of the PSI score over trailing weeks. Identifies models where drift is accelerating before a threshold breach occurs, enabling proactive governance action. A model with PSI 0.18 that increased by 0.04 last week is more concerning than one with PSI 0.22 that has been stable for 6 weeks.
Formula
drift_delta(w) = PSI(week_w) - PSI(week_w-1)
drift_accel = drift_delta(w) - drift_delta(w-1)
rising_signal = COUNT(weeks WHERE drift_delta > 0) ≥ 3 in trailing 4 weeks
Unit
Float (signed delta per week); Trend indicator: STABLE / RISING / ACCELERATING
Thresholds
RISING drift_delta > 0.01 for 3 of last 4 weeks ACCELERATING drift_accel > 0 for 2 consecutive weeks
Regulatory Mapping
CPG 234 §35 Early warning for model degradation requiring proactive response
Dashboard Location
Drift Monitor (sparkline trend indicator, TREND ↑ alert badge)
Description
The proportion of decisions producing each outcome label (APPROVE / DECLINE / FLAG / ESCALATE / REFER / HOLD) in a period, for a given model. Monitors for unexpected shifts in approval/decline rates, which may indicate model drift, population shift, or systematic bias. Also used for fairness auditing by cohort (e.g. approval rate by postcode, age band, or product type).
Formula
outcome_pct(label) = COUNT(decisions WHERE model_decision = label AND period AND model)
/ COUNT(decisions WHERE period AND model) × 100
Unit
Percentage (0–100%) per outcome label; displayed as stacked bar chart
Thresholds
Compared to baseline period. A shift of >10 percentage points in any single outcome label within a quarter triggers an Outcome Distribution Shift alert. Absolute thresholds are model-specific (e.g. a fraud model should have a very different distribution from a credit model).
Regulatory Mapping
RG 274 §22 Monitoring for discriminatory or anomalous AI decision patterns
Dashboard Location
Drift Monitor (outcome distribution bars per model)
Description
The magnitude of change in outcome distribution between two periods (typically current quarter vs prior quarter or vs registration baseline). Quantifies whether the model is approving/declining significantly more or fewer decisions than it did historically. Complements PSI (which measures input drift) with a measure of output drift.
Formula
shift(label) = |outcome_pct(label, current) - outcome_pct(label, baseline)|
total_shift = SUM(shift(label)) / COUNT(labels)
Unit
Percentage points (absolute shift per label and mean shift across all labels)
Thresholds
OK shift(any label) < 5pp WARN 5–10pp ALERT >10pp on any label
Regulatory Mapping
RG 274 §22CPG 234 §35 Detection of systematic model output changes requiring governance review
Dashboard Location
Drift Monitor (outcome distribution comparison — baseline vs current)
Description
The proportion of all decisions classified as MATERIAL (above the model's configured materiality threshold). Confirms that the entity's materiality classification logic is operating as expected and allows tracking of changes in the mix of material vs non-material decisions over time.
Formula
materiality_rate = COUNT(decisions WHERE regulatory_class = 'MATERIAL' AND period)
/ COUNT(decisions WHERE period) × 100
Thresholds
No universal threshold — compared to historical baseline per model. A sudden shift in materiality rate (e.g. dropping from 80% to 20%) likely indicates a change in the calling system's classification logic, not a genuine change in business activity.
Regulatory Mapping
CPS 230 §43 Correct classification of material decisions for reporting purposes
Dashboard Location
Decisions table (regulatory_class filter), Board Report (implied in oversight metrics)
Description
Identifies systematic patterns in human override behaviour — specifically, whether a particular reviewer, model, outcome type, or cohort shows a statistically anomalous override rate relative to peers. A reviewer with a 40% override rate while peers average 3% may indicate rubber-stamp reversal (gaming the governance process) or genuine systematic model failure in a specific segment. Used in Internal Audit and fairness investigations.
Formula
OPI(reviewer) = override_rate(reviewer) / AVG(override_rate, all reviewers)
OPI(model, outcome) = override_rate(model, model_decision=X)
/ override_rate(model, all outcomes)
Unit
Ratio (1.0 = average; >2.0 = high anomaly; <0.2 = suspiciously low)
Thresholds
INVESTIGATE OPI > 2.0 or OPI < 0.2 for any reviewer over a rolling 4-week window
Regulatory Mapping
CPS 230 §44RG 271 Quality of human oversight; fairness monitoring
Dashboard Location
Internal Audit export (CSV), Board Report Key Findings (when anomaly detected)
Description
An estimate of the proportion of actual AI inferences that produced a Proofmark Decision Record. 100% completeness means every model inference is logged. Any shortfall indicates missed logging — decisions that occurred but are not in the audit trail. Computed by comparing logged decision count against the model's expected daily volume (configured at model registration).
Formula
completeness = COUNT(decisions logged, period) / expected_decisions(model, period) × 100
Thresholds
OK ≥ 98% WARN 90–98% CRITICAL < 90% — a critical alert fires and is surfaced in the APRA Readiness panel
Regulatory Mapping
CPS 230 §43 Completeness and reliability of the AI decision audit trail
Dashboard Location
APRA Readiness panel ("Audit trail completeness"), Board Report compliance posture
Description
The proportion of Decision Records whose SHA-256 immutability hash matches the hash stored at write time. Should always be 100%. Any value below 100% indicates either storage corruption, database tampering, a botched migration, or an application bug. Each failing record is logged individually as an integrity event.
Formula
integrity_score = COUNT(decisions WHERE recomputed_hash = stored_hash)
/ COUNT(decisions) × 100
Unit
Percentage (0–100%); should always be 100%
Thresholds
VERIFIED 100% CRITICAL ALERT Any value < 100% — triggers immediate security incident response
Regulatory Mapping
ASIC RG 271CPS 230 §43 Record integrity and tamper-evidence for regulatory admissibility
Dashboard Location
Decision Detail ("Hash Verified" badge), APRA Readiness panel, Board Report compliance posture
Description
A composite governance health indicator computed from the seven most critical regulatory compliance dimensions. Not a single number — displayed as a checklist of pass/warn/fail ratings across the seven dimensions, suitable for board reporting and regulator briefing. Each dimension maps to a specific APRA/ASIC obligation.
Components
1. Audit trail completeness (M14 ≥ 98%) — CPS 230 §43
2. Review rate — all decisions (M02 ≥ 90%) — CPS 230 §44
3. High-consequence review rate (M03 ≥ 95%) — CPS 230 §44 / RG 271
4. Model drift — all models within tolerance (M08: no breaches) — CPG 234 §35
5. Outcome distribution anomalies (M11: no alerts) — RG 274 §22
6. Immutability integrity (M15 = 100%) — ASIC RG 271
7. Board reporting — generated this period (report produced within 90 days)
Unit
Pass / Warn / Fail per dimension (not a weighted score — each dimension is binary for regulatory purposes)
Regulatory Mapping
CPS 230CPG 234RG 271RG 274 Composite across all primary obligations
Dashboard Location
Dashboard (APRA Readiness panel), Board Report (compliance posture checklist)
The Population Stability Index (PSI) is the industry-standard metric for measuring input feature drift in deployed machine learning models. It quantifies how much the distribution of a variable has changed between a baseline (training or registration) period and a current monitoring period. Proofmark computes PSI weekly for each registered model using the output confidence score as the primary PSI variable (output PSI), and optionally accepts input-level PSI computed by the calling system.
Methodology
PSI requires two distributions of the same variable: the baseline (expected) and the current (actual). Both distributions are discretised into bins (typically 10 equal-width or equal-frequency bins). For each bin, the proportion of observations in the baseline and current distributions is compared using a symmetric KL-divergence formula.
Worked Example
CreditAI v2.3 was registered with a baseline month of October 2023. The output confidence score distribution over that month is the "Expected" distribution. PSI is computed weekly against this baseline.
| Bin | Score Range | Expected % (Baseline) | Actual % (Current) | Actual − Expected | ln(Actual / Expected) | Contribution |
|---|
| 1 | 0.00 – 0.10 | 1.2% | 1.8% | +0.6% | 0.405 | 0.0024 |
| 2 | 0.10 – 0.20 | 2.1% | 2.4% | +0.3% | 0.134 | 0.0004 |
| 3 | 0.20 – 0.30 | 3.8% | 4.1% | +0.3% | 0.077 | 0.0002 |
| 4 | 0.30 – 0.40 | 5.2% | 5.8% | +0.6% | 0.110 | 0.0007 |
| 5 | 0.40 – 0.50 | 8.4% | 9.0% | +0.6% | 0.069 | 0.0004 |
| 6 | 0.50 – 0.60 | 12.1% | 11.6% | -0.5% | -0.042 | 0.0002 |
| 7 | 0.60 – 0.70 | 18.4% | 17.2% | -1.2% | -0.067 | 0.0008 |
| 8 | 0.70 – 0.80 | 22.3% | 20.8% | -1.5% | -0.070 | 0.0011 |
| 9 | 0.80 – 0.90 | 18.9% | 18.1% | -0.8% | -0.043 | 0.0003 |
| 10 | 0.90 – 1.00 | 7.6% | 9.2% | +1.6% | 0.191 | 0.0031 |
| PSI Total | 0.0096 |
Result: PSI = 0.0096 → STABLE
CreditAI v2.3's output confidence distribution is highly stable against the October 2023 baseline. No governance action required.
PSI Interpretation Scale
| PSI Range | Classification | Proofmark Status | Governance Action |
|---|
| PSI < 0.10 | No significant change | STABLE | No action required. Continue standard monitoring. |
| 0.10 ≤ PSI < 0.20 | Some shift detected | MONITOR | Investigate cause. Increase review frequency. Document in model risk log. |
| 0.20 ≤ PSI < 0.25 | Significant shift | NEAR LIMIT | Root cause analysis required. Consider model review. Notify model risk owner. |
| 0.25 ≤ PSI < threshold | Major shift | NEAR LIMIT | Escalate to CRO. Consider suspending automated decisions pending model review. |
| PSI ≥ threshold (default 0.30) | Population has changed materially | BREACH | Mandatory governance response. Notify risk committee. Suspend or retrain. Document APRA response. |
Configurable Threshold
The PSI breach threshold is configurable per model (default 0.30). A fraud detection model operating in a high-velocity, adversarially-driven environment may tolerate a higher threshold (0.40) than a stable home loan credit model (0.20). Threshold changes require approval through the entity's model risk management process and are documented in Proofmark's model configuration audit trail.
Primary Regulatory Obligations
| Obligation | Standard | Requirement | Proofmark Feature | Metric |
|---|
| AI decision logging |
CPS 230 §43 |
Entities must maintain records of material decisions made by or with AI assistance, sufficient for post-incident review |
Decision Record (6 fields), append-only audit store, CSV export |
M01, M14 |
| Human oversight |
CPS 230 §44 |
Material AI-assisted decisions must have a documented human oversight mechanism with recorded outcomes |
Review action logging (Field 4), review queue, REVIEW_REQUIRED flag routing |
M02, M03, M05 |
| Board reporting |
CPS 230 §47 |
The Board must receive regular reporting on AI model governance, including risk indicators and oversight metrics |
Board Reports view, director attestation, print-to-PDF |
M16 |
| Model risk management |
CPG 234 §31–34 |
Models must be inventoried, governed through a defined lifecycle, and subject to ongoing performance monitoring |
Model registration, version tracking, API key management, model archive |
M01, M06, M07 |
| Drift monitoring |
CPG 234 §35 |
Entities must monitor deployed models for performance degradation and population shift, with defined response thresholds |
Drift Monitor view, PSI calculation, sparkline trend, drift alert table |
M08, M09 |
| Internal dispute resolution |
ASIC RG 271 §22 |
Financial services licensees must be able to retrieve and explain AI-assisted decisions in response to customer complaints |
Decision Detail view (all 6 fields), decision ID, customer_id lookup, full input replay |
M01, M02 |
| Credit decision records |
ASIC RG 274 |
Credit licensees must document the basis for automated credit decisions and monitor for fairness and distribution anomalies |
Inputs field (full feature capture), outcome distribution monitoring, override tracking |
M10, M11, M04 |
| Record integrity |
ASIC RG 271CPS 230 §43 |
Decision records must be tamper-evident and admissible as evidence |
SHA-256 immutability hash, DB trigger immutability, integrity verification on read |
M15 |
| View | Audience | What It Shows | Metrics Displayed |
|---|
| Dashboard |
All roles |
Total decisions (M01), human review rate (M02), high-consequence review rate (M03), flagged decisions count (M07), recent 8 decisions table, APRA Readiness panel (M16), model status strip with drift scores (M08) |
M01, M02, M03, M07, M08, M16 |
| Decisions |
Compliance Analyst, Internal Audit |
Paginated, filterable audit log. Filters: model, outcome, review status, confidence flag, date range, customer ID. CSV export. Each row shows decision ID, model, outcome badge, value, confidence, review status, timestamp. |
M01, M02, M07, M12 |
| Decision Detail |
Compliance Analyst, Legal, Auditor |
Full 6-field Decision Record for a single decision. Inputs (JSON), model output (raw score + label), confidence (score + flag + thresholds), human oversight (reviewer + action + override), final outcome (applied decision + override diff), audit metadata (ID + hash + regulatory class). Integrity verification badge. |
M15 (per record) |
| AI Models |
Head of AI Governance, ML Engineers |
Model registry: all models with current drift score vs threshold, decision count, last active timestamp, API key count, status. Register new model modal. Per-model drift visual. |
M01, M08 |
| Model Detail |
ML Engineer, Model Risk |
Model configuration (thresholds, use case, materiality), API key management (generate, revoke, prefix display), integration code snippet, decision volume and last active. |
M01 |
| Drift Monitor |
Head of AI Governance, CRO, CDO |
PSI KPI cards (breach / near limit / OK / active alerts), per-model 7-week sparkline trend charts (M08, M09), outcome distribution stacked bars (M10), drift alert table with governance actions required. |
M08, M09, M10, M11 |
| Board Reports |
CRO, Board Risk Committee, External Auditor |
Executive governance summary, key findings (colour-coded by severity), APRA/ASIC compliance posture checklist (M16 components), model performance summary table (M01, M02, M04, M08), director attestation section with signature lines. Print-to-PDF export. |
M01, M02, M03, M04, M08, M15, M16 |
Starter
Observe
1–2 models · up to 50K decisions/month
- Decision ingest API (POST /v1/decisions)
- Full 6-field Decision Record storage
- Decisions dashboard and detail view
- Human oversight logging (Field 4)
- CSV export
- 2 user seats
Most Common
Govern
Up to 5 models · up to 500K decisions/month
- Everything in Observe
- Drift Monitor (M08, M09, M10, M11)
- Board Reports (one-click PDF)
- APRA Readiness panel (M16)
- Confidence flag routing
- 10 user seats
Enterprise
Assure
Unlimited models · unlimited decisions
- Everything in Govern
- External auditor portal (read-only)
- Override Pattern Index (M13)
- Fairness cohort analysis
- Private cloud / on-prem option
- Custom reporting periods
- Unlimited seats · dedicated support
Pricing basis: Per-model per-month fee + per-decision volume charge above tier limits. Pricing anchored to regulatory risk exposure — organisations with higher-value materiality thresholds (e.g. >$100K per decision) pay a premium rate reflecting the compliance value delivered. Volume pricing available for >5M decisions/month.
- Decision ingest API (POST /v1/decisions)
- Human review API (POST /:id/review)
- Multi-tenant PostgreSQL with RLS
- JWT + API key authentication
- 6-field Decision Record + immutability hash
- Admin portal (model + key management)
- Decisions dashboard + detail view
- AI Models registry view
- CSV export
- Drift Monitor (PSI + outcome distribution)
- Board Reports (print-ready PDF)
- APRA Readiness panel
- Register proofmark.com.au
- Deploy concept website
- Deploy API to Hetzner VPS
- Run migration 001 on live PostgreSQL
- Wire dashboard to live API (replace mock data)
- SSL / HTTPS configuration
- Environment management (.env.production)
- Land first design partner
- Pilot with real model and real decisions
- Pricing model finalised
FUTURE
Phase 3 — Enterprise
- External auditor portal (scoped read-only)
- Override Pattern Index (M13) automation
- Fairness cohort analysis (by demographic / product)
- Alerting — email / Slack on drift breach
- Custom report periods (financial year, regulatory cycle)
- Private cloud / on-premises deployment
- APRA supervisory report export format
- SDK / library wrappers (Python, Java)
- Confidence score calibration monitoring
Gate: Design Partner Required Before Phase 2 Deployment
Do not deploy and market Proofmark without first securing a design partner — an APRA-regulated entity (bank, insurer, super fund) willing to pilot the platform with real AI decisions in exchange for roadmap influence and a preferential commercial arrangement. The white paper is the door-opener. The design partner conversation validates pricing, confirms which regulatory obligations are the most acute, and ensures Phase 2 is built for real operational conditions, not assumed ones.
Document Details
Document
Proofmark Platform Blueprint
Version
1.0 — Initial Release
Classification
Commercial In Confidence