DEMO / 05 · Demo ready
Model monitoring lab
Drift, calibration and data-quality monitoring for prediction streams with a documented alert state machine and delayed-label handling.
What decision is supported
Whether a deployed scoring model can still be trusted this week: has the input distribution moved, are the scores still calibrated where labels have arrived, and should the alert state change.
Who uses it
A model owner responsible for a scoring model in a fictional lending or fraud setting, and an analyst who needs an alert to carry the evidence that raised it.
What data enters
A stream of prediction records with features, scores and labels that arrive after a delay, plus a reference window from training. The demo uses synthetic streams only.
What is computed
Population stability and per-feature drift statistics against the reference window, data-quality checks (missingness, range, type), calibration on the labelled subset only, and an alert state machine with stated transitions (ok, watch, alert, acknowledged) and the evidence attached to each transition.
What action is suggested
Keep, watch, or escalate the model, with the windows and statistics that drove the state shown next to the recommendation.
What evidence supports it
Drift statistics shown in the model explorer are computed in the browser from the synthetic demo stream and labelled demo.
No metric is claimed for this project. Anything computed by the repository on its synthetic fixture carries the label demo.
What fails or is uncertain
- Thresholds for the alert state machine are configuration, not learned, and are documented as such.
- Calibration is only measured where labels exist, so a recent window can look fine simply because its labels have not arrived.
- Drift statistics detect change, not harm; a drifted feature may not affect decisions.