On this page
Overfitting means a model has learned sample-specific noise, so its training result looks strong while its performance on genuinely unseen data deteriorates. Underfitting means the model is too restricted to capture a useful relationship, so it performs poorly even on the data used to fit it. For an FRM® Part I candidate, compare training and validation behaviour before deciding which remedy fits the evidence.
Why this matters in risk models
A risk model is meant to help interpret new cases, not only reproduce records already in a database. In credit risk, for example, a classifier may separate observations in its development sample beautifully because it has discovered accidental quirks rather than a stable relationship. A new borrower population or a later economic period may not contain those quirks. Conversely, a very rigid model may miss a genuinely curved relationship between a driver and a risk outcome. Both failures appear as weak new-data performance, but they call for different responses.
GARP discusses the need to split samples when checking the reproducibility of a model and highlights overfitting in machine learning. Its discussion of risk managers applying artificial intelligence also identifies regularisation and cross-validation as ways to balance fit against predictive performance. These sources establish why the topic belongs in risk practice. The exam task is more specific: read the evidence in the train and validation results, identify the likely fit problem, and choose a repair that targets its cause.
Start with the comparison, not the model label
Training error measures performance on examples that influenced the fitted parameters. Validation error measures performance on a separate set used to compare specifications. A model can always be made more flexible enough to reduce its training error, but that alone says little about generalisation. If training error is small and validation error much larger, the model has a generalisation gap consistent with overfitting. If both are large and similar, the model is failing even where it was trained, which is consistent with underfitting.
Neither pattern proves the whole diagnosis in isolation. A validation set with a different distribution may expose a changed environment; a leaked feature may make both results look deceptively strong. The candidate should read the setup: were data split before feature selection, were observations independent, and was the error measure appropriate? A useful comparison holds the evaluation method fixed across models. Otherwise, differences may reflect a changed metric or dataset rather than a change in model complexity.
The gap is also not itself a score. A small gap between two poor results does not make a model good; it says the model is consistently weak. A large gap between two excellent results may still warrant scrutiny if the absolute validation result meets the decision need, but it remains evidence that the fitted model is sensitive to its training sample. The decision depends on what the model is for, how costly mistakes are, and whether the validation data resemble the future cases the model will face.
Bias, variance and the complexity choice
Bias describes systematic error from restrictions that stop a model representing the relationship of interest. A straight line cannot express a pronounced curve without additional terms or a different functional form. Variance describes how much fitted predictions change when the training sample changes. A highly flexible model may trace minor fluctuations in one sample, so a replacement sample produces substantially different fitted predictions. Underfitting is commonly associated with high bias; overfitting is commonly associated with high variance. These are tendencies in the exam framework, not measurements obtained merely by looking at one score.
As complexity rises, in-sample fit usually improves because a more flexible specification can mimic simpler ones and can exploit extra detail. Validation error may initially fall as the model captures a real pattern, then rise once added flexibility chiefly follows noise. That is the bias–variance trade-off: too little flexibility misses signal, while too much makes fitted conclusions unstable. There is no magic complexity setting independent of the amount and quality of data. The appropriate choice must be judged on held-out information and the intended risk decision.
An illustrative comparison with round numbers
Suppose a risk team predicts a loss amount, measured in invented units, using the same training and validation samples for three candidate specifications. A very simple model has a training mean absolute error of 20 units and a validation error of 21. A medium model has training error of 11 and validation error of 12. A highly flexible model has training error of 2 and validation error of 19. These numbers are entirely illustrative and show error, so lower is better. No real bank or historical dataset is represented.
The simple model is likely underfitting because it makes sizeable mistakes even on data used to estimate it, and the held-out result is similarly poor. The medium model improves on both samples and has only a small gap. The flexible model wins the training contest decisively but loses much of that advantage on validation: its error rises from 2 to 19, a gap of 17 units. It is the overfitting candidate. Selecting it because 2 is the smallest displayed number would mistake training fit for predictive usefulness.
A sensible next experiment for the simple model is to add a relevant feature or allow a limited nonlinear relationship, then compare validation error again. For the highly flexible model, the team could reduce complexity, impose regularisation, or obtain more representative training observations. It should preserve an untouched evaluation set for the final assessment rather than repeatedly tuning to the same validation observations. These are proposed diagnostic steps, not guarantees that any chosen specification will work in a later market regime.
Why time and leakage complicate the diagnosis
Risk records are often ordered in time. A random split can place information from a later period into model development while a supposedly unseen earlier observation sits in validation. Even without an explicit date field, a feature calculated using future data contaminates the comparison. A validation score may then look good because the model has received information unavailable at the decision moment. GARP’s discussion of out-of-sample and out-of-time testing is relevant here: a separate later period can test whether a relationship survives beyond the development window.
A later-period test is valuable, but it can also contain a genuine economic shift. If all candidate models worsen there, the issue may include changed conditions rather than simply excess model capacity. The candidate should distinguish the purpose of each split: training estimates parameters, validation helps choose a specification, and a final untouched test offers a less contaminated check of that choice. Reusing a final test to make repeated adjustments makes it part of development, so its independence is lost.
Exam traps and a short decision routine
Do not equate a complex machine learning method automatically with overfitting or a conventional regression automatically with underfitting. A particular model can be well calibrated, poorly specified, or affected by unrepresentative data. Diagnose from evidence. Similarly, adding parameters is a plausible response to underfitting but usually aggravates a variance problem. Removing features indiscriminately may erase genuine signal. The remedy must match the observed failure and then be checked on data the revised model did not train on.
A useful sequence is to confirm the metric and split, compare training with validation error, ask whether absolute errors meet the decision need, and then choose a targeted change. When both errors are high, inspect missing relationships and data quality. When training error is very low but validation error is high, constrain flexibility and investigate leakage. When both are acceptably low, test stability across meaningful periods before making a practical claim. The FRM program page locates the qualification, while the Omni methodology page provides study context.
The candidate’s clearest written answer links symptom, mechanism and remedy. “Low training error and high validation error” is a symptom; learning quirks of the development sample is the mechanism; simplifying or regularising and checking a fresh sample is a possible remedy. “High error in both samples” points to a representation problem; the response is to consider a richer relevant relationship and test whether it actually improves unseen performance. That chain is more useful than memorising the words high bias and high variance without reading the data.
Sources
- Statistical Hypothesis Skepticism: Implications for Credit Risk | GARP · garp.org · Retrieved
- How Risk Managers Can Most Effectively Apply AI | GARP · garp.org · Retrieved



