Part two of a two part article on PD modelling by Sajit Roshan, Shahnawaz Alam, Imran Essuf and Sriraghavan Rajamannar
The probability of default (PD) is the likelihood that a loan will not be repaid and will fall into default. PD is calculated for each client who has a loan for wholesale banking, or for a portfolio of clients with similar attributes for retail banking. The credit history of the counterparty portfolio and nature of the investment are taken into account to calculate the PD.
There are many alternatives for estimating the probability of default. Default probabilities may be estimated from a historical data base of actual defaults using modern techniques like logistic regression. Default probabilities may also be estimated from the observable prices of credit default swaps, bonds, and options on common stock. We continue to explain the different tests used to ascertain PD.
X-HEAD: Portfolio stability tests
Stability tests assess the shifts in the underlying portfolio. We aim to check whether the portfolio that we are applying the model on still has the same characteristics as the original sample that was used to build the model. If, for example, the population has changed significantly, the model may still be good but we will not get accurate results since we are applying it to a set of customers that it was not designed for.
Portfolio tests are useful in two ways:
Portfolio stability tests help verify the population similarity or dissimilarity, and hence act as early warning systems for scorecard validation;
Where the rank ordering or calibration tests show poor model performance, portfolio stability tests help identify the underlying reasons, revealing the changes in the factors that could have led to the degradation in the model's predictive ability.
The data required for these tests are different from the first two classes of tests we covered in last month's article, rank ordering tests and calibration tests. Portfolio stability tests for an application scorecard require all the applications of a particular period along with their respective scores and the scorecard factors. Similarly portfolio stability tests for a behaviour scorecard require all the accounts at a particular point in time along with their scores and scorecard factors. These tests do not need the 12-month actual performance and hence are quite effective in building an early warning system.
X-HEAD: Population stability index
The population stability index measures the magnitude of the shift between the population today and the population that was considered at the time of building the model, across score ranges.
For a behaviour model, the distribution of the portfolio in a given time period is compared with that of the model development time period across the score ranges as shown in table one.
Table one: Population stability index
In this illustration, the score ranges are determined in such a way that there is exactly 10 per cent of model time period observations in each bucket. An alternative approach could use fixed score ranges like 0-100, 101-200 ...and 901-999.
Expected percentage (E) is the population distribution of the model development sample and actual percentage (A) is the population distribution in the recent time period across the score ranges.
The index calculated as:
(A - E) * ln (A / E)
The summation of the indices across all score ranges results in the overall population stability index.
A population stability index of zero implies that the current population is distributed in an identical way to the original population. An index of 0.1 to 0.25 implies a small change in the distribution and an index greater than 0.25 implies a significant change in the distribution of the population. These thresholds are commonly accepted industry benchmarks.
The population stability index is compared over various time periods to detect the shifting trends in the population.
While the population stability index tells us whether a shift has occurred or not, and provides an indication of the magnitude of the shift, it does not tell us about the reasons for the shift.
X-HEAD: Characteristic stability index:
The characteristic stability index is a measure that quantifies the shift in the distribution of a characteristic in the population today as compared to the distribution of that characteristic at the time of building the model, across the attributes of the characteristic.
For a behaviour model, the distribution of the characteristic, for example, years of service in a given time period, is compared with that of the model development time period across the attributes of the characteristic. An illustration is provided in table two.
Table two: Characteristic stability index
X-HEAD: Model drill down tests
This is a general test that may be used to test any major assumptions or definitions that have been used when building the model. If these assumptions have an effect on the predictability of the model today, then a test should be conducted to check its validity.
X-Head: Model revision tests
While there are various methodologies followed for building a PD model, they broadly have the following steps:
Business problem definition
Scorecard design
Identifying the master list of significant predictors
Variable reduction
Variable transformation
Correlation study and variable clustering
Assigning weights to factors and attributes
Calibration
A considerable amount of analyses and judgmental input goes into each of these steps. While these analyses are thoroughly conducted during the initial scorecard development, a quick re-visit of these steps, periodically, would help boost the confidence of the various stakeholders in the internal rating systems. It would also help scorecard developers know whether the model needs just a minor fix-up or a complete revamp. These tests also provide a greater insight into model performance and remedial actions.
Sajit Roshan is working as a quantitative analyst for Mashreqbank. Shahnawaz Alam and Imran Essuf are also quantative analysts and were assisted in producing this article by Sriraghavan Rajamannar.
Banker Middle East 2007
