Backtesting

Banks that manage market risk with the internal models approach use backtesting every day to check that VaR is consistent with actual profit and loss. The result feeds into the multiplier applied to regulatory capital and the model validation unit's review includes backtesting.

Why VaR is checked against actual profit and loss

VaR (value at risk) is an estimate derived statistically from past observed data. After it is measured, it therefore has to be compared with actual profit and loss to see whether the estimate was off. This check is called backtesting¹. The definition of VaR and how to measure it are covered in Value at Risk (VaR).

Backtesting counts exceptions, the days on which the loss exceeded VaR. For VaR at a 99% confidence level, the loss is expected to exceed VaR on 1 day in 100. The test checks whether the number of exceptions is well above this expectation¹.

Under Japan's capital adequacy notice for banks, banks that calculate their market risk capital requirement with their own models (called banks adopting the internal models approach in the notice) also perform backtesting as a regulatory requirement². The notice defines backtesting as the procedure of comparing VaR with daily hypothetical P&L and actual P&L³. In the notice, a backtesting exception means that the hypothetical P&L or the actual P&L exceeds VaR⁴.

How the daily comparison works and how exceptions are counted

Under the notice, banks using the internal models approach make the following comparison. It covers daily VaR and daily profit and loss over the most recent 250 business days, including the evaluation date⁴.

1. At the end of each business day, the bank measures VaR on its positions with a one-day holding period and a 99% confidence level.
2. At the end of the next business day, the bank calculates the profit or loss for that day (the next section covers which profit and loss are compared).
3. If the loss exceeds the previous day's VaR, the bank records that day as an exception. Days with a profit, and days with a loss no larger than VaR, do not count as exceptions.
4. The bank counts the exceptions over the most recent 250 business days and judges the result with the criteria in the following sections.

Consider a hypothetical example in which Bank X has the following profit and loss over five business days. Amounts are in billions of yen.

On day 3, the loss of 1.2 billion yen exceeds the previous day's VaR of 1.1 billion yen, so it is an exception. On day 5, the loss of 1.15 billion yen is below the VaR of 1.2 billion yen, so the day is not an exception even though it is a loss. Days 1 and 4 show profits, so they have nothing to do with exceptions.

When profit and loss or VaR cannot be observed on a day, the exception counts are adjusted. If only the actual P&L cannot be observed, one is added to the actual P&L exception count. If only the hypothetical P&L cannot be observed, one is added to the hypothetical P&L exception count. In any other case where something cannot be observed, one is added to both⁴. If an exception is caused by a non-modelable risk factor (a factor that moves the value of a position), the bank can exclude it from the count by notifying the Commissioner of the Financial Services Agency⁴.

The profit and loss compared (hypothetical P&L and actual P&L)

The notice sets out two kinds of profit and loss for the comparison, hypothetical P&L and actual P&L. Hypothetical P&L is the daily profit or loss from revaluing the positions held at the close of the previous day with market data at the close of the current day. It excludes commissions, fees, intraday trading, and new and amended trades⁵. Actual P&L is the actual profit or loss from the daily profit and loss calculation, and it includes intraday trading and new and amended trades. It excludes commissions and fees⁵.

Hypothetical P&L shows the price moves on the previous day's positions, while actual P&L also includes the results of the day's trading. The bank counts exceptions separately for each and uses the higher count as the backtesting result⁴.

Judging the exception count with the binomial distribution

If the model is correct and each day has an independent 1% chance of an exception, the number of exceptions $K$ over 250 days follows a binomial distribution⁷. The probability that $K$ equals $k$ is given by the following formula.

$$P(K = k) = {}_{250}C_{k}\,(0.01)^{k}\,(0.99)^{250-k}$$

There are ${}_{250}C_{k}$ ways to choose the $k$ exception days out of 250 days. The average number of exceptions is $250 \times 0.01 = 2.5$.

Figure description

This figure joins the "probability of exactly that number" column of the table above with a line, with the number of exceptions on the horizontal axis. The probability rises from 8.11% at 0 exceptions to a maximum of 25.74% at 2, then falls to 21.49% at 3, 13.41% at 4, and 6.66% at 5, and reaches 0.02% at 10. Vertical lines mark 5 and 10 exceptions, and the "probability of that number or more" in the table is 10.78% at 5 and 0.03% at 10. Between the points, the line simply joins the values in the table, and those in-between values are not in the table. Even a correct model most often produces around 2 exceptions. Ten or more, which fall in the right tail, rarely occur.

The judgment follows the logic of a statistical test. If you assume the model is correct, the probability of 10 or more exceptions in 250 days is only 0.03%. When 10 or more exceptions occur, the assumption is judged to be wrong, meaning the model has a problem⁸.

On the other hand, even a correct model has a 10.78% chance of five or more exceptions. Five to nine exceptions point to a problem with the model but are not conclusive⁹. If Bank X has six exceptions in 250 days, the probability that a correct model produces six or more is 4.12%. Three exceptions would occur with a probability of 45.68% even with a correct model.

The notice's three-zone approach and the multiplier

The notice sorts the bank-wide backtesting result into three zones by the number of exceptions over the most recent 250 business days. This method is called the three-zone approach⁹. The notice applies the multiplier for the zone to the market risk capital requirement (called the market risk equivalent amount in the notice)¹¹.

Figure description

This figure is for bank-wide backtesting. It looks at the number of exceptions through two questions in order. It first asks whether there are 9 or fewer, and if the answer is no, the result is the red zone. If yes, it asks whether there are 4 or fewer, and if the answer is no, the result is the amber zone, and if yes, the green zone. The multiplier values are those in the table above. As exceptions increase, the zone moves up and the multiplier rises.

The notice defines the three zones as follows (our translation)¹¹:

• Green zone: "a result indicating no problem with the quality and accuracy of the model"
• Amber zone: "a result suggesting a possible problem with the model, but not conclusively"
• Red zone: "a result indicating a very high likelihood of a problem with the model"

The Bank of Japan's 2012 training material calls the middle zone the yellow zone, and the 2016 overview by the Financial Services Agency and the Bank of Japan also calls it the yellow zone⁹.

The multiplier is applied to the average over the most recent 60 business days of the amount calculated with expected shortfall (see Expected Shortfall). In the notice's formula, $C_A$, the part of the capital requirement calculated with the internal model excluding default risk, is calculated as follows¹².

$$C_A = \max\left(IMCC_{t-1} + SES_{t-1},\; m_c \cdot IMCC_{avg} + SES_{avg}\right)$$

$IMCC$ is the amount calculated with expected shortfall for modelable risk factors, and $SES$ is the amount calculated with stress scenarios for non-modelable risk factors. The subscript $t-1$ denotes the value on the calculation reference date, and $avg$ denotes the average over the most recent 60 business days, including the reference date. $m_c$ is the multiplier in the table above, and the Commissioner of the Financial Services Agency may add a qualitative add-on to it¹².

Consider a hypothetical example in which Bank X has an $IMCC_{avg}$ of 10 billion yen, an $SES_{avg}$ of 2 billion yen, and an $IMCC$ of 11 billion yen and an $SES$ of 2 billion yen on the reference date. With three exceptions (green zone), the value $(1.50) \times 10 + 2 = 17$ billion yen exceeds the reference-date total of 13 billion yen, so $C_A$ is 17 billion yen. With six exceptions (amber zone), it is $(1.76) \times 10 + 2 = 19.6$ billion yen, an increase of 2.6 billion yen.

Desk-level checks and the P&L attribution test

The Fundamental Review of the Trading Book (FRTB) adds checks for each trading desk (a group of traders who set a business strategy and trade under it) to bank-wide backtesting¹³. Under the current notice, only desks that pass both desk-level backtesting and the P&L attribution test can use the internal models approach².

Desk-level backtesting compares one-day VaR at the 97.5% and 99% confidence levels with hypothetical P&L and actual P&L over the most recent 250 business days. A desk with more than 30 exceptions at 97.5% or more than 12 at 99% in the most recent 12 months calculates its market risk with the standardized approach¹⁴.

The P&L attribution test compares risk-theoretical P&L (the profit or loss calculated with the desk's risk management model) with hypothetical P&L day by day. It uses two metrics, rank correlation (Spearman's rank correlation) and closeness of the distributions (the Kolmogorov-Smirnov test, or KS test). The results sort each desk into three zones¹⁵.

Desk-level backtesting and the P&L attribution test are carried out every quarter, and the classification of desks based on the results is updated¹⁶.

The capital requirement is calculated with expected shortfall, but backtesting uses VaR. According to the 2016 overview by the Financial Services Agency and the Bank of Japan, "testing with a risk measure over the liquidity horizon or testing with ES is difficult" (our translation), so the existing framework of counting VaR exceptions is extended instead¹⁷.

What to do after an exception and how validation is organized

The Bank of Japan's 2012 training material says that when an exception occurs, it is important to analyze the cause and background, and that the analysis can be used to identify stress events that VaR does not capture and to improve the VaR model. The main causes are as follows¹⁸.

• A stress event
• A rise in volatility after VaR was measured
• A distribution of actual profit and loss with fatter tails than the normal distribution
• Trends or autocorrelation in profit and loss
• Too little observed data (VaR is unstable)
• The choice of observation period (old data with small price moves carries weight)

When profit and loss have a trend or autocorrelation, the square-root conversion from one-day VaR to VaR over the holding period no longer fits¹⁸. The conversion is covered in Probability and Statistics Basics.

Stress events that are not in the observation period do not show up in VaR. The Stress Testing page covers this limitation¹⁹.

The notice also sets out what banks using the internal models approach must do after exceptions. Exceptions in bank-wide and desk-level backtesting are documented with their details and causes, and the records are kept⁴. The bank notifies the Commissioner of the Financial Services Agency without delay in either of two cases. One is when bank-wide backtesting falls in the amber or red zone. The other is when desk-level backtesting has more than 30 exceptions at the 97.5% confidence level or more than 12 at the 99% confidence level in the most recent 12 months. In both cases, the bank attaches a document that sets out the causes and the planned response²⁰. If a notified classification such as a zone is recognized as including events caused by special market factors or the like, the bank can cancel the exceptions used for that classification²¹.

The notice also sets requirements for how validation is organized. The bank sets up a unit that manages market risk. A model validation unit independent of the design and operation of the internal models approach validates the model at approval and at least once a year after that¹⁶. Model validation includes backtesting and the P&L attribution test, and backtesting validation also covers how hypothetical P&L is calculated²².

Notes

1. Bank of Japan Financial System and Bank Examination Department, Center for Advanced Financial Technology, "Ⅱ．市場リスクの把握と管理" (II. Identifying and managing market risk) (June 2012), "４．バックテストによるＶａＲの検証" (4. Checking VaR by backtesting), page 39 (in Japanese). ↩ ↩²

2. Financial Services Agency, "銀行法第十四条の二の規定に基づき、銀行がその保有する資産等に照らし自己資本の充実の状況が適当であるかどうかを判断するための基準" (Standards for determining whether a bank's capital adequacy is appropriate in light of the assets it holds, under Article 14-2 of the Banking Act, our translation; FSA Notice No. 19 of 2006, consolidated version updated on August 24, 2026), Article 275 (page 459) (in Japanese). A trading desk is defined in Article 1, item 89 (page 24). ↩ ↩²

3. The same Notice, Article 271-3, paragraph 3, item 7 (page 436). The term "内部モデル方式採用行" (bank adopting the internal models approach) appears in Article 275 and elsewhere. ↩

4. The same Notice, Article 275-2 (bank-wide backtesting, pages 460–461): paragraph 1 on the comparison, paragraph 2 on the definition of an exception and adjustments for unobservable days, paragraph 3 on exclusions, paragraph 6 on records, and paragraph 7 on confidence levels other than 99%. Desk-level records are in Article 275-3, paragraph 5 (page 462). ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

5. The same Notice, Article 1, item 92 (hypothetical P&L) and item 95 (actual P&L) (pages 24–25). Both exclude valuation adjustments whose treatment is set separately in the capital adequacy calculation and valuation adjustments deducted from Common Equity Tier 1 capital and other capital. ↩ ↩²

6. The same Notice, Article 275-5, paragraphs 3 to 5 and paragraph 7 (page 463). Article 1, items 92 and 95, which define hypothetical P&L and actual P&L, use the word "評価調整" (valuation adjustment), while Article 275-5 uses "時価調整" (fair value adjustment). ↩ ↩²

7. Bank of Japan Financial System and Bank Examination Department, Center for Advanced Financial Technology, "Ⅱ．市場リスクの把握と管理" (June 2012), "ＶａＲを超過する損失が発生する回数（Ｋ）とその確率" (Number of losses exceeding VaR (K) and their probabilities), page 41, and "バックテスト（２項検定）" (Backtesting (binomial test)), page 42 (in Japanese). The probabilities in the table are the same as those in the table on page 42 and match a recalculation from the formula. ↩

8. Bank of Japan Financial System and Bank Examination Department, Center for Advanced Financial Technology, "Ⅱ．市場リスクの把握と管理" (June 2012), page 43 (in Japanese). ↩

9. Bank of Japan Financial System and Bank Examination Department, Center for Advanced Financial Technology, "Ⅱ．市場リスクの把握と管理" (June 2012), "バーゼル銀行監督委員会の３ゾーン・アプローチ" (The Basel Committee's three-zone approach), page 40 (in Japanese; the material says the source is the Basel Committee's January 1996 framework). Financial Services Agency and Bank of Japan, "「マーケット・リスクの最低所要自己資本」の概要" (Overview of "Minimum capital requirements for market risk") (April 2016), "４－⑫" (4-12), page 22 (in Japanese). ↩ ↩² ↩³

10. Financial Services Agency and Bank of Japan, "「マーケット・リスクの最低所要自己資本」の概要" (April 2016), "４－⑭ モデル検証" (4-14 Model validation), page 24 (in Japanese). ↩

11. Financial Services Agency, "銀行法第十四条の二の規定に基づき、銀行がその保有する資産等に照らし自己資本の充実の状況が適当であるかどうかを判断するための基準", Article 275-2, paragraphs 4 and 5 (pages 460–461) (in Japanese). ↩ ↩²

12. The same Notice, Article 279 (pages 477–478). The symbols $C_A$, $IMCC$, $SES$, and $m_c$ in the text are those used in the notice. ↩ ↩²

13. Financial Services Agency and Bank of Japan, "「マーケット・リスクの最低所要自己資本」の概要" (April 2016), "１―① 最終文書公表の経緯" (1-1 Background to the publication of the final document), page 2, "３－① 標準的方式と内部モデル方式の関係" (3-1 Relationship between the standardized approach and the internal models approach), point 3, page 8, and "４－⑨ デスク・レベルでの検証（２）：バックテスト" (4-9 Desk-level validation (2): backtesting), page 19 (in Japanese). ↩

14. Financial Services Agency, "銀行法第十四条の二の規定に基づき、銀行がその保有する資産等に照らし自己資本の充実の状況が適当であるかどうかを判断するための基準", Article 275-3, paragraphs 1, 2, and 4 (pages 461–462) (in Japanese). Paragraph 1 describes the comparison period as the "most recent 250 business days," while paragraph 4 describes the period for the count criteria as the "most recent 12 months" (our translation). ↩

15. The same Notice, Article 1, item 94 (risk-theoretical P&L, page 24), Article 275-4, paragraph 1 (page 462), Article 275-7 (page 465), and Article 275-8 (pages 465–466). ↩

16. The same Notice, Article 272-6, item 10(b)(3) (quarterly testing and updating of the classification, page 443), item 11 (the market risk management unit, page 443), and item 15 (the model validation unit, page 444). ↩ ↩²

17. Financial Services Agency and Bank of Japan, "「マーケット・リスクの最低所要自己資本」の概要" (April 2016), "４－⑨ デスク・レベルでの検証（２）：バックテスト" (4-9 Desk-level validation (2): backtesting), page 19 (in Japanese). ↩

18. Bank of Japan Financial System and Bank Examination Department, Center for Advanced Financial Technology, "Ⅱ．市場リスクの把握と管理" (June 2012), "バックテストの分析・活用" (Analyzing and using backtesting results), page 44, and "ＶａＲ超過損失の発生原因・背景" (Causes and background of losses exceeding VaR), page 45 (in Japanese). ↩ ↩²

19. Bank of Japan Financial System and Bank Examination Department, Center for Advanced Financial Technology, "Ⅱ．市場リスクの把握と管理" (June 2012), "５．ＶａＲの補完とストレステスト" (5. Supplementing VaR and stress testing), page 46 (in Japanese). ↩

20. Financial Services Agency, "銀行法第十四条の二の規定に基づき、銀行がその保有する資産等に照らし自己資本の充実の状況が適当であるかどうかを判断するための基準", Article 275-9 (pages 466–467) (in Japanese). A bank also notifies the Commissioner when the P&L attribution test falls in the amber or red zone. ↩

21. The same Notice, Article 275-10 (page 467). What can be cancelled are the backtesting exceptions and the failed P&L attribution test results used for the classification. ↩

22. The same Notice, Article 272-7, paragraph 2, item 2 (page 446). ↩