On this page
A CAIA® hypothesis test asks whether an observed estimate is too unusual to attribute readily to sampling noise under a stated null hypothesis. It does not establish that a manager has skill when the underlying sample is biased. A sound answer checks both the statistical decision and the way observations entered the data set. The second check can overturn confidence in the first.
What the test actually answers
Suppose a fund reports positive estimated alpha. That number is an estimate from particular observations and a particular model, not a direct reading of an unobservable quality called skill. The null hypothesis might be that true alpha is zero; the alternative might be that it differs from zero. A test uses a measure of uncertainty around the estimate to judge whether the sample is surprising if the null holds. It cannot answer whether the model captured every relevant risk exposure.
The official CAIA Level I curriculum companion names the test sequence and specifically includes the danger of accepting a hypothesis, economic versus statistical significance, distribution assumptions and sampling problems. The 2026 exam handbook says the curriculum is revised for an exam cycle and is the source for testable content. Use the current official materials for your sitting; the earlier companion here supports the concepts explained, not a claim about unchanged wording in a later cycle.
Set out the question before seeing the result. A two-sided alternative asks whether the parameter differs in either direction from the null value; a one-sided alternative asks about a specified direction. Decide which one matches the decision you need to make. Changing direction after seeing the estimate turns a planned test into a search for a favourable conclusion. The interpretation of a p-value also depends on this design choice.
The four decisions in a basic test
First, state the null and alternative clearly, including the parameter and benchmark. Second, choose a significance threshold and an appropriate test statistic before examining the result. Third, calculate the statistic from the sample estimate and its standard error. Fourth, compare the statistic or its p-value with the decision threshold, then explain the conclusion in words. The conclusion should say whether evidence is sufficient to reject the null, rather than claiming a hypothesis has been proven.
A significance level is a rule about the long-run frequency of a false rejection when the null is true under the model assumptions. It is not the probability that the null is true after observing the data. A p-value asks how extreme the observed statistic would be under the null and the assumed sampling model. It is not the probability that the result will repeat. Candidates lose the logic of the test when they substitute either of those attractive but unsupported interpretations.
The distinction between errors is also practical. Rejecting a true null is a false positive, conventionally called a Type I error. Failing to reject a false null is a false negative, called a Type II error. A stricter rejection threshold can reduce the first risk while making the second more likely if the sample and effect remain fixed. Larger, genuinely informative samples can improve precision, but merely adding more observations from a distorted database does not repair its selection rule.
A worked example with invented returns
Consider a wholly illustrative fund estimate of annual alpha of 2.0 percentage points with a standard error of 1.0 percentage point. Test a zero-alpha null using a two-sided rule with an illustrative critical value of 1.96. The test statistic is the difference between 2.0 and zero divided by 1.0, which equals 2.0. Because 2.0 exceeds 1.96, this simple calculation crosses that chosen threshold. These values are invented to teach the operation; they are not a claim about any real fund.
Now change only the standard error to an illustrative 1.5 percentage points because the estimate is noisier. The test statistic becomes about 1.33, which does not cross 1.96. The estimated economic return is still 2.0 percentage points. This contrast shows that a large point estimate and strong evidence are different things. It also shows why the answer must include the benchmark, uncertainty and decision rule rather than reciting the alpha estimate alone.
The calculation is valid only if the chosen sampling model and uncertainty estimate are appropriate. Serial dependence, unusual tails, stale valuations and a model that omits relevant exposures can distort a standard error or the meaning of alpha. In alternative investments, irregular valuations can make a return series appear smoother than the underlying economic exposure. An exam case may give you a simple statistic to compute, but a professional interpretation should still note what assumptions make that statistic informative.
Even when the first calculation crosses its threshold, it does not establish that the fund adds value after fees, costs and implementation constraints. Statistical significance addresses evidence against a specified null; economic significance asks whether the effect matters for the decision. An estimate can be measured precisely and still be too small to matter. Conversely, a potentially meaningful estimate can be too noisy to support a confident conclusion. Keep those questions separate.
Where biased samples enter
Survivorship bias appears when failed or closed funds disappear from a database while continuing funds remain. A comparison calculated only from survivors can make the opportunity set look stronger than the one an investor actually faced at the start. Self-selection is related but occurs when managers choose whether to report at all. If reporting is associated with good results, the reported set is not representative. A tight confidence interval around its average does not solve that problem.
Backfill bias arises when a fund begins reporting to a database and adds a selected earlier return history. If entry becomes more attractive after a good run, the imported history can make an index appear stronger than a contemporaneous investable set. Look-ahead bias is different: an analyst uses information that was unavailable on the historical decision date. Both can improve a backtest on paper, but they enter at different stages. Identify the stage before naming the bias.
The CAIA companion explicitly lists selection, self-selection, survivorship, data dredging, backtesting and backfilling as syllabus concepts. A CAIA-hosted discussion of alternative investment data also notes that removing funds that ceased to exist can bias findings upward. Neither source licenses an automatic numerical correction. The size and direction of a particular bias depend on database rules and the missing observations, which are often not visible to the analyst.
Many searches can manufacture a discovery
Suppose an analyst tries twenty independent, ineffective signals and applies a 5% false-positive threshold to each. The expected number of chance rejections is one. Under the stated independence assumption, the probability of at least one false positive is one minus 0.95 raised to the twentieth power, about 64.2%. These are invented assumptions and a calculated illustration, not an observed success rate. They show why a single reported winner is weaker evidence when it was chosen from a large unreported search.
Data mining can be useful when hypotheses have a clear economic rationale and are checked on data reserved for validation. Data dredging hides the number of tries and presents an accidental winner as though it were the only planned question. Overfitting compounds the issue by tuning many choices to past noise. Out-of-sample checks, a pre-specified question and full reporting of attempted variants help reveal how much selection took place. None guarantees future returns, but each makes the evidence easier to judge.
A reliable order for an exam explanation
When a case gives an impressive test result, first identify the null, alternative, statistic and formal decision. Next ask how funds were chosen, which funds vanished, when reporting began and whether historical information was available at the time. Then distinguish a statistical rejection from a useful economic effect. If the sample is unrepresentative, say that the arithmetic may be correct for that sample while its inference about the wider population is unreliable. This is a stronger answer than either dismissing all tests or trusting every significant result.
The CAIA programme page sets the broader study context; Omni’s methodology explains its teaching approach. In revision, practise pairing each bias with the point where information enters or leaves a data set. That habit makes the bias names easier to distinguish and keeps the final interpretation tied to the actual evidence rather than the attractiveness of a reported number.
Sources
- CAIA Level I Curriculum Companion | CAIA Association · caia.org · Retrieved
- 2026 CAIA Exam Handbook | CAIA Association · caia.org · Retrieved
- Alternative Investment Analyst Review | CAIA Association · caia.org · Retrieved



