Statistical significance is extremely important when drawing conclusions from noisy data. Noise in the data is problematic when investors overemphasize what might have been a period-specific outcome rather than a robust and repeatable result that is likely to persist going forward. Manager selection is a process that requires analyzing noisy performance data, so it should lean heavily on the notion of statistical significance.
When advisors, consultants, or investors select a manager, their approach often involves a combination of qualitative and quantitative analysis. The latter, by necessity, requires the use of past performance, as it is essentially all we have to evaluate. The focus is typically on managers with a positive "alpha" (i.e., outperformance relative to a benchmark or asset pricing model). Although past performance represents what actually happened, new investors cannot get those returns; therefore, the objective of the analysis is to determine whether outperformance in the past is any indication of skill, or simply good luck. In other words, is the positive alpha likely to persist in the future, after the manager is selected?
This task is more difficult than many investors might think. Several notable studies of manager performance found little persistence in past performance.1 Regardless, investors rarely hire managers with a history of poor relative performance, so the manager selection merry-go-round often becomes:
- Hire managers who have outperformed in the past.
- Fire managers who underperform in the future.
- Repeat.
This activity meets Einstein's definition of insanity, as many investors do the same thing over and over again while expecting different results. There are many reasons why this is an exercise in futility:
- It generally takes a long track record for a manager's alpha to be statistically significant.
- Positive alphas, even statistically significant ones, may not be an indication of skill.
- By the time you are confident there is skill, it's probably too late to benefit.
- Past performance is (really) no indication of future performance.
- If you know for sure the manager has skill, you likely won't benefit because the scarce resource captures the rent!
It Takes Time
Finding managers with positive alphas is like looking up the box scores from last night's games, but picking those who will outperform in the future is anything but trivial. To begin with, a quantitative analysis of past performance should incorporate tests of statistical significance to determine the likelihood their true alpha is not zero. In most cases, a long track record is required for the manager's alpha to be statistically significant. Given the average alpha and the standard deviation of the alpha, we can determine the track record required (in years) to obtain a t-stat of 2.
Table 1. Minimum track record for a statistically significant alpha (t-stat > 2)
| Average Alpha | |||||
| 1% | 2% | 3% | 4% | ||
|---|---|---|---|---|---|
| Standard Deviation of Alpha | 4% | 64 | 16 | 7 | 4 |
| 6% | 144 | 36 | 16 | 9 | |
| 8% | 256 | 64 | 28 | 16 | |
A track record of outperforming a benchmark or asset pricing model by an average of 2% per year (net of fees) over the life of the fund would get the attention of many investors, especially when you consider that the equity premium might only be around 5%. A representative standard deviation of alpha in the Morningstar universe of actively managed US equity mutual funds is approximately 6%.2 As illustrated in the table above, a 2% average alpha and a 6% standard deviation of the alpha requires a track record of thirty-six years before you can be 95% sure that the true alpha is not in fact zero (i.e., there was no skill at all). Based on these parameters, by the time you are reasonably confident there is some amount of skill, the manager is likely retired and on her yacht!
The Effects of Chance
Identifying a skillful manager involves more than simply narrowing down the universe to funds with positive alphas and a t-stat of 2 or more. This approach ignores the effects of chance. There is still a 2.5% probability the outperformance was due to good luck, and the true alpha of the manager is zero. Said another way, one out of forty managers is expected to have a positive alpha with a t-stat of 2 by chance. With so many funds in the universe, many will have statistically significant alphas even when there is no skill at all.
For example, in a 5,000-fund universe, 125 managers are expected to have a positive alpha with a t-stat greater than 2, even if their true alpha is zero. Unfortunately for investors, the opportunity is not in sorting through the many managers with statistically significant alphas, but rather in finding these managers because there are too few of them. Fama and French recently studied 3,156 US equity funds and compared their performance to a simulated universe of funds in which the true alpha for every fund was zero. Their results identified fewer funds with statistically significant alphas than you would expect to find by chance.3
By Then, It's Too Late
An investor concluding that a statistically significant alpha is evidence of skill could be guilty of data mining, meaning he is making inferences from what might have been a chance outcome limited to that time period. To counter these claims, academics conduct out-of-sample tests to confirm a statistically significant result. For example, out-of-sample data can be obtained by repeating the experiment using an independent time period (e.g., 1926–1962 rather than 1963–1992) or a different data set from an overlapping time period (e.g., international rather than US market data).
Practitioners should also conduct out-of-sample tests when analyzing manager performance to help rule out that a statistically significant alpha didn't occur by chance. On the surface, conducting out-of-sample tests might seem like an overly cautious approach akin to wearing a belt and suspenders. However, even if this were true, when you consider the consequences and what is at stake, it sure beats getting caught with your pants down.
The only way to test out-of-sample data when doing performance analysis is by using totally independent time periods. Accordingly, the number of years required in Table 1 gets multiplied by the number of independent periods you're comfortable with before you have faith there is some amount of robust and repeatable skill.
Table 2. Minimum track record for two independent periods with statistically significant alpha (t-stat > 2)*
| Average Alpha | |||||
| 1% | 2% | 3% | 4% | ||
|---|---|---|---|---|---|
| Standard Deviation of Alpha | 4% | 128 | 32 | 14 | 8 |
| 6% | 288 | 72 | 32 | 18 | |
| 8% | 512 | 128 | 56 | 32 | |
* Assumes the average alpha and standard deviation is the same in both time periods.
Using the prior example of a 2% average alpha and a 6% standard deviation of the alpha means you need a track record of seventy-two years if you're satisfied with a statistically significant result from only two independent time periods. But, in this instance, by the time you think there is evidence of skill, it's too late—the manager may be dead!
Persistence
Having said that, let's assume you've found a manager with statistically significant alpha in multiple independent periods and she is not yet retired or deceased. The question still remains: will the positive alpha continue? You cannot rule out luck because of the effects of chance noted above, but more important, many performance studies conclude that winners do not continue to win, and even when there is alpha in the extremes, it does not persist. The only slight indication of persistence is among the extreme losers, and it is mostly explained by high fees and high turnover.4
If we eliminate these funds from consideration, manager selection becomes a random draw, but whether investors know they are picking them at random is another question. Their goal is often to achieve top quartile performance, and pursuit of this goal usually boils down to choosing from managers among the top quartile in the past. However, excluding the persistent losers noted above, yesterday's top quartile performers have the same 25% probability of being in tomorrow's top quartile as every other manager!
Scarce Resources
Ironically, the predicament is not only for the investor trying to identify skill but for managers trying to prove they have it. A fundamental of economics is that the scarce resource captures the rent. If capital is freely floating and perfectly liquid, then the scarce resource is not the investor's money but the manager's skill! There is an enormous economic incentive for managers to indisputably prove they are in possession of this elusive ability.
Let's assume a manager has a twenty-year track record of outperforming by 4% each year. In this extreme example, a t-stat is meaningless since the standard deviation of the alpha is zero. If we rule out the possibility of a Ponzi scheme, the manager surely has undeniable skill. However, as soon as she proves her unique ability, she can capture the rent by increasing her fees to nearly 4%. An increase in fees of this magnitude may draw the ire of her investors, so, alternatively, she could raise more and more assets, thereby distributing her alpha over a larger asset base, which would dilute investor results.5 This latter approach may largely go unnoticed by investors, but either way they lose as their alpha subsequently becomes zero.
Making Progress
So, herein lies the paradox of skill. Many investors are searching for the Holy Grail of fund management. Their goal is to identify a skillful manager with certainty and participate in future returns. But confirming skill takes an investment lifetime, and you can never be fully confident that the alpha is not random. Even if you could identify skill ahead of time, you probably would not benefit. Winning managers hike their fees or attract large volumes of new investment long before their skill is statistically confirmed—and both actions can dilute returns.
But this paradox is not a case of "damned if you do and damned if you don't" for all investors. You can get off the manager selection merry-go-round and start making progress toward a successful investment experience by following these simple principles:
- Diversify by asset class rather than by fund manager, broker, or advisor.
- Buy into markets, not managers, and let capitalism be your guru.
- Focus on what you can control—costs, asset allocation, risks, and discipline. Ignore what you cannot control—the media, prognosticators, market returns, and your gut.
- Work with an advisor who understands these principles and can help you apply them.
The comments of Weston Wellington are gratefully acknowledged.
1. Noteworthy studies include: Mark Carhardt, "On Persistence in Mutual Fund Performance," Journal of Finance 52, no. 1( March 1997). Garrett Quigley and Rex A. Sinquefield, "Performance of UK Equity Unit Trusts," Journal of Asset Management 1, 72-92. James L. Davis, "Mutual Fund Performance and Manager Style," Financial Analysts Journal 57, no. 1 (Jan/Feb 2001).
2. Source: Index Funds Advisors.
3. Eugene F. Fama and Kenneth R. French, "Luck Versus Skill in the Cross Section of Mutual Fund Returns," Journal of Finance 65, no. 5 (October 2010): 1965–1947.
4. Carhardt, "On Persistence in Mutual Fund Performance." Fama and French, "Luck Versus Skill in the Cross Section of Mutual Fund Returns."
5. Jonathan Berk and Richard C. Green, "Mutual Fund Flows and Performance in Rational Markets," NBER Working Paper No. W9275, October 2002.
