Methodology

ORTHOGONAL MEASURES ABSOLUTE SKILL

FundAttribution’s mission is to serve the needs of investors, large and small, by delivering objective and actionable analytics.

Typically, fund evaluation systems assign stars or measure if a fund is in the top quartile.  Investors are increasingly questioning the value delivered by active fund managers in light of numerous studies. It is no longer sufficient to compare an active manager against a peer group of other active managers.

The Orthogonal Attribution Engine isolates the skill delivered by managers in excess of what is available through investable passive alternatives and other indices.  It considers various aspects of manager skill and their consistency over time

The left chart depicts the distribution of one key skill for active equity managers in the US mutual fund universe over a recent three year period.  This suggests the average manager has slight skill before expenses. Note that his presentation does not adjust for survivorship bias (roughly 7% of funds disappear in a given year) and funds flow, that our universe may not be complete,  and that our model is subject to revision over time.

 

 

 

Active equity managers in our universe also demonstrated some skill in sector selection compared with passive indices. Structured /quantitative funds displayed a small degree of skill before expenses.

ORTHOGONAL RELATES SKILL TO EXPENSE LEVELS

Funds with strong performance come at a price. We are introducing some metrics which estimate the probability the portfolio manager will deliver value over the next 12 months which justifies his or her expense structure.

 

ADAPTIVE BENCHMARKING ENABLES BETTER JUDGMENT

Classic fund attribution systems attribute the difference in return between the fund and predetermined benchmark to a variety of factors

This method is adequate when analyzing a single fund, but less useful when comparing a series of funds against each other. 

  • Often the benchmark is assigned based on the fund’s stated objective or based on an arbitrary category or style assignment. If the fund outperforms or underperforms its benchmark, it begs the question: “Was the investment manager skillful or is the performance explained by a difference between the fund’s benchmark and what it is actually doing?”  
  • Many funds vary their investment mix or rotate sectors over time, which makes accurate comparisons even more difficult.
  • Comparing funds with disparate benchmarks is particularly difficult

The Orthogonal Attribution Engine solves these issues using adaptive benchmarking.  Orthogonal’s benchmarks are not based on a predetermined assignment. Rather, we assess what the fund is actually doing. We do this at every point of time. We base this on both the reported holdings history of the fund and regression-based inference techniques.  The holdings and regression results are blended to achieve a highly accurate model of the manager’s activity.  We consider the characteristics of the fund’s holdings and the expected impact of the manager’s decisions on fund performance.

 

DATA INTENSITY

Orthogonal considers data from a variety of sources and goes back in time as far as possible. Generally speaking, three years of strong performance has some predictive value but is not enough to confidently extrapolate a manager's skill. Some funds are easier to extrapolate than others.

  

FORWARD-LOOKING METRICS

We look at consistency, persistency, managerial changes, and trends. Volatility does not affect Orthogonal's assessment of skill but it does affect our confidence that a manager's past performance will continue. So the most skillful manager does not always receive the strongest recommendation. The chart below illustrates the impact of mean skill and consistency on our confidence.  We consider many skills, and our confidence is affected by the length of the manager's track record and other factors. 

 

 

 

 

DIFFERENTIATED

Orthogonal's analytics yield fresh insights compared with existing methods. This chart compares Orthogonal's assessment of the probability of future success with a prominent national fund evaluation service. This comparison is based on 10,500 public equity fund classes where Orthogonal and third party ratings are available.  How to read the chart:  of funds rated two stars, 22% have 30-39% confidence of success. Success means the fund will generate enough gross return through skill over the next twelve months to justify its active expense structure.

The funds rated highest by the third party service are, on average, more likely to be successful (50% vs only 24% for their lowest category). The funds with higher star ratings are, on average, more likely to be successful. However, those ratings do not tell investors the full story. Only a quarter of the funds carrying the highest rating are worthy of investment, based on our hurdle of 60% confidence.

  

 

 

 

Across all rating categories, fewer than 10% of funds merit investment. Investors and allocators need advanced tools like Orthogonal to distinguish the elite funds which are statistically likely to outperform and justify their fees.

 

INTEGRATED

The Orthogonal Attribution Engine is an integrated solution.  There is no need to license fund data or indices separately. We rely our own proprietary databases of returns, holdings, and fundamental data to offer comprehensive coverage

 

FLEXIBILITY AND RANGE

The current implementation features an attribution model called Total Market Indexing (TMI). The Orthogonal Attribution Engine currently supports three additional models and can be customized for the needs of an organization.

The current website implementation allows screening and graphic comparison based on one skill (stock selection.) Ultimately, clients will be able to make comparisons based on a variety of skills.

The product supports funds outside the US. The analytics support both US and global investors.

While the current model is geared to equity funds, credit funds are modelled and will be better addressed in future versions of the product.

Separate Accounts and private funds can be modelled upon request.

 

FUND SCREENING

Orthogonal provides asset allocators and advisors with tools to efficiently identify the most skillful managers which meet a set of parameters.

 

REPORTING

Clients can export graphic fund comparisons. Additional reporting features will be added over time.

  

CATEGORIZATION

Although our fund assessments do not rely on arbitrary category assignments, we do categorize funds as an accommodation. These category assignments are useful for screening and reporting purposes, but they do not affect the underlying analysis.  Our categorization reflects in part which group a given fund correlates with most strongly and may not always conform to  the fund’s name or stated objective

 

FUNDATTRIBUTION FIXED INCOME MODEL

In April of 2016 we introduced a model tailored to fixed income funds

The model measures the skill of a fixed income fund using a multi-variate regression model. Factors considered include the fund’s duration and exposure to various risk classes and countries. The funds are compared to indices constructed from the average performance of different fund categories derived by FundAttribution.

Our fixed income attribution model is a streamlined version FundAttribution’s Orthogonal Attribution Engine model tailored to and has some important differences. Among those:

  • The current version of the model is neither forward-looking nor probabilistic. We haven’t attempt to backtest, extrapolate past success forward, or assess the distribution of future skill
  • The model does not utilize historic holdings data
  • The model captures performance of broad credit classes using a limited number of factors.  Some niche funds may have low r-squareds and produce skill measurements which are less meaningful.
  • The model is not adept at comparing funds whose inception dates differ markedly
  • The model doesn’t consider shifts by the portfolio manager over time

 

Each class of the fund is rated based on a comparison of skill and expense structure. Roughly 10% of fund classes demonstrate enough skill to justify further consideration.

 

Optimal portfolios are constructed using fund classes available to the client to maximize the overall Sharpe Ratio.

For the April edition of Mutual Fund Observer where we select an optimal Short Duration fixed income portfolio for a hypothetical institutional client. We identified 38 fund classes with evidence of skill which met the criteria. The model portfolio included 3 of those 38 classes.

 

We apply our understanding of the fund to categorize it appropriately, but our categorization does not affect our evaluation. Similarly, Duration statistics (self-reported) may be useful in fund screening but do affect fund ratings.

 

 

 FREQUENTLY ASKED QUESTIONS

Q: What is the Orthogonal Attribution Engine?

A: The Orthogonal Attribution Engine is a model used by the website to calculate forward looking projections and probabilities. The term orthogonal refers to the process of separating manager performance into many independent and uncorrelated factors. Consistent good decisions over a sufficient period may be interpreted as skill. 

The OAE is the first product from FundAttribution. The product is hosted on the domain www.fundattribution.com

Q: What benchmarks or indices do you use?

A: In general a fund is compared to a custom-fitted, time-blended benchmarks. These benchmarks use a number of proprietary indices developed by FundAttribution. Some of these represent the average gross return of financial instruments with specified characteristics. These may include baskets of individual securities, active or passive funds, or published indices.  We weight different sources and splice them when required.

Q: How much history is required to develop confidence a portfolio manager has skill?

A: It varies considerably. We don’t assign metrics until three years have elapsed and we prefer six. Some track records extrapolate better than others.  Sometimes the model interprets success sustained over several years as one run of good luck.

Q: What is your data frequency?

A: We use a combination of daily returns and, where possible, quarterly holdings. Other fund level data is updated at least quarterly.

Q: How far back does your data go?

A: We use data back to 1995.

Q: Why is there a lag?

A: With our current processes, the data used to evaluate a fund may lag by several months.  And the recent data may be less accurate. As a result, we tend to put less weight on the most recent results in any forward looking analyses.

Q: How do you deal with new classes or clones of established funds?

A: We reflect the history of the oldest class.

Q: How successful is the model in making useful predictions?

A: Backtesting studies (summarized on the site) indicate the skill predictions are valid with 99% confidence.

Q: How do you deal with changes in fund management?

A: For projections and probabilistic calculations, we ignore the fund’s track record prior to the appointment of the current lead manager. The website calculates and displays average skill over the manager’s tenure but may also display data from prior periods.

Sometimes these issues require customized evaluation, available upon request

Q: Isn’t your model predicated on the assumption that the future will resemble the past?

A: Historically, managers who have demonstrated skill continue to be skillful.  We are assuming that continues. Sometimes there is a known discontinuity in the investment process; in those circumstances it may make sense to vary the approach.

Q: What are the limitations of the demo version? Why can I only compare funds based on Security Selection skill?

A: Users will have access only to specified categories. Some pulldown menus are disabled. Screening will return only ten queries. Enhancements including support for other skills will be added in the future.

Q: When will the product be commercially available?

A: Currently the Orthogonal Attribution Engine is undergoing beta testing. FundAttribution is undertaking customized analyses and special projects upon request.

Q: What are sS and sR?

sS is a fund manager's excess return in a given period after normalizing all factor exposures. It is typically expressed as an annualized return. Skill is sometimes inferred if sS is consistenly high for a sufficiently long period.

sR represents contribution from overweighting certain industry sectors over time. In some fund categories, it is helpful to look at sS and sR combined.

Q: What is the TMI Model?  Why can’t I choose the other models?

A: TMI (or Total Market Index) is a method of attribution which relates the fund’s performance to the global equity universe based on differences in the portfolio’s beta, market cap, geographic positioning, growth/value orientation, sector rotation, and security selection.  Some other models are under development and not supported in Orthogonal 2.0.

Q: What are “Baselines?”

A: Every portfolio is compared to a baseline return which reflects its allocation between sectors over time.  The baseline indicates what an unskilled, market-weighted portfolio would have returned over a given time period. The default baseline is a global US$ portfolio. The US baseline presents the comparison against the US market. The category baseline reflects the average portfolio in the assigned category.

Q: How do you categorize funds?

A: FundAttribution categories funds based on a combination of the fund’s stated objectives, holdings, correlation analyses, and third party ratings. Currently 87 categories are in use. Category assignments are reviewed frequently and our schema will evolve over time.

Although we categorize funds for the convenience of users, the categories have no impact on our assessment.

Q: How do you deal with survivorship bias?

A: Survivorship bias is an important factor when we study skill across a population of managers. The average active portfolio may exhibit a small degree of skill. This reflects that funds with less successful histories tend to be merged or liquidated.  It is also a consideration in our index construction, we try to avoid comparing funds to a reference portfolio which may have bias.

Q: How do you deal with loads? Hidden expenses?

A: Currently front-loads are amortized over seven years.  Whenever possible we adjust for indirect expenses.

Q: How do you deal with funds with multiple strategies? How do you deal with style drift?

A: The Orthogonal Attribution Engine strives to understand the manager’s strategy over time and adjusts the baseline to reflect the most appropriate comparison.

Q: How do you replicate funds?

A: We strive to use a belt and suspenders approach based on analysis of historic holdings and regression.  Some funds lend themselves more to one approach than the other and we take that into account.

Q: I evaluated a passive fund using your system. Why did the fund show skill?

A: Our aim is for passive funds to have skill near zero.

  • The fund may track a different index than Orthogonal uses which happened to perform differently. For example, in small cap there are several competing indices which sometimes diverge.
  • The fund may track a subsector more specific than our indices. For example a uranium ETF may not be well explained by our commodity baskets; if uranium underperforms the nearest comparison, it will be reflected as negative skill

During beta testing, some other issues may affect skill evaluation

  • There are issues causing funds with fixed income exposure to show positive skill
  • There are some issues with index construction prior to 2005.
  • The passive fund may not be replicated or attributed accurately.
  • Our returns data may be incorrect.   

Q: Why is the typical probability well below 50%?

A: In a word, costs. If all funds had low expenses, the median probability would be around 50%

Q: How are quantitative managers analyzed?

A: Currently, these funds are evaluated using the same methodology