How markets become efficient (answer: they don't)

A staggering amount of effort has been spent -- and wasted -- exploring the idea of market efficiency. The notoriously malleable efficient markets hypothesis (EMH) claims (in its weakest form) that markets are "information efficient" -- market movements are unpredictable because smart investors keep them that way. They should quickly -- even, "instantaneously" in some statements -- pounce on any predictable pattern in the market, and by profiting will act to wipe out that pattern.

I've written too many times (here, here, here, for example) about the masses of evidence against this idea. It's not that predictable patterns don't attract investors who often act in ways that tend to wipe out those patterns through arbitrage. Part of the problem is that investors often act in ways that amplify the pattern (following trends, for example). Moreover, there are fundamental limits to arbitrage -- "the markets can stay irrational longer than you can stay solvent." Still, the EMH stumbles onward like a zombie -- dead, proven incorrect and misleading, yet still taking center place in the way many people think of markets.

I found an illuminating new perspective on the matter in this recent paper by Doyne Farmer and Spyros Skouras, which explore analogies between finance and ecology. This analogy is itself deeply suggestive. They note, for example, how the interactions between hedge funds can be useful viewed in ecological terms -- funds sometimes act as direct competitors (the profits of one reducing opportunities for another), and in other cases as predator and prey or as symbiotic partners. But I want to look specifically at an effort they make to give a rough estimate of the timescale over which the actions of sophisticated arbitragers might reasonably be expected to wipe out a new predictable pattern in the market. That is, if the market for whatever reason is temporarily inefficient -- showing a predictable pattern -- how quickly should it be returned to efficiency? How long is the time to relaxation that the EMH claims is "instantaneous" or close to it?

The gist of their idea is very simple. Before you can exploit a predictable pattern, you first have to identify it. If you're going to invest money trading against it, you need to be fairly sure you've identified a real pattern, not just a statistical fluke. If you're going to invest somebody else's money, you have to convince them. This takes some time. The stronger the pattern, the more it stands out and the less time it should take to be sure. Weaker signals will be hidden by more noise, and reliable identification will take longer. Looking at how much time it should take to get good statistics should give an order of magnitude of how long a pattern should persist before any smart investor can begin reliably trading against it and (perhaps) erasing it.

Here's the specific argument, expressed using the Sharpe ratio (ratio of expected return to standard deviation of a strategy exploiting the pattern):

This makes obvious intuitive sense. If S is very large, making the pattern obvious, more deterministic and easier to exploit, then the time over which it might be expected to vanish is smaller. Truly obvious patterns can be expected to vanish quickly. But if S is small, the timescale for identification and exploitation grows.

As Farmer and Skouras note, successful investment strategies often have Sharpe ratios of about S = 1, so this gives a result of about 10 years. [This is the result if one makes the analysis on an annual timescale, with the Sharpe ratio calculated on a yearly basis. If we're talking about fast algorithmic trading, then the analysis takes place on a shorter timescale.]

So, 10 years is the order of magnitude estimate -- which is a rather peculiar interpretation of the word "instantaneous." Perhaps that word should be replaced in the EMH with "very slowly," although that somewhat dampens the appeal of the idea: "The EMH asserts that sophisticated investors will very slowly identify and exploit any inefficiencies in the market, tending the erase those inefficiencies over a few decades or so." Given that new inefficiencies can be expected the arise in the mean time, you might as well call this more plausible hypothesis the PIMH: the perpetually inefficient markets hypothesis.

And their estimate, Farmer and Skouras point out, is actually optimistic:

We should stress that this estimate is based on idealized assumptions, such as log-normal returns – heavy tails, autocorrelations, and other effects will tend to make the timescale even longer.... As a given inefficiency is exploited, it will become weaker and the Sharpe ratio of investment strategies associated with it drops. As the Sharpe ratio becomes smaller the fluctuations in its returns become bigger, which can generate uncertainty about whether or not the strategy is still viable. This slows down the approach to inefficiency even more.

Of course, as I mentioned above, this analysis depends on timescale. Take t in years and we're thinking about predictable patterns emerging on the usual investment horizon of a year or longer, patterns exploited by hedge funds and mutual funds of the more traditional (not high frequency) kind.  Here we see that the time to expect predictable patterns to be wiped out is very long indeed. If 10 years is the order of magnitude, then it's likely some of these patterns persist for several decades -- getting up to the time of a typical investing career. Hardcore supporters of the EMH should learn to speak more honestly: "We have every reason to expect that predictable market inefficiencies should be wiped out fairly quickly, at least on the timescale of a human investment career."

All in all, this way of estimating the time for relaxation back to the "efficient equilibrium" suggests that the relaxation is anything but fast, and often very slow. The EMH may be right that there likely aren't any obvious patterns, but more subtle predictable patterns will likely persist for long periods of time, even while they present real profit opportunities. The market is not in equilibrium. And with no mechanism to prevent them, new predictable patterns and "inefficiencies" should be emerging all the time.

Overconfidence is adaptive?

A fascinating paper in Nature from last week suggests that overconfidence may actually be an adaptive trait. This is interesting as it strikes at one of the most pervasive assumptions in all of economics -- the idea of human rationality, and the conviction that being rational must always be more adaptive than being irrational. Quite possibly not:

Humans show many psychological biases, but one of the most consistent, powerful and widespread is overconfidence. Most people show a bias towards exaggerated personal qualities and capabilities, an illusion of control over events, and invulnerability to risk (three phenomena collectively known as ‘positive illusions’)^{2, 3, 4, 14}. Overconfidence amounts to an ‘error’ of judgement or decision-making, because it leads to overestimating one’s capabilities and/or underestimating an opponent, the difficulty of a task, or possible risks. It is therefore no surprise that overconfidence has been blamed throughout history for high-profile disasters such as the First World War, the Vietnam war, the war in Iraq, the 2008 financial crisis and the ill-preparedness for environmental phenomena such as Hurricane Katrina and climate change^{9, 12, 13, 15, 16}.

If overconfidence is both a widespread feature of human psychology and causes costly mistakes, we are faced with an evolutionary puzzle as to why humans should have evolved or maintained such an apparently damaging bias. One possible solution is that overconfidence can actually be advantageous on average (even if costly at times), because it boosts ambition, morale, resolve, persistence or the credibility of bluffing. If such features increased net payoffs in competition or conflict over the course of human evolutionary history, then overconfidence may have been favoured by natural selection^{5, 6, 7, 8}.

However, it is unclear whether such a bias can evolve in realistic competition with alternative strategies. The null hypothesis is that biases would die out, because they lead to faulty assessments and suboptimal behaviour. In fact, a large class of economic models depend on the assumption that biases in beliefs do not exist¹⁷. Underlying this assumption is the idea that there must be some evolutionary or learning process that causes individuals with correct beliefs to be rewarded (and thus to spread at the expense of individuals with incorrect beliefs). However, unbiased decisions are not necessarily the best strategy for maximizing benefits over costs, especially under conditions of competition, uncertainty and asymmetric costs of different types of error^{8, 18, 19, 20, 21}. Whereas economists tend to posit the notion of human brains as general-purpose utility maximizing machines that evaluate the costs, benefits and probabilities of different options on a case-by-case basis, natural selection may have favoured the development of simple heuristic biases (such as overconfidence) in a given domain because they were more economical, available or faster.

 The paper studies this question in a simple analytical model of an evolutionary environment in which individuals compete for resources. If the resources are sufficiently valuable, the authors find, overconfidence can indeed be adaptive:

Here we present a model showing that, under plausible conditions for the value of rewards, the cost of conflict, and uncertainty about the capability of competitors, there can be material rewards for holding incorrect beliefs about one’s own capability. These adaptive advantages of overconfidence may explain its emergence and spread in humans, other animals or indeed any interacting entities, whether by a process of trial and error, imitation, learning or selection. The situation we model—a competition for resources—is simple but general, thereby capturing the essence of a broad range of competitive interactions including animal conflict, strategic decision-making, market competition, litigation, finance and war.

Very interesting. But I just had a thought -- perhaps this may also explain why many economists seem to exhibit such irrational exuberance over the value of neo-classical theory itself?

Leverage control for market stability

I listened today to a number of extremely informative talks at a workshop in Durham (UK) on Tipping Points in Financial Systems (description here part way down the page). I'll make some comments on the various talks in coming days. But it might be worth noting a few observations on some further progress on a model of market volatility -- and its inherent link to leverage -- achieved by Stefan Thurner and colleagues.

I wrote about this work several years ago in an OpEd for the New York Times, and also in this thing for Nature, but today learned about some further developments which seem particularly important. The model developed in this work makes the point that saavy participants in speculative markets (call them "hedge funds," but they could banks or just one individual) can use leverage to deliver higher returns and thereby attract more investors. This is obvious and natural. Many details aside, however, the model showed that the competition between funds to attract investors drives a race to higher leverage, increasing market volatility, and the eventual probability of violent market crashes. Leverage is dangerous and comes with systemic costs.

This can be seen (in an abstract way, sorry) from the figure below from the paper. In a long simulation of the market, this shows that the likelihood of finding market returns (absolute value of the logarithm of prices differences over a short time) exceeding a value R. The red is how the market works when leverage is low -- the probability to see really big market movements, R > 0.1 or so, is extremely small. But as hedge funds evolve to use significant leverage, the market moves into a regime described instead by the blue curve -- the probability of tail events and extreme movements becomes orders of magnitude larger.

The implication is clear: leverage causes volatility.

But Thurner suggested today that intermediate levels of leverage actually reduce market volatility, because it makes it easier for the saavy hedge fund investors to pounce on and wipe out market mispricings. This is an interesting point and one worth pondering. I haven't yet digested the latter parts of the updated paper, which now considers several policy moves and how they influence volatility, but the results have the wonderful ambiguity that one learns to expect in confronting complex systems. For example, capping leverage at intermediate levels (factors of around 10) is in some case worse than capping it at higher levels (around 15). Controls on the capital reserves held by the funds (or banks) also have some ambiguous results -- in some cases, making them hold higher reserves can lead to more volatility in the market, not less. Weird.

I'll try to digest this new work and report on it's implications once I understand them more clearly, but they already demonstrate the point that our intuition isn't so good at seeing the link between interventions in markets and the likely consequences. I'm certainly guilty on occasion of thinking that if the financial industry is against any proposed regulation, then it must be a good one. Often that's not a bad rule of thumb. But if we're really going to make progress in making markets work for everyone, we need to think very carefully -- and back up proposals with hard evidence. This work is developing such evidence.

How derivatives make markets unstable: Part I

I posted a while back on some of the dirty secrets of the derivatives industry. I promised then to give a little more discussion at some point of two terrifically important pieces of research -- still not widely known, especially in mainstream finance -- which show how adding more derivatives to a market can make it less stable, not more stable. This goes directly against the received wisdom of economic (equilibrium) theory which claims that markets become more efficient as they become more complete, i.e. as it becomes possible to take essentially any kind of market position by virtue of a dense spectrum of financial instruments.

One of the papers I had in mind was this landmark study from several years ago in which William Brock, Cars Hommes and Florian Wagener considered the question of whether, in the run up to the recent crisis, "... highly leveraged positions using complex financial instruments may have amplified market volatility." The answer to which their analysis leads is -- yes, quite probably. More generally, they illustrate how more derivatives in general should make markets more unstable, increasing volatility.

Their paper is a little technical, but worth a read. I'll outline the gist of their argument, which starts with several straightforward observations and moves to a not-so-obvious conclusion:

Observation 1: They start by noting that people aren't the hyper-rational automatons of Milton Friedman's (or other neo-classical economists') favorite fantasies. Rather, people in the real world form their expectations and craft their behaviour in an adaptive way -- that is, they learn from experience.

Observation 2: They also note that people aren't identical. We not only learn, but our brains are different and we've all had different experiences in the past, so, at any moment, we've probably learned different things and have slightly different expectations (heterogeneous expectations, in economic lingo) about the future.

Observation 3: People are generally risk averse -- if they're willing to bet $100 on a gamble that could pay off, but involves risks, they'll be willing to bet more than $100 in the same gamble if you reduce the risks. In other words, people shy away from gambles more the riskier they are. This is basic empirical psychology.

Starting from these observations, Brock and colleagues then consider an "intertemporal" asset market (economist-speak meaning a market in which time exists) in which a lot of people look to past prices and try to predict future prices, buying and selling as they see fit. This market contains both risky and non-risky things to invest in -- stocks and risk-free bonds (which are guaranteed to increase in value by a factor R>1 over each interval of time). Stocks might rise more, but are less certain and hence riskier. In addition, the people can buy derivatives -- instruments which act like pure bets and give a pay off in certain circumstances.

What this all amounts to is that people in this market can 1) play it safe by buying bonds, 2) gamble more by buying stocks, and also 3) buy derivatives if they want which (in this model) have no effect except to offset some of the risks involved in buying stocks.

What Brock and colleagues then show is that the combination of the derivatives, the risk aversion of investors, and their tendency to learn by "reinforcement" -- to be more likely to follow strategies which have paid off in the past -- leads directly to trouble. I'll describe how in a moment, but one final thing before I do: the strength of reinforcement learning in the model (how quickly people shift to use better performing strategies) is controlled by one parameter β; bigger β means faster switching. In previous work, Brock and Hommes have shown that in an asset market in which people learn by the reinforcement process, there is a natural "tipping point" -- at a certain critical value of β -- where the market goes from being stable to being unstable. Intuitively, when people switch too quickly, taking even scanty short term evidence as proof of a strategy's superiority, fluctuations in the market become much stronger.

OK, so what happens in this market when you currently have, say, 15 possible derivatives covering lots of different possible outcomes, and now add a 16th derivative to cover other outcomes (i.e. we have derivatives on stocks and commodities, and suddenly invent some new ones to cover mortgage bonds)? Brock and colleagues show that the addition of this one new derivative makes the market go unstable more quickly, i.e. at a lower value of β. The mechanism involves a simple interplay of reduced risk and human confidence. This new derivative, by making it possible for investors to lower the risks associated with investments, leads them to invest more money. They take bigger bets. These bigger bets naturally amplify how quickly the bets that turn out to be correct amass profits. So, there are bigger differences in the payoffs to recent winning and losing strategies, which draws more followers to the winners more quickly (even if the fundamental switching rate of people haven't changed).

In brief: by the very act of reducing the risk of some strategies, the derivative invites more vigorous gambling on that strategy, leading to faster flows of people from one strategy to another. The extra derivative makes the market more volatile.

This model doesn't involve many questionable assumptions. It's a very basic model of the most central facts of any market, respecting some realities of human psychology. It suggests that derivatives hold inherent dangers. Yet as far as I can see, the ongoing discussion of regulating derivatives isn't taking this perspective into account. As Satyajit Das notes, the drive toward greater returns that is an essential part of the dynamics in the Brock, Hommes and Wagener model is a very real force in today's derivatives markets:

Investors searching for return drive speculation. Concerned about stagnant real incomes and inadequate retirement savings, individual investors seek out higher yielding investment structures, often based on derivatives. Pension funds and other institutional investors use derivatives to enhance returns to fully fund and meet their contracted liabilities. In an environment of diminishing returns and fierce competition for attractive investments, fund managers use derivative strategies to enhance returns through readily accessible leverage and capacity to create risk “cocktails”.

Facing increased pressure on earnings, corporations have increasingly “financialised”, resorting to speculative derivative trading to meet profit expectations. ... [Such] seculative activity amplifies rather than reduces volatility and systemic risks. Perversely, this may impede capital formation and also increase the cost of capital for companies.

What happens in the real world backs up the lesson of this simple model. Derivatives reduce risks only in a very narrow and restricted sense, while undermining the functioning of markets more generally. Of course, there's lots of money to be made by the people selling derivatives, so don't expect them to admit (or care about) any of this.