Markets are rational even if they're irrational

I promise very soon to stop beating on the dead carcass of the efficient markets hypothesis (EMH). It's a generally discredited and ill-defined idea which has done a great deal, in my opinion, to prevent clear thinking in finance. But I happened recently on a defense of the EMH by a prominent finance theorist that is simply a wonder to behold -- its logic a true empirical testament to the powers of human rationalization. It also illustrates the borderline Orwellian techniques to which diehard EMH-ers will resort to cling to their favourite idea.

The paper was written in 2000 by Mark Rubinstein, a finance professor at University of California, Berkeley, and is entitled "Rational Markets: Yes or No. The Affirmative Case." It is Rubinstein's attempt to explain away all the evidence against the EMH, from excess volatility to anomalous predictable patterns in price movements and the existence of massive crashes such as the crash of 1987. I'm not going to get into too much detail, but will limit myself to three rather remarkable arguments put forth in the paper. They reveal, it seems to me, the mind of the true believer at work:

1. Rubinstein asserts that his thinking follows from what he calls The Prime Directive. This commitment is itself interesting:

When I went to financial economist training school, I was taught The Prime Directive. That is, as a trained financial economist, with the special knowledge about financial markets and statistics that I had learned, enhanced with the new high-tech computers, databases and software, I would have to be careful how I used this power. Whatever else I would do, I should follow The Prime Directive:

Explain asset prices by rational models. Only if all attempts fail, resort to irrational investor behavior.

One has the feeling from the burgeoning behavioralist literature that it has lost all the constraints of this directive – that whatever anomalies are discovered, illusory or not, behavioralists will come up with an explanation grounded in systematic irrational investor behavior.

Rubinstein here is at least being very honest. He's going to jump through intellectual hoops to preserve his prior belief that people are rational, even though (as he readily admits elsewhere in the text) we know that people are not rational. Hence, he's going to approach reality by assuming something that is definitely not true and seeing what its consequences are. Only if all his effort and imagination fails to come up with a suitable scheme will he actually consider paying attention to the messy details of real human behaviour.

What's amazing is that, having made this admission, he then goes on to criticize behavioural economists for having found out that human behaviour is indeed messy and complicated:

The behavioral cure may be worse than the disease. Here is a litany of cures drawn from the burgeoning and clearly undisciplined and unparsimonious behavioral literature:

Reference points and loss aversion (not necessarily inconsistent with rationality):
Endowment effect: what you start with matters
Status quo bias: more to lose than to gain by departing from current situation
House money effect: nouveau riche are not very risk averse

Overconfidence:
Overconfidence about the precision of private information
Biased self-attribution (perhaps leading to overconfidence)
Illusion of knowledge: overconfidence arising from being given partial information
Disposition effect: want to hold losers but sell winners
Illusion of control: unfounded belief of being able to influence events

Statistical errors:
Gambler’s fallacy: need to see patterns when in fact there are none
Very rare events assigned probabilities much too high or too low
Ellsberg Paradox: perceiving differences between risk and uncertainty
Extrapolation bias: failure to correct for regression to the mean and sample size
Excessive weight given to personal or antidotal experiences over large sample statistics
Overreaction: excessive weight placed on recent over historical evidence
Failure to adjust probabilities for hindsight and selection bias

Miscellaneous errors in reasoning:Violations of basic Savage axioms: sure-thing principle, dominance, transitivity
Sunk costs influence decisions
Preferences not independent of elicitation methods
Compartmentalization and mental accounting
“Magical” thinking: believing you can influence the outcome when you can’t
Dynamic inconsistency: negative discount rates, “debt aversion”
Tendency to gamble and take on unnecessary risks
Overpricing long-shots
Selective attention and herding (as evidenced by fads and fashions)
Poor self-control
Selective recall
Anchoring and framing biases
Cognitive dissonance and minimizing regret (“confirmation trap”)
Disjunction effect: wait for information even if not important to decision
Time-diversification
Tendency of experts to overweight the results of models and theories
Conjunction fallacy: probability of two co-occurring more probable than a single one

Many of these errors in human reasoning are no doubt systematic across individuals and time, just as behavioralists argue. But, for many reasons, as I shall argue, they are unlikely to aggregate up to affect market prices. It is too soon to fall back to what should be the last line of defense, market irrationality, to explain asset prices. With patience, the anomalies that appear puzzling today will either be shown to be empirical illusions or explained by further model generalization in the context of rationality.

Now, there's sense in the idea that, for various reasons, individual behavioural patterns might not be reflected at the aggregate level. Rubinstein's further arguments on this point aren't very convincing, but at least it's a fair argument. What I find more remarkable is the a priori decision that an explanation based on rational behaviour is taken to be inherently superior to any other kind of explanation, even though we know that people are not empirically rational. Surely an explanation based on a realistic view of human behaviour is more convincing and more likely to be correct than one based on unrealistic assumptions (Milton Friedman's fantasies notwithstanding). Even if you could somehow show that market outcomes are what you would expect if people acted as if they were rational (a dubious proposition), I fail to see why that would be superior to an explanation which assumes that people act as if they were real human beings with realistic behavioural quirks, which they are.

But that's not how Rubinstein sees it. Explanations based on a commitment to taking real human behaviour into account, in his view, have "too much of a flavor of being concocted to explain ex-post observations – much like the medievalists used to suppose there were a different angel providing the motive power for each planet." The people making a commitment to realism in their theories, in other words, are like the medievalists adding epicycles to epicycles. The comparison would seem more plausibly applied to Rubinstein's own rational approach.

2. Rubinstein also relies on the wisdom of crowds idea, but doesn't at all consider the many paths by which a crowd's average assessment of something can go very much awry because individuals are often strongly influenced in their decisions and views by what they see others doing. We've known this going all the way back to the famous 1950s experiments of Solomon Asch on group conformity. Rubinstein pays no attention to that, and simply asserts that we can trust that the market will aggregate information effectively and get at the truth, because this is what group behaviour does in lots of cases:

The securities market is not the only example for which the aggregation of information across different individuals leads to the truth. At 3:15 p.m. on May 27, 1968, the submarine USS Scorpion was officially declared missing with all 99 men aboard. She was somewhere within a 20-mile-wide circle in the Atlantic, far below implosion depth. Five months later, after extensive search efforts, her location within that circle was still undetermined. John Craven, the Navy’s top deep-water scientist, had all but given up. As a last gasp, he asked a group of submarine and salvage experts to bet on the probabilities of different scenarios that could have occurred. Averaging their responses, he pinpointed the exact location (within 220 yards) where the missing sub was found. 

Now I don't doubt the veracity of this account or that crowds, when people make decisions independently and have no biases in their decisions, can be a source of wisdom. But it's hardly fair to cite one example where the wisdom of the crowd worked out, without acknowledging the at least equally numerous examples where crowd behaviour leads to very poor outcomes. It's highly ironic that Rubinstein wrote this paper just as the dot.com bubble was collapsing. How could the rational markets have made such mistaken valuations of Internet companies? It's clear that many people judge values at least in part by looking to see how others were valuing them, and when that happens you can forget the wisdom of the crowds.

Obviously I can't fault Rubinstein for not citing these experiments  from earlier this year which illustrate just how fragile the conditions are under which crowds make collectively wise decisions, but such experiments only document more carefully what has been obvious for decades. You can't appeal to the wisdom of crowds to proclaim the wisdom of markets without also acknowledging the frequent stupidity of crowds and hence the associated stupidity of markets.

3. Just one further point. I've pointed out before that defenders of the EMH in their arguments often switch between two meanings of the idea. One is that the markets are unpredictable and hard to beat, the other is that markets do a good job of valuing assets and therefore lead to efficient resource allocations. The trick often employed is to present evidence for the first meaning -- markets are hard to predict -- and then take this in support of the second meaning, that markets do a great job valuing assets. Rubinstein follows this pattern as well, although in a slightly modified way. At the outset, he begins making various definitions of the "rational market":

I will say markets are maximally rational if all investors are rational.

This, he readily admits, isn't true:

Although most academic models in finance are based on this assumption, I don’t think financial economists really take it seriously. Indeed, they need only talk to their spouses or to their brokers.

But he then offers a weaker version:

... what is in contention is whether or not markets are simply rational, that is, asset prices are set as if all investors are rational.

In such a market, investors may not be rational, they may trade too much or fail to diversify properly, but still the market overall may reflect fairly rational behaviour:

In these cases, I would like to say that although markets are not perfectly rational, they are at least minimally rational: although prices are not set as if all investors are rational, there are still no abnormal profit opportunities for the investors that are rational.

This is the version of "rational markets" he then tries to defend throughout the paper. Note what has happened: the definition of the rational market has now been weakened to only say that markets move unpredictably and give no easy way to make a profit. This really has nothing whatsoever to do with the market being rational, and the definition would be improved if the word "rational" were removed entirely. But I suppose readers would wonder why he was bothering if he said "I'm going to defend the hypothesis that markets are very hard to predict and hard to beat" -- does anyone not believe that? Indeed, this idea of a "minimally rational"  market is equally consistent with a "maximally irrational" market. If investors simply flipped coins to make their decisions, then there would also be no easy profit opportunities, as you'd have a truly random market.

Why not just say "the markets are hard to predict" hypothesis? The reason, I suspect, is that this idea isn't very surprising and, more importantly, doesn't imply anything about markets being good or accurate or efficient. And that's really what EMH people want to conclude -- leave the markets alone because they are wonderful information processors and allocate resources efficiently. Trouble is, you can't conclude that just from the fact that markets are hard to beat. Trying to do so with various redefinitions of the hypothesis is like trying to prove that 2 = 1. Watching the effort, to quote physicist John Bell in another context, "...is like watching a snake trying to eat itself from the tail. It becomes embarrassing for the spectator long before it becomes painful for the snake."

The limitations of markets

Economists aren't often as vocal as they should be about the limitations of markets -- especially the extreme assumptions required for them to deliver superior outcomes and some kind of "efficiency." I've documented here before some of the exuberant cheer-leading for the wonders of modern markets that was the norm before the financial crisis of 2008. No self doubt or balanced criticism about the dangers of markets there.

Now, only a few years after the crisis -- and with a global economic crisis just looming up before us -- the old hysteria is again getting underway with calls (especially from US politicians) for more privatization to get the damned inefficient government out of everything. For an intelligent, fact-based perspective, I'm simply going to quote the following extended discussion from economist Mark Thoma. He deserves a medal for saying what most other economists ought to be saying every day to everyone they meet:

To listen to some commentators is to believe that markets are the solution to all of our problems. Health care not working? Bring in the private sector. Need to rebuild a war-torn country? Send in the private contractors. Emergency relief after earthquakes, hurricanes, and tornadoes? Wal-Mart with a contract is the answer.

Whatever the problem, the private sector - markets and their magic - beats government every time. Or so we are told. But this is misplaced faith in markets. There is nothing special about markets per se - they can perform very badly in some circumstances. It is competitive markets that are magic, though even then we have to remember that markets have no concern whatsoever with equity, only efficiency, and sometimes equity can be an overriding concern.

In order to work their magical efficiency, markets need very special conditions to be present. There must be full information available to all participants. Product quality, locations and prices of alternative suppliers, every relevant piece of information must be known. Not quite sure if the wine is good or not? That's an information problem. Not sure if the used car has problems? Don't know where any gas stations are except the ones beside the freeway in a strange town? No way to monitor the quality of the building built in Iraq with U.S. aid? No way to be sure if consultants are worth the amount they are being paid? Information problems are common and they can cause substantial departures from the perfectly competitive, ideal outcome.

There also must be numerous buyers and sellers, enough so that no single buyer or seller's decisions can affect the market price. For example, if a firm can affect the market price by threatening to limit supply, the market does not satisfy this condition. If, as some claim, CEOs are in such short supply that they can individually negotiate their compensation, then the market is not producing an efficient outcome. Whenever there are a small number of participants on either side of the market - suppliers or demanders - this is potentially problematic.

In order for markets to work their magic, the product must be homogeneous. That is, the product or input to production sold by all firms in the market must be perfectly substitutable so that as far as the buyer is concerned, one is as good as the other. If some buyers favor one brand over another, if CEOs are perceived to have different and unique talents, this condition does not hold. In many cases the variety may be worth the inefficiency, not many of us would want just one style and color of shirt to be available in stores, but the inefficiency is there nonetheless.

In order for markets to work their magic there must be free entry and exit. Most people understand free entry, but free exit is sometimes less evident, so let me try to give an example. Starting a blog on Blogger or TypePad is easy. Entry is a snap and you can be up and running in no time at all. It's easy to join the competition and start supplying posts. But suppose that later you decide you want to switch to, say, TypePad from Blogger (or the other way around). That is not so easy. There is no way, at least no simple and convenient way, to export all of your old posts from Blogger and import them into TypePad, a significant barrier to exit if a large number of posts must be moved. Whenever barriers exist in markets that prevent free movement into and out of the marketplace or between firms within a market (on either side - there are sometimes barriers to purchasing as well), markets will underperform.

The list goes on and on. In order for markets to work their magic, there can be no externalities, no public goods, no false market signals, no moral hazard, no principle agent problems, and, importantly, property rights must be well-defined (and I probably missed a few). In general, the incentives that the market provides must be consistent with perfect competition, or nearly so in practical applications. When the incentives present in the marketplace are inconsistent with a competitive outcome, there is no reason to expect the private sector to be efficient.

Markets don't work just because we get out of the way. When government contracts are moved to the private sector without ensuring the proper incentives are in place, there will be problems - waste, inefficiency, higher prices than needed, etc. There is nothing special about markets that guarantees that managers or owners of companies will have an incentive to use public funds in a way that maximizes the public rather than their own personal interests. It is only when market incentives direct choices to coincide with the public interest that the two sets of interests are aligned.

If there is no competition, or insufficient competition in the provision of government services by private sector firms, there is no reason to expect the market to deliver an efficient outcome, an outcome free of waste and inefficiency. Why would we think that giving a private sector firm a monopoly in the provision of a public service would yield an efficient outcome? If the projects are of sufficient scale, or require specialized knowledge so that only one or a few private sector firms are large enough or specialized enough to do the job, why would we expect an ideal outcome just because the private sector is involved? If cronyism limits the participants in the marketplace, why would we expect an outcome that maximizes the public interest?

There is nothing inherent in markets that guarantees a desirable outcome. A market can be a monopoly, a market can be perfectly competitive, a market can be lots of things. Markets with bad incentives produce bad outcomes, markets with good incentives do better.

I believe in markets as much as anyone. But the expression free markets is often misinterpreted to mean that unregulated markets are all that is required for markets to work their wonders and achieve efficient outcomes. But unregulated is not enough, there are many, many other conditions that must be present. Deregulation or privatization may even move the outcome further from the ideal competitive benchmark rather than closer to it, it depends upon the characteristics of the market in question.

For government goods and services, when incentives consistent with a competitive outcome are present, we should get government out of the way and privatize, and there are lots of circumstances where this will be appropriate. There is no reason at all for the government to produce its own pencils and pens, buying them from the private sector is more efficient so long as the bids are competitive.

When competitive conditions are not met but can be regulated, the regulations should be put in place and the private sector left to do its thing (e.g.  mandating that sellers disclose problems with a house to prevent asymmetric information or mandating that government funded projects be subject to competitive bidding and monitoring to ensure contract terms are met). There's no reason for government to do anything except ensure that the incentives to motivate competitive behavior are in place and enforced.

But rampant privatization based upon some misguided notion that markets are always best, privatization that does not proceed by first ensuring that market incentives are consistent with the public interest, doesn't do us any good. There are lots of free market advocates out there and I am with them so long as we understand that free does not mean the absence of government intervention, regulation, or oversight, even libertarians agree that governments must intervene to ensure basics like private property rights. Free means that the conditions for perfect competition are approximated as much as possible and sometimes that means the presence - rather than the absence - of government is required.

Efficiency versus stability

UPDATED BELOW

I had an opinion piece published today in Bloomberg Views looking at the relationship between market efficiency and stability, a topic which hasn't received much attention in the economics literature until recently. The point of the essay was to explore two distinct recent studies which suggest that adding more derivative instruments to markets tends to make them less stable, even if they do push markets toward the ideal of market completeness and efficiency.

I wanted to make available here some further technical information on the two studies I mentioned, but as publication arrived very quickly and I've been pressed with other deadlines I haven't yet managed to write the post as I wanted. However, I can at least offer some information with the idea of updating it very shortly (later today, Thursday 25 August).

I've given some extensive discussion of the first study I mentioned, by economists William Brock, Cars Hommes and Florian Wagener, in an earlier post.

The second study by Matteo Marsili is quite technical and relies for parts of its analysis on ideas and techniques imported from physics. I will tomorrow try to give some simplified discussion of the gist of this argument. What makes this particularly fascinating is that it works fully within the confines of standard general equilibrium models, and examines how market stability should evolve as the market approaches the ideal of market completeness. Agents are assumed to be fully rational, there are no problems with asymmetric information, etc. Even here, however, Marsili finds that the equilibrium becomes more and more unstable as the ideal is approached. Efficient markets are also unstable markets.

UPDATE

Marsili's argument is one he has been developing in a series of papers (with various co-authors) over several years. This paper from last year offers what is perhaps the most concise argument. It looks at a market with informed (fundamentalist) traders and non-informed (noise) traders, and shows, first, that the market becomes efficient as the number of informed traders grows. They are assumed in the model to have different kinds of private information about market outcomes, and the market becomes efficient, roughly speaking, once there are enough traders to cover the space of outcomes so all private information gets aggregated into market prices. The paper then introduces a non-informed trader -- a chartist or trend follower -- and shows that this trader has a maximum impact on the market precisely at the point at which it becomes efficient. The conclusion is very much against standard economic thinking:

[The results suggest} that information efficiency might be a necessary condition for bubble phenomena - induced by the behavior of non-informed traders...

Another paper from two years ago approaches the problem from a slightly different angle. This study looks explicitly at how the proliferation of financial instruments (derivatives) provides more means for diversifying and sharing risks and takes the market to an efficient state. However, it finds that this state is what physicists refer to as a "critical state", which is a state characterized by extreme (essentially infinite) susceptibility to small disturbances. Any small noise stirs up huge fluctuations. Again, efficiency trails instability in its wake. As the paper asserts:

This suggests that the hypothesis of Arbitrage Pricing Theory (the notion that arbitrage works to keep market in an efficient state) may not be compatible with a stable market dynamics.

This paper also makes the important point that market stability really ought to be thought of as a public good because well functioning markets do help everyone. But like most public goods, private individuals acting in their own interests will not likely provide it.

Finally, the paper I discussed in the Bloomberg article is from last year and analyses a model set up specifically so as to include the finance sector. It is very much akin to standard general equilibrium models, and includes essentially two components:

1. There are investors who aim to take their current wealth and preserve it (or make it grow) into the future. They do this by investing in various instruments provided by a sector of financial firms. These investors are assumed to be rational and have full information and they invest their wealth optimally over the set of possible investments.

2. There are financial firms who create the investment instruments and take on risks in supplying them. They also act optimally, and they hedge their risks by trading between themselves. Again, the firms are rational and have full information.

Marsili then studies what happens to this world of investors and financial firms optimally making decisions as the number of different financial instruments grows. The first result confirms expectations -- the financial firms are ever more successful in hedging their risks and they can provide the financial instruments more cheaply. Investors can therefore invest more effectively. The market becomes efficient.

But there are also two unexpected consequences. As Marsili describes them,

As markets approach completeness, however, two "unintended consequences" also arise: equilibrium portfolios develop a marked susceptibility to idiosynchratic shocks and/or parameter uncertainty and hedging engenders divergent trading volumes in the interbank market. Combining these, suggests an inverse relation between financial stability and the size of the financial sector...

In other words, the character of the optimum portfolios for both the investors and the financial firms becomes hugely sensitive to tiny shocks to the economy. As the efficient state is approached, these agents have to work ever harder to adjust their holdings to remain in the optimal condition. The market only remains efficient through an ever faster and more vigorous churning of investment positions. This shows up in the hedging done by the financial firms, where the volume of trading required to remain optimally hedged actually becomes infinite as the market reaches efficiency.

All three of these papers show much the same thing -- efficiency bringing instability along with it. But this latter paper may be the most interesting as it shows directly how the size of the financial sector also naturally explodes as this efficient-unstable regime is approached. The effect sounds suspiciously like what has happened in the past 30 years or so with massive growth in the financial industries in most developed nations.

What I find really remarkable, however, is that all of this comes from the very models that economists have been using for a long time to make arguments about market efficiency. Why did it take a physicist to look at what happens to stability at the same point? This seems bizarre indeed.

Algorithmic trading -- the positive side

In researching a forthcoming article, I happened upon this recent empirical study in the Journal of Finance looking at some of the benefits of algorithmic trading. I've written before about natural instabilities inherent to high-frequency trading, and I think we still know very little about the hazards presented by dynamical time-bombs linked to positive feed backs in the ecology of algorithmic traders. Still, it's important not to neglect some of the benefits algorithms and computer trading do bring; this study highlights them quite well.

This paper asks the question: "Overall, does AT (algorithmic trading) have salutary effects on market quality, and should it be encouraged?" The authors claim to give "the first empirical analysis of this question." The ultimate message coming out is that "algorithmic trading improves liquidity and enhances the informativeness of quotes." In what follows I've given a few highlights -- some points being obvious, others less obvious:

From a starting point near zero in the mid-1990’s, AT (algorithmic trading) is thought to be responsible for as much as 73% of trading volume in the U.S in 2009.

That's no longer news, of course. By now, mid-2011, I expect that percentage has risen to closer to 80%.

Generally, when I think of automated trading, I think of two activities: market makers (such as GETCO) and statistical arbitrage high-frequency traders, of which there are many (several hundred) firms. But this article rightly emphasizes that automated trading now runs through the markets at every level:

There are many different algorithms, used by many different types of market participants. Some hedge funds and broker-dealers supply liquidity using algorithms, competing with designated market-makers and other liquidity suppliers. For assets that trade on multiple venues, liquidity demanders often use smart order routers to determine where to send an order (e.g., Foucault and Menkveld (2008)). Statistical arbitrage funds use computers to quickly process large amounts of information contained in the order flow and price moves in various securities, trading at high frequency based on patterns in the data. Last but not least, algorithms are used by institutional investors to trade large quantities of stock gradually over time.

One very important point the authors make is that it is not at all obvious that algorithmic trading should improve market liquidity. Many people seem to think this is obvious, but there are many routes by which algorithms can influence market behaviour, and they work in different directions:

... it is not at all obvious a priori that AT and liquidity should be positively related. If algorithms are cheaper and/or better at supplying liquidity, then AT may result in more competition in liquidity provision, thereby lowering the cost of immediacy. However, the effects could go the other way if algorithms are used mainly to demand liquidity. Limit order submitters grant a trading option to others, and if algorithms make liquidity demanders better able to identify and pick off an in-the-money trading option, then the cost of providing the trading option increases, and spreads must widen to compensate. In fact, AT could actually lead to an unproductive arms race, where liquidity suppliers and liquidity demanders both invest in better algorithms to try to take advantage of the other side, with measured liquidity the unintended victim.

This is the kind of thing most participants in algorithmic trading do not emphasize when raving about the obvious benefits it brings to markets.

However, the most important part of the paper comes in an effort to track the rise of algorithmic trading (over roughly a five year period, 2001-2006) and to compare this to changes in liquidity. This isn't quite as easy as it might seem because algorithmic trading is just trading and not obviously distinct in market records from other trading:

We cannot directly observe whether a particular order is generated by a computer algorithm. For cost and speed reasons, most algorithms do not rely on human intermediaries but instead generate orders that are sent electronically to a trading venue. Thus, we use the rate of electronic message traffic as a proxy for the amount of algorithmic trading taking place.

 The figure below shows this data, recorded for stocks with differing market capitalization (sorted into quintiles, Q1 being the largest fifth). Clearly, the amount of electronic traffic in the trading system has increased by a factor of at least five over a period of five years:

The paper then compares this to data on the effective bid-ask spread for this same set of stocks, again organized by quintile, over the same period. The resulting figure indeed shows a more or less steady decrease in the spread, a measure of improving liquidity:

So, there is a clear correlation. The next question, of course, is whether this correlation reflects a causal process or not. I won't get into details but what perhaps sets this study apart from others (see, for example, any number of reports by the Tabb Group, which monitors high-frequency markets) is an effort to get at this causal link. The authors do this by studying a particular historical event that increased the amount of algorithmic trading in some stocks but not others.The results suggest that there is a causal link.

The conclusion, then, is that algorithmic trading (at least in the time period studied, in which stocks were generally rising) does improve market efficiency in the sense of higher liquidity and better price discovery. But the paper also rightly ends with a further caveat:

While we do control for share price levels and volatility in our empirical work, it remains an open question whether algorithmic trading and algorithmic liquidity supply are equally beneficial in more turbulent or declining markets. Like Nasdaq market makers refusing to answer their phones during the 1987 stock market crash, algorithmic liquidity suppliers may simply turn off their machines when markets spike downward.

This resonates with a general theme across all finance and economics. When markets are behaving "normally", they seem to be more or less efficient and stable. When they go haywire, all the standard theories and accepted truths go out the window. Unfortunately, "haywire" isn't as unusual as many theorists would like it to be.

** UPDATE **

Someone left an interesting comment on this post, which for some reason hasn't shown up below. I had an email from Puzzler183 saying:

"I am an electronic market maker -- a high frequency trader. I ask you: why should I have to catch the falling knife? If I see that it isn't not a profitable time to run my business, why should I be forced to, while no one else is?

You wouldn't force a factory owner to run their plant when they couldn't sell the end product for a profit. Why am I asked to do the same?

During normal times, bid-ask spreads are smaller than ever. This is directly a product of automation improving the efficiency of trading."

This is a good point and I want to clarify that I don't think the solution is to force anyone to take positions they don't want to take. No one should be forced to "catch the falling knife." My point is simply that in talking about market efficiency, we shouldn't ignore the non-normal times. An automobile engine which uses half the fuel of any other when working normally wouldn't be considered efficient if it exploded every few hours. Judgments of the efficiency of the markets ought to include consideration of the non-normal times as well as the normal.

An important issue is to explore if there is a trade-off between efficiency in "normal times" as reflected in low spreads, and episodes of explosive volatility (the mini flash crashes which seem ever more frequent). Avoiding the latter (if we want to) may demand throwing some sand into the gears of the market (with trading speed limits or similar measures).

But I certainly agree with Puzzler183: no one should be forced to take on individual risks against their wishes.

Making markets (appear) safe -- through more vigorous lobbying

It's as predictable as the Sun rising not long after it sets -- financial firms rightly criticized for creating dangerous systemic risks will do what is natural to protect their turf. No, not by looking deeply at their practices and asking if they actually do create greater risk, but by hiring a slew of lobbyists and image consultants to change the debate and stop any potential regulation in its tracks. As this article in the New York Times describes, now it's the turn of the high-frequency traders to follow this time-honored path (thanks to Alex Bentley at the University of Durham, UK for pointing me to this).

I learned last year that writing about finance isn't like writing about science, which I've been doing for 15 years. Scientists get touchy if you criticize their work, but generally respond with reasons and try to convince you you're wrong. Financial firms respond with threats of lawsuits. I found this out last year when I wrote this article for Wired UK on high-frequency trading and its potential systemic perils. I sent an early draft to the then PR person for GETCO, one big HFT firm, asking for her comments and help so I didn't misrepresent anything. I often find that showing interested parties early drafts of articles gets them to voice their criticisms early, so I can take them into account in later drafts. In this case it didn't work, as the PR person didn't respond with any reasoned argument.. Instead, she went quite ballistic. Even though I hadn't criticized GETCO at all in the piece -- I merely mentioned them as HFT traders, and argued that HFT trading in general may present new kinds of systemic risks -- she threatened to get the lawyers involved if I mentioned GETCO in the article at all.

GETCO is one of the firms mentioned in the NYT article as now hiring lots of lobbyists to prevent any new legislation which might hurt their profits, to hell with the stability of markets as a whole.

To be clear, I don't think these people are evil in any sense. They're trading in a legal way, and what they do brings some clear benefits to markets -- it has lowered spreads over the past decade and has indeed made it possible for many smaller traders to compete with the larger banks. But the HFT traders ought to be honest about that fact that no one -- absolutely no one -- currently knows what kinds of new systemic risks enter a market when it becomes dominated by algorithms making thousands of trades a second. This is new territory, and human intuition just isn't up working out what is likely to happen. Paul Wilmott made this point quite eloquently in an NYT OpEd well before the Flash Crash of 6 May, 2010 proved his concerns to be valid.

Since then, as I've mentioned before, we've had lots of smaller flash crashes, and a really devastating one may strike any day and possibly bring deep damage to the larger economy. Personally, it would seem sensible to put in place a speed limit of one trade per second and be done with it. Do we really need to trade faster than that?

What's Efficient About the Efficient Markets Hypothesis?

The infamous Efficient Markets Hypothesis (EMH) has been the subject of rancorous and unresolved debate for decades. It's often used to assert that markets don't need regulation or oversight because they have a remarkable power to get prices just about right (stocks, bonds and other assets have their correct "fundamental values"), and so never get too much out of balance. Somehow the idea still gets lots of attention even after the recent crisis. Financial Times columnist Tom Harford recently suggested that the EMH gets some things right (markets are "mostly efficient") even if it is also supports unjustified faith in market stability. In a talk, economist George Akerlof took on the question of whether the EMH can be seen to have caused the crisis, and concludes that yes, it could, although there are plenty of other causes as well.

Others have defended the EMH as being unfairly maligned. Jeremy Siegel, for example, argues that the EMH actually doesn't imply anything about prices being right, and insists that, recent dramatic evidence to the contrary, "our economy is inherently more stable" than it was before -- precisely because of modern financial engineering and the wondrous ability of markets to aggregate information into prices. Robert Lucas asserted much the same thing in The Economist, as did Alan Greenspan in the Financial Times. Lucas asserted his view (equivalent to the EMH) that the market really does know best:

The main lesson we should take away from the EMH for policy making purposes is the futility of trying to deal with crises and recessions by finding central bankers and regulators who can identify and puncture bubbles. If these people exist, we will not be able to afford them.

That debate over the EMH persists half century after it was first stated seems to reflect tremendous confusion and disagreement over what the hypothesis actually asserts. As Andrew Lo and Doyne Farmer noted in a paper from a decade ago, it's not actually a well-defined hypothesis that would permit clear and objective testing:

One of the reasons for this state of affairs is the fact that the EMH, by itself, is not a well posed and empirically refutable hypothesis. To make it operational, one must specify additional structure: e.g., investors’ preferences, information structure, etc. But then a test of the EMH becomes a test of several auxiliary hypotheses as well, and a rejection of such a joint hypothesis tells us little about which aspect of the joint hypothesis is inconsistent with the data.

So what does the EMH assert?

In trying to bring some order to the topic, one useful technique is to identify distinct forms of the hypothesis reflecting different shades of meaning frequently in use. This was originally done in 1970 by Eugene Fama, who introduced a "weak" form, a "semi-strong" form and a "strong" form of the hypothesis. Considering these in turn is useful, and helps to expose a rhetorical trick -- a simple bait and switch -- that defenders of the EMH (such as those mentioned above) often use. One version of the EMH makes an interesting claim -- that markets always work very efficiently (and rapidly) in bringing information to bear on prices which therefore take on accurate values. This (as we'll see below) is clearly false. Another version makes the uninteresting and uncontroversial claim that markets are hard to predict. The rhetorical trick is to mix these two in argument and to defend the interesting one by giving evidence for the uninteresting one. In his Economist article, for example, Lucas cites as evidence for information efficiency the fact that markets are hard to predict, when these are very much not the same thing.

Let's look at this in a little more detail. The Weak form of the EMH merely asserts that asset prices fluctuate in a random way so that there's no information in past prices which can be used to predict future prices. As it is, even this weak form appears to be definitively false if it is taken to apply to all asset prices. In their 1999 book A Non-random Walk Down Wall St, Andrew Lo and Craig MacKinley documented a host of predictable patterns in the movements of stocks and other assets. Many of these patterns disappeared after being discovered -- presumably because some market agents began trading on these strategies -- but there existence for a short time proves that markets have some predictability.

Other studies document the same thing in other ways. The simplest argument for the randomness of market movements is that any patterns that exist should be exploited by market participants to make profits. The trading they do should act to remove these patterns. Is this true? Take a look at Figure 1 below, taken from a paper from 2008 by Doyne Farmer and John Geanakoplos. Back in the 1970s, Farmer and others at a financial firm called The Prediction Company identified numerous market signals they could use to try to predict market movements in the future. The figure shows the correlation between one such trading signal and market prices two weeks in advance, calculated from data over a 23 year period. In 1975, this correlation was as high as 15%, and it was still persisting at a level of roughly 5% as of 2008. This signal -- I don't know what it is, as it is a proprietary signal of The Prediction Company -- has long been giving reliable advance information on market movements.

One might try to argue that this data shows that the pattern is indeed gradually being wiped out, but this is hardly anything like the rapid or "nearly instantaneous" action generally supposed by efficient market enthusiasts. Indeed, there's not much reason to think this pattern will be entirely wiped out for another 50 years.

This persisting memory in price movements can also be analyzed more systematically. Physicist Jean-Philippe Bouchaud and colleagues from the hedge fund Capital Fund management have explored the subtle nature of how new market orders arrive in the market and initiate trades. A market order is a request by an investor to either buy or sell a certain volume of an asset. In the view of the EMH, these orders should arrive in markets at random, driven by the randomness of arriving news. If one piece of news is positive for some stock, influencing someone to place a market buy order, there's no reason to expect that the next piece of news is therefore more likely also to be positive and to trigger another. So there shouldn't be any observed correlation in the times when buy or sell orders enter the market. But there is.

What Bouchaud and colleagues found (originally in 2003, but improved on since then) is that the arrivals of these order are correlated and remain so over very long times -- even over months. This means that the sequence of buy or sell market orders isn't at all just a random signal, but is highly predictable. As Bouchaud writes in a recent and beautifully written review: "Conditional on observing a buy trade now, one can predict with a rate of success a few percent above 1/2 that the sign of the 10,000th trade from now (corresponding to a few days of trading) will be again positive."

Hardly the complete unpredictability claimed by EMH enthusiasts. To look at just one more piece of evidence -- from a very long list of possibilities -- we might take an example discussed recently by Gavyn Davies in the Financial Times. He refers to a study by Andrew Haldane of the Bank of England. As Davies writes,

Andy Haldane conducts the following experiment. He estimates the results of an investment strategy in US equities which is based entirely on the past direction of the stockmarket. If the market rises in the period just ended, the strategy buys stocks for the next period, and vice versa. In other words, the strategy simply extrapolates the recent trend in the market. The result? According to Andy, if you had been wise enough to start this procedure with $1 in 1880, you would have consistently shifted in and out of stocks at the right times, and you would now possess over $50,000. Not bad for a strategy which could have been designed in a kindergarten.

Next, Andy tries an alternative strategy based on value. This calculates whether the stockmarket is fundamentally over or undervalued, and buys the market only when value gives a positive signal. The criterion for measuring value is the dividend discount model, first devised by Robert Shiller. If you had been clever enough to devise this measure of value investing in 1880, and had invested $1 at the time, the procedure would have left you with a portfolio now worth the princely sum of 11 cents.

That, according to the weak version of the EMH, shouldn't be possible.

If weakened still further you might salvage some form of the weak hypothesis by saying that "most or many asset prices are difficult to predict," which seems to be true. We might call this the Absurdly Weak form of the EMH, and it seems ridiculous to form such a puffed-up "hypothesis" at all. Does anyone doubt that markets are hard to predict?

But the more serious point with regard to the weak (or absurdly weak) forms of the EMH is that the word "efficient" really has no business being present at all. This word seems to go back to a famous paper by Paul Samuelson, the originator (along with Eugene Fama) of the EMH, who established that prices should fluctuate randomly and be impossible to predict in a market that is "informationally efficient," i.e. in which participants bring all possible information to bear in trying to anticipate the future. If such efficient information processing goes on in the market, then prices will fluctuate randomly. Informational efficiency is what Lucas and others claim the market does, and they take the difficulty of predicting markets as evidence. But it is not, in fact, evidence of anything of the sort.

Think carefully about this. The statement that information efficiency implies random price movements in no way implies the opposite -- that random price movements imply that information is being processed efficiently, although many people seem to want to draw this conclusion. Just suppose (to illustrate the point) that investors in some market make their decisions to buy and sell by flipping coins. Their actions would bring absolutely no information into the market, yet prices would fluctuate randomly and the market would be hard to predict. It would be far better and more honest to call the weak form of the EMH the Random Market Hypothesis or the Market Unpredictability Hypothesis. It is strictly speaking false, as we just noted, although still a useful, crude first approximation. It's about as true as it is to say that water doesn't flow uphill. Yes, mostly, but then, ordinary waves do it at the seaside every day.

So the weak version of the EMH isn't very useful. Perhaps it has some value in dissuading casual investors from thinking it ought to be easy to beat the market, but it's more metaphor than science.

Next up is the "semi-strong" version of the EMH. This asserts that the prices of stocks or other assets (in the market under consideration) reflect all publicly available information, so these assets have the correct values in view of this information.That is, investors quickly pounce on any new information that becomes public, buy or sell accordingly, and the supply and demand in the market works its wonders so prices take their fundamental values (instantaneously, it is often said, or at least very quickly). This version has one big advantage already over the weak form of the EMH -- it actually makes an assertion about information, and so might plausibly say something about the efficiency with which the market absorbs and processes information. However, there are many vague terms here. What do we mean precisely by "public"? How quickly are the prices supposed to reflect the new information? Minutes? Days? Weeks? This isn't specified.

Notice that a hypothesis formulated this way -- as a positive statement that a market always behaves in a certain way -- cannot possibly ever be proven. Evidence that a market works this way today doesn't mean it will tomorrow or did yesterday. Asserting that the hypothesis is true is asserting the truth of an infinite number of propositions -- efficiency for all stocks, for example, and all information at all times. No finite amount of evidence goes any distance whatsoever toward establishing this infinite set of propositions. The only thing that can be tested is whether it is sometimes -- possibly often or even frequently -- demonstrably false that a market is efficient in this sense.

This observation puts into a context an enormous body of studies which purport to give "evidence for" the EMH, going back to Fama's 1970 review. What they all mean is "evidence consistent with" the EMH, but not in any sense "evidence for." In science, you test hypotheses by trying to prove they are wrong, not right, and the most useful hypotheses are those that turn out hardest to find any evidence against. This is very much not the case for the semi-strong EMH.

If markets move quickly to absorb new information, then they should settle down and remain inert in the absence of new information. This seems to be very much not the case. Nearly two decades ago, a classic economic study by Lawrence Summers and others found that of the 50 largest single-day price movements since World War II, most happened on days when there was no significant news, and that news in general seemed to account for only about a third of the overall variance in stock returns. A similar study more recently (2002) found much the same thing: "Many large stock price changes have no events associated with them."

But if we leave aside the most dramatic market events, what about price movements over short times during a single day? Here too the evidence rather strongly contradicts the semi-strong EMH. Bouchaud and his colleagues at Capital Fund Management recently used data for high-frequency trading to test the alleged EMH link between news and price movements far more precisely. Their idea was to study possible links between sudden jumps in the prices of stock prices and possible news items appearing in electronic news feeds, which might, for example, announce new information about a company. Without entering into the technical points, they found that most sudden price jumps took place without any conceivably causal news arriving on the feeds. To be sure, the news entering did cause price movements in many cases, but most large movements happened in the absence of such news.

Finally, we can immediately also dismiss -- with the evidence just cited -- the strong version of the EMH which claims that markets rapidly reflect not only all public information, but all private information as well. In such a market insider trading would be impossible, because insider information gives no one an advantage. If I'm a government regulator about to issue a drilling permit to Exxon for a wildly lucrative new oil field, even my personal knowledge won't permit be to profit by buying Exxon stock in advance of announcing my decision. The market, in effect, can read my mind and tell the future. This is clearly ridiculous.

So it appears that the two stronger versions of the EMH -- which make real claims about how the markets process information -- are demonstrably (or obviously ) false. The weak version is also falsified by masses of data -- there are patterns in the market which can be used to make profits. People are doing it all the time.

The one statement close to the EMH which does have empirical support is that market movements are very difficult to predict because prices do move in a highly erratic, essentially random fashion. Markets sometimes and perhaps even frequently process new information fairly quickly and that information gets reflected in prices. But frequently they do not. And frequently markets move even though there appears to be no new information at all -- as if they simply have rich internal dynamics driven by the expectations, fears and hopes of market participants.

All in all, the EMH then doesn't tell us much. Perhaps Emanuel Dermin, a former physicist who has worked on Wall St. as a "quant" for many years, puts it best: you shouldn't take the thing too seriously, he suggests, but only take it to assert that "it's #$&^ing difficult or well-nigh impossible to systematically predict what's going to happen next." But this, of course, has nothing at all to do with "efficiency." Many economists, lured by the desire to prove some kind of efficiency for markets, have gone a lot further, absurdly so, even trying to make a strength of its own ignorance about markets, indeed enshrining its ignorance as if it were a final infallible theory. Dermin again:

The EMH was a kind of jiu-jitsu response on the part of economists to turn weakness into strength. "I can't figure out how things work, so I'll make that a principle." 

In this sense, on the other hand, I have to admit that the word "efficient" fits here after all. Maybe the word is meant to apply to "hypothesis" rather than "markets." Measured for its ability to wrap up a universe of market complexity and rich dynamic possibilities in a sentence or two, giving the illusion of complete and final understanding on which no improvement can be made, the efficient markets hypothesis is indeed remarkably efficient.