Networks in finance

Just over a week ago, the journal Nature Physics published an unusual issue. In addition to the standard papers on technical physics topics, this issue contained a section with a special focus on finance, especially on complex networks in finance. I'm sure most readers of this blog won't have access to the papers in this issue, so I thought I'd give a brief summary of the papers here.

It's notable that these aren't papers written just by physicists, but represent the outcome of collaborations between physicists and a number of prominent economists (Nobel Prize winner Joseph Stiglitz among them) and several regulators from important central banks. The value of insight coming out of physics-inspired research into the collective dynamics of financial markets is really starting to be recognized by people who matter (even if most academic economists won't wake up to this probably for several decades).

I've written about this work in my most recent column for Bloomberg, which will be published on Sunday night EST. I was also planning to give here some further technical detail on one very important paper to which I referred in the Bloomberg article, but due to various other demands in the past few days I haven't quite managed that yet. The paper in question, I suspect, is unknown to almost all financial economists, but will, I hope, gain wide attention soon. It essentially demonstrates that the theorists' ideal of complete, arbitrage free markets in equilibrium isn't a nirvana of market efficiency, as is generally assumed. Examination of the dynamics of such a market, even within the neo-classical framework, shows that any approach to this efficient ideal also brings growing instability and likely market collapse. The ideal of complete markets, in other words, isn't something we should be aiming for. Here's some detail on that work from something I wrote in the past (see the paragraphs referring to the work of Matteo Marsili and colleagues).

Now, the Nature Physics special issue.

The first key paper is "Complex derivatives," by Stefano Battiston, Guido Caldarelli, Co-Pierre Georg, Robert May and Joseph Stiglitz. It begins by noting that the volume of derivatives outstanding fell briefly following the crisis of 2008, but is now increasing again. According to usual thinking in economics and finance, this growth of the market should be a good thing. If people are entering into these contacts, it must be for a reason, i.e. to hedge their risks or to exploit opportunities, and these deals should lead to beneficial economic exchange. But, as Battiston and colleagues note, this may not actually be true:

By engaging in a speculative derivatives market, players can potentially amplify their gains, which is arguably the most plausible explanation for the proliferation of derivatives in recent years. Needless to say, losses are also amplified. Unlike bets on, say, dice — where the chances of the outcome are not affected by the bet itself — the more market players bet on the default of a country, the more likely the default becomes. Eventually the game becomes a self-fulfilling prophecy, as in a bank run, where if each party believes that others will withdraw their money from the bank, it pays each to do so. More perversely, in some cases parties have incentives (and opportunities) to precipitate these events, by spreading rumours or by manipulating the prices on which the derivatives are contingent — a situation seen most recently in the London Interbank Offered Rate (LIBOR) affair.

Proponents of derivatives have long argued that these instruments help to stabilize markets by distributing risk, but it has been shown recently that in many situations risk sharing can also lead to instabilities.

The bulk of this paper is devoted to supporting this idea, examining several recent independent lines of research which indicate the more derivatives can make market less stable. This work shares some ideas with theoretical ecology, where it was once thought (40 years ago) that more complexity in an ecology should generally confer stability. Later work suggested instead that complexity (at least too much of it) tends to breed instability. According to a number of recent studies, the same seems to be true in finance:

It now seems that the proliferation of financial instruments induces strong fluctuations and instabilities for similar reasons. The basis for pricing complex derivatives makes several conventional assumptions that amount to the notion that trading activity does not feed back on the dynamical behaviour of markets. This idealized (and unrealistic) model can have the effect of masking potential instabilities in markets. A more detailed picture, taking into account the effects of individual trades on prices, reveals the onset of singularities as the number of financial instruments increases.

The remainder of the paper goes on to explore various means that may be taken, through regulations, to try to manage the complexity of the financial network and encourage its stability. Stability isn't something we should expect to occur on its own. It demands real attention to detail. Blind adherence to the idea that "more derivatives is good" is a recipe for trouble.

The second paper in the Nature Physics special issue is "Reconstructing a credit network," by Guido Caldarelli, Alessandro Chessa, Andrea Gabrielli, Fabio Pammolli and Michelangelo Puliga. This work addresses an issue that isn't quite as provocative as the value of the derivatives industry, but the topic may be of extreme importance in future efforts to devise effective financial regulations. The key insight coming from network science is that the architecture of a network -- its topology -- has a huge impact on how influences (such as financial distress) spread through the network. Hence, global network topology is intimately linked up with system stability; knowledge of global structure is absolutely essential to managing systemic risk. Unfortunately, the history of law and finance is such that much of the information that would be required to understand the real web of links between financial institutions remains private, hidden, unknown to the public or to regulators.

The best way to overcome this is certainly to make this information public. When  financial institutions undertake transactions among themselves, the rest of us are also influenced and our economic well being potentially put at risk. This information should be public knowledge, because it impacts upon financial stability, which is a public good. However, in the absence of new legislation to make this happen, regulators can right now turn to more sophisticated methods to help reconstruct a more complete picture of global financial networks, filling in the missing details. This paper, written by several key experts in this technical area, reviews what is now possible and how these methods might be best put to use by regulators in the near future.

Finally, the third paper in the Nature Physics special issue is "The power to control," by Marco Galbiati, Danilo Delpini and Stefano Battiston. "Control" is a word you rarely hear in the context of financial markets, I suppose because the near religion of the "free market" has made "control" seem like an idea of "communists" or at least "socialists" (whatever that means). But regulation of any sort, laws, institutions, even social norms and accepted practices, all of these represent some kind of "control" placed on individuals and firms in the aim, for society at large, of better outcomes. We need sensible control. How to achieve it?

Of course, "control" has a long history in engineering science where it is the focus of an extensive and quite successful "control theory." This paper reviews some recent work which has extended control theory to complex networks. One of the key questions is if the dynamics of large complex networks might be controlled, or at least strongly steered, by influencing only a small subset of the elements making up the network, and perhaps not even those that seem to be the most significant. This is, I think, clearly a promising area for further work. Let's take the insight of a century and more of control theory and ask if we can't use that to help prevent, or give early warnings of, the kinds of disasters that have hit finance in the past decade.

Much of the work in this special issue has originated out of a European research project with the code name FOC, which stands for, well, I'm not exactly sure what it stands for (the project describes itself as "Forecasting Financial Crises" which seems more like FFC to me). In any event, I know some of these people and apart from the serious science they have a nice sense of humor. Perhaps the acronym FOC was even chosen for another reason. As I recall, one of their early meetings a few years ago was announced as "Meet the FOCers." Humor in no way gets in the way of good science.

Why diversification doesn't work

You're standing in your canoe, on a beautiful Canadian lake, taking photos of the wildlife, occasionally fishing. Why standing, not sitting? Well, you've read about those disturbing studies that show how sitting is really bad for your long term health; how every hour of television viewing, for example, takes about 20 minutes off your life expectancy, and why the same is probably true for sitting at the computer, sitting reading a book, whatever. So you're standing and that's OK because you're balanced and stable, with your weight distributed uniformly.

Of course, anyone with even a few minutes of experience in a canoe knows this isn't as safe as it seems. What really matters isn't how well-balanced you are when the canoe rests peacefully, but what happens when a few waves come along, kicked up by rednecks passing in a souped-up bass trawler (I lived in rural Virginia for several years, so I know the experience). As you shift your stance to stay upright, and the boat shifts, that balanced distribution vanishes and you can easily tip. Stability demands balance in the midst of the boat's dynamics, not only in the static peace beforehand.

As it turns out, this same lesson applies to investment portfolios -- a new paper in Nature Scientific Reports shows just how important this insight may be.

Famously, of course, Harry Markowitz introduced the idea of diversification into investing back in the 1950s (at least he formalized the idea, which was probably around long before). Using information on the mathematical correlations between the returns of the different stocks in a portfolio, you can choose a weighted portfolio to minimize the overall portfolio of volatility for any expected return. This is maybe the most basic of all results in mathematical finance.

But it doesn't work; it suffers from the same problem as the balanced man in the canoe. This is clear from any number of studies over the past decade which show that the correlations between stocks change when markets move up or down. If the market suddenly plunges downward, you would hope that your well-diversified portfolio, invested as it is in stocks that tend to move unlike one another, would be OK. But when markets move significantly down (or up), it turns out, the correlations are no longer what they were. Trending markets induce strong correlations among stocks that aren't there beforehand, and aren't obvious from long-term averages. So the risks to a portfolio are actually much larger than the simple diversification analysis suggests -- just as the risk of a canoe tipping is much more than it seems to a man standing balanced on a peaceful lake.

The new paper by physicist Tobias Preis and colleagues makes this point with probably the largest data set used so far, looking at the stocks in the DJIA over about 70 years. It's a fairly simple analysis (modulo some nitty gritty details). Roughly, they look at the correlations between different stocks in the DJIA and see how these correlations depend on the recent average return of the DJIA. Are the correlations stable? Or do they go up as the market begins to move? The figure below showing the average correlation coefficient versus the return indicates that the result is clearly the latter: a trending market, in either direction, induces significant correlations among the DJIA stocks.

One of the interesting things here is that this link holds on many different timescales, from 10 days up through two months. The worrying thing for an investor, of course, is that these correlations make the risks of large losses significantly larger than they would appear to be on the basis of long-term correlations alone. As the authors conclude:

... a “diversification breakdown” tends to occur when stable correlations are most needed for portfolio protection. Our findings, which are qualitatively consistent with earlier findings^{42, 44} but quantitatively different, could be used to anticipate changes in mean correlation of portfolios when financial markets are suffering significant losses. This would enable a more accurate assessment of the risk of losses.

 As any canoeist knows, dynamics really matter.

High-frequency trading, the downside -- Part II

In this post I'm going to look a little further at Andrew Haldane's recent Bank of England speech on high-frequency trading. In Part I of this post I explored the first part of the speech which looked at evidence that HFT has indeed lowered bid-ask spreads over the past decade, but also seems to have brought about an increase in volatility. Not surprisingly, one measure doesn't even begin to tell the story of how HFT is changing the markets. Haldane explores this further in the second part of the speech, but also considers in a little more detail where this volatility comes from.

In well known study back in 1999, physicist Parameswaran Gopikrishnan and colleagues (from Gene Stanley's group in Boston) undertook what was then the most detailed look at market fluctuations (using data from the S&P Index in this case) over periods ranging from 1 minute up to 1 month. This early study established a finding which (I believe) has now been replicated across many markets -- market returns over timescales from 1 minute up to about 4 days all followed a fat-tailed power law distribution with exponent α close to 3. This study found that the return distribution became more Gaussian for times longer than about 4 days. Hence, there seems to be rich self-similarity and fractal structure to market returns on times down to 1 around second.

What about shorter times? I haven't followed this story for a few years. It turns out that in 2007, Eisler and Kertesz looked at a different set of data -- for total transactions on the NYSE between 2000 and 2002 -- and found that the behaviour at short times (less than 60 minutes) was more Gaussian. This is reflected in the so-called Hurst exponent H having an estimated value close to 0.5. Roughly speaking, the Hurst exponent describes -- based on empirical estimates -- how rapidly a time series tends to wander away from its current value with increasing time. Calculate the root mean square deviation over a time interval T and for a Gaussian random walk (Brownian motion) this should grow in proportion to T to the power H= 1/2. A Hurst exponent higher than 1/2 indicates some kind of interesting persistent correlations in movements.

However, as Haldane notes, Reginald Smith last year showed that stock movements over short times since around 2005 have begun showing more fat-tailed behaviour with H above 0.5. That paper shows a number of figures showing H rising gradually over the period 2002-2009 from 0.5 to around 0.6 (with considerable  fluctuation on top of the trend). This rise means that the market on short times has increasingly violent excursions, as Haldane's chart 11 below illustrates with several simulations of time series having different Hurst exponents:

The increasing wildness of market movements has direct implications for the risks facing HFT market makers, and hence, the size of the bid-ask spread reflecting the premium they charge. As Haldane notes, the risk a market maker faces -- in holding stocks which may lose value or in encountering counterparties with superior information about true prices -- grows with the likely size of price excursions over any time period. And this size is directly linked to the Hurst exponent.

Hence, in increasingly volatile markets, HFTs become less able to provide liquidity to the market precisely because they have to protect themselves:

This has implications for the dynamics of bid-ask spreads, and hence liquidity, among HFT firms. During a market crash, the volatility of prices (σ) is likely to spike. From equation (1), fractality heightens the risk sensitivity of HFT bid-ask spreads to such a volatility event. In other words, liquidity under stress is likely to prove less resilient. This is because one extreme event, one flood or drought on the Nile, is more likely to be followed by a second, a third and a fourth. Reorganising that greater risk, market makers’ insurance premium will rise accordingly.

This is the HFT inventory problem. But the information problem for HFT market-makers in situations of stress is in many ways even more acute. Price dynamics are the fruits of trader interaction or, more accurately, algorithmic interaction. These interactions will be close to impossible for an individual trader to observe or understand. This algorithmic risk is not new. In 2003, a US trading firm became insolvent in 16 seconds when an employee inadvertently turned an algorithm on. It took the company 47 minutes to realise it had gone bust.

Since then, things have stepped up several gears. For a 14-second period during the Flash Crash, algorithmic interactions caused 27,000 contracts of the S&P 500 E-mini futures contracts to change hands. Yet, in net terms, only 200 contracts were purchased. HFT algorithms were automatically offloading contracts in a frenetic, and in net terms fruitless, game of pass-the-parcel. The result was a magnification of the fat tail in stock prices due to fire-sale forced machine selling.

These algorithmic interactions, and the uncertainty they create, will magnify the effect on spreads of a market event. Pricing becomes near-impossible and with it the making of markets. During the Flash Crash, Accenture shares traded at 1 cent, and Sotheby’s at $99,999.99, because these were the lowest and highest quotes admissible by HFT market-makers consistent with fulfilling their obligations. Bid-ask spreads did not just widen, they ballooned. Liquidity entered a void. That trades were executed at these “stub quotes” demonstrated algorithms were running on autopilot with liquidity spent. Prices were not just information inefficient; they were dislocated to the point where they had no information content whatsoever.

This simply follow from the natural dynamics of the market, and the situation market makers find themselves in. If they want to profit, if they want to survive, they need to manage their risks, and these risks grow rapidly in times of high volatility. Their response is quite understandable -- to leave the market, or least charge much more for their service. 

Individually this is all quite rational, yet the systemic effects aren't likely to benefit anyone. The situation, Haldane notes, resembles a Tragedy of the Commons in which individually rational actions lead to a collective disaster, fantasies about the Invisible Hand notwithstanding:

If the way to make money is to make markets, and the way to market markets is to make haste, the result is likely to be a race – an arms race to zero latency. Competitive forces will generate incentives to break the speed barrier, as this is the passport to lower spreads which is in turn the passport to making markets. This arms race to zero is precisely what has played out in financial markets over the past few years.

Arms races rarely have a winner. This one may be no exception. In the trading sphere, there is a risk the individually optimising actions of participants generate an outcome for the system which benefits no-one – a latter-day “tragedy of the commons”. How so? Because speed increases the risk of feasts and famines in market liquidity. HFT contribute to the feast through lower bid-ask spreads. But they also contribute to the famine if their liquidity provision is fickle in situations of stress.

Haldane then goes on to explore what might be done to counter these trends. I'll finish with a third post on this part of the speech very soon. 

But what is perhaps most interesting in all this is how much of Haldane's speech refers to recent work done by physicists -- Janos Kertesz, Jean-Philippe Bouchaud, Gene Stanley, Doyne Farmer and others -- rather than studies more in the style of neo-classical efficiency theory. It's encouraging to see that at least one very senior banking authority is taking this stuff seriously.

Efficiency versus stability

UPDATED BELOW

I had an opinion piece published today in Bloomberg Views looking at the relationship between market efficiency and stability, a topic which hasn't received much attention in the economics literature until recently. The point of the essay was to explore two distinct recent studies which suggest that adding more derivative instruments to markets tends to make them less stable, even if they do push markets toward the ideal of market completeness and efficiency.

I wanted to make available here some further technical information on the two studies I mentioned, but as publication arrived very quickly and I've been pressed with other deadlines I haven't yet managed to write the post as I wanted. However, I can at least offer some information with the idea of updating it very shortly (later today, Thursday 25 August).

I've given some extensive discussion of the first study I mentioned, by economists William Brock, Cars Hommes and Florian Wagener, in an earlier post.

The second study by Matteo Marsili is quite technical and relies for parts of its analysis on ideas and techniques imported from physics. I will tomorrow try to give some simplified discussion of the gist of this argument. What makes this particularly fascinating is that it works fully within the confines of standard general equilibrium models, and examines how market stability should evolve as the market approaches the ideal of market completeness. Agents are assumed to be fully rational, there are no problems with asymmetric information, etc. Even here, however, Marsili finds that the equilibrium becomes more and more unstable as the ideal is approached. Efficient markets are also unstable markets.

UPDATE

Marsili's argument is one he has been developing in a series of papers (with various co-authors) over several years. This paper from last year offers what is perhaps the most concise argument. It looks at a market with informed (fundamentalist) traders and non-informed (noise) traders, and shows, first, that the market becomes efficient as the number of informed traders grows. They are assumed in the model to have different kinds of private information about market outcomes, and the market becomes efficient, roughly speaking, once there are enough traders to cover the space of outcomes so all private information gets aggregated into market prices. The paper then introduces a non-informed trader -- a chartist or trend follower -- and shows that this trader has a maximum impact on the market precisely at the point at which it becomes efficient. The conclusion is very much against standard economic thinking:

[The results suggest} that information efficiency might be a necessary condition for bubble phenomena - induced by the behavior of non-informed traders...

Another paper from two years ago approaches the problem from a slightly different angle. This study looks explicitly at how the proliferation of financial instruments (derivatives) provides more means for diversifying and sharing risks and takes the market to an efficient state. However, it finds that this state is what physicists refer to as a "critical state", which is a state characterized by extreme (essentially infinite) susceptibility to small disturbances. Any small noise stirs up huge fluctuations. Again, efficiency trails instability in its wake. As the paper asserts:

This suggests that the hypothesis of Arbitrage Pricing Theory (the notion that arbitrage works to keep market in an efficient state) may not be compatible with a stable market dynamics.

This paper also makes the important point that market stability really ought to be thought of as a public good because well functioning markets do help everyone. But like most public goods, private individuals acting in their own interests will not likely provide it.

Finally, the paper I discussed in the Bloomberg article is from last year and analyses a model set up specifically so as to include the finance sector. It is very much akin to standard general equilibrium models, and includes essentially two components:

1. There are investors who aim to take their current wealth and preserve it (or make it grow) into the future. They do this by investing in various instruments provided by a sector of financial firms. These investors are assumed to be rational and have full information and they invest their wealth optimally over the set of possible investments.

2. There are financial firms who create the investment instruments and take on risks in supplying them. They also act optimally, and they hedge their risks by trading between themselves. Again, the firms are rational and have full information.

Marsili then studies what happens to this world of investors and financial firms optimally making decisions as the number of different financial instruments grows. The first result confirms expectations -- the financial firms are ever more successful in hedging their risks and they can provide the financial instruments more cheaply. Investors can therefore invest more effectively. The market becomes efficient.

But there are also two unexpected consequences. As Marsili describes them,

As markets approach completeness, however, two "unintended consequences" also arise: equilibrium portfolios develop a marked susceptibility to idiosynchratic shocks and/or parameter uncertainty and hedging engenders divergent trading volumes in the interbank market. Combining these, suggests an inverse relation between financial stability and the size of the financial sector...

In other words, the character of the optimum portfolios for both the investors and the financial firms becomes hugely sensitive to tiny shocks to the economy. As the efficient state is approached, these agents have to work ever harder to adjust their holdings to remain in the optimal condition. The market only remains efficient through an ever faster and more vigorous churning of investment positions. This shows up in the hedging done by the financial firms, where the volume of trading required to remain optimally hedged actually becomes infinite as the market reaches efficiency.

All three of these papers show much the same thing -- efficiency bringing instability along with it. But this latter paper may be the most interesting as it shows directly how the size of the financial sector also naturally explodes as this efficient-unstable regime is approached. The effect sounds suspiciously like what has happened in the past 30 years or so with massive growth in the financial industries in most developed nations.

What I find really remarkable, however, is that all of this comes from the very models that economists have been using for a long time to make arguments about market efficiency. Why did it take a physicist to look at what happens to stability at the same point? This seems bizarre indeed.

Algorithmic trading -- the positive side

In researching a forthcoming article, I happened upon this recent empirical study in the Journal of Finance looking at some of the benefits of algorithmic trading. I've written before about natural instabilities inherent to high-frequency trading, and I think we still know very little about the hazards presented by dynamical time-bombs linked to positive feed backs in the ecology of algorithmic traders. Still, it's important not to neglect some of the benefits algorithms and computer trading do bring; this study highlights them quite well.

This paper asks the question: "Overall, does AT (algorithmic trading) have salutary effects on market quality, and should it be encouraged?" The authors claim to give "the first empirical analysis of this question." The ultimate message coming out is that "algorithmic trading improves liquidity and enhances the informativeness of quotes." In what follows I've given a few highlights -- some points being obvious, others less obvious:

From a starting point near zero in the mid-1990’s, AT (algorithmic trading) is thought to be responsible for as much as 73% of trading volume in the U.S in 2009.

That's no longer news, of course. By now, mid-2011, I expect that percentage has risen to closer to 80%.

Generally, when I think of automated trading, I think of two activities: market makers (such as GETCO) and statistical arbitrage high-frequency traders, of which there are many (several hundred) firms. But this article rightly emphasizes that automated trading now runs through the markets at every level:

There are many different algorithms, used by many different types of market participants. Some hedge funds and broker-dealers supply liquidity using algorithms, competing with designated market-makers and other liquidity suppliers. For assets that trade on multiple venues, liquidity demanders often use smart order routers to determine where to send an order (e.g., Foucault and Menkveld (2008)). Statistical arbitrage funds use computers to quickly process large amounts of information contained in the order flow and price moves in various securities, trading at high frequency based on patterns in the data. Last but not least, algorithms are used by institutional investors to trade large quantities of stock gradually over time.

One very important point the authors make is that it is not at all obvious that algorithmic trading should improve market liquidity. Many people seem to think this is obvious, but there are many routes by which algorithms can influence market behaviour, and they work in different directions:

... it is not at all obvious a priori that AT and liquidity should be positively related. If algorithms are cheaper and/or better at supplying liquidity, then AT may result in more competition in liquidity provision, thereby lowering the cost of immediacy. However, the effects could go the other way if algorithms are used mainly to demand liquidity. Limit order submitters grant a trading option to others, and if algorithms make liquidity demanders better able to identify and pick off an in-the-money trading option, then the cost of providing the trading option increases, and spreads must widen to compensate. In fact, AT could actually lead to an unproductive arms race, where liquidity suppliers and liquidity demanders both invest in better algorithms to try to take advantage of the other side, with measured liquidity the unintended victim.

This is the kind of thing most participants in algorithmic trading do not emphasize when raving about the obvious benefits it brings to markets.

However, the most important part of the paper comes in an effort to track the rise of algorithmic trading (over roughly a five year period, 2001-2006) and to compare this to changes in liquidity. This isn't quite as easy as it might seem because algorithmic trading is just trading and not obviously distinct in market records from other trading:

We cannot directly observe whether a particular order is generated by a computer algorithm. For cost and speed reasons, most algorithms do not rely on human intermediaries but instead generate orders that are sent electronically to a trading venue. Thus, we use the rate of electronic message traffic as a proxy for the amount of algorithmic trading taking place.

 The figure below shows this data, recorded for stocks with differing market capitalization (sorted into quintiles, Q1 being the largest fifth). Clearly, the amount of electronic traffic in the trading system has increased by a factor of at least five over a period of five years:

The paper then compares this to data on the effective bid-ask spread for this same set of stocks, again organized by quintile, over the same period. The resulting figure indeed shows a more or less steady decrease in the spread, a measure of improving liquidity:

So, there is a clear correlation. The next question, of course, is whether this correlation reflects a causal process or not. I won't get into details but what perhaps sets this study apart from others (see, for example, any number of reports by the Tabb Group, which monitors high-frequency markets) is an effort to get at this causal link. The authors do this by studying a particular historical event that increased the amount of algorithmic trading in some stocks but not others.The results suggest that there is a causal link.

The conclusion, then, is that algorithmic trading (at least in the time period studied, in which stocks were generally rising) does improve market efficiency in the sense of higher liquidity and better price discovery. But the paper also rightly ends with a further caveat:

While we do control for share price levels and volatility in our empirical work, it remains an open question whether algorithmic trading and algorithmic liquidity supply are equally beneficial in more turbulent or declining markets. Like Nasdaq market makers refusing to answer their phones during the 1987 stock market crash, algorithmic liquidity suppliers may simply turn off their machines when markets spike downward.

This resonates with a general theme across all finance and economics. When markets are behaving "normally", they seem to be more or less efficient and stable. When they go haywire, all the standard theories and accepted truths go out the window. Unfortunately, "haywire" isn't as unusual as many theorists would like it to be.

** UPDATE **

Someone left an interesting comment on this post, which for some reason hasn't shown up below. I had an email from Puzzler183 saying:

"I am an electronic market maker -- a high frequency trader. I ask you: why should I have to catch the falling knife? If I see that it isn't not a profitable time to run my business, why should I be forced to, while no one else is?

You wouldn't force a factory owner to run their plant when they couldn't sell the end product for a profit. Why am I asked to do the same?

During normal times, bid-ask spreads are smaller than ever. This is directly a product of automation improving the efficiency of trading."

This is a good point and I want to clarify that I don't think the solution is to force anyone to take positions they don't want to take. No one should be forced to "catch the falling knife." My point is simply that in talking about market efficiency, we shouldn't ignore the non-normal times. An automobile engine which uses half the fuel of any other when working normally wouldn't be considered efficient if it exploded every few hours. Judgments of the efficiency of the markets ought to include consideration of the non-normal times as well as the normal.

An important issue is to explore if there is a trade-off between efficiency in "normal times" as reflected in low spreads, and episodes of explosive volatility (the mini flash crashes which seem ever more frequent). Avoiding the latter (if we want to) may demand throwing some sand into the gears of the market (with trading speed limits or similar measures).

But I certainly agree with Puzzler183: no one should be forced to take on individual risks against their wishes.

Leverage Control -- A Subtle Story

I mentioned recently some work (in progress) by Stefan Thurner and colleagues exploring how leverage influences stability (price volatility) in a competitive, speculative market. Thurner spoke about this at a meeting on Tipping Points in Durham, UK. What I find most appealing about this work is that is explores this question with a model that is rich enough to exhibit many of the basic features we see in speculative markets -- competition between hedge funds and other investment firms to attract investors' funds, the use of leverage to amplify potential gains, the monitoring of leverage by banks who lend to the investment firms, occasional abrupt crashes and bankruptcies, etc.

Is it a perfect model? Of course not, there is no such thing; models are tools for thinking. But it is arguably better than anything else we currently have for running "policy experiments" to test what might happen in such a market if regulators take this or that step -- establishing tight limits to allowed leverage, for example. 

Stefan kindly sent me the slides from his talk, a few of which I'd like to mention here. As I said, this is work in progress, so these are preliminary results. They're interesting because they suggest that avoiding dangerous market instability through leverage limits comes with costs, and that our intuition isn't at all a reliable guide -- we need these kinds of models in which we can discover surprising outcomes (before we discover them in reality).

I won't give a detailed description of the model; it can be found in an early draft of the paper available here. Thurner and colleagues have been working to improve the model over several years, and it now reproduces a number of realistic market behaviors quite naturally. Thurner summarized these as follows:

 

In other words, the hedge funds act to eliminate mis-pricings (taking volatility out of the market), and profit by doing so. Funds have to be aggressive to survive in the face of stuff competition, but suffer if they get too large. Risks shorten the lifetime of a fund. Overall, the models also reproduces the right statistical fluctuations in the market.

As I discussed in my earlier post in this work, competition between hedge funds leads naturally to increasing leverage and drives the market to have a fat-tailed distribution of returns; it becomes subject (like real market) to large price fluctuations as a matter of course driven by its own internal dynamics (no external impacts required). In this condition, the market is highly prone to catastrophic crashes triggered by nothing by small price fluctuations linked to noise traders (unsophisticated investors buying and selling more or less at random). The figure below shows a typical example, plotting the wealth of various funds versus time, with a dramatic crash that affects all funds at once (different colors for different funds):

Now, a natural question is -- could these kinds of events be avoided with proper regulations? One idea would be to restrict the amount of leverage allowed with the aim of keeping the market returns in a mode Gaussian regime, i.e. eliminating fat tails. People could probably argue for decades about whether this would work or not without coming to an answer; this model makes it possible to do an experiment to find out, which is what Thurner and colleagues have done.

Two figures (below) show some of the results, and require some explanation. The different colors correspond to different possible regulatory regimes, and show how behavior changes with maximum allowed hedge fund leverage : BLUE (no other regulations), PALE GREEN (regulations akin to Basel I and II, in which banks loaning to hedge funds are restricted by capital requirements) and RED (a situation in which banks monitor hedge funds and reduce a hedge fund's allowed leverage below the maximum when the volatility in its assets grows; a kind of adaptive leverage control). 

First, consider a figure showing how how the action of hedge funds, and their use of volatility, actually benefits the market -- making it more efficient (in one sense). 

The figure shows the mean square price volatility versus allowed leverage. Increasing leverage lets the hedge funds pounce on opportunities more aggressively and wipe out mis-pricings more effectively. Die hard free market people should love this as it shows that the effect is strongest in the absence of any regulation. The regulated markets require higher leverage to get the same reduction in volatility.

But this isn't the whole story. Now consider another figure for the probability (per unit time) of a failure of one of the hedge funds:

Here the pure free market solution isn't so good, as this probability rises rapidly with increasing leverage. There is a relatively low value of leverage (around 5 in the model's units) where the market benefits of leverage have already been realized, and more leverage only leads to more failures (because it takes the market into the regime of fat-tailed returns; this can happen even if the mean square volatility remains small).

The regulated markets in this case perform marginally better -- the regulations reduce the number of failures, and the cost for this is marginally increased volatility.

A surprising outcome is that these same regulations, in the regime of very high leverage, actually do worse than no regulations at all -- they lead to higher market volatility AND more failures as well, a truly perverse regime.

All in all, then, this model offers a sobering perspective on how regulators might go about trying to avoid crashes linked to fat tails by limiting leverage. Some limitation clearly seems to be good. But too much can be bad, especially when coupled with other market regulations. You can't test out one idea in isolation, because they interact in surprising ways.

I'll probably have some further comments on this in the near future. It's a work in progress, as is my understanding of it -- and of what it means for the bigger picture.