How do you view the relationship between Technological Innovation and Job Creation?  A net positive, a net negative, net neutral?

The relationship between technological innovation and job creation is threefold:

1. Technological innovation creates (relatively few) high-paying jobs for people who develop, deploy, and maintain the new technology (including not only engineers, but salesmen and marketers as well),
2. Technological innovation makes certain types of jobs obsolete, but
3. The well-paid engineers, salesmen and marketers have a demand for everything a well-paid person tends to buy, from houses and cars to designer coffees and sushi restaurants to modern health care. Hence, the relevant industries experience job growth. Note that this job growth can occur in both unskilled (retail salespeople) and highly skilled (doctors and nurses) occupations…

As to whether it is a “net positive” or “net negative” process, the answer is, IN TERMS OF WHAT? The sheer number of jobs has steadily increased, as technological progress tends to lead to workforce shifting out of more productive (meaning, highly automated) industries into less productive (meaning, hard to automate) ones.

The effect on wages, meanwhile, has been diverging. Workers who operate expensive machinery (say, locomotive operators) and workers who require extensive training (as in, for example, registered nurses) see their wages increase over time; those who don’t, face wage stagnation in nominal terns and thus its slow deterioration in real terms…

for /f %%a in ('xcopy "%source%" "%destination%" /L /Y') do (
  if not exist "%destination%.\%%~nxa" xcopy "%%a" "%destination%" /Y
)

Neat…

Inside the Machine: A Journey into the World of High-Frequency Trading

10 Jun 2010
Michael Peltz

An editor’s journey into the world of high-speed trading and proprietary algorithms that make or break markets.

At 2:45p.m. on Thursday, May 6, George (Gus) Sauter received a frantic call from one of his traders to get in front of a Bloomberg terminal. The Dow Jones industrial average, already down 3.9 percent that day on fears about Greece, was in free fall. In just five minutes the index plunged 573 points. Less than two minutes later, the Dow had rocketed back up 543 points, going on to finish the day down 3.2 percent.

“It was just crazy,” Sauter, chief investment officer of mutual fund giant Vanguard Group, told me a few days later. “I had to go to our fixed-income building, about a five-minute walk from my office. By the time I got there, the market had rallied.”

Crazy, indeed. The aptly named “flash crash” temporarily wiped out more than a half trillion dollars in equity value, shaking what little faith nervous investors had in U.S. markets. Shares of Dow component Procter & Gamble Co., the ultimate defensive blue-chip stock, dropped more than one third in a matter of minutes before recovering almost as quickly, all for no apparent reason. A few other large U.S. companies, including accounting firm Accenture, saw their stocks trade as low as a penny a share, only to close not far from where they had begun the day (nearly $42 a share in the case of Accenture) — again, on no news. By the time the dust settled, a whopping 19.3 billion shares had changed hands, more than twice the average daily U.S. equity market volume this year and the second-biggest trading day ever.

But for me, the single most amazing fact about the flash crash was that no one had a clue as to what had triggered it. Not that I should have been surprised, based on the conversation I’d had two days earlier with Mary Schapiro, chairman of the Securities and Exchange Commission. Schapiro, whose organization is charged with maintaining “fair and orderly” markets, explained to me how the SEC did a detailed study after the October 1987 crash to reconstruct what had happened. “We’ve lost some of the capacity to do that given the dramatic volumes of trading that exist today,” Schapiro said. “But we need to be able to do that to understand where are the vulnerabilities in our marketplace and what are the practices that have the potential to hurt investors and the marketplace in the long run.”

In 1987 the SEC had a much easier task because the vast majority of listed U.S. equities were traded in one place — on the floor of the New York Stock Exchange, where specialists employed by the Big Board’s member firms made a market based on an open-outcry auction system. Today, as a result of a series of regulatory changes designed to increase competition and make the market fairer for mom-and-pop investors, only about one quarter of all U.S. equity trading occurs through the now publicly held NYSE Euronext. And the majority of that trading is done electronically, either by the new NYSE floor specialists, called designated market makers, or on the fully automated NYSE Arca platform. The rest of the trading in U.S. equities is spread across a wide range of venues, including the three other major exchanges (Nasdaq Stock Market, BATS Exchange and Direct Edge) and dozens of broker-dealer-operated trading systems, electronic communications networks (ECNs) and dark pools, where buyers and sellers are matched up anonymously.

[Click here to view the Top 25 Most-Favored Stocks In High-Frequency Trading.]

The past decade of fragmentation and automation has given rise to a whole new type of professional trading firm: one that uses sophisticated computer algorithms, often running on servers housed right next to exchanges’ own machines, and high-speed market data feeds to buy and sell securities in rapid-fire fashion. Some of these high frequency traders place hundreds of millions, even billions, of buy and sell orders a day, continually canceling and replacing them, and are likely to be on the other side of your trade. Not that you’d know who they are — proprietary trading firms are not required to disclose their identity — or recognize their names. The bulge bracket of high frequency trading includes firms like Allston Trading, DRW Holdings, Global Electronic Trading Co. (Getco), Hudson River Trading, Quantlab Financial, RGM Advisors, Sun Trading, Tower Research Capital and Tradebot Systems.

High frequency trading has become a multibillion-dollar business, accounting for an estimated 50 to 70 percent of the total U.S. equity market volume on any given day. Since last summer it has also been a lightning rod for the populist anger directed at Wall Street, despite the fact that most of the largest high frequency firms operate far from the canyons of lower Manhattan, in places like Chicago, Kansas City and Austin, Texas. Critics accuse high frequency traders of being fair-weather market makers who, unlike the former NYSE specialists they’ve largely replaced, don’t have a legal obligation to trade during periods of stress. They also say that the growth in high frequency trading has created a two-tiered market of technology haves and have-nots that is unfair to long-term investors and poses potential systemic risks.

I grew interested in high frequency trading last year when I was writing a feature on hedge fund firm Citadel Investment Group (more on that later). As an editor, however, it wasn’t until January that I was able to dig into what I soon learned is an incredibly arcane world. My first stop was a company called Pragma Securities, an agency-only brokerage firm that aggregates more than 40 different dark pools, electronic trading venues and open market destinations into a single liquidity source for clients. Douglas Rivelli and David Mechner, Pragma’s co-CEOs, spent two hours at the firm’s spacious New York offices taking me through that world.

High frequency traders, Rivelli and Mechner explained, generally fall into one of two camps: proprietary trading shops that act as electronic market makers, using computers to generate and adjust buy and sell orders automatically throughout the day, and hedge funds that specialize in statistical arbitrage, seeking to exploit pricing inefficiencies among different securities and asset classes. The distinctions between the two sometimes blur, however, as proprietary trading firms often try to capitalize on some of the same buy and sell signals that statistical arbitrageurs use and hedge funds trade on ever-shorter time horizons. High frequency firms are best known for trading equities, but they also trade futures, options and foreign exchange — basically, anything that can be traded electronically. High frequency trading is also an increasingly global phenomenon, gaining ground in both Europe and Asia.

One thing is clear: Hedge funds don’t like to be called high frequency traders, as I quickly discovered after visiting with some of the biggest quantitative managers, including AQR Capital Management in Greenwich, Connecticut, and D.E. Shaw & Co. and Renaissance Technologies Corp. in New York.

In the wake of the flash crash, as people scrambled to determine what had triggered the market plunge, it didn’t take much longer than the 400 to 600 microseconds (millionths of a second) that high frequency traders typically need to identify and place a trade for fingers to start pointing at them. “The potential for giant high-speed computers to generate false trades and create market chaos reared its head again today,” Delaware Senator Ted Kaufman said in a statement released that same afternoon. When I caught up with the Democratic lawmaker a week later, he was even more incensed, pointing out that regulators still didn’t know what had caused the flash crash.

“We have a 300-pound gorilla in the room, and we’re saying that we’re going to keep it in a cage somewhere,” he told me. “This thing will be 600 pounds.”

“But isn’t part of the problem that there are 300 gorillas?” I asked, referring to the fact that an estimated 200 to 400 firms do high frequency trading.

“Good point,” he replied. “We have all these gorillas, and guess what? We put them in zoos where the people running the zoos don’t have enough information and authority to take care of them.”

Kaufman’s interest in high frequency trading predates mine. When he was sworn into office in January 2009 to fill the Senate seat of his former boss Joe Biden, Kaufman was hell-bent on making sure that everybody responsible for the 2008 market meltdown paid for their actions. He soon focused on short-selling, urging the SEC to reinstate the uptick rule requiring short sales to be filled at a higher price; the rule had been eliminated in 2007. He told me that when he was in business school in the 1960s, it was “an article of faith” that the uptick rule was “one of the two or three things that helped deal with predatory bear raids.” As a result of his interest in short-selling, Kaufman said, his office started getting calls from some fairly sophisticated people, including former Wall Streeters, telling him that if he thought that practice was bad, he should look at high frequency trading.

Kaufman likes to draw an analogy between high frequency trading and the swaps market. “With synthetic derivatives, you had a lot of money at stake, no transparency and then a major meltdown,” he explained to me. “If you look at high frequency trading, I think the same Kaufman formula works.”

A graduate of the Wharton School of the University of Pennsylvania, the 71-year-old Kaufman is a quick study and understands markets. If I were a high frequency trader, I’d take him seriously.

Kaufman has been high frequency trading’s loudest critic. But he’s far from alone. Seth Merrin, founder and CEO of Liquidnet Holdings, which operates an electronic marketplace that provides block trading for institutional investors, likes to compare high frequency traders to the American army during the Revolutionary War. “The institutions are the equivalent of the British army, walking down the battlefield wearing bright red,” he told me back in March in his glass-enclosed office at Liquidnet’s sleek midtown Manhattan headquarters. “The high frequency traders are the Americans hiding in the woods in camouflage, picking them off. If the British army hadn’t changed its tactics, they would have lost every subsequent war.”

Even Duncan Niederauer, who as the pragmatic CEO of NYSE Euronext has been retooling his exchange to attract more business from high frequency traders, took a swipe at them. “We as an industry have to say how much is too much of this technology,” he said during an interview on CNBC after the flash crash, undoubtedly causing some consternation among the folks at NYSE Euronext who are selling space in the company’s new, 400,000-square-foot data center and co-location facility in Mahwah, New Jersey.

High frequency traders say that any efforts to rein in technology would be misplaced. Although speed is important to what they do, the quality of a firm’s computer models for analyzing markets and identifying where and at what price to buy and sell securities is what really determines success or failure, they argue. In their defense, high frequency traders say that they increase liquidity, lower trading costs, improve price discovery and reduce risk by dampening short-term volatility.

“High frequency trading is the liquidity backbone of the equity markets,” Manoj Narang, the founder, CEO and chief investment strategist of Tradeworx, told me when I first met him, in early March. “Long-term investors are the ones who cause bubbles, as well as liquidity crises when these bubbles burst.”

Narang, 40, is one of only a handful of proprietary traders I found willing to talk openly with a journalist about what they do. Most prefer to operate in the shadows, both to protect their valuable algorithms and to avoid regulatory scrutiny. But Narang, who left Wall Street in 1999 to start Tradeworx, sought me out when he heard through a public relations contact this winter that I was working on a story on high frequency trading. His 25-person firm, which operates out of an office above a Restoration Hardware store in Red Bank, New Jersey, trades about 40 million shares a day on about $6 million in proprietary capital. Tradeworx also runs a $500 million statistical arbitrage hedge fund (which trades another 40 million shares a day) and owns a subsidiary, Thesys Technologies, which licenses its high-performance trading platform to other investors.

Narang lifted his profile on May 6 when he revealed to the Wall Street Journal that his firm turned off its high frequency trading computers during the flash crash. Tradeworx wasn’t the only one to do so. Kansas City–based Tradebot, started by BATS founder David Cummings, also stopped trading. Tradebot is one of the world’s two largest high frequency firms, reportedly trading as many as 1 billion shares a day in U.S. equities. Only Chicago-based Getco is thought to be bigger. Although Getco won’t comment on its daily trading volume, a spokeswoman for the firm did tell me that it continued to provide a two-sided market on all the electronic exchanges during the flash crash.

Most high frequency traders, in fact, kept their computers running, according to Jeffrey Wecker, president and CEO of Lime Brokerage. Wecker should know. His firm, which accounts for as much as 5 percent of the daily equity trading volume in the U.S., is the oldest and largest provider of high-speed trading solutions and access to all major U.S. exchanges for high frequency traders.

The high frequency firms that did stop trading on May 6 have been criticized for contributing to the decline by pulling liquidity from the market when it was needed most. But Narang told me that his firm had no choice because the exchanges were likely to cancel, or break, trades that were clearly erroneous (like selling Accenture at a penny a share). “If the exchanges broke all our buys and not our sells, we could have exceeded our capital requirements,” he explained. “We didn’t want to take the risk. The high frequency traders who continued to trade that afternoon made a fortune.”

IN JANUARY THE SEC PUBLISHED A CONCEPT RELEASE on equity market structure, seeking public comment on everything from high frequency trading strategies and systemic risks to co-location and dark pools. At 74 pages, the report might seem like a real snoozer, but it’s actually a great primer on how the U.S. equity markets have responded to regulatory changes, starting in 1996 with the adoption of “order-handling” rules. These new rules, which were designed to make the markets fairer following the Nasdaq price-fixing scandal in the mid-’90s, created ECNs and gave them the power to publish their stock quotes publicly alongside those of the listed markets. In 1999, Regulation Alternative Trading System (ATS) went into effect, enabling ECNs to operate as market centers without having to register as exchanges. By the following year ECNs like Island and Archipelago had taken about one third of market makers’ volume in Nasdaq-listed stocks. But it wasn’t until after April 2001 — the deadline the SEC mandated for all U.S. exchanges to switch from fractions to decimals — that electronic trading really started to take off.

As bid-offer spreads shrunk and competition increased, ECNs and exchanges adopted a “maker-taker” pricing scheme to attract liquidity. Under the maker-taker model, market participants that offer to provide, or make, liquidity by posting an order to buy or sell a certain number of shares at a particular price receive a rebate. Those that execute against that order — that is, take the liquidity — have to pay a fee. Exchanges earn the difference between the rebate they pay and the fee they charge. The SEC limits taker fees to 0.30 cents a share; rebates tend to be lower for economic reasons, but for high frequency firms trading millions of shares a day, they can make for a pretty good living.

“The maker-taker model created an arbitrage that provided incentive for those firms that could properly blend together knowledge of trading, financial economics and computers all into a single, scalable system that could handle high volumes of transactions,” Lime Brokerage CEO Wecker told me back in April when we met at his firm’s Greenwich Village offices.

The final major regulatory change was Regulation NMS (for “National Market System”), which was passed by the SEC in 2005 and went into effect in 2007. One of the key pieces of Reg NMS is the “trade-through” rule prohibiting any exchange from executing a trade at an inferior price to one quoted at another trading venue. Trade-through protection is critical for high frequency traders; it ensures that if they are the first to post the best price for a stock, they’ll get the trade.
“High frequency trading is a product of Reg NMS, decimalization and technology improvements,” says John Knuff, general manager of global financial markets for Equinix, a Foster City, California–based company that runs 87 data centers in 35 key metropolitan areas around the world. “High frequency traders are the democratic enforcers of Reg NMS’s trade-through rules.”

Under Reg NMS, exchanges are required to handle electronic orders immediately or risk having them redirected to other venues. Once the rule was adopted, the NYSE — which even after decimalization had been clinging desperately to its manual specialist system — had no choice but to embrace automation. In 2007 the NYSE switched to a system it called the Hybrid Market, expanding its automatic execution facility, Direct+, and giving specialists the power to create their own algorithms to quote and trade electronically. The hybrid system included circuit breakers, called liquidity replenishment points, that would be triggered if a stock experienced a large price swing, at which time automated trading would stop and human specialists would step in.

That’s exactly what happened on the afternoon of May 6, when the NYSE imposed a trading slowdown in Big Board stocks like P&G and Accenture. Investors who wanted to sell or buy were forced to go to other electronic exchanges or ECNs. Although Nasdaq OMX Group CEO Robert Greifeld and other competitors criticized the NYSE on May 6 for making the meltdown worse, NYSE Euronext CEO Niederauer staunchly defended its actions, pointing out that the exchange did exactly what it should have under Reg NMS. He was vindicated two weeks later when the SEC proposed instituting similar circuit breakers for all exchanges that would pause trading in any stock in the Standard & Poor’s 500 index if its price moved 10 percent or more in a five-minute period. The new circuit breaker rule, which is likely to be approved by the SEC this month, would go into effect on a pilot basis through December 10, at which point the regulator could expand it to other stocks and exchange-traded funds.

“When a stock gets overheated, some form of stock-specific circuit breaker is a very effective means for letting information repopulate in the marketplace,” Lime’s Wecker told me a few days after the flash crash. He advocates an initial short-term cooling-off period measured in seconds or minutes; if a stock suffers a subsequent steep decline in price, the next halt would be longer.

Like many of the people I have interviewed about high frequency trading over the past five months, Wecker is concerned that regulators could try to rein in the practice by putting limits on technology. After all, his business is built on speed. “The challenge with speed bumps is that you are slowing down innovation to accommodate the players who have no interest in investing in innovation,” says Wecker, 47, who spent 11 years at Goldman, Sachs & Co., including six in its famed quantitative trading group, before eventually moving to Lehman Brothers to build its electronic trading group. He left Lehman in April 2008, five months before the investment bank filed for bankruptcy, and was hired by Lime founder Mark Gorton that November.

Lime handles hundreds of millions of trade orders a day. This spring it introduced a product called LimeInside that enables customers to place an order with Nasdaq, NYSE Arca and BATS in less than ten microseconds, on average — including real-time pretrade risk checks. That’s blazingly fast, confirms Tradeworx CEO Narang. It takes his group about 20 microseconds to do a trade from the moment a stock quote enters its system, triggers a signal, determines an order and passes through risk controls. Besides Lime, the only firms that are faster than Tradeworx, Narang says, are Tradebot and Getco.

Trading in the single-digit microseconds would be impossible for firms like Lime, Tradebot and Getco if they didn’t house their algorithms near the computer matching engines that power exchanges, ECNs and other electronic marketplaces. Brokerages, proprietary trading firms, hedge funds and other asset managers can lease co-location space in exchange-owned facilities (such as the NYSE’s Mahwah data center) or those owned and operated by third-party providers like Equinix.

Co-location has been a hot-button issue for critics of high frequency trading. I wonder, however, how many have actually visited a co-location facility. In March I got the chance to do just that at Equinix’s 340,000-square-foot NY4 data center in Secaucus, New Jersey. My host was Brandon Travan, head of the information technology infrastructure, co-location and cloud-hosting services divisions for Gravitas Technology, one of the growing number of companies that provide turnkey technology solutions for high frequency traders — and one of the first tenants at NY4.

From the outside, the white, two-story, unmarked building, located 11 miles west of downtown Manhattan in an area known best for outlet shopping, looks like a suburban medical office or law firm. (I later learned that it had been an eyeglass factory.) The lobby is equally nondescript, if not a little odd — there’s not much more than a phone, a plain steel door and a biometric hand scanner. After dialing in a personal code matched against the geometry of his hand, Travan got us into an inner lobby, where three security guards sat behind bulletproof glass and Kevlar-reinforced walls. I gave them my driver’s license (which, I was told, I’d get back when I left), and after two more sets of hand scans and steel doors, we entered the co-location area.

I was glad that I had decided to wear a light coat over my suit that day, because Equinix keeps the facility at a cool 65 to 72 degrees Fahrenheit. The design itself is also very cool. Built on a concrete slab with 45-foot-high ceilings, the building is organized along a rectangular grid system, with rows and rows of servers housed in metal cages for as far as the eye can see. Yellow trays snake above the cages, carrying all types of cables. Orange “innerducts” transport fiber-optic connections within the cages. Giant air-conditioning units, located outside the co-location area in case of a water leak, pump air through large green ducts that come down over the cages, creating a giant convection loop that sends the heat from the servers and networking equipment up toward the ceiling and out through the roof.

The facility is kept dark, both to improve its energy efficiency and to protect the anonymity of Equinix’s tenants. Each cage has lights that come on automatically when someone enters, I learned when Travan typed in the code to unlock his cage. Gravitas has about 35 clients operating out of its space, including one large hedge fund firm that spent more than $1 million on computer hardware, software and setup costs. The majority of Gravitas’s clients, however, are small. The company provides all of its clients with direct high-speed connections to all the market data providers and trade execution networks, including other NY4 tenants, like Direct Edge and the International Securities Exchange.

“We make the electronic trading community of Equinix available to smaller players by taking advantage of economies of scale — helping them get in with the technology they need with almost no up-front capital,” Travan told me. “If you’ve got a coder and the next best algo for trading equities, currencies or other vehicles, instead of needing a quarter-million dollars to start up, you can simply install your code on our hosting platform or send us servers to plug in, and you’re ready to go.”

Not everybody, however, buys the democratization argument. James McCaughan, CEO of Principal Global Investors, says co-location gives high frequency traders an unfair advantage. “Co-location creates an informational asymmetry that is fundamentally acting against the interests of long-term investors,” McCaughan told me exactly one week after my visit to the Equinix data center. “I have no problem with people developing algorithms for high frequency trading as long as they’re doing it with the same information as everybody else.”

McCaughan, whose team manages $215 billion in mostly 401(k) and other retirement assets for Principal Financial Group, considers himself to be a pretty savvy investor. His equity-trading desk has six people at the company’s Des Moines, Iowa, headquarters; two each in London and Singapore; and one in Tokyo, as well as access to state-of-the-art trade-execution algorithms offered by all the leading brokerage firms and third-party vendors. Although there’s nothing stopping Principal from using co-location, McCaughan told me that for a long-term investor, it’s probably not worth the effort. “As a large, sophisticated investor, we can compete,” he said. “But it is a weaker market if you have to be that sophisticated to compete.”

HIGH FREQUENCY TRADING BURST into the public consciousness last summer when news broke that a former Goldman Sachs Group computer programmer had been arrested by Federal Bureau of Investigation agents at Newark Liberty International Airport for allegedly stealing software code. According to the FBI, the programmer, Sergey Aleynikov, transferred thousands of files related to Goldman’s proprietary trading program to his home computers with the intention of using them to help his new employer build a high frequency trading platform. It didn’t take long for that employer, Chicago-based Teza Technologies, to cut its ties with Aleynikov. But Teza’s co-founders soon landed in hot water themselves when just six days after Aleynikov’s arrest, hedge fund firm Citadel sued them for violating a noncompete agreement and trying to steal its trade secrets. Last fall Citadel finally got the injunction against Teza that it was seeking, but by the time the judgment was rendered, the nine-month noncompete period had nearly expired.

The Citadel-Teza lawsuit provides an illuminating window into the world of high frequency trading. The person at the center of the drama is Mikhail Malyshev, a brilliant Russian émigré with a Ph.D. in plasma physics who joined Citadel’s high frequency ­trading group in 2003. During Malyshev’s six years at Citadel, the firm spent hundreds of millions of dollars researching and developing high frequency trading models and building out the IT infrastructure and systems to implement them. Its market data system, for example, contains roughly 100 times the amount of information in the Library of Congress. Citadel uses this historical data to test its models, which attempt to forecast changes in the prices of securities by analyzing statistical pricing patterns, supply and demand imbalances and other factors. The signals, or alphas, that prove to have predictive power are then translated into computer algorithms, which are integrated into Citadel’s master source code and electronic trading program.

Malyshev oversaw all aspects of Citadel’s nearly 60-person high frequency business, including the approval of trading strategies. He resigned on February 16, 2009; the next day his top lieutenant, Jace Kohlmeier, left too. By April they had incorporated Teza. Last fall, while I was reporting my story on Citadel, Kenneth Griffin, the firm’s billionaire founder and CEO, told me that before the arrest of Aleynikov, he had no idea what Malyshev and Kohlmeier were doing at Teza. After all, he said, the two were still on Citadel’s payroll as part of their noncompete agreements.

Like most hedge fund firms, Citadel is secretive about its investment strategies. The fact that Griffin would pursue a very public lawsuit with Teza’s founders says a lot about the importance of high frequency trading to the $12 billion-in-assets Citadel. In 2008 its high frequency group made $1.15 billion, compared with gains of $892 million the previous year and $75 million in 2005, according to Malyshev’s testimony. The 2008 performance was especially impressive given how poorly Citadel’s large multistrategy funds did that year: Its flagship Kensington and Wellington funds were each down about 55 percent.

Benjamin Blander, who joined the firm in 2004 from Banc One Corp., now leads the high frequency group, which manages a portion of Citadel’s $1.8 billion Tactical Trading fund. Citadel, however, was the only large hedge fund firm I could find that was willing to admit that it does high frequency trading. Most say they use many of the same tools as high frequency traders — employing high-speed computer programs and co-location services to generate, route and execute orders — but that their strategies have a longer time horizon.

“Even the slowest high frequency traders are turning over their portfolios at least a half dozen times a day,” AQR principal Michael Mendelson told me when I met with him in January at the firm’s Greenwich headquarters. “We tend to hold things for at least a day or two, and usually longer.”

Mendelson was head of quantitative equity trading at Goldman Sachs before joining AQR in 2005; he oversees the firm’s statistical arbitrage strategies. He explained to me that high frequency trading doesn’t require much in the way of capital — in fact, it would be hard-pressed to put hundreds of millions of dollars to work. The typical high frequency firm, he added, is likely to have about $5 million in proprietary capital and trade a few million shares a day through a specialized brokerage firm like Lime or Wedbush Securities.

High frequency trading, I learned, is a very low-margin, low-risk strategy. Traders earn less than a penny a share and rarely hold overnight positions. Profits are measured in hundredths of a cent, or “mils,” to use the industry parlance. According to Narang, high frequency traders typically earn about 10 mils, or 0.1 cent, a share trading U.S. equities. One of the attractions of the strategy is its consistency. High frequency traders rarely have losing days. They also tend to do very well during periods of high volatility. May was likely a great month for them.

Narang and other high frequency traders I spoke with gripe about the press, saying that it has often misrepresented what they do and grossly inflated the profitability of their business. They’ve got a legitimate beef. Last summer, a few weeks after the Goldman software-theft news broke, the New York Times ran a front-page story by Charles Duhigg describing how a handful of traders use powerful computers to “reap billions at everyone else’s expense.” The article went on to say that “these systems are so fast they can outsmart or outrun other investors, humans and computers alike,” using flash orders to step in front of those investors. (Under Reg NMS, exchanges were given the ability to “flash” marketable orders electronically for a split second to some professional traders before they are displayed to the broad public.) The article included the bold assertion that high frequency traders generated some $21 billion in profits in 2008.

The source of the data was TABB Group, a New York– and London-based research firm. The problem with the Times story, as I discovered when I met with Larry Tabb at his Wall Street office in early March, was that the $21 billion included a lot more than just high-frequency market making. “That number included anybody following an equities-related time-sensitive strategy that doesn’t take a significant end-of-day position,” the TABB founder and CEO explained. “It is pairs trading, options market making, futures and cash arbitrage, exchange-traded funds.” The profits for the virtual market makers in U.S. equities, he said, were roughly $8 billion in 2008 and $7.2 billion in 2009 and likely to be lower this year because of the drop in volatility and trading volume. (Tradeworx’s Narang says that high frequency trading in U.S. equities generates no more than $2 billion to $4 billion a year in profits overall.)

New York Senator Charles Schumer probably didn’t ask Larry Tabb for clarification after he read the Times article. The very same day it appeared, the longtime lawmaker fired off a letter to Mary ­Schapiro demanding that the SEC do something about high frequency traders and their ability to view order-flow information before the general public by using flash orders. “This kind of unfair access seriously compromises the integrity of our markets and creates a two-tiered system where a privileged group of insiders receives preferential treatment,” he wrote. If the SEC failed to curb flash trades, Schumer threatened to introduce legislation that would.

THE SEC’S NEW HEADQUARTERS IS a marvel of modern architecture. Conveniently located next to Washington’s historic Union Station, the building has a spectacular glass-and-steel atrium where visitors can watch the news on a giant flat-screen television while waiting to make their way into the agency’s inner sanctum. That’s exactly what I was doing one cloudy Tuesday afternoon in late March when John Nester, the SEC’s director of public affairs, came to get me. I’d taken the train down from New York that day to meet with James Brigagliano, deputy director of the agency’s Division of Trading and Markets, and several other members of his team to discuss high frequency trading. The problem was, I didn’t know in advance who was going to be in the meeting, and as Brigagliano and three other staffers from the SEC division filed into the room and quickly introduced themselves, I didn’t catch all of their names. I handed out business cards to each of them in hopes of getting them in return, but only one person (assistant director John Roeser) had brought a card.

It took me about 20 minutes into the interview to piece together the two missing names — “Dan” (who I later found out was market structure counsel Daniel Gray) and “Dave” (associate director David Shillman). Despite my initial confusion, I was impressed with the SEC staff’s knowledge of high frequency trading. We had a wide-ranging discussion, spanning everything from market structure and regulation to Washington politics and financial reform. The SEC’s interest in high frequency trading, I learned, long preceded Schumer’s famous letter to Schapiro last summer (and a similar call from Senator Kaufman for the SEC to do a comprehensive review of market structure). Brigagliano told me that the SEC began hearing about high frequency trading not long after it passed Reg NMS and that his group started working on the equity market structure concept release early last year.

The SEC has a tripartite mission: to protect investors, maintain fair and orderly markets and facilitate capital formation. Although the mandates can sometimes be at odds with one another, getting the first two right can go a long way toward ensuring the third. “We see confidence in the markets as essential for capital formation,” ­Brigagliano told me. “Investors are not going to commit capital if they think that the system is rigged.”

By the time I met with Brigagliano and his team, the SEC had proposed several new rules to help safeguard the market, including the elimination of flash orders and a prohibition against broker-dealers providing clients with unfiltered, or “naked,” access to exchanges and other alternative trading systems. Naked access has been popular among some speed-conscious high frequency traders because it enables them to place buy and sell orders directly with an exchange or trading venue without being slowed down by pretrade credit and risk checks. The SEC’s rule proposal, if approved, would require brokerage firms to create and maintain strict risk management controls for clients that are given direct sponsored access to electronic trading venues.

Of course, one of the biggest problems the SEC and other regulators have faced is that they simply haven’t had the tools or the data to track the billions and billions of trades that fire across the electronic exchanges and trading platforms each day. Although several of the largest high frequency players, like Getco, are registered broker-dealers and have the reporting requirements that go along with that, the vast majority of firms operate in anonymity. In April the SEC looked to change that when it voted to propose the creation of a large-trader reporting system. If the proposal is approved, any firm or individual that trades 2 million shares or $20 million in exchange-listed securities in a day, or 20 million shares or $200 million in securities in a month, will be required to identify itself to the SEC. The agency will then assign that trader a number, which its broker-dealer will use to tag all its transactions, reporting them, upon request, to the SEC.

“The large-trader reporting system will be simple to put in place,” the SEC’s Shillman told me during my March meeting at the agency. “Creating a consolidated audit trail is more complicated, and could take several years, because it requires systems changes at exchanges and broker-dealers.”

The SEC began working on a consolidated audit-trail proposal last summer, consulting with exchanges and broker-dealers on what would need to be done and how much it would cost. On May 26 the agency unveiled its new rule, which would create a consolidated order-tracking system that would enable the agency to access in real time most of the data needed to reconstruct a market dislocation like the flash crash. But progress doesn’t come cheap: The SEC estimates that it will initially cost about $4 billion to build the system and an additional $2.1 billion a year to maintain it. Taxpayers, however, won’t have to worry about footing the bill; the costs would be borne by broker-dealers, exchanges and other trading venues.

The large-trader tagging and consolidated audit-trail proposals are likely to be approved by the commission over the coming months. Although it’s wishful thinking to expect the SEC to ever be able to match the technological sophistication of high frequency traders, the new rules should eventually give the regulator the necessary tools to monitor and police their activity. I’m sure that is going to make some high frequency traders very nervous, but they shouldn’t be. Unlike some of the more vocal critics of high frequency trading, SEC chairman Schapiro knows that the technological clock cannot be turned back.

“While technology has provided benefits to the market, it has also created real issues,” she told me in early May. “We want to be very careful and thoughtful about how we approach it. The idea that you can say, ‘Let’s just unplug everything’ or ‘Let’s put something into the machines that makes everything go slower’ is probably not realistic.”

In other words, in the battle of man versus machine, both sides could end up winning.

Investors Want to Plug Into China’s Smart-Grid Market

01 Apr 2010
Xiang Ji

Beijing’s massive conversion to a smart power grid could just one-up the rest of the world. Savvy entrepreneurs and investors are looking to get a piece of the green action.

When Beijing announced a $585 billion stimulus package in November 2008, Jeffrey Kang spotted an opportunity. The package’s vast investment mandates included one aimed at upgrading the country’s electricity distribution system to a smart grid that would use high-tech meters to precisely match supply with demand in households and offices. The energy savings would be an obvious boon for the planet — but savvy entrepreneurs and investors like Kang also wanted a piece of the green action.

China Electricity Council, a national power industry association, estimates that total spending on the smart grid will hit $40 billion by 2011, although the entire project likely won’t be completed until 2020. An estimated 300 million old electricity meters are to be replaced by smart meters that encourage lower energy use by displaying usage prominently. The meters can also track household energy patterns and adjust distribution accordingly.

The whole smart-grid system — comprising ultrahigh-voltage transmission lines, sensors and smart meters, all connected through computer networks — enhances energy efficiency not only by matching supply and demand, but also by more efficiently managing intermittent renewable energy sources.

Entrepreneurs and investors see great prospects in the conversion to a smart grid. Kang, who is CEO of Nasdaq-listed, Shenzhen-based module supplier Cogo Group, signed a deal in April 2009 to acquire China’s Mega Smart Group, a supplier of parts for smart-meter makers. Kang estimates the deal will generate $20 million in sales in the first year, or about 7 percent of Cogo’s total revenue. And that’s only the start. “Smart meters will be a key driver in our growth going forward,” says Kang. Just last month Yale University said it had invested in Redwood City, California’s Silver Spring Networks, a smart-meter manufacturer planning an IPO in 2010.

SBI Energy, a Rockville, Maryland–based market research firm, forecasts that the smart-grid market will grow from $90 billion in 2009 to $171 billion in 2014. SBI says government and corporate mandates to convert to climate-friendly energy systems will drive the boom.

Vinod Khosla, founder of Khosla Ventures, a Menlo Park, California–based venture capital firm, predicts that the world’s electricity grid will eventually be set up “so that smart transformers are feeding information to smart way stations and talking to smart meters.” VC firms’ interest in smart grids emerged only recently, with investments as of last year totaling $414 million. By contrast, solar power has attracted $1.2 billion of VC funds, according to consulting firm Cleantech Group.

London-based VC firm WHEB Ventures not long ago made a capital injection of an undisclosed sum in PassivSystems, a Berkshire, U.K., company that makes energy management systems that fit into smart grids. “Though it’s still in early stages, smart grid represents a potentially vast global market,” says Megan Bingham-Walker, an associate at WHEB Ventures, which manages £114 million ($170.2 million). President Barack Obama last October granted $3.4 billion in stimulus money to develop smart-grid technology and install upgraded meters in the U.S.; utilities are to match these funds. Europe, meanwhile, has mandated that 20 percent of its energy must come from renewable sources by 2020.

Still, investors looking to plug directly into the smart-grid market may find it difficult to do so. For instance, Robert Metcalfe, a partner at Polaris Venture Partners, a $3 billion, Waltham, Massachusetts–based private equity firm, has been screening energy-management software developers but has yet to write a check. “One of the challenges is that there is no standard to root for, making it hard to recognize the winner,” he laments.

Prior to version 0.4, the plugin retrieved external data by using file_get_contents() function.  Some Web hosting companies, however, disable the use of this function on remote URLs for security reasons by setting PHP configuration directive allow_url_fopen to Off; consequently, include_HTML wouldn’t work on systems configired this way.

The alternative, of course, is to use the Client URL (cURL) extension, which would work regardless of allow_url_fopen setting, but may or may not be available on a specific system.

Starting from version 0.4, include_HTML begins by checking if the cURL extension is available on the host system.  If it is, the plugin relies on it to retrieve external data.  If it isn’t, the plugin falls back on the old file_get_contents() mechanism.

It is hoped that this modification would improve the plugin’s portability.  A big thank-you to VickiLH2 for suggesting and testing out the cURL data retrieval.

September 6, 2009

How Did Economists Get It So Wrong?

By PAUL KRUGMAN

I. MISTAKING BEAUTY FOR TRUTH

It’s hard to believe now, but not long ago economists were congratulating themselves over the success of their field. Those successes — or so they believed — were both theoretical and practical, leading to a golden era for the profession. On the theoretical side, they thought that they had resolved their internal disputes. Thus, in a 2008 paper titled “The State of Macro” (that is, macroeconomics, the study of big-picture issues like recessions), Olivier Blanchard of M.I.T., now the chief economist at the International Monetary Fund, declared that “the state of macro is good.” The battles of yesteryear, he said, were over, and there had been a “broad convergence of vision.” And in the real world, economists believed they had things under control: the “central problem of depression-prevention has been solved,” declared Robert Lucas of the University of Chicago in his 2003 presidential address to the American Economic Association. In 2004, Ben Bernanke, a former Princeton professor who is now the chairman of the Federal Reserve Board, celebrated the Great Moderation in economic performance over the previous two decades, which he attributed in part to improved economic policy making.

Last year, everything came apart.

Few economists saw our current crisis coming, but this predictive failure was the least of the field’s problems. More important was the profession’s blindness to the very possibility of catastrophic failures in a market economy. During the golden years, financial economists came to believe that markets were inherently stable — indeed, that stocks and other assets were always priced just right. There was nothing in the prevailing models suggesting the possibility of the kind of collapse that happened last year. Meanwhile, macroeconomists were divided in their views. But the main division was between those who insisted that free-market economies never go astray and those who believed that economies may stray now and then but that any major deviations from the path of prosperity could and would be corrected by the all-powerful Fed. Neither side was prepared to cope with an economy that went off the rails despite the Fed’s best efforts.

And in the wake of the crisis, the fault lines in the economics profession have yawned wider than ever. Lucas says the Obama administration’s stimulus plans are “schlock economics,” and his Chicago colleague John Cochrane says they’re based on discredited “fairy tales.” In response, Brad DeLong of the University of California, Berkeley, writes of the “intellectual collapse” of the Chicago School, and I myself have written that comments from Chicago economists are the product of a Dark Age of macroeconomics in which hard-won knowledge has been forgotten.

What happened to the economics profession? And where does it go from here?

As I see it, the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. Until the Great Depression, most economists clung to a vision of capitalism as a perfect or nearly perfect system. That vision wasn’t sustainable in the face of mass unemployment, but as memories of the Depression faded, economists fell back in love with the old, idealized vision of an economy in which rational individuals interact in perfect markets, this time gussied up with fancy equations. The renewed romance with the idealized market was, to be sure, partly a response to shifting political winds, partly a response to financial incentives. But while sabbaticals at the Hoover Institution and job opportunities on Wall Street are nothing to sneeze at, the central cause of the profession’s failure was the desire for an all-encompassing, intellectually elegant approach that also gave economists a chance to show off their mathematical prowess.

Unfortunately, this romanticized and sanitized vision of the economy led most economists to ignore all the things that can go wrong. They turned a blind eye to the limitations of human rationality that often lead to bubbles and busts; to the problems of institutions that run amok; to the imperfections of markets — especially financial markets — that can cause the economy’s operating system to undergo sudden, unpredictable crashes; and to the dangers created when regulators don’t believe in regulation.

It’s much harder to say where the economics profession goes from here. But what’s almost certain is that economists will have to learn to live with messiness. That is, they will have to acknowledge the importance of irrational and often unpredictable behavior, face up to the often idiosyncratic imperfections of markets and accept that an elegant economic “theory of everything” is a long way off. In practical terms, this will translate into more cautious policy advice — and a reduced willingness to dismantle economic safeguards in the faith that markets will solve all problems.

II. FROM SMITH TO KEYNES AND BACK

The birth of economics as a discipline is usually credited to Adam Smith, who published “The Wealth of Nations” in 1776. Over the next 160 years an extensive body of economic theory was developed, whose central message was: Trust the market. Yes, economists admitted that there were cases in which markets might fail, of which the most important was the case of “externalities” — costs that people impose on others without paying the price, like traffic congestion or pollution. But the basic presumption of “neoclassical” economics (named after the late-19th-century theorists who elaborated on the concepts of their “classical” predecessors) was that we should have faith in the market system.

This faith was, however, shattered by the Great Depression. Actually, even in the face of total collapse some economists insisted that whatever happens in a market economy must be right: “Depressions are not simply evils,” declared Joseph Schumpeter in 1934 — 1934! They are, he added, “forms of something which has to be done.” But many, and eventually most, economists turned to the insights of John Maynard Keynes for both an explanation of what had happened and a solution to future depressions.

Keynes did not, despite what you may have heard, want the government to run the economy. He described his analysis in his 1936 masterwork, “The General Theory of Employment, Interest and Money,” as “moderately conservative in its implications.” He wanted to fix capitalism, not replace it. But he did challenge the notion that free-market economies can function without a minder, expressing particular contempt for financial markets, which he viewed as being dominated by short-term speculation with little regard for fundamentals. And he called for active government intervention — printing more money and, if necessary, spending heavily on public works — to fight unemployment during slumps.

It’s important to understand that Keynes did much more than make bold assertions. “The General Theory” is a work of profound, deep analysis — analysis that persuaded the best young economists of the day. Yet the story of economics over the past half century is, to a large degree, the story of a retreat from Keynesianism and a return to neoclassicism. The neoclassical revival was initially led by Milton Friedman of the University of Chicago, who asserted as early as 1953 that neoclassical economics works well enough as a description of the way the economy actually functions to be “both extremely fruitful and deserving of much confidence.” But what about depressions?

Friedman’s counterattack against Keynes began with the doctrine known as monetarism. Monetarists didn’t disagree in principle with the idea that a market economy needs deliberate stabilization. “We are all Keynesians now,” Friedman once said, although he later claimed he was quoted out of context. Monetarists asserted, however, that a very limited, circumscribed form of government intervention — namely, instructing central banks to keep the nation’s money supply, the sum of cash in circulation and bank deposits, growing on a steady path — is all that’s required to prevent depressions. Famously, Friedman and his collaborator, Anna Schwartz, argued that if the Federal Reserve had done its job properly, the Great Depression would not have happened. Later, Friedman made a compelling case against any deliberate effort by government to push unemployment below its “natural” level (currently thought to be about 4.8 percent in the United States): excessively expansionary policies, he predicted, would lead to a combination of inflation and high unemployment — a prediction that was borne out by the stagflation of the 1970s, which greatly advanced the credibility of the anti-Keynesian movement.

Eventually, however, the anti-Keynesian counterrevolution went far beyond Friedman’s position, which came to seem relatively moderate compared with what his successors were saying. Among financial economists, Keynes’s disparaging vision of financial markets as a “casino” was replaced by “efficient market” theory, which asserted that financial markets always get asset prices right given the available information. Meanwhile, many macroeconomists completely rejected Keynes’s framework for understanding economic slumps. Some returned to the view of Schumpeter and other apologists for the Great Depression, viewing recessions as a good thing, part of the economy’s adjustment to change. And even those not willing to go that far argued that any attempt to fight an economic slump would do more harm than good.

Not all macroeconomists were willing to go down this road: many became self-described New Keynesians, who continued to believe in an active role for the government. Yet even they mostly accepted the notion that investors and consumers are rational and that markets generally get it right.

Of course, there were exceptions to these trends: a few economists challenged the assumption of rational behavior, questioned the belief that financial markets can be trusted and pointed to the long history of financial crises that had devastating economic consequences. But they were swimming against the tide, unable to make much headway against a pervasive and, in retrospect, foolish complacency.

III. PANGLOSSIAN FINANCE

In the 1930s, financial markets, for obvious reasons, didn’t get much respect. Keynes compared them to “those newspaper competitions in which the competitors have to pick out the six prettiest faces from a hundred photographs, the prize being awarded to the competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole; so that each competitor has to pick, not those faces which he himself finds prettiest, but those that he thinks likeliest to catch the fancy of the other competitors.”

And Keynes considered it a very bad idea to let such markets, in which speculators spent their time chasing one another’s tails, dictate important business decisions: “When the capital development of a country becomes a by-product of the activities of a casino, the job is likely to be ill-done.”

By 1970 or so, however, the study of financial markets seemed to have been taken over by Voltaire’s Dr. Pangloss, who insisted that we live in the best of all possible worlds. Discussion of investor irrationality, of bubbles, of destructive speculation had virtually disappeared from academic discourse. The field was dominated by the “efficient-market hypothesis,” promulgated by Eugene Fama of the University of Chicago, which claims that financial markets price assets precisely at their intrinsic worth given all publicly available information. (The price of a company’s stock, for example, always accurately reflects the company’s value given the information available on the company’s earnings, its business prospects and so on.) And by the 1980s, finance economists, notably Michael Jensen of the Harvard Business School, were arguing that because financial markets always get prices right, the best thing corporate chieftains can do, not just for themselves but for the sake of the economy, is to maximize their stock prices. In other words, finance economists believed that we should put the capital development of the nation in the hands of what Keynes had called a “casino.”

It’s hard to argue that this transformation in the profession was driven by events. True, the memory of 1929 was gradually receding, but there continued to be bull markets, with widespread tales of speculative excess, followed by bear markets. In 1973-4, for example, stocks lost 48 percent of their value. And the 1987 stock crash, in which the Dow plunged nearly 23 percent in a day for no clear reason, should have raised at least a few doubts about market rationality.

These events, however, which Keynes would have considered evidence of the unreliability of markets, did little to blunt the force of a beautiful idea. The theoretical model that finance economists developed by assuming that every investor rationally balances risk against reward — the so-called Capital Asset Pricing Model, or CAPM (pronounced cap-em) — is wonderfully elegant. And if you accept its premises it’s also extremely useful. CAPM not only tells you how to choose your portfolio — even more important from the financial industry’s point of view, it tells you how to put a price on financial derivatives, claims on claims. The elegance and apparent usefulness of the new theory led to a string of Nobel prizes for its creators, and many of the theory’s adepts also received more mundane rewards: Armed with their new models and formidable math skills — the more arcane uses of CAPM require physicist-level computations — mild-mannered business-school professors could and did become Wall Street rocket scientists, earning Wall Street paychecks.

To be fair, finance theorists didn’t accept the efficient-market hypothesis merely because it was elegant, convenient and lucrative. They also produced a great deal of statistical evidence, which at first seemed strongly supportive. But this evidence was of an oddly limited form. Finance economists rarely asked the seemingly obvious (though not easily answered) question of whether asset prices made sense given real-world fundamentals like earnings. Instead, they asked only whether asset prices made sense given other asset prices. Larry Summers, now the top economic adviser in the Obama administration, once mocked finance professors with a parable about “ketchup economists” who “have shown that two-quart bottles of ketchup invariably sell for exactly twice as much as one-quart bottles of ketchup,” and conclude from this that the ketchup market is perfectly efficient.

But neither this mockery nor more polite critiques from economists like Robert Shiller of Yale had much effect. Finance theorists continued to believe that their models were essentially right, and so did many people making real-world decisions. Not least among these was Alan Greenspan, who was then the Fed chairman and a long-time supporter of financial deregulation whose rejection of calls to rein in subprime lending or address the ever-inflating housing bubble rested in large part on the belief that modern financial economics had everything under control. There was a telling moment in 2005, at a conference held to honor Greenspan’s tenure at the Fed. One brave attendee, Raghuram Rajan (of the University of Chicago, surprisingly), presented a paper warning that the financial system was taking on potentially dangerous levels of risk. He was mocked by almost all present — including, by the way, Larry Summers, who dismissed his warnings as “misguided.”

By October of last year, however, Greenspan was admitting that he was in a state of “shocked disbelief,” because “the whole intellectual edifice” had “collapsed.” Since this collapse of the intellectual edifice was also a collapse of real-world markets, the result was a severe recession — the worst, by many measures, since the Great Depression. What should policy makers do? Unfortunately, macroeconomics, which should have been providing clear guidance about how to address the slumping economy, was in its own state of disarray.

IV. THE TROUBLE WITH MACRO

“We have involved ourselves in a colossal muddle, having blundered in the control of a delicate machine, the working of which we do not understand. The result is that our possibilities of wealth may run to waste for a time — perhaps for a long time.” So wrote John Maynard Keynes in an essay titled “The Great Slump of 1930,” in which he tried to explain the catastrophe then overtaking the world. And the world’s possibilities of wealth did indeed run to waste for a long time; it took World War II to bring the Great Depression to a definitive end.

Why was Keynes’s diagnosis of the Great Depression as a “colossal muddle” so compelling at first? And why did economics, circa 1975, divide into opposing camps over the value of Keynes’s views?

I like to explain the essence of Keynesian economics with a true story that also serves as a parable, a small-scale version of the messes that can afflict entire economies. Consider the travails of the Capitol Hill Baby-Sitting Co-op.

This co-op, whose problems were recounted in a 1977 article in The Journal of Money, Credit and Banking, was an association of about 150 young couples who agreed to help one another by baby-sitting for one another’s children when parents wanted a night out. To ensure that every couple did its fair share of baby-sitting, the co-op introduced a form of scrip: coupons made out of heavy pieces of paper, each entitling the bearer to one half-hour of sitting time. Initially, members received 20 coupons on joining and were required to return the same amount on departing the group.

Unfortunately, it turned out that the co-op’s members, on average, wanted to hold a reserve of more than 20 coupons, perhaps, in case they should want to go out several times in a row. As a result, relatively few people wanted to spend their scrip and go out, while many wanted to baby-sit so they could add to their hoard. But since baby-sitting opportunities arise only when someone goes out for the night, this meant that baby-sitting jobs were hard to find, which made members of the co-op even more reluctant to go out, making baby-sitting jobs even scarcer. . . .

In short, the co-op fell into a recession.

O.K., what do you think of this story? Don’t dismiss it as silly and trivial: economists have used small-scale examples to shed light on big questions ever since Adam Smith saw the roots of economic progress in a pin factory, and they’re right to do so. The question is whether this particular example, in which a recession is a problem of inadequate demand — there isn’t enough demand for baby-sitting to provide jobs for everyone who wants one — gets at the essence of what happens in a recession.

Forty years ago most economists would have agreed with this interpretation. But since then macroeconomics has divided into two great factions: “saltwater” economists (mainly in coastal U.S. universities), who have a more or less Keynesian vision of what recessions are all about; and “freshwater” economists (mainly at inland schools), who consider that vision nonsense.

Freshwater economists are, essentially, neoclassical purists. They believe that all worthwhile economic analysis starts from the premise that people are rational and markets work, a premise violated by the story of the baby-sitting co-op. As they see it, a general lack of sufficient demand isn’t possible, because prices always move to match supply with demand. If people want more baby-sitting coupons, the value of those coupons will rise, so that they’re worth, say, 40 minutes of baby-sitting rather than half an hour — or, equivalently, the cost of an hours’ baby-sitting would fall from 2 coupons to 1.5. And that would solve the problem: the purchasing power of the coupons in circulation would have risen, so that people would feel no need to hoard more, and there would be no recession.

But don’t recessions look like periods in which there just isn’t enough demand to employ everyone willing to work? Appearances can be deceiving, say the freshwater theorists. Sound economics, in their view, says that overall failures of demand can’t happen — and that means that they don’t. Keynesian economics has been “proved false,” Cochrane, of the University of Chicago, says.

Yet recessions do happen. Why? In the 1970s the leading freshwater macroeconomist, the Nobel laureate Robert Lucas, argued that recessions were caused by temporary confusion: workers and companies had trouble distinguishing overall changes in the level of prices because of inflation or deflation from changes in their own particular business situation. And Lucas warned that any attempt to fight the business cycle would be counterproductive: activist policies, he argued, would just add to the confusion.

By the 1980s, however, even this severely limited acceptance of the idea that recessions are bad things had been rejected by many freshwater economists. Instead, the new leaders of the movement, especially Edward Prescott, who was then at the University of Minnesota (you can see where the freshwater moniker comes from), argued that price fluctuations and changes in demand actually had nothing to do with the business cycle. Rather, the business cycle reflects fluctuations in the rate of technological progress, which are amplified by the rational response of workers, who voluntarily work more when the environment is favorable and less when it’s unfavorable. Unemployment is a deliberate decision by workers to take time off.

Put baldly like that, this theory sounds foolish — was the Great Depression really the Great Vacation? And to be honest, I think it really is silly. But the basic premise of Prescott’s “real business cycle” theory was embedded in ingeniously constructed mathematical models, which were mapped onto real data using sophisticated statistical techniques, and the theory came to dominate the teaching of macroeconomics in many university departments. In 2004, reflecting the theory’s influence, Prescott shared a Nobel with Finn Kydland of Carnegie Mellon University.

Meanwhile, saltwater economists balked. Where the freshwater economists were purists, saltwater economists were pragmatists. While economists like N. Gregory Mankiw at Harvard, Olivier Blanchard at M.I.T. and David Romer at the University of California, Berkeley, acknowledged that it was hard to reconcile a Keynesian demand-side view of recessions with neoclassical theory, they found the evidence that recessions are, in fact, demand-driven too compelling to reject. So they were willing to deviate from the assumption of perfect markets or perfect rationality, or both, adding enough imperfections to accommodate a more or less Keynesian view of recessions. And in the saltwater view, active policy to fight recessions remained desirable.

But the self-described New Keynesian economists weren’t immune to the charms of rational individuals and perfect markets. They tried to keep their deviations from neoclassical orthodoxy as limited as possible. This meant that there was no room in the prevailing models for such things as bubbles and banking-system collapse. The fact that such things continued to happen in the real world — there was a terrible financial and macroeconomic crisis in much of Asia in 1997-8 and a depression-level slump in Argentina in 2002 — wasn’t reflected in the mainstream of New Keynesian thinking.

Even so, you might have thought that the differing worldviews of freshwater and saltwater economists would have put them constantly at loggerheads over economic policy. Somewhat surprisingly, however, between around 1985 and 2007 the disputes between freshwater and saltwater economists were mainly about theory, not action. The reason, I believe, is that New Keynesians, unlike the original Keynesians, didn’t think fiscal policy — changes in government spending or taxes — was needed to fight recessions. They believed that monetary policy, administered by the technocrats at the Fed, could provide whatever remedies the economy needed. At a 90th birthday celebration for Milton Friedman, Ben Bernanke, formerly a more or less New Keynesian professor at Princeton, and by then a member of the Fed’s governing board, declared of the Great Depression: “You’re right. We did it. We’re very sorry. But thanks to you, it won’t happen again.” The clear message was that all you need to avoid depressions is a smarter Fed.

And as long as macroeconomic policy was left in the hands of the maestro Greenspan, without Keynesian-type stimulus programs, freshwater economists found little to complain about. (They didn’t believe that monetary policy did any good, but they didn’t believe it did any harm, either.)

It would take a crisis to reveal both how little common ground there was and how Panglossian even New Keynesian economics had become.

V. NOBODY COULD HAVE PREDICTED . . .

In recent, rueful economics discussions, an all-purpose punch line has become “nobody could have predicted. . . .” It’s what you say with regard to disasters that could have been predicted, should have been predicted and actually were predicted by a few economists who were scoffed at for their pains.

Take, for example, the precipitous rise and fall of housing prices. Some economists, notably Robert Shiller, did identify the bubble and warn of painful consequences if it were to burst. Yet key policy makers failed to see the obvious. In 2004, Alan Greenspan dismissed talk of a housing bubble: “a national severe price distortion,” he declared, was “most unlikely.” Home-price increases, Ben Bernanke said in 2005, “largely reflect strong economic fundamentals.”

How did they miss the bubble? To be fair, interest rates were unusually low, possibly explaining part of the price rise. It may be that Greenspan and Bernanke also wanted to celebrate the Fed’s success in pulling the economy out of the 2001 recession; conceding that much of that success rested on the creation of a monstrous bubble would have placed a damper on the festivities.

But there was something else going on: a general belief that bubbles just don’t happen. What’s striking, when you reread Greenspan’s assurances, is that they weren’t based on evidence — they were based on the a priori assertion that there simply can’t be a bubble in housing. And the finance theorists were even more adamant on this point. In a 2007 interview, Eugene Fama, the father of the efficient-market hypothesis, declared that “the word ‘bubble’ drives me nuts,” and went on to explain why we can trust the housing market: “Housing markets are less liquid, but people are very careful when they buy houses. It’s typically the biggest investment they’re going to make, so they look around very carefully and they compare prices. The bidding process is very detailed.”

Indeed, home buyers generally do carefully compare prices — that is, they compare the price of their potential purchase with the prices of other houses. But this says nothing about whether the overall price of houses is justified. It’s ketchup economics, again: because a two-quart bottle of ketchup costs twice as much as a one-quart bottle, finance theorists declare that the price of ketchup must be right.

In short, the belief in efficient financial markets blinded many if not most economists to the emergence of the biggest financial bubble in history. And efficient-market theory also played a significant role in inflating that bubble in the first place.

Now that the undiagnosed bubble has burst, the true riskiness of supposedly safe assets has been revealed and the financial system has demonstrated its fragility. U.S. households have seen $13 trillion in wealth evaporate. More than six million jobs have been lost, and the unemployment rate appears headed for its highest level since 1940. So what guidance does modern economics have to offer in our current predicament? And should we trust it?

VI. THE STIMULUS SQUABBLE

Between 1985 and 2007 a false peace settled over the field of macroeconomics. There hadn’t been any real convergence of views between the saltwater and freshwater factions. But these were the years of the Great Moderation — an extended period during which inflation was subdued and recessions were relatively mild. Saltwater economists believed that the Federal Reserve had everything under control. Fresh­water economists didn’t think the Fed’s actions were actually beneficial, but they were willing to let matters lie.

But the crisis ended the phony peace. Suddenly the narrow, technocratic policies both sides were willing to accept were no longer sufficient — and the need for a broader policy response brought the old conflicts out into the open, fiercer than ever.

Why weren’t those narrow, technocratic policies sufficient? The answer, in a word, is zero.

During a normal recession, the Fed responds by buying Treasury bills — short-term government debt — from banks. This drives interest rates on government debt down; investors seeking a higher rate of return move into other assets, driving other interest rates down as well; and normally these lower interest rates eventually lead to an economic bounceback. The Fed dealt with the recession that began in 1990 by driving short-term interest rates from 9 percent down to 3 percent. It dealt with the recession that began in 2001 by driving rates from 6.5 percent to 1 percent. And it tried to deal with the current recession by driving rates down from 5.25 percent to zero.

But zero, it turned out, isn’t low enough to end this recession. And the Fed can’t push rates below zero, since at near-zero rates investors simply hoard cash rather than lending it out. So by late 2008, with interest rates basically at what macroeconomists call the “zero lower bound” even as the recession continued to deepen, conventional monetary policy had lost all traction.

Now what? This is the second time America has been up against the zero lower bound, the previous occasion being the Great Depression. And it was precisely the observation that there’s a lower bound to interest rates that led Keynes to advocate higher government spending: when monetary policy is ineffective and the private sector can’t be persuaded to spend more, the public sector must take its place in supporting the economy. Fiscal stimulus is the Keynesian answer to the kind of depression-type economic situation we’re currently in.

Such Keynesian thinking underlies the Obama administration’s economic policies — and the freshwater economists are furious. For 25 or so years they tolerated the Fed’s efforts to manage the economy, but a full-blown Keynesian resurgence was something entirely different. Back in 1980, Lucas, of the University of Chicago, wrote that Keynesian economics was so ludicrous that “at research seminars, people don’t take Keynesian theorizing seriously anymore; the audience starts to whisper and giggle to one another.” Admitting that Keynes was largely right, after all, would be too humiliating a comedown.

And so Chicago’s Cochrane, outraged at the idea that government spending could mitigate the latest recession, declared: “It’s not part of what anybody has taught graduate students since the 1960s. They [Keynesian ideas] are fairy tales that have been proved false. It is very comforting in times of stress to go back to the fairy tales we heard as children, but it doesn’t make them less false.” (It’s a mark of how deep the division between saltwater and freshwater runs that Cochrane doesn’t believe that “anybody” teaches ideas that are, in fact, taught in places like Princeton, M.I.T. and Harvard.)

Meanwhile, saltwater economists, who had comforted themselves with the belief that the great divide in macroeconomics was narrowing, were shocked to realize that freshwater economists hadn’t been listening at all. Freshwater economists who inveighed against the stimulus didn’t sound like scholars who had weighed Keynesian arguments and found them wanting. Rather, they sounded like people who had no idea what Keynesian economics was about, who were resurrecting pre-1930 fallacies in the belief that they were saying something new and profound.

And it wasn’t just Keynes whose ideas seemed to have been forgotten. As Brad DeLong of the University of California, Berkeley, has pointed out in his laments about the Chicago school’s “intellectual collapse,” the school’s current stance amounts to a wholesale rejection of Milton Friedman’s ideas, as well. Friedman believed that Fed policy rather than changes in government spending should be used to stabilize the economy, but he never asserted that an increase in government spending cannot, under any circumstances, increase employment. In fact, rereading Friedman’s 1970 summary of his ideas, “A Theoretical Framework for Monetary Analysis,” what’s striking is how Keynesian it seems.

And Friedman certainly never bought into the idea that mass unemployment represents a voluntary reduction in work effort or the idea that recessions are actually good for the economy. Yet the current generation of freshwater economists has been making both arguments. Thus Chicago’s Casey Mulligan suggests that unemployment is so high because many workers are choosing not to take jobs: “Employees face financial incentives that encourage them not to work . . . decreased employment is explained more by reductions in the supply of labor (the willingness of people to work) and less by the demand for labor (the number of workers that employers need to hire).” Mulligan has suggested, in particular, that workers are choosing to remain unemployed because that improves their odds of receiving mortgage relief. And Cochrane declares that high unemployment is actually good: “We should have a recession. People who spend their lives pounding nails in Nevada need something else to do.”

Personally, I think this is crazy. Why should it take mass unemployment across the whole nation to get carpenters to move out of Nevada? Can anyone seriously claim that we’ve lost 6.7 million jobs because fewer Americans want to work? But it was inevitable that freshwater economists would find themselves trapped in this cul-de-sac: if you start from the assumption that people are perfectly rational and markets are perfectly efficient, you have to conclude that unemployment is voluntary and recessions are desirable.

Yet if the crisis has pushed freshwater economists into absurdity, it has also created a lot of soul-searching among saltwater economists. Their framework, unlike that of the Chicago School, both allows for the possibility of involuntary unemployment and considers it a bad thing. But the New Keynesian models that have come to dominate teaching and research assume that people are perfectly rational and financial markets are perfectly efficient. To get anything like the current slump into their models, New Keynesians are forced to introduce some kind of fudge factor that for reasons unspecified temporarily depresses private spending. (I’ve done exactly that in some of my own work.) And if the analysis of where we are now rests on this fudge factor, how much confidence can we have in the models’ predictions about where we are going?

The state of macro, in short, is not good. So where does the profession go from here?

VII. FLAWS AND FRICTIONS

Economics, as a field, got in trouble because economists were seduced by the vision of a perfect, frictionless market system. If the profession is to redeem itself, it will have to reconcile itself to a less alluring vision — that of a market economy that has many virtues but that is also shot through with flaws and frictions. The good news is that we don’t have to start from scratch. Even during the heyday of perfect-market economics, there was a lot of work done on the ways in which the real economy deviated from the theoretical ideal. What’s probably going to happen now — in fact, it’s already happening — is that flaws-and-frictions economics will move from the periphery of economic analysis to its center.

There’s already a fairly well developed example of the kind of economics I have in mind: the school of thought known as behavioral finance. Practitioners of this approach emphasize two things. First, many real-world investors bear little resemblance to the cool calculators of efficient-market theory: they’re all too subject to herd behavior, to bouts of irrational exuberance and unwarranted panic. Second, even those who try to base their decisions on cool calculation often find that they can’t, that problems of trust, credibility and limited collateral force them to run with the herd.

On the first point: even during the heyday of the efficient-market hypothesis, it seemed obvious that many real-world investors aren’t as rational as the prevailing models assumed. Larry Summers once began a paper on finance by declaring: “THERE ARE IDIOTS. Look around.” But what kind of idiots (the preferred term in the academic literature, actually, is “noise traders”) are we talking about? Behavioral finance, drawing on the broader movement known as behavioral economics, tries to answer that question by relating the apparent irrationality of investors to known biases in human cognition, like the tendency to care more about small losses than small gains or the tendency to extrapolate too readily from small samples (e.g., assuming that because home prices rose in the past few years, they’ll keep on rising).

Until the crisis, efficient-market advocates like Eugene Fama dismissed the evidence produced on behalf of behavioral finance as a collection of “curiosity items” of no real importance. That’s a much harder position to maintain now that the collapse of a vast bubble — a bubble correctly diagnosed by behavioral economists like Robert Shiller of Yale, who related it to past episodes of “irrational exuberance” — has brought the world economy to its knees.

On the second point: suppose that there are, indeed, idiots. How much do they matter? Not much, argued Milton Friedman in an influential 1953 paper: smart investors will make money by buying when the idiots sell and selling when they buy and will stabilize markets in the process. But the second strand of behavioral finance says that Friedman was wrong, that financial markets are sometimes highly unstable, and right now that view seems hard to reject.

Probably the most influential paper in this vein was a 1997 publication by Andrei Shleifer of Harvard and Robert Vishny of Chicago, which amounted to a formalization of the old line that “the market can stay irrational longer than you can stay solvent.” As they pointed out, arbitrageurs — the people who are supposed to buy low and sell high — need capital to do their jobs. And a severe plunge in asset prices, even if it makes no sense in terms of fundamentals, tends to deplete that capital. As a result, the smart money is forced out of the market, and prices may go into a downward spiral.

The spread of the current financial crisis seemed almost like an object lesson in the perils of financial instability. And the general ideas underlying models of financial instability have proved highly relevant to economic policy: a focus on the depleted capital of financial institutions helped guide policy actions taken after the fall of Lehman, and it looks (cross your fingers) as if these actions successfully headed off an even bigger financial collapse.

Meanwhile, what about macroeconomics? Recent events have pretty decisively refuted the idea that recessions are an optimal response to fluctuations in the rate of technological progress; a more or less Keynesian view is the only plausible game in town. Yet standard New Keynesian models left no room for a crisis like the one we’re having, because those models generally accepted the efficient-market view of the financial sector.

There were some exceptions. One line of work, pioneered by none other than Ben Bernanke working with Mark Gertler of New York University, emphasized the way the lack of sufficient collateral can hinder the ability of businesses to raise funds and pursue investment opportunities. A related line of work, largely established by my Princeton colleague Nobuhiro Kiyotaki and John Moore of the London School of Economics, argued that prices of assets such as real estate can suffer self-reinforcing plunges that in turn depress the economy as a whole. But until now the impact of dysfunctional finance hasn’t been at the core even of Keynesian economics. Clearly, that has to change.

VIII. RE-EMBRACING KEYNES

So here’s what I think economists have to do. First, they have to face up to the inconvenient reality that financial markets fall far short of perfection, that they are subject to extraordinary delusions and the madness of crowds. Second, they have to admit — and this will be very hard for the people who giggled and whispered over Keynes — that Keynesian economics remains the best framework we have for making sense of recessions and depressions. Third, they’ll have to do their best to incorporate the realities of finance into macroeconomics.

Many economists will find these changes deeply disturbing. It will be a long time, if ever, before the new, more realistic approaches to finance and macroeconomics offer the same kind of clarity, completeness and sheer beauty that characterizes the full neoclassical approach. To some economists that will be a reason to cling to neoclassicism, despite its utter failure to make sense of the greatest economic crisis in three generations. This seems, however, like a good time to recall the words of H. L. Mencken: “There is always an easy solution to every human problem — neat, plausible and wrong.”

When it comes to the all-too-human problem of recessions and depressions, economists need to abandon the neat but wrong solution of assuming that everyone is rational and markets work perfectly. The vision that emerges as the profession rethinks its foundations may not be all that clear; it certainly won’t be neat; but we can hope that it will have the virtue of being at least partly right.

Paul Krugman is a Times Op-Ed columnist and winner of the 2008 Nobel Memorial Prize in Economic Science. His latest book is “The Return of Depression Economics and the Crisis of 2008.”

This article has been revised to reflect the following correction:

Correction: September 6, 2009
Because of an editing error, an article on Page 36 this weekend about the failure of economists to anticipate the latest recession misquotes the economist John Maynard Keynes, who compared the financial markets of the 1930s to newspaper beauty contests in which readers tried to correctly pick all six eventual winners. Keynes noted that a competitor did not have to pick “those faces which he himself finds prettiest, but those that he thinks likeliest to catch the fancy of the other competitors.” He did not say, “nor even those that he thinks likeliest to catch the fancy of other competitors.”

/**
 * Disable Our Feed Urls
 */
function disable_our_feeds() {
	wp_die( __('<strong>Error:</strong>' .
		'No RSS Feed Available, ' .
		'Please visit our <a href="'.
		get_bloginfo('url') .
		'">homepage</a>.'));
}

add_action('do_feed', 'disable_our_feeds', 1);
add_action('do_feed_rdf', 'disable_our_feeds', 1);
add_action('do_feed_rss', 'disable_our_feeds', 1);
add_action('do_feed_rss2', 'disable_our_feeds', 1);
add_action('do_feed_atom', 'disable_our_feeds', 1);

Problem: an atypical Black Screen of Death.  In most cases, the Black Screen of Death appears at boot; often, the mouse cursor is visible and draggable.  In this case, however, the Black Screen of Death would appear randomly during the computer’s operation, causing the computer to freeze, power button being the only way to regain control.  This would happen about once a day.  Typically, Vista would reboot in normal mode without a problem, but every once in a while it would have to go through a lengthy restore process before booting up (I think the behavior at reboot depended on whether the computer was doing anything sensitive when it froze).

Possible solutions that didn’t work: (1) disabling event logging; (2) setting processor minimum to 100%.

Solution that worked: upgrading BIOS software to the latest version available from Toshiba Support.  So far, over a week without a single black-screen incident.