Preface

As a 12-year-old boy I was befuddled when my father—a finance professor—gave me trading authority over a small brokerage account. At the time I didn't understand what the stock market was, and I had no idea how to proceed. He educated me on how to read stock tables in the daily newspaper (this was 1985), call a broker, and place an order. I was set free with my limited knowledge and zero experience with the goal of growing the balance.

To select investments, I first turned to the local newspaper. I reviewed the micro-text of the stock tables. The numbers didn't make sense to me—my first dead-end. For Plan B I visited the library, and the librarian referred me to dusty books from the 1960s that extolled the virtues of 'tronics stocks and Dow Theory. “Nothing for me here,” I thought. I wanted to know what to buy right now, not to learn ancient theory.

Next I went to a bookstore. A young attendant directed me to the magazine section, and the first magazine I picked up listed the Top 10 Growth Stocks of 1985. “Perfect!” I thought. I went home, called up the broker, and dictated the top 10 names to him, buying shares in each.

Over the next few months, I didn't pay attention to the stocks' performance. About a year later I figured it would be a good time to check in. I expected to hear that I had made big gains. In fact, I fantasized that the broker would soon be calling me for investment advice. When I opened an account statement I saw—to my disbelief—that the account was down 20 percent.

Confused, I went back to the bookstore. I related my tale to another attendant, and he condescendingly informed me, “Clearly you bought the wrong magazine.” “He's right!” I realized. This new, wiser guide helped me find a magazine extolling the Top 10 Most Innovative stocks of 1986. I went home, invested in a few of the top 10, and waited. I paid more attention this time, and I noticed that the first three monthly account statements were positive. I felt good, back on track, and I imagined I would be redeemed as a genius stock investor.

A year later, after a nine-month hiatus, I opened the latest account statement. The damage was worse—my account was now down nearly 50 percent from where it had started. “This can't be right,” I thought. I sheepishly called the broker. He confirmed the loss.

I wanted to understand what the experts knew that I did not, so I started reading books by investing by gurus such as Benjamin Graham and Peter Lynch. I noticed that these books were teaching me not only about fundamentals, but also about psychology. It seemed that many of history's most successful investors used an understanding of investor behavior. Baron Nathan von Rothschild, an early scion of the Rothschild banking dynasty, in 1812 guided investors to, “Buy to the sound of cannons, sell to the sound of trumpets.” Benjamin Graham wrote, “We buy from pessimists, and we sell to optimists.” Warren Buffett modernized the saying as, “Be fearful when others are greedy and greedy when others are fearful.” This advice seemed like useful guidance, but it wasn't specific or easily actionable.

Psychology-based investing advice seemed too vague. I wanted more concrete guidance, and as I embarked on engineering coursework in college, I found what I believed was a true advantage in investing—a deeper understanding of mathematics and models.

Mathematical Mayhem

While the markets had beaten me as a 12-year-old investor, in college I vowed to learn their tricks and recover my losses. Working through a university degree in electrical engineering, the solidity of mathematics—of software development and machine learning algorithms—seemed like the best path to resurrecting the now-dormant brokerage account. I obtained long price and volume data histories, reserved CPU time on the engineering department's fast RISC machines, and wrote code to identify patterns in the prices.

The predictive systems I built seemed promising at first. The algorithms found basic patterns in prices, and they had decent accuracy in their out-of-sample predictions. As a result I decided to use the systems in live trading. Over the following three years, I traded S&P 500 futures contracts based on these systems' directional signals.

While they were initially successful, two problems emerged as I used these predictive systems. First, I saw what quantitative analysts call alpha decay, the phenomenon in which good mathematical trading systems gradually fade in their profitability. The models worked well on their “training set” in the 1980s and early 1990s, but every year through the late 1990s the profitability declined. Perhaps other traders were finding the same patterns and arbitraging them out of the markets, or perhaps markets were changing.

The second problem with these trading systems was more personal. Sometimes they indicated I should buy stocks as the prices were plummeting and the news was exceptionally negative. Other times, they told me to sell stocks as the market was charging higher. Such signals felt wrong to me. They required trading against the herd, and they were emotionally difficult to execute. Even though I intellectually knew that cherry-picking trading signals is a bad habit, too often I made excuses and deviated from the plan. After retrospectively analyzing my behavior, it seemed that the best trading signals were the most emotionally difficult to follow. I would have to battle my own human nature in order to trade well.

Considering these two problems together, it occurred to me that the most enduring edge in markets—the one least likely to suffer alpha decay—might lie in identifying the information and feelings (sentiments) that compel traders to move as a herd, too often at their own peril. Sentiment has a way of pulling traders in, of fooling them again and again. I wanted a way to quantify sentiment so I could use it without simultaneously being its victim. I became a scholar of sentiment to gain perspective on it.

Through four years of medical school and then four more years as a psychiatry resident, I researched the biology of decision making. During residency training I began coaching investors, and it was through coaching that I gained a deep appreciation of the very human, and very diverse, natures of successful traders.

Near the end of psychiatry residency I started neuroeconomics post-doctoral studies at Stanford University with Brian Knutson. Knutson's lab studies subjects undertaking financial risks, using tools such as brain fMRI and psychometric testing. I wrote extensively about this research in the book Inside the Investor's Brain.1 Researchers such as Knutson have demonstrated, even when the expected value of a risk is fixed, differences in the presentation and description of the potential gain or loss predictably alter behavior. I wondered how to quantify such “soft” factors in the information stream. Financial social media and news seemed like a good place to start.

Framing the Issue

The field of text analytics—quantifying sentiment, topics, and tones in investment-related language—is the quantitative basis for this book. While it's self-evident that the release of information such as corporate earnings leads to gyrations in stock prices, the impact is sometimes counterintuitive. For example, a company might beat consensus earnings estimates but the stock price immediately loses value. Given the preponderance of online stock conversations where earnings estimates and other opinions are shared, it seemed that if one could measure and quantify the important content in those messages, then perhaps a predictive edge could be identified in markets.

In pursuit of such a predictive edge, in 2004 the MarketPsych team built financial text analytics software. First, we built search engine technology to gather news and social media as quickly as articles were published. Then we built text analyzers to quantify influential characteristics in text. We created time series of each high-impact factor—factors such as fear or excitement—for each stock over time. Finally, we tested the data statistically to determine its correlation with future price action. We found promising results, and we resolved to start trading.

Trading on Sentiment

Using our early text analysis engines on social media, we set up simple investment strategies. These strategies ran automatically and posted their results online. Over 18 months through 2007 the strategies earned a 34 percent absolute return on paper. The strategies were written up in Popular Science in February 2008,2 and that article stoked interest from investors. We raised a small fund to trade a market-neutral hedge fund strategy. As far as we know, the fund was the first social media–based hedge fund.

On September 2, 2008, we launched the fund with $1 million under management and dreams of big growth. Three days after we started trading, Fannie Mae and Freddie Mac failed. Lehman Brothers and AIG went under the following weekend. Fortunately, we had prepared for the increased chaos by creating primarily fear-based strategies. Despite the onset of the financial crisis, the fund's returns held up through mid-2009.3,4 Over its first 12 months, the fund was up 40 percent net, putting it in the top 1 percent of hedge funds through the financial crisis.

We had built trading models that made money in volatile, emotional...