From "Big Data" to "Big Insights"

A story by Jason Palmer, Science and Technology Reporter for BBC News, offers an interesting account of the possible benefits of "big data" studies, which seems to be one of the prevailing trends when it comes to getting institutions to put big bucks into technology. Palmer's article is based on a report that appeared today on the Scientific Reports Web site maintained by Nature, basically a platform for a more rapid turnaround process for making scientific results available to the public than his been achieved through print publication. (This particular paper was received on February 25, accepted through a peer review process on April 3, and showed up on the Web site this morning.) The title of the paper is "Quantifying Trading Behavior in Financial Markets Using Google Trends;" and the authors are Tobias Preis. Helen Susannah Moat, and H. Eugene Stanley. Since this article is publicly available at no charge, it seems worth while to reproduce the abstract:

Crises in financial markets affect humans worldwide. Detailed market data on trading decisions reflect some of the complex human behavior that has led to these crises. We suggest that massive new data sources resulting from human interaction with the Internet may offer a new perspective on the behavior of market participants in periods of large market movements. By analyzing changes in Google query volumes for search terms related to finance, we find patterns that may be interpreted as “early warning signs” of stock market moves. Our results illustrate the potential that combining extensive behavioral data sets offers for a better understanding of collective human behavior.

This definitely makes for an interesting read; and I can imagine that it is going to get a lot of citations at future "big data" conferences, particularly when potential funders are sitting in the audience. Nevertheless, Palmer's article includes a quote that Moat gave that may be cause for a bit of reflection:

We were intrigued by the idea that stock market data serves as a really large record of all the actions people take in the stock market, but don't necessarily tell us much about how people decided to take those actions.

This seems to imply that market behavior is based on "the actions people take," which appeals to that classic democratic image of the little old lady with five shares of AT&T sitting attentively at the annual stockholders meeting.

That image is, of course, a myth. It is quite a stretch to suggest that market behavior reflects the actions of individual agents. Most transactions that move market indicators (such as the standard averages reported in the news every day) are the actions of large funds buying and selling shares in quantities beyond the wildest dreams of that little old lady. Furthermore, these days they tend to be the actions of software, rather than individuals who assume the responsibility of managing such funds.

Of course there have been stories about the unpleasant consequences of automated trading for at least the last two decades. Now we have the possibility that one automated system will be trawling Google for big data for the sake of telling another automated system how to make large transactions. If we accept the premise that all software systems have undetected bugs, are we really prepared for that possibility?