Big data stocks have been big winners recently, few more so than Splunk (SPLK). The company, which is best known for software that harnesses and then analyzes machine data to extract meaning, value, and ultimately intelligence, has seen its shares rise about 8% over the past three months.
Yet the good times may not last. Demand for big data tools is outpacing the supply of talented scientists capable of using them effectively. Worse, a new McKinsey study predicts that, by 2018, the U.S. could face a shortage of 1.5 million big data specialists.
Even so, there's good reason to like the big data trend. A digital world throws off so much information that harnessing it all can make for an astoundingly complex task. Just look at how much Google (GOOGL 1.87%) spends on the ocean-boiling task of indexing the Web.
Think of it making a high capacity digital brain from scratch, and with a fast memory to boot. Nothing like that had existed previously, so the search king invented a programming model, called MapReduce, which has since inspired the creation of open-source file systems Apache Hadoop and Apache Cassandra. These are the technologies that make it possible to collate and make meaning out of what would otherwise be machine-readable gobbledygook.
Amazon.com (AMZN 0.03%) already has a web service that leverages MapReduce, while IBM (IBM -0.14%) uses Hadoop in powering its InfoSphere BigInsights analytics software. And then, of course, there's Splunk, which some customers are using to query data stored in Hadoop and Cassandra archives.
Executives are willing to spend on translators. According to research firm Gartner, big data is already on track to drive $28 billion in worldwide IT spending this year before rising to $34 billion in 2013.
And that's great news. But it also might be too early to go all-in, especially since analyst estimates are just that: estimates. Software and tools are only as good as the specialists who use them, and they're not easy to find right now.
