Big Data Needs Big Search

No doubt about it. The era of big data is upon us.

“The data tsunami we’ve seen in the past five years will look mild compared to what’s coming,” said David Vellante, president of Wikibon.org, a leading IT Think Tank.  Leading industry analyst Gartner predicts that enterprise data will increase by 650 percent by 2015—a doubling of the world’s data every 18 months.

And a big part of the big data story is the growth of unstructured data, like email and social media. According to Pingdom, email is, in fact, growing at a clip faster than social media, as these 2010 statistics reflect:

Email:

• 1.88 billion: The number of email users worldwide
• 480 million: New email users since the year before
• 2.9 billion: The number of email accounts worldwide

Social Media:

• 175 million: People on Twitter
• 100 million: New accounts added on Twitter
• 600 million: People on Facebook
• 150 million: New people on Facebook (2010)

So what?

Now that there is so much data, it is time to unlock its value.

Companies cannot afford to keep purchasing storage to merely keep up – they need to leverage solutions that not only optimize storage, but also let users harness its power. And in order to harness the power of Big Data, you need intelligent ways to index the data and make it accessible.

At LiveOffice, we’re combining next-generation Big Data technologies, like Hadoop (which distributes workload, bandwidth and computational power across multiple nodes) and Lucene, with our cloud archiving platform to render sub-second search results – even with complex search queries.

Ultimately, the real power of Big Data will boil down to big search--the ability to quickly search obscene amounts of data to find specific messages of interest. More importantly, these new indexing/search technologies can unearth previously invisible trends and deliver powerful business insights.