New technologies such as Hadoop in the Big Data space bring potential advantages, but significant complexity and disruption often accompany the beneficial impact. To create a benchmark of exactly what is happening with Hadoop, Sand Hill Group in September-October conducted a survey of individuals directly involved in evaluating and implementing Hadoop in their organization. How they apply Hadoop, implementation challenges they encountered and the quality of their experience to date are revealed in our report published this week: “Do you Hadoop? - A Survey of Big Data Practitioners.”

While most people agree that Hadoop is in the early stages of adoption in the enterprise, our survey indicates that Hadoop has moved well beyond the “science project” status. Eleven percent of the participants were in the process of developing their first Hadoop-based solution with another 16 percent currently conducting a formal proof of concept (POC). Nearly 40 percent of the survey respondents stated they identified a Big Data business imperative as the driver of their investment in Hadoop initiatives.

Key findings

1. Advanced analytics moving into the mainstream

Traditional statistical modeling drove the predictive analytics science in recent decades, but it functioned with a paucity of data. Big Data technologies such as Hadoop enable organizations to handle larger data sets and significantly improve advanced analytics capabilities.

Our survey reveals an expanding implementation of advanced analytics such as risk propensity, optimization and predictive analytics. Nearly one in four respondents stated that advanced analytics will be their #1 Hadoop-based initiative in the next 12-18 months.

2. Increasing use of new data types

Among Big Data’s defining three “Vs” (volume, variety and velocity), an increasing advantage is the variety of data sets that organizations can use to enrich the quality of analytics. Business analytics professionals have always wanted to use disparate data sets to enrich the quality of their analyses. Typical limitations to this requirement were the availability of data and the technological capability to create a sustainable process for synthesizing data.

Our survey shows an increased adoption of Hadoop-based technologies to gather new types of data. Twenty percent of the respondents stated that over the next 12-18 months they plan to incorporate new data types such as streaming data and geographic data into their Hadoop environment to support new advanced analytics.

3. Moderating emphasis on basic analytics

In the enterprise environment, business intelligence teams were the first to be exposed to Hadoop as a tool to solve traditional business analytics problems. They sought Big Data technologies to solve more complex problems beyond basic analytics and dashboards with an order of magnitude reduction in the cost of processing very large volumes of data.

Ramping up their use of advanced analytics is the goal over the next 18-months for 32 percent of our study respondents. They expressed plans to move away from basic analytics such as statistics, patterns and search as they increase their use of advanced analytics to enrich their business perspective and improve results.

This survey is a precursor to Sand Hill Group’s study entitled “Big Data Prescriptions for Transforming Business,” which focuses on how executive management views the deployment of advanced analytics technologies to impact shareholder value. Like Sand Hill’s past reports on cloud and mobility, this will provide a qualitative perspective from the executive suite and is based on in-depth interviews with top-tier decision makers that will influence investment on Big Data in the enterprise in the next few years. Stay tuned for more on this study.

Click here to download the report, “Do you Hadoop? - A Survey of Big Data Practitioners.”

(Originally published in SandHill.com)

M. R. Rangaswami is co-founder and CEO of Sand Hill Group and the publisher of SandHill.com.