Hadoop is now on the minds of executives who care deeply about the power of their rapidly accumulating data. It has already inspired a broad range of big data experiments, established a beachhead as a production system in the enterprise and garnered tremendous optimism for expanded use.

However, it is also starting to create tremendous frustration. A recent analyst report showed less enthusiasm for Hadoop pilots this year than last. Many companies are getting lost on their way to big data glory. Instead, they find themselves in a confusing place of complexity and befuddlement. What’s going on?

While there are heady predictions that by 2020, 75 percent of the Fortune 2000 will be running a 1,000-node Hadoop cluster, there is also evidence that Hadoop is not being adopted as easily as one would think. In 2013, six years after the birth of Hadoop, Gartner said that only 10 percent of the organizations it surveyed were using Hadoop. According to the most recent Gartner survey, less than 50 percent of 284 respondents have invested in Hadoop technology or even plan to do so.

The current attempts to transform Hadoop into a full-blown enterprise product only accomplish the basics and leave the most challenging activities, the operations part, to the users, who, for good reason, wonder what to do next. Now we get to the problem. Hadoop is still complex to run at scale and in production.

Once you get Hadoop running, the real work is just beginning. In order to provide value to the business you need to maintain a cluster that is always up and high performance while being transparent to the end-user. You must make sure the jobs don’t get in each other’s way. You need to support different types of jobs that compete for resources. You have to monitor and troubleshoot the work as it flows through the system. This means doing all sorts of work that is managed, controlled, and monitored by experts. These tasks include diagnosing problems with users’ jobs, handling resource contention between users, resolving problems with jobs that block each other, etc.

How can companies get past the painful stage and start achieving the cost and big data benefits that Hadoop promises? When we look at the advanced practitioners, those companies that have ample data and ample resources to pursue the benefits of Hadoop, we find evidence that the current ways of using Hadoop still require significant end-customer involvement and hands-on support in order to be successful.

For example, Netflix created the Genie project to streamline the use of Amazon Elastic MapReduce by its data scientists, whom Netflix wanted to insulate from the complexity of creating and managing clusters. The Genie project fills the gaps between what Amazon offers and what Netflix actually needs to run diverse workloads in an efficient manner. After a user describes the nature of a desired workload by using metadata, Genie matches the workload with clusters that are best suited to run it, thereby granting the user’s wish.

Once Hadoop finds its “genie,” it can solve the problem of turning Hadoop into a useful tool that can be run at scale and in production. The reason Hadoop adoption and the move into production is going slowly is that these hard problems are being figured out over and over again, stalling progress. By filling this gap for Hadoop, users can do just what they want to do, and learn things about data, without having to waste time learning about Hadoop.