Like many others, we were quite surprised at the news that the proverbial guns had been lowered between Cloudera and Hortonworks after a decade of intense rivalry.  The news of their mega-merger percolated through social media circles with admissions of shock but not horror. And although I was reminded of a prior premonition, I was as surprised as anyone.

The truth is, we shouldn’t have been surprised.  Hadoop Distributors have been consolidating from 2011 onwards.  The market has grown as the vendors have contracted – from IntelIBM, PivotalMapRHortonworks and Cloudera to what is now Cloudera and an increasingly threatened MapR.

This is a great move for Cloudera and an even better move for Hortonworks.  We would expect the onsite/hybrid/multi-cloud buyers to benefit greatly as the platform software available to them becomes more comprehensive and easier to manage both technically and commercially.  The near and medium term will see many large-scale companies sustaining and growing onsite data platforms, which the newly formed Cloudera are perfectly placed to excel in this market.

However, in the long term, the deployment model for data platforms will ultimately default to public cloud as issues with security, residency and inertia are reduced to insignificance.  Those service providers that offer the best features at the lowest price will win that battle. Future mergers or acquisition could see Cloudera become equally successful in this market – Microsoft seems the most obvious suitor at present.

There will be significant engineering challenges ahead to consolidate products, but with the depth of technical talent in both companies this is only a matter of when, not if, this comes together. Some core platform projects will coalesce relatively easily, such as Apache Hive and Apache Hadoop HDFS. Others may require tough calls to be made, such as in the case of Apache Tez and Apache Spark.

This is good news for existing Cloudera and Hortonworks customers who will avail of single-enterprise support for a wider mix of products – for example Apache NiFi and Cloudera Data Science Workbench maintained and managed by the same product support team.  Adding IoT capability to an existing Cloudera installation or Workload Management to an existing Hortonworks installation – all supported by the same organisation – has until now been aspirational only, but will soon become possible.

The merger simplifies the market for new customers, giving them more power, fewer decisions to make, and less management overhead. With this reconstituted platform, user-friendliness and ease of deployment and management will be more important than ever as customer expectations will be raised.  We expect a simpler, more consistent and more feature-rich experience to be delivered.

An obvious benefit also arises for technology partners who can rationalise their own certification and integration obligations with one fewer Hadoop distribution to maintain against.

This merger strengthens the position of the newly-merged Cloudera to a point where a medium-term, almost philosophical, battle will emerge: onsite/hybrid/multi-cloud (Cloudera) on one side and the hyper-scale, pure-cloud triumvirate (AWS, Microsoft and Google) on the other. So perhaps the guns will be raised again, but this time not at each-other.

Thanks to contributions from Joseph McKavanagh and image credited to whywekiss.