Why Cloudera is saying ‘Goodbye, MapReduce’ and ‘Hello, Spark’

Derrick Harris

<a href="http://www.cloudera.com/content/cloudera/en/home.html">Cloudera</a>, a company that helped popularize Hadoop as a platform for analyzing huge amounts of data when it was founded in 2008, is overhauling its core technology. The One Platform Initiative the company announced Wednesday lays out Cloudera&#8217;s plan to officially replace MapReduce with <a href="http://spark.apache.org/">Apache Spark</a> as the default processing engine for Hadoop.<div class="paywall-selector"></div><div class="paywall">
Cloudera chief technologist Eli Collins said the company is &#8220;at best&#8221; halfway through the process from a technology standpoint and should be done in about a year. When complete, Spark should have similar levels of security, manageability, and scalability as MapReduce, and should be equally integrated with the rest of the technologies that comprise the ever-expanding Hadoop platform.
Collins said Spark&#8217;s existing weaknesses are &#8220;OK for early adopters, but really not acceptable to our customer base&#8221; as a whole. Cloudera says it has more than 100 customers running Spark in production—including <a href="https://fortune.com/company/equifax/" target="_blank">Equifax</a>, Experian, and CSC—but realizes that broader adoption and an improved Spark experience are a chicken-or-egg type of problem.
&#8220;If Spark is everywhere, then it’s a safe technology choice,&#8221; Collins explained. &#8220;And if it’s a safe technology choice, we can move the ecosystem.&#8221;
<figure id="attachment_1" aria-describedby="caption-attachment-1" style="width: 728px" class="wp-caption alignnone"><a href="https://content.fortune.com/wp-content/uploads/2015/09/oneplatform.jpg?quality=80"><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-1294932" src="https://content.fortune.com/wp-content/uploads/2015/09/oneplatform.jpg?quality=80" alt="oneplatform" width="728" height="673" /></a><figcaption id="caption-attachment-1" class="wp-caption-text">Cloudera</figcaption></figure>
The history of the move to Spark is in some ways as old Hadoop itself. <a href="https://fortune.com/company/alphabet/" target="_blank">Google</a> (GOOG) created MapReduce in the early 2000s as a faster, easier implementation of existing parallel processing approaches, and the creators of Hadoop <a href="https://gigaom.com/2013/03/04/the-history-of-hadoop-from-4-nodes-to-the-future-of-data/">developed an open source version of Google&#8217;s work</a>. However, while MapReduce proved revolutionary for early big data workloads (nearly every major web company is a heavy Hadoop user), its limitations became more clear as Hadoop and big data became mainstream technology movements.
Large enterprises, technology startups and other potential Hadoop users saw the potential in storing lots of data using the Hadoop file system and in analyzing that data, but they wanted something faster and more flexible than MapReduce. It was designed for indexing the web at places like Google and <a href="https://fortune.com/company/yahoo/" target="_blank">Yahoo</a> (YHOO), a batch-processing job where latency was measured in hours rather than milliseconds. MapReduce is also notoriously difficult to program, a problem that helped exacerbate the &#8220;big data skills gap&#8221; to which analyst firms and consultants have been pointing for years.
When Spark was created a few years ago at the University of California, Berkeley, it was the solution Hadoop vendors, Hadoop users, and venture capitalists alike needed to resolve their MapReduce woes. Spark is significantly faster and easier to program than MapReduce, meaning it can handle a much broader array of jobs. In fact, the project includes libraries for real-time data analysis, interactive SQL analysis, and machine learning, in addition to its core MapReduce-style engine.
And better yet, Spark is designed to integrate with Hadoop&#8217;s native file system. This means Hadoop users don&#8217;t have to move their terabytes or even petabytes of data elsewhere in order to take advantage of Spark. By 2013, major VC firms had began putting millions of dollars into <a href="https://databricks.com/">Databricks</a>, a startup founded by the creators of Spark, and major Hadoop vendors Cloudera, MapR, and Hortonworks (HDP) were beginning to integrate Spark into their Hadoop distributions.
<figure id="attachment_1" aria-describedby="caption-attachment-1" style="width: 819px" class="wp-caption alignnone"><a href="https://content.fortune.com/wp-content/uploads/2015/09/spark.jpg?quality=80"><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-1294935" src="https://content.fortune.com/wp-content/uploads/2015/09/spark.jpg?quality=80" alt="spark" width="819" height="452" /></a><figcaption id="caption-attachment-1" class="wp-caption-text">Databricks</figcaption></figure>
&#8220;Spark is one of the few components where you’ve seen 100% adoption and 100% investment [from the Hadoop community],&#8221; Cloudera&#8217; Collins said. Even Yahoo, which sponsored and drove Hadoop&#8217;s development early on and from which Hortonworks spun out, should be off of MapReduce within a year, he added. And IBM (IBM) <a href="https://fortune.com/2015/06/10/ibm-big-data-tech/">announced in June a $300 million commitment</a> to help develop Spark as the future of seemingly all analytic workloads.
That&#8217;s saying something in a technology market that has been characterized by corporate one-upmanship and flat-out insults over the years.
So now, Cloudera and the greater Hadoop community are trying to take the Spark transition over the finish line by making sure it works where MapReduce works and can handle as much (or nearly as much) data as MapReduce can. The latter is challenging—large Spark users might run it on hundreds of nodes, whereas large MapReduce users might run it on tens of thousands of nodes—but heavy Spark users and developers are working to close the gap.
&#8220;If we really want to replace MapReduce, we have to dot all those Is and cross all those Ts,&#8221; Collins said.
If the Hadoop community does it right, it could reap serious rewards as the technology, and &#8220;big data&#8221; in general, begins really catching on among mainstream businesses. &#8220;The vast majority of people who’ll use Hadoop in the next 10 years haven’t seen it yet,&#8221; Collins said. &#8220;…and Spark will be there [when they do.]&#8221;
For more about the business value of data analytics, watch this Fortune video:
<div class="vodvideocontainer" style="position: relative; padding-bottom: 56.25%;"><iframe src="//fortune.com/videos/embed/35fc0129-83dc-493c-bee9-5743417f6a70" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; overflow: hidden;" scrolling="no" frameborder="0" allowfullscreen></iframe></div>
<a href="https://fortune.com/getdatasheet/">Sign up for Data Sheet</a>, Fortune’s daily newsletter about the business of technology.
&nbsp;</div>

Cloudera wants to make Apache Spark the default processing engine in the Hadoop big data platform, replacing the longstanding MapReduce.

Cloudera plans to replace Hadoop MapReduce with Apache Spark

Hadoop allows some of the world's largest companies to store and process datasets on clusters of commodity hardware.

legacy

Future of Work

Tech

Cloudera plans to replace Hadoop MapReduce with Apache Spark | Fortune

Paolo Confino

Here’s when AI will launch a decade-long cycle of economic growth and productivity gains

/2025/03/14/ai-labor-market-unemployment-jobs-hiring-goldman-sachs-productivity/

Christiaan Hetzner

Tesla CEO Elon Musk introduces the newly unveiled Cybertruck in 2019.

Elon Musk’s Tesla reportedly halts Cybertruck deliveries as owners complain of metal sides falling off

/2025/03/14/elon-musk-tesla-cybertruck-delivery-halt-owners-complain-of-metal-sides-falling-off/

Preston Fore

Nvidia surpasses Tesla as the No. 1 held stock on Robinhood, fueled by Gen Z interest

/2025/03/14/nvidia-stock-gen-z-investment-robinhood/

Chris Morris

Facebook ramps up TikTok battle by letting creators monetize their Stories

/2025/03/14/facebook-creators-monetization/

Elon Musk arrives for US President Donald Trump's address to a joint session of Congress

Tesla may have no new car coming after all as report reveals plan to launch a stripped down Model Y 

/2025/03/14/elon-musk-tesla-may-have-no-new-car-coming-after-all-model-y/

Beatrice Nolan

Elon Musk and Donald Trump stand in front of a red Tesla outside the White House

Tesla warns White House over tariffs in unsigned letter: ‘It’s a polite way to say that the bipolar tariff regime is screwing over Tesla’

/2025/03/14/tesla-warns-trump-administration-tariffs-company-exposed-retaliation/

Eleanor Pringle

If you think the current outlook is bad, just wait until the White House can't find anyone to buy its debt, warns Ray Dalio

https://fortune.com/2025/03/12/national-debt-burden-ray-dalio-foreign-government-pressure/?itm_source=parsely-api

The Associated Press

She became a billionaire overnight after years as a hospice nurse. Now a philanthropist, she is challenging America’s richest 'to redistribute’ their wealth

https://fortune.com/article/billionaire-overnight-philanthropist-challenging-america-richest-redistribute-wealth/?itm_source=parsely-api

Jessica Mathews

In a striking change of tone, Tesla mega bull Dan Ives warns Elon Musk that patience is ‘wearing thin’

https://fortune.com/2025/03/11/tesla-bull-wedbush-dan-ives-change-of-tone-warns-elon-musk-patience-wearing-thin/?itm_source=parsely-api

DOGE says it's saved the federal government $115 billion. Experts say the figures don't stack up.

https://fortune.com/2025/03/13/doge-saved-federal-government-115-billion-experts-doubt-figures-elon-musk/?itm_source=parsely-api

Marco Quiroz-Gutierrez

Tesla is getting pummeled in the stock market, but Elon Musk's other companies are having a field day in the secondary market—especially xAI

https://fortune.com/2025/03/14/elon-musk-private-company-valuations-soar-tesla-stock-falls/?itm_source=parsely-api

Nicholas Gordon

Tesla loses $127 billion in one-day market bloodbath as China car group questions Musk’s pursuit of ‘personal glory’

https://fortune.com/asia/2025/03/11/tesla-shares-drop-china-byd-ev/?itm_source=parsely-api