STAC Report: Spark Resource Managers, Phase 1

IBM Platform Conductor for Spark comes out on top in this first test of throughput and fairness.

29 March 2016

With the growth in popularity of Apache Spark as a framework for big data analytics, organizations are now looking to offer Spark clusters that supply shared services to many users with diverse workloads, datasets, and business priorities. How well a Spark system utilizes cluster resources in such multi-tenant scenarios depends on the resource manager--i.e., the software layer that determines how much computing resource to make available to each Spark executor or task.

Several resource managers are now available, including open source products like Apache YARN, Apache Mesos, and commercial alternatives like IBM Platform Conductor for Spark.

At the request of IBM, STAC recently embarked on a two-phase project to analyze the performance of Spark using these three resource managers in an environment that was otherwise the same. Phase 1 of the project uses a single job type (a small sort job) and a single job-submission pattern. Phase 2 will have more job types and submission patterns.

The STAC Report from Phase 1 is now available. The configuration details for Spark and each resource manager are available to premium members on the same page as the report.

Key findings from this test environment under the chosen workload include:

• Apache YARN and Apache Mesos showed similar throughput (YARN was about 12% higher than Mesos), but IBM Platform Conductor for Spark had substantially higher throughput than both: 58% higher than Mesos and 41% higher than YARN.
• Apache YARN and Apache Mesos had the same average run time per job of 94 seconds, but IBM Platform Conductor for Spark was 59% faster at 59 seconds.
• IBM Platform Conductor for Spark and Apache YARN demonstrated much better fairness than Apache Mesos (with fairness defined as the standard deviation of per-job run times as a percent of the mean). Conductor had the best fairness of 22%, with YARN at 27% and Mesos at 235%.