Spark Resource Manager Comparison of IBM Spectrum Conductor with Spark, Apache YARN and Apache Mesos – Phase 2

Type: Audited

Stack under test:

  • IBM Spectrum Conductor with Spark 2.1.0 vs Apache YARN 2.7.3 vs Apache Mesos 1.0.1
  • Spark v2.0.1/2.0.2 with HDFS v2.7.3
  • Red Hat Enterprise Linux 7.1
  • 11 x Lenovo x 3630 M4 servers, 14 x 7200 RPM drives
  • 2 x 8-core Intel Xeon E5-2450 @ 2.10GHz
  • Mellanox MT27500 ConnectX-3 10GbE Adapters
  • IBM BNT RackSwitch G8124-E 10GbE Switch

Please log in to see file attachments. If you are not registered, you may register for no charge.

Big data has become a big topic at STAC. All it takes to confirm this is a quick glance at discussions and presentations at STAC Summits over the past few years (particularly NY and London).