Big Data Evaluator Homepage


News

25/11/2016 BDEv 2.3 is released!

BDEv 2.3 is a point release for the BDEv 2.x line

  • Added support for Flame-MR version 0.10.0
  • Added support for the Flink 1.1.x line (tested with version 1.1.3)
  • Added support for the SLURM job scheduler
  • Updated versions of some frameworks
    • Hadoop 2.6.5
    • Hadoop 2.7.3
    • RDMA-Hadoop-2 1.1.0
    • Spark 1.6.3
    • Flink 1.0.3
  • Memory settings for mappers and reducers can now be set separately
  • Added configuration variable in etc/core-default.sh to control the Hadoop parameter "io.file.buffer.size"
  • Virtual and physical memory limits in YARN containers as well as the ratio between them can now be configured in etc/yarn.default.sh
  • Fix critical bug when running Flink under YARN related with JobManager memory
  • Minor bug fixes

04/07/2016 BDEv 2.2 is released!

BDEv 2.2 is a point release for the BDEv 2.x line

  • Add support for multiple Workers/TaskManagers per node in Spark and Flink
  • Enhancements in several Flink workloads
    • Grep
    • PageRank (with delta iterations)
    • Connected Components (with Gelly)
    • K-Means
  • Enhancements in several Spark workloads
    • TeraSort
    • K-Means (with MLlib)
  • Bug fixes

05/05/2016 BDEv 2.1 is released!

BDEv 2.1 is a point release for the BDEv 2.x line

  • Allow to start the job history server for Spark-based frameworks
  • Added support for some new frameworks
    • Hadoop 2.6.4
    • RDMA-Hadoop-2 0.9.9
    • Spark 1.6.1
    • Flink 1.0.2
  • Added support for Mahout 0.11.2 and 0.12.0
  • Added new benchmarks for Spark
    • PageRank
    • Connected Components
    • K-Means
    • Bayes
    • Hive SQL queries:
      • Aggregation
      • Join
      • Scan
  • Added new benchmarks for Flink
    • PageRank
    • Connected Components
    • K-Means
  • Multiple bug fixes

02/03/2016 BDEv 2.0 is released!

BDEv 2.0 is a major release which contains a number of significant enhancements. Starting with this release, the MapReduce Evaluator (MREv) tool has been renamed to Big Data Evaluator (BDEv).

  • Added support for Flink version 0.10.2 on YARN and standalone modes
  • Added support for RDMA-Spark version 0.9.1 on YARN and standalone modes
  • Added support for Spark on standalone mode
  • Updated versions of some frameworks
    • Hadoop-2.7.2
    • RDMA-Hadoop-2 0.9.8
    • Spark 1.5.2
    • Spark 1.6.0
  • New data set generation using the HiBench DataGen tool
  • Added new benchmarks for Hadoop-based frameworks
    • Grep
    • K-Means
    • Connected Components
    • Hive SQL queries:
      • Aggregation
      • Join
      • Scan
  • Added new benchmarks for Spark and Flink
    • WordCount
    • Sort
    • Grep
    • TeraSort
  • Added new benchmark for DataMPI
    • Grep
  • Allow to start the job history server for Hadoop-based frameworks
  • Input datasize for WordCount, Sort and TeraSort can now be different
  • Enhanced configuration of parameters
    • Separated configuration files for HDFS, MapReduce and YARN
    • Solution-specific parameters in a separated file
    • Many new configuration parameters
  • Support for multiple disks
  • Network interfaces are now optional parameters. If not specified, IPs are determined using the hostfile
  • Updated iLO scripts to version 4.70
  • Allow to configure IP, user name and password for iLO interfaces
  • Automatically download Apache Mahout and Apache Hive on demand
  • Optimized code refactorization and simplified internal configuration
  • Multiple bug fixes

10/08/2015 MREv 1.1 is released!

MREv 1.1 has been released and can be obtained from the Downloads section. The main new features of MREv 1.1 are:

  • Updated versions of the frameworks
    • Hadoop-2.7.1-GbE
    • Hadoop-2.7.1-IPoIB
    • Hadoop-2.7.1-UDA
    • RDMA-Hadoop-2-0.9.7-GbE
    • RDMA-Hadoop-2-0.9.7-IPoIB
    • Spark-1.4.1-YARN-GbE
    • Spark-1.4.1-YARN-IPoIB
  • Configurable timeout for the workloads
  • User-defined batch command
  • Separate configuration and log directories for each evaluation
  • Enhanced resource configuration

09/12/2014 MREv 1.0 is released!

Version 1.0 of MREv has been released and can be obtained from the Downloads section.