Merge branch 'master' of github.com:intel-hadoop/HiBench

Conflicts:
	bin/build-all.sh
	bin/functions/load-config.py
	bin/functions/workload-functions.sh
	src/autogen/src/main/java/org/apache/hadoop/fs/dfsioe/IOMapperBase.java

Author: lvsoft

README.md

@@ -18,7 +18,7 @@
 
 This benchmark suite contains 10 typical micro workloads. This benchmark suite also has options for users to enable input/output compression for most workloads with default compression codec (zlib). Some initial work based on this benchmark suite please refer to the included ICDE workshop paper (i.e., WISS10_conf_full_011.pdf).
 
-Note: 
+Note:
  1. Since HiBench-2.2, the input data of benchmarks are all automatically generated by their corresponding prepare scripts.
  2. Since HiBench-3.0, it introduces Yarn support
  3. Since HiBench-4.0, it consists of more workload implementations on both Hadoop MR and Spark. For Spark, three different APIs including Scala, Java, Python are supportive.
@@ -32,7 +32,7 @@ Note:
 2. WordCount (wordcount)
 
     This workload counts the occurrence of each word in the input data, which are generated using RandomTextWriter. It is representative of another typical class of real world MapReduce jobs - extracting a small amount of interesting data from large data set.
-	
+
 3. TeraSort (terasort)
 
     TeraSort is a standard benchmark created by Jim Gray. Its input data is generated by Hadoop TeraGen example program.
@@ -52,7 +52,7 @@ Note:
 6. PageRank (pagerank)
 
     This workload benchmarks PageRank algorithm implemented in Spark-MLLib/Hadoop (a search engine ranking benchmark included in pegasus 2.0) examples. The data source is generated from Web data whose hyperlinks follow the Zipfian distribution.
-	
+
 7. Nutch indexing (nutchindexing)
 
     Large-scale search indexing is one of the most significant uses of MapReduce. This workload tests the indexing sub-system in Nutch, a popular open source (Apache project) search engine. The workload uses the automatically generated Web data whose hyperlinks and words both follow the Zipfian distribution with corresponding parameters. The dict used to generate the Web page texts is the default linux dict file /usr/share/dict/linux.words.
@@ -74,13 +74,15 @@ Note:
 10. enhanced DFSIO (dfsioe)
 
     Enhanced DFSIO tests the HDFS throughput of the Hadoop cluster by generating a large number of tasks performing writes and reads simultaneously. It measures the average I/O rate of each map task, the average throughput of each map task, and the aggregated throughput of HDFS cluster. Note: this benchmark doesn't have Spark corresponding implementation.
-    
+
 **Supported hadoop/spark release:**
 
   - Apache release of Hadoop 1.x and Hadoop 2.x
   - CDH4/CDH5 release of MR1 and MR2.
+  - HDP2.3
   - Spark1.2
   - Spark1.3
+ Note : No version of CDH supports SparkSQL. Please download SparkSQL from Apache-spark official release page if you are using it.
 
 ---
 ### Getting Started ###
@@ -92,39 +94,41 @@ Note:
      Download/checkout HiBench benchmark suite
 
      Run `<HiBench_Root>/bin/build-all.sh` to build HiBench.
-      
+
      Note: Begin from HiBench V4.0, HiBench will need python 2.x(>=2.6) .
 
 2. HiBench Configurations.
 
      For minimum requirements: create & edit `conf/99-user_defined_properties.conf`：
-     
-          cd conf 
+
+          cd conf
           cp 99-user_defined_properties.conf.template 99-user_defined_properties.conf
-     
+
      And Make sure below properties has been set:
 
           hibench.hadoop.home      The Hadoop installation location
           hibench.spark.home       The Spark installation location
           hibench.hdfs.master      HDFS master
           hibench.spark.master     SPARK master
-	  
+
      Note: For YARN mode, set `hibench.spark.master` to `yarn-client`. (`yarn-cluster` is not supported yet)
 
+     To run HiBench on HDP, please specify `hibench.hadoop.mapreduce.home` to the mapreduce home, normally it should be "/usr/hdp/current/hadoop-mapreduce-client". Also please specify `hibench.hadoop.release` to "hdp".
+
 3. Run
 
    Execute the `<HiBench_Root>/bin/run-all.sh` to run all workloads with all language APIs with `large` data scale.
 
 4. View the report:
-   
+
    Goto `<HiBench_Root>/report` to check for the final report:
       - `report/hibench.report`: Overall report about all workloads.
       - `report/<workload>/<language APIs>/bench.log`: Raw logs on client side.
       - `report/<workload>/<language APIs>/monitor.html`: System utilization monitor results.
       - `report/<workload>/<language APIs>/conf/<workload>.conf`: Generated environment variable configurations for this workload.
       - `report/<workload>/<language APIs>/conf/sparkbench/<workload>/sparkbench.conf`: Generated configuration for this workloads, which is used for mapping to environment variable.
       - `report/<workload>/<language APIs>/conf/sparkbench/<workload>/spark.conf`: Generated configuration for spark.
-      
+
    [Optional] Execute `<HiBench root>/bin/report_gen_plot.py report/hibench.report` to generate report figures.
 
    Note: `report_gen_plot.py` requires `python2.x` and `python-matplotlib`.
@@ -134,12 +138,12 @@ Note:
 
 1. Parallelism, memory, executor number tuning:
 
-          hibench.default.map.parallelism       Mapper numbers in MR, 
+          hibench.default.map.parallelism       Mapper numbers in MR,
                                                 partition numbers in Spark
-          hibench.default.shuffle.parallelism   Reducer numbers in MR, shuffle 
+          hibench.default.shuffle.parallelism   Reducer numbers in MR, shuffle
                                                 partition numbers in Spark
           hibench.yarn.executors.num            Number executors in YARN mode
-          hibench.yarn.executors.cores          Number executor cores in YARN mode 
+          hibench.yarn.executors.cores          Number executor cores in YARN mode
           spark.executors.memory                Executor memory, standalone or YARN mode
           spark.driver.memory                   Driver memory, standalone or YARN mode
 
@@ -149,11 +153,11 @@ Note:
 
           hibench.compress.profile              Compression option `enable` or `disable`
           hibench.compress.codec.profile        Compression codec, `snappy`, `lzo` or `default`
-     
+
 3. Data scale profile selection:
 
           hibench.scale.profile                 Data scale profile, `tiny`, `small`, `large`, `huge`, `gigantic`, `bigdata`
-                                                  
+
    You can add more data scale profiles in `conf/10-data-scale-profile.conf`. And please don't change `conf/00-default-properties.conf` if you have no confidence.
 
 4. Configure for each workload or each language API:
@@ -165,7 +169,7 @@ Note:
               workloads/<workload>/<language APIs>/.../*.conf     Configure for various languages
 
      2. For configurations in same folder, the loading sequence will be
-     sorted according to configure file name. 
+     sorted according to configure file name.
 
      3. Values in latter configure will override former.
 
@@ -188,7 +192,7 @@ Note:
           hibench.spark.version          spark1.3
 
 6. Configures for running workloads and language APIs:
-  
+
   The `conf/benchmarks.lst` file under the package folder defines the
   workloads to run when you execute the `bin/run-all.sh` script under
   the package folder. Each line in the list file specifies one
@@ -226,7 +230,7 @@ Note:
    You'll need to install numpy (version > 1.4) in master & all slave nodes.
 
    For CentOS(6.2+):
-     
+
      `yum install numpy`
 
    For Ubuntu/Debian:
@@ -238,7 +242,7 @@ Note:
    You'll need to install python-matplotlib(version > 0.9).
 
    For CentOS(6.2+):
-     
+
      `yum install python-matplotlib`
 
    For Ubuntu/Debian:

bin/build-all.sh

@@ -27,7 +27,7 @@ cd $DIR/src/sparkbench && \
 for mr in MR1 MR2; do
         for spark_version in 1.2 1.3 1.4 1.5; do
                 cp target/*-jar-with-dependencies.jar jars
-                mvn clean package -D spark$spark_version -D$mr
+                mvn clean package -D spark$spark_version -D $mr
                 if [ $? -ne 0 ]; then
                         echo "Build failed for spark$spark_version and $mr, please check!"
                         exit 1

bin/functions/execute_with_log.py

@@ -77,22 +77,23 @@ def execute(workload_result_file, command_lines):
     count = 100
     last_time=0
     log_file = open(workload_result_file, 'w')
-    while True:
+    # see http://stackoverflow.com/a/4417735/1442961
+    lines_iterator = iter(proc.stdout.readline, b"")
+    for line in lines_iterator:
         count += 1
         if count > 100 or time()-last_time>1: # refresh terminal size for 100 lines or each seconds
             count, last_time = 0, time()
             width, height = get_terminal_size()
             width -= 1
 
         try:
-            line = proc.stdout.readline().rstrip()
+            line = line.rstrip()
             log_file.write(line+"\n")
             log_file.flush()
         except KeyboardInterrupt:
             proc.terminate()
             break
         line = line.decode('utf-8')
-        if not line: break
         line = replace_tab_to_space(line)
         #print "{Red}log=>{Color_Off}".format(**Color), line
         lline = line.lower()