Table of Contents

Spark - History Server

About

job history server

For map reduce job, see history server

It lists the following jobs type:

Management

Start

./sbin/start-history-server.sh

URL

web interface: http://server:18080 by default

For azure hdinsight: https://clusterName.azurehdinsight.net/sparkhistory/

Configuration

(see spark.history.provider below),

file-system provider

spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs

Documentation / Reference