Spark - Web UI (Driver UI)

About

Each driver program has a web UI, typically on port 4040, that displays information about:

running tasks,
executors,
and storage usage.

The Spark UI will tell you which DataFrames and what percentages are in memory. Spark - (Executor) Cache

This UI is only available for the duration of the application. The cluster manager UI's (master en worker) are always available. Example with the built-in standalone cluster manager, the UI by default are available at the port 8080 for the master and 8081 for the workers.

Articles Related

Management

URL

http://<driver-node>:4040

where:

Spark - Driver