About
distinct([numTasks])) is a transformation that returns a new data set (RDD) that contains the distinct elements of the source data set.
Articles Related
Example
rdd2 = sc.parallelize([1,4,2,2,3])
rdd2.distinct()
[1,4,2,2,3] → [1,4,2,3]
Spark - Distinct
