RDD - Partition

About

Partition in RDD

Articles Related

Managememnt

set

• parrallelize. (Example for two)

rdd = sc.parallelize([1, 2, 3, 4], 2)

get

• PySpark

rdd.getNumPartitions

mapPartitions

Return a new RDD by applying a function to each partition of this RDD.

• Scala
• Java
• PySpark