Skip to content

L — Looping

Rules that detect DataFrame operations inside Python loops. Iterating over a DataFrame or calling DataFrame transformations in a loop prevents Spark from building an optimal query plan and causes plan bloat.

Rule Title
L001 Avoid looping without .localCheckpoint() or .checkpoint()
L002 Avoid while loops with DataFrames
L003 Avoid calling withColumn() inside a loop