PySpark coding practice

Hands-on DataFrame problems that come up in data-engineering interviews — windows, joins, skew, sessionization, complex types, performance. Each card has the task; click Show solution to reveal idiomatic code and the gotcha interviewers probe.


  


  
Concepts at a glance

  

    

      
Window: partition → order → frame

      
A window picks a partition, sorts it, then a moving frame feeds the aggregate.

      
partitionBy("dept") · orderBy("ts") · rows unboundedPreceding → currentRow

      

      
blue = partition · green = frame · gold = current row

    


    

      
Data skew → salting

      
One hot key overloads a task. Split it into N salted sub-keys so work spreads.

      
before — one task does everything

      
key="US"

      
after — salt = rand()%N, join on (key, salt)

      

    


    

      
Sessionization (30-min gap)

      
New session starts when the gap from the previous event exceeds the threshold.

      

        

      

      
a gap > 30 min flips new_session=1; a running sum of those flips = session id

    


    

      
Narrow vs wide (shuffle)

      
Narrow = data stays on its partition. Wide = data crosses the network (shuffle).

      
narrow — map, filter, withColumn, union

      
P0→P0
        P1→P1

      
wide — groupBy, join, distinct, repartition, orderBy

      
P0P1
        ⇄ shuffle ⇄P0'P1'