Arena · Data Engineering · Coding · advanced

PySpark transformation

Write a PySpark job that: (1) reads a JSON file from S3, (2) filters out rows where "status" is null, (3) adds a "processed_date" column with today's date, (4) writes the output as Parquet partitioned by "country". Use the DataFrame API.

Sign in to see what a strong answer covers and to get AI feedback on your own.

Practice this question on PrepGraph