PySpark Interview Questions for Data Engineers (with Answers)

PySpark interviews for data engineering roles test whether you understand how Spark actually runs your code: lazy evaluation, stages and shuffles, partitions, join strategies and skew, reliable Delta Lake writes, and a method for debugging slow jobs. The 20 questions below come with short, correct answers and code you should be able to type from memory, checked against the Apache Spark 4.2 and Delta Lake documentation.

At a glance

How Spark runs your code

1. What is lazy evaluation, and why does Spark use it?

Transformations such as filter, select and join do not run when you call them. Spark records them in a plan, and only an action (count, show, collect or a write) starts a job. Waiting lets the optimiser see the whole pipeline first, so it can push filters down to the file scan, read only the columns you use and combine steps.

orders = spark.read.parquet("/lake/bronze/orders")    # reads metadata only
paid = orders.filter(orders.status == "PAID").select("order_id", "amount")
paid.explain()    # prints the plan, still no job
paid.count()      # action: Spark now runs a job

2. What is the difference between narrow and wide transformations?

In a narrow transformation each output partition depends on one input partition: filter, select, withColumn. Spark chains these together inside one stage. A wide transformation needs rows with the same key brought together from many partitions: groupBy, distinct, repartition and most joins. That requires a shuffle, and each shuffle starts a new stage.

3. Explain job, stage and task.

Each action creates a job. Spark splits the job into stages at shuffle boundaries. Each stage runs as a set of tasks, one task per partition, spread across executor cores. A stage with 200 partitions on 40 cores therefore runs in about five waves of tasks.

4. What is a shuffle, and why is it expensive?

A shuffle redistributes data so that it is grouped differently across partitions. The Spark RDD programming guide calls it costly because it involves disk I/O, data serialisation and network I/O. A good answer adds how to avoid unnecessary shuffles: filter rows and drop columns early, broadcast small tables, and never repartition without a reason.

5. What is the difference between cache() and persist()?

Both keep a computed result for reuse. cache() uses the default storage level, while persist() lets you choose one. For DataFrames the default is MEMORY_AND_DISK_DESER, so partitions that do not fit in memory spill to disk; for RDDs, cache() means MEMORY_ONLY. Caching is lazy too: nothing is stored until the first action. Cache only what you reuse across several actions, and release it afterwards.

active = silver_df.filter("is_active = true").cache()
active.count()                      # first action fills the cache
active.groupBy("city").count().show()
active.unpersist()

Partitions

6. How does Spark decide the number of partitions?

When reading files, Spark packs up to spark.sql.files.maxPartitionBytes (128 MB by default) into each partition. After a shuffle it uses spark.sql.shuffle.partitions, which defaults to 200. Adaptive Query Execution (AQE), enabled by default since Spark 3.2, then coalesces small post-shuffle partitions at runtime using real sizes. Mention that a fixed 200 suits neither a tiny job nor a huge one, which is exactly the problem AQE addresses.

7. repartition() or coalesce()?

coalesce(n) only reduces the number of partitions, merging existing ones without a full shuffle. Use it to cut the number of output files after heavy filtering. repartition(n) or repartition(n, "col") performs a full shuffle and can increase or decrease partitions, giving evenly sized, hash-partitioned output. Use it when partitions are badly unbalanced.

df.rdd.getNumPartitions()
fewer = df.coalesce(8)                    # fewer files, no full shuffle
by_state = df.repartition(64, "state")    # full shuffle, grouped by state

8. What is the small files problem, and how do you fix it?

Thousands of tiny files slow every read, because each file adds open and planning overhead. Common causes are writing with too many partitions, frequent streaming micro-batches, and partitionBy on a high-cardinality column such as customer_id. Partition on low-cardinality columns such as a date, coalesce before writing, and on Delta tables run OPTIMIZE to compact files. For tables filtered on high-cardinality columns, liquid clustering (Delta Lake 3.1 and above) is the newer option; it cannot be combined with partitioning or ZORDER.

OPTIMIZE silver.orders ZORDER BY (customer_id);
CREATE TABLE gold.sales (order_id BIGINT, store_id INT) USING DELTA CLUSTER BY (store_id);

Joins and skew

9. Which join strategies does Spark use, and when does it broadcast?

The main ones are broadcast hash join, sort-merge join and shuffle hash join. If one side is smaller than spark.sql.autoBroadcastJoinThreshold (10 MB by default), Spark copies it to every executor and the large side is never shuffled. AQE can also switch a sort-merge join to a broadcast join at runtime once it sees the real size. You can request a strategy with a hint, but the performance tuning guide is clear that Spark may not follow it for every join type.

from pyspark.sql.functions import broadcast
enriched = orders.join(broadcast(stores), "store_id", "left")

10. How do you detect data skew?

In the Spark UI, look for a stage where most tasks finish in seconds and a few run for minutes, or where the maximum task input is far above the median. Then confirm it in the data:

from pyspark.sql import functions as F
orders.groupBy("store_id").count().orderBy(F.desc("count")).show(10)

A frequent culprit is a placeholder key, such as null or "UNKNOWN", shared by a huge number of rows.

11. How do you fix a skewed join?

Work through the options in this order:

1. Check that AQE skew handling is on (spark.sql.adaptive.skewJoin.enabled, true by default). It splits skewed partitions in sort-merge joins. By default a partition counts as skewed when it is over five times the median partition size and larger than 256 MB.
2. Broadcast the smaller side if it fits in memory.
3. Handle null keys separately; in an inner join they never match anyway.
4. Salt the hot key: add a random number on the big side and replicate the small side once per salt value.

N = 8
big = orders.withColumn("salt", (F.rand() * N).cast("long"))
small = stores.crossJoin(spark.range(N).withColumnRenamed("id", "salt"))
joined = big.join(small, ["store_id", "salt"]).drop("salt")

12. Your join returns more rows than the left table. Why?

The key is not unique on the right side, so each left row matches several right rows. That is a data problem, not a Spark bug. Check uniqueness before joining and deduplicate the dimension (question 13 shows how).

stores.groupBy("store_id").count().filter("count > 1").show()

Expect the follow-up on nulls: null never equals null in a normal join condition, so use eqNullSafe when nulls should match each other.

DataFrame code you should write from memory

13. Keep only the latest record for each customer.

from pyspark.sql import Window
w = Window.partitionBy("customer_id").orderBy(F.col("updated_at").desc())
latest = (customers
          .withColumn("rn", F.row_number().over(w))
          .filter("rn = 1")
          .drop("rn"))

Point out that dropDuplicates(["customer_id"]) is not a substitute: it keeps one row per key, but not necessarily the latest one.

14. Why should you avoid Python UDFs?

A regular Python UDF moves data between the JVM and a Python worker, and the optimiser cannot see inside it. Use built-in functions first, because they run inside the engine. If you genuinely need Python logic, a pandas UDF processes whole batches through Apache Arrow instead of one row at a time. The Arrow in PySpark guide notes that from Spark 4.2, Arrow optimisation is on by default for regular Python UDFs, but those still run row by row.

gross = F.round(F.col("net") * (1 + F.col("gst_rate")), 2)    # built-in, no UDF
with_tax = sales.withColumn("gross", gross)

15. row_number, rank or dense_rank?

All three number rows within a window. With a tie for second place, row_number gives 1, 2, 3, 4; rank gives 1, 2, 2, 4; dense_rank gives 1, 2, 2, 3. Use row_number for deduplication and dense_rank for questions like "top three salaries per department".

Delta Lake

16. What does Delta Lake add on top of Parquet?

A transaction log (_delta_log) that records every commit. That gives you ACID transactions, so readers never see half-written data; schema enforcement on write; time travel to earlier versions; and MERGE, UPDATE and DELETE on files in a data lake. Delta is also the table format of the Microsoft Fabric lakehouse, which is why Fabric interviews ask about it.

17. Write an upsert into a Delta table.

from delta.tables import DeltaTable
target = DeltaTable.forName(spark, "silver.customers")
(target.alias("t")
   .merge(updates.alias("s"), "t.customer_id = s.customer_id")
   .whenMatchedUpdateAll()
   .whenNotMatchedInsertAll()
   .execute())

The usual follow-up: a merge can fail when several source rows match the same target row, because Delta cannot tell which one should win. Deduplicate the source first with the pattern from question 13.

18. Schema enforcement versus schema evolution.

Delta validates every write by default. If the DataFrame has a column the table lacks, or a data type does not match, the write fails instead of quietly corrupting the table. Evolution is opt-in for each write: mergeSchema adds the new columns. Replacing a table's schema is a separate, deliberate step using overwriteSchema.

(new_batch.write.format("delta")
   .mode("append")
   .option("mergeSchema", "true")
   .saveAsTable("silver.customers"))

19. How do time travel and VACUUM interact?

Every commit creates a version that you can inspect, query and restore:

DESCRIBE HISTORY silver.customers;
SELECT * FROM silver.customers VERSION AS OF 12;
RESTORE TABLE silver.customers TO VERSION AS OF 12;

VACUUM deletes data files that versions inside the retention period no longer need; the default retention is 7 days. After a vacuum you cannot time travel further back than that, and restoring a version whose files are gone fails. Table history itself is kept for 30 days by default, and the Delta utility commands page notes that vacuum is not triggered automatically.

Debugging

20. A job that took 10 minutes now takes an hour. What do you do?

Describe a method, not a guess:

1. Compare the inputs. Did the data grow, or did the number of files explode?
2. Open the Spark UI. Find the slowest stage and check the spread of task times (skew), shuffle read size and spill to disk.
3. Read the plan. df.explain(mode="formatted") shows whether filters reach the scan and which join strategy was chosen. A broadcast join that became a sort-merge join after a table grew is a classic cause.
4. Check partition counts. Too few leaves cores idle; thousands of tiny partitions waste time on scheduling.
5. Change one thing, rerun and compare. Then write down what fixed it.

Preparing for PySpark interviews in India

If a job description mentions Microsoft Fabric, Azure Databricks or Azure Synapse, you will run Spark on a managed platform, so know its version. Microsoft's Apache Spark runtimes in Fabric page lists Runtime 2.0, the current generally available runtime, with Apache Spark 4.1 and Delta Lake 4.2, while Runtime 1.3 (Spark 3.5) has reached the end-of-support-announced stage. Defaults change between versions, so say which version your answer assumes.

Three habits make the biggest difference in preparation:

• Practise on a dataset you care about, end to end: raw files in, a clean Delta table out. Our guide to the medallion architecture in Fabric gives you a structure to follow.
• Explain trade-offs out loud. "I would broadcast here because the dimension is small, but if it grows past the threshold Spark falls back to sort-merge" beats a memorised definition.
• Know when not to use Spark. Interviewers often ask when Pandas is the better tool; Pandas vs PySpark covers it, and the data engineer roadmap for India shows where PySpark sits among the other skills.

How CertTulen Academy can help

If you are preparing for data engineering interviews, our Practical Data Engineering with Python and PySpark course works through these topics in hands-on labs: partitions, joins, skew and Delta Lake merges. If your target roles mention Microsoft Fabric, the Microsoft Fabric Data Engineer (DP-700) course adds lakehouses, pipelines and the certification itself, with an exam voucher, mock practice tests and lab access included (see the course page for details). Classes are live, online and MCT-led. Contact us for a batch that suits your time zone, or WhatsApp +91 93618 83583.