CloudGuild · Blog · Cheat sheets · Lessons · Certifications
Databricks Certified Data Engineer Associate Exam Question Walkthrough
Understand a common question about reading JSON data and writing to Delta tables in PySpark for the DBX-DEA exam.
You may find that many candidates trip over the specific operations for reading and writing data in PySpark. Knowing the precise syntax and methods is essential for success.
The question
You are using PySpark to read data from a JSON file and write it into a Delta table. Which of the following statements correctly represents this operation?
- A. df.write.format('json').save('/delta/table')
- B. df.write.format('delta').saveAsTable('table_name')
- C. spark.read.json('/path/to/file.json').write.save('/delta/table')
- D. spark.read.json('/path/to/file.json').write.format('delta').saveAsTable('table_name')
Think before you scroll
Before you choose an answer, consider the requirements for writing data to a Delta table. Pay attention to the methods and formats used in each option. Not all combinations of read and write operations will yield a valid result.
The answer
The correct option is D. It correctly reads the JSON file and writes it to a Delta table using the saveAsTable() method, which is specifically designed for creating managed tables in Databricks.
Why the other options lose
- A: This option uses
df.write.format('json'), which is incorrect because it specifies the output format as JSON instead of Delta. Delta is the required format for writing to a Delta table. - B: Here, while it mentions
format('delta'), it does not read from a file. It assumes thatdfis already defined, but it fails to show the JSON read operation that must occur first. - C: This option reads the JSON file correctly, but it uses
write.save('/delta/table'). This is incorrect because it does not specify the Delta format or usesaveAsTable(), which is necessary for saving as a Delta table.
The concept behind it
When working with PySpark and Delta tables, it is crucial to read the data in the correct format and then use appropriate methods for writing. The saveAsTable() method is specifically tailored for saving data into Delta tables, while formats must align with the types of tables being created or modified.
Exam trap to remember
Always remember: to write to a Delta table, use saveAsTable() with the correct format specified. Reading and writing operations must align perfectly for successful data management in Databricks.