Databricks Certified Data Engineer Associate: Certified Data Engineer Associate

Incorrect. Time travel (querying a table AS OF VERSION/TIMESTAMP) is a read operation that uses the transaction log to access older snapshots. It does not delete data files. If time travel fails, it’s because the referenced files are missing (typically due to VACUUM), not because time travel itself removed anything.

Incorrect. “DELETE HISTORY” is not a typical Delta Lake command used to remove historical versions and physical files in Databricks. Delta history is maintained in the transaction log, and physical cleanup is handled by VACUUM. While you can limit log retention via configuration, the standard mechanism that deletes data files is still VACUUM.

Incorrect. OPTIMIZE compacts many small files into fewer larger files for performance. It creates new optimized files and marks old files as removed in the transaction log, but those old files are not physically deleted immediately. They remain available for time travel until VACUUM is run and the retention period has elapsed.

Incorrect. HISTORY (DESCRIBE HISTORY) only displays the table’s commit history and operation metadata. It is purely informational and does not modify the table, transaction log, or underlying data files. It cannot cause data files to disappear or prevent restoring an older version.

Spark SQL array functions are not primarily about working with “a variety of types at once.” They operate on array-typed columns (arrays of a single element type, possibly complex like struct). While arrays can contain complex elements, the benefit is not generic multi-type processing; it’s targeted manipulation of array data (access, explode, transform, filter).

Working within partitions and windows is the domain of window functions (for example: row_number, rank, lag/lead, sum over(partition by ... order by ...)). Array functions do not define partitioning semantics or window frames. You might use arrays in combination with window results, but the partition/window capability itself is not provided by array functions.

Time-related intervals are handled by Spark SQL datetime functions and time-window constructs (date_add, add_months, datediff, timestampadd, window for event-time aggregations). Array functions do not provide interval arithmetic or time bucketing. This option confuses array operations with temporal processing features in Spark SQL.

Spark SQL does not provide an “array of tables” concept for procedural automation via array functions. Automation is typically done with Databricks Workflows/Jobs, notebooks, DLT pipelines, or external orchestration tools. Array functions operate on array columns within a dataset, not on collections of tables as procedural objects.

Incorrect. Reproducibility is achieved by controlling code and environment: using Databricks Repos/Git, pinning the Databricks Runtime version, managing libraries (e.g., requirements), and using consistent job configurations. Cluster pools do not guarantee identical execution environments; they only speed up cluster provisioning by reusing warm instances.

Incorrect. Testing to identify errors is primarily a software engineering/CI concern: unit/integration tests, job runs in a test workspace, and automated pipelines. While pools can make test clusters start faster, they are not the key feature for testing correctness; they address performance/latency of cluster startup, not test capability.

Incorrect. Version control across multiple collaborators is handled through Databricks Repos with Git providers (GitHub, Azure DevOps, GitLab, etc.) and collaboration workflows (branches, PRs). Cluster pools are unrelated to code versioning; they manage compute instance availability to reduce cluster start times.

Incorrect. Making a report runnable by all stakeholders is about permissions and sharing (workspace access, SQL Warehouses, dashboards, jobs permissions, Unity Catalog privileges). Pools can be shared, but they do not inherently make artifacts runnable by stakeholders; access control and appropriate compute endpoints are the determining factors.

The ability to manipulate the same data using a variety of languages is primarily a Databricks/Spark capability (SQL, Python, Scala, R) rather than a Delta Lake feature. Delta Lake defines the table/storage format and transaction log, but language flexibility comes from Spark APIs and Databricks notebooks supporting multiple languages against the same underlying data.

Real-time collaboration on a single notebook is a Databricks Workspace feature (collaborative editing, comments, permissions). Delta Lake is a storage layer and does not provide notebook collaboration capabilities. This option can be tempting because it’s a “Lakehouse Platform” benefit, but it is not attributable to Delta Lake specifically.

Setting up alerts for query failures is handled by orchestration/monitoring features such as Databricks Jobs notifications, Databricks SQL alerts, and external observability tools. Delta Lake does not provide alerting; it provides transactional storage and table management features. Failures can be detected via job runs/logs, not via Delta Lake itself.

Distributing complex data operations is a core capability of Apache Spark’s distributed compute engine and the Databricks runtime (clusters, parallelism, shuffle, etc.). Delta Lake complements Spark by adding ACID and table reliability on object storage, but it is not the component responsible for distributing computation.

Incorrect. Delta tables are not stored in a single file. Delta Lake relies on many Parquet data files plus a transaction log directory (_delta_log) containing multiple log and checkpoint files. A single-file representation would not support Delta’s scalable ACID transactions, concurrent writes, and incremental commits.

Incorrect. Delta does not store all data in a single file. Data is stored across many Parquet files (often partitioned). While metadata/history is stored as files in _delta_log, it is not typically described as being in a separate location from the table; it is a subdirectory within the same table path.

Incorrect. Delta tables are not only data files. While the data itself is stored as Parquet files, Delta’s defining feature is the transaction log (_delta_log) that stores metadata and history (add/remove actions, schema, properties, protocol). Without these additional files, it would just be a plain Parquet dataset, not a Delta table.

Incorrect. This describes neither Delta nor typical lakehouse storage. Delta tables are not single-file and do not store only data. They require multiple files for scalable storage and a transaction log for ACID transactions, versioning, and metadata management.

SELECT * FROM my_table WHERE age > 25 returns only rows matching the condition in the query result, but it does not change the underlying Delta table. This is a common confusion between filtering data during retrieval versus modifying stored data. The table remains unchanged after the query completes, so it does not “remove rows” or “save the updated table.”

UPDATE my_table WHERE age > 25 is not valid SQL syntax for updating rows because UPDATE requires a SET clause (for example, UPDATE ... SET col = value WHERE ...). Even if corrected, UPDATE modifies values in existing rows rather than deleting them. Therefore it cannot be used to remove rows from the table.

UPDATE my_table WHERE age <= 25 is also invalid syntax due to the missing SET clause. More importantly, UPDATE changes data values rather than removing rows. Even if the intent were to keep only age <= 25, UPDATE cannot delete the other rows; you would still need DELETE with the opposite predicate or a rewrite operation.

DELETE FROM my_table WHERE age <= 25 is valid Delta Lake syntax, but it deletes the wrong set of rows. The question asks to remove rows where age > 25; this option would remove rows with age 25 or less, leaving the age > 25 rows behind. It’s a predicate-direction trap.

Commit can be performed within Databricks Repos. You can stage changes from notebooks/files in the repo and create a commit that records those changes in the local repo state within the Databricks workspace. This is a core supported workflow: edit notebook/code in Repos, commit the changes, and then push them to the remote repository.

Pull is supported in Databricks Repos to bring down the latest changes from the remote repository into the Databricks workspace copy. This is commonly used after teammates merge changes via PR/MR in the Git provider. While pull may incorporate remote updates, it does not replace the need for performing explicit branch merges outside Repos.

Push is supported in Databricks Repos. After committing changes in your Databricks workspace repo, you can push those commits to the remote Git provider. This enables collaboration and integration with CI/CD systems. Note that pushing may be restricted by branch protections in the Git provider, which is handled outside Databricks.

Clone is supported in Databricks Repos and is typically the first step: linking a remote Git repository and creating a workspace copy. Databricks Repos provides UI-driven cloning by specifying the repository URL and credentials/integration. This operation is explicitly part of the Repos feature set.

Storing structured and unstructured data is a hallmark of data lakes and lakehouses, but it does not inherently improve data quality. Traditional data lakes already store many data types. Quality issues (duplicates, partial writes, inconsistent reads) can still occur without transactional guarantees and table management features.

SQL support improves accessibility and analytics usability, but it does not guarantee higher data quality. You can query low-quality or inconsistent data with SQL just as easily as high-quality data. Quality improvements require controls on how data is written and managed (e.g., transactions, constraints, enforcement).

Open formats (e.g., Parquet) improve interoperability and reduce vendor lock-in, but they do not by themselves prevent inconsistent or partial writes, nor do they enforce correctness. Many traditional data lakes already use open formats; quality issues still arise without transactional and governance features.

Enabling ML/AI workloads is a capability of the platform, not a mechanism that improves data quality. ML can even amplify data quality issues if the underlying data is inconsistent or corrupted. Quality improvements come from reliable ingestion, transactional storage, and enforcement/governance features.

A Databricks account representative is not part of your workspace’s security boundary and does not administer your Unity Catalog objects. They can help with licensing, support escalation, and account-level guidance, but they do not log in to your environment to change table ownership. Exam pattern: vendor reps are almost never the correct choice for operational governance tasks.

This transfer is possible. Databricks governance is designed to handle offboarding and prevent orphaned data assets. Administrators can manage securables and reassign ownership or adjust privileges even when the original owner is no longer present. If you see “not possible,” it’s usually a distractor unless the question explicitly states there is no admin access at all.

The new lead data engineer may be the desired recipient of ownership, but they are not necessarily authorized to perform the transfer. Unless they also hold an administrative role (workspace admin/metastore admin) or have been delegated the necessary privileges, they cannot override current ownership. The question asks who must perform the transfer, which points to an admin.

The original data engineer cannot transfer ownership because the scenario states they no longer have access. Even if they were the owner, ownership transfer requires them to be able to authenticate and execute the change. This option is included to test whether you notice the constraint that the original owner is unavailable.

SELECT * FROM sales is a SQL query, not a PySpark command by itself. In a Python notebook or script, this text alone would not execute unless passed into spark.sql as a string. The question asks for a command the team could use in PySpark, so this option is incomplete and not valid as written. A valid PySpark equivalent would be spark.sql("SELECT * FROM sales").

This statement is false because Databricks supports interoperability between SQL and PySpark through the same Spark engine and shared metastore or catalog. A table created or queried in SQL can also be accessed from Python if the user has the necessary permissions. Delta tables are specifically designed to be usable across Spark APIs and languages. Therefore, sharing data between SQL and PySpark is a normal and common workflow.

spark.sql("sales") is not valid because spark.sql expects a complete SQL statement, not just a table name. Passing only "sales" would cause a SQL parsing error since it is not a standalone query. If the team wanted to use spark.sql, they would need to write something like spark.sql("SELECT * FROM sales"). This makes the option incorrect as written.

DESCRIBE LOCATION customer360 is not a standard Spark SQL/Databricks SQL command for retrieving a database’s location. Learners sometimes confuse this with commands that describe objects or with Delta’s DESCRIBE DETAIL. Because the syntax is not valid for database metadata inspection, it will not reliably return the database location.

DROP DATABASE customer360 removes the database (schema) and optionally its contents (depending on CASCADE/RESTRICT). It is a destructive DDL operation and does not return the database location. On an exam, any DROP option is a strong signal it’s not meant for “return information” questions.

ALTER DATABASE customer360 SET DBPROPERTIES ('location' = '/user'} attempts to set a custom property named 'location', but DBPROPERTIES are not the same as the database’s actual Location field. Also, the option contains a syntax issue (mismatched braces/quotes). Even if corrected, it would not be the right way to retrieve the location; it modifies metadata instead of returning it.

USE DATABASE customer360 changes the current database context for subsequent SQL statements. It does not display or return the database’s location. While it’s commonly used before creating tables or running queries, it is not a metadata inspection command.

INNER JOIN is not the right operation for appending rows. A join combines tables horizontally (adds columns) based on a join condition; without an ON clause it becomes invalid SQL in most dialects or a cross join in some contexts, which would multiply rows. It would not simply stack March and April transactions into one table and could drastically change row counts.

OUTER JOIN is also a horizontal combination of two tables, intended to match rows by keys and preserve non-matching rows from one or both sides. It requires a join condition to be meaningful. Even with a condition, it would produce a wider table (more columns) and potentially null-extended rows, not a single unified list of transactions.

INTERSECT returns only rows that appear in both tables (the overlap). Since the question states there are no duplicate records between the tables, the intersection would be empty (or near empty), which is the opposite of the goal. INTERSECT is used for finding common records, not for combining all records.

MERGE is a DML operation used to upsert into an existing target table based on a matching condition (WHEN MATCHED/WHEN NOT MATCHED). It is not a set operator for combining two SELECT statements into a new table via CTAS. Even conceptually, MERGE requires keys and match logic; it’s overkill and incorrect for simply appending two monthly transaction tables.

Incorrect. Spark SQL/Databricks does not use a 10 GB threshold (or any size threshold) to decide whether DROP TABLE deletes data. Data deletion behavior is determined by table ownership (managed vs external), not by the amount of data stored.

Incorrect. Table size being smaller than 10 GB is not relevant to DROP TABLE semantics. DROP TABLE removes the metastore entry; whether data is deleted depends on whether the table is managed (owned) or external (not owned).

Incorrect. Not having an explicit LOCATION does not imply that data will remain after DROP TABLE. In fact, tables without a specified LOCATION are commonly managed tables (stored in the default warehouse location), and dropping a managed table typically deletes both metadata and data.

Incorrect. Managed tables are owned by the metastore. For managed tables, DROP TABLE generally deletes both the metadata and the underlying data files. The observed behavior (files still present) is the opposite of what you’d expect for a managed table.

A database, also called a schema in many contexts, is a logical container used to organize tables, views, and other objects. It does not itself represent a derived dataset created from combining source tables. While a database can contain the final object, it does not materialize query results to storage. Therefore it does not satisfy the requirement to create a reusable physical data entity.

A function encapsulates reusable logic, such as a scalar or table-valued computation, but it does not create and persist a dataset. Functions are invoked at query time and return computed results rather than storing data files in a physical location. Even if shared across sessions, they are not the right abstraction for a saved data entity built from tables. This fails the physical persistence requirement in the question.

A standard view is a persistent metadata object that stores only the SQL definition of a query, not the query results themselves. Other engineers can use it across sessions, but each query against the view reads from the underlying tables and recomputes the result. Because the data is not materialized to a physical storage location, a view does not meet the stated requirement. Candidates sometimes confuse persistent metadata with persisted data, which is the key distinction here.

A temporary view is limited to the current Spark session or notebook context and disappears when that session ends. It is useful for intermediate transformations during development, but it is not available to other engineers in other sessions. It also does not write the resulting dataset to a physical storage location. For both scope and persistence reasons, it is not correct here.

Unity Catalog focuses on governance: centralized permissions, data discovery, auditing, and lineage across workspaces. While lineage and auditing can help investigate quality issues, Unity Catalog does not provide native, automated data quality rule evaluation (expectations) with pass/fail metrics during pipeline execution. It’s complementary to quality tools but not the primary solution for monitoring quality levels automatically.

Data Explorer is primarily a user interface experience for exploring data, running queries, and visualizing results. It can be used manually to inspect data quality trends, but it does not automate quality checks or enforce/track data quality rules as part of ingestion and transformation. For exam purposes, it’s not considered a data quality monitoring automation tool.

Delta Lake provides reliability features such as ACID transactions, schema enforcement/evolution, and table constraints (e.g., CHECK constraints, NOT NULL). These can prevent invalid data from being written, which improves quality enforcement. However, Delta Lake alone doesn’t provide pipeline-level automated monitoring dashboards/metrics for quality levels over time in the way DLT expectations and event logs do.

Auto Loader is optimized for incremental file ingestion from cloud storage with features like schema inference, schema evolution, and scalable directory listing/notification services. It helps ingest data reliably and efficiently, but it does not provide a native framework for defining and monitoring data quality rules with pass/fail metrics. Auto Loader is often used with DLT, where DLT handles quality monitoring.

Incorrect because it claims compute resources will persist to allow for additional testing. That behavior aligns more with Development mode expectations (interactive iteration/debugging). In Production mode, the pipeline is meant to run operationally; resources are not kept around for “testing” once you stop the pipeline. The continuous updating part is plausible, but the compute persistence rationale is wrong.

Incorrect because Continuous mode does not run “once and then persist without processing.” Continuous pipelines keep processing as new data arrives (after handling any backlog). Even if there is a temporary lull in new data, the pipeline remains active and ready to process, rather than being a one-shot update that idles indefinitely by design.

Incorrect because it describes a triggered/one-time execution: update once and shut down, terminating compute. That is characteristic of Triggered Pipeline Mode (or a manual one-shot update), not Continuous mode. The question explicitly states Continuous Pipeline Mode, so the pipeline should not automatically shut down after a single update.

Incorrect because it combines a one-time update/shutdown (triggered behavior) with compute persisting for testing (development-like behavior). Continuous mode contradicts the “updated once and shut down” claim, and Production mode contradicts “persist to allow for additional testing.” This option mixes two incorrect assumptions.

Incorrect. Structured Streaming does record offset ranges for each trigger when checkpointing is enabled. Without checkpointing, recovery is limited and progress may be lost, but the engine is explicitly designed to persist offsets/metadata to durable storage for fault tolerance. The statement contradicts fundamental Structured Streaming design.

Incorrect for this question. Replayable sources and idempotent sinks are important for end-to-end exactly-once behavior, but they are not the two approaches Spark uses to record the offset range per trigger. Offset recording is done via checkpointing and offset/commit logs; source/sink properties affect whether replays cause duplicates.

Incorrect. Write-ahead/offset logs are part of the progress tracking story, but “idempotent sinks” are not the mechanism Spark uses to record offset ranges. Idempotent sinks help tolerate reprocessing without duplicate outputs; they do not replace checkpointing, which is where the offset logs are stored and managed.

Incorrect. Checkpointing is essential, and idempotent sinks help with output correctness, but the question asks specifically about recording offset ranges per trigger. Spark records offsets via checkpointing plus offset/commit (WAL-style) logs. Idempotent sinks are complementary for exactly-once output, not the offset-recording mechanism.

Incorrect. While Gold tables are often more directly aligned to business use cases, “valuable” is subjective and not a defining property of the Medallion layers. Silver can be extremely valuable as the reusable, conformed foundation for many products. The architecture defines refinement and purpose, not an absolute measure of value.

Incorrect. Gold is generally more refined and more purpose-built than Silver, not less. Silver is the cleaned/conformed layer, whereas Gold is the curated serving layer (often with business logic, dimensional modeling, and aggregates). A “less refined view” would more closely describe Bronze, not Gold.

Incorrect. Gold does not necessarily contain more data than Silver. Because Gold frequently includes aggregations and curated subsets for specific use cases, it often has fewer rows than Silver. Silver commonly retains detailed, conformed records that can feed many Gold tables, so it can be larger in volume.

Incorrect. “Truthful” is not guaranteed by being in Gold. Data quality and correctness depend on validation rules, expectations, and governance applied across the pipeline. Silver is often where many quality controls are enforced; Gold may add business logic and aggregation but is not inherently more truthful than Silver.

Incorrect. Bronze tables are not defined by containing less data than the raw files. In many implementations, Bronze preserves all source records and may even include extra metadata columns such as ingestion timestamps, file names, or batch identifiers. The relationship is about structure and manageability, not reduced volume.

Incorrect. Bronze tables are not inherently more truthful than raw data. They are intended to preserve the source data as landed, but they may still contain duplicates, malformed records, late-arriving data, and other source-system issues. The key distinction is that Bronze makes raw data queryable with schema, not that it improves correctness.

Incorrect. Aggregated data belongs more commonly in Gold tables, and sometimes in later Silver transformations depending on the use case. Bronze tables are meant to capture detailed source-level records with minimal transformation rather than summarized or rolled-up data. Therefore this does not describe the Bronze-to-raw-data relationship.

Incorrect. Raw data is the least refined representation because it is typically stored exactly as received from the source, often in files such as JSON or CSV. Bronze tables are slightly more refined because they organize that raw data into a managed table with schema and often ingestion metadata. So Bronze is not less refined than raw data; it is raw data in a structured table form.