Practice Test #1

Incorrect. A hyperparameter grid is typically built with ParamGridBuilder and used by tuning components like CrossValidator or TrainValidationSplit. The grid itself is not an Estimator; it’s a configuration object listing parameter combinations to try. Estimators are the algorithms being tuned (e.g., LogisticRegression), not the grid describing candidate settings.

Incorrect. Chaining multiple algorithms together to specify an ML workflow describes a Pipeline (an Estimator) or a PipelineModel (a Transformer) depending on whether it is fit yet. While a Pipeline is indeed an Estimator, the statement is not the definition of an Estimator in general; it describes a specific workflow container rather than the core Estimator concept.

Incorrect. This describes a Transformer (often a trained model) because it takes a DataFrame and outputs a new DataFrame with additional columns such as predictions, probability, or transformed features. In Spark ML, the trained model returned by an Estimator’s fit method is usually a Transformer (e.g., LogisticRegressionModel).

Incorrect. An evaluation tool is an Evaluator (e.g., RegressionEvaluator, MulticlassClassificationEvaluator, BinaryClassificationEvaluator). Evaluators compute metrics from predictions and labels but do not train models and do not produce Transformers. They are commonly used alongside Estimators in tuning workflows like CrossValidator.

Incorrect. .loc is a pandas DataFrame indexer used for label-based selection and boolean masking. Spark DataFrames do not support .loc, because Spark DataFrames are not indexed the same way and operate via distributed query plans. This option reflects pandas syntax, not PySpark/Databricks Spark DataFrame operations.

This explanation is inaccurate. In PySpark, DataFrame.__getitem__ supports passing a Column expression, so spark_df[spark_df["discount"] <= 0] can return a filtered DataFrame. Although filter()/where() is the more idiomatic and exam-expected syntax, saying this form is invalid or not a Spark API pattern is technically misleading.

Incorrect. This is another pandas-style .loc usage (and the parentheses/argument structure also resembles pandas). Spark DataFrames do not provide .loc for row/column selection. In Spark, you filter rows with filter()/where() and select columns with select(), not with pandas indexers.

Incorrect. mapInPandas is a DataFrame method used as df.mapInPandas(func, schema) to transform partitions and return pandas DataFrames, not a Column expression usable inside withColumn. Also, mapInPandas requires an explicit output schema and is not invoked as a standalone function in this context.

Incorrect. Iterator(spark_df) is not a valid way to supply Spark DataFrame data to a Pandas UDF. Spark controls the iterator of pandas batches internally during execution. You never manually wrap a Spark DataFrame in an Iterator to call a Pandas UDF; instead you pass Column expressions and Spark handles batching and distribution.

Incorrect. This mixes APIs incorrectly. mapInPandas is not used inside withColumn, and predict(spark_df.columns) is not a valid call because spark_df.columns is a Python list of strings, not Spark Columns. Even if predict were called, it would not return a Spark Column expression suitable for withColumn.

Incorrect. Passing spark_df.columns provides a single Python list argument (of strings) to the UDF, not multiple Spark Column arguments. Scalar Pandas UDFs require Spark Column inputs; they cannot accept a Python list of column names as a single argument in a DataFrame expression. The correct approach is to expand the list: predict(*spark_df.columns).

Incorrect. Databricks Feature Store exposes feature table metadata programmatically. The FeatureStoreClient.get_table method returns a FeatureTable object that includes metadata such as the description. This option may seem plausible if you assume descriptions are only visible in the UI, but they are accessible via the API.

Incorrect. create_training_set is used to create a TrainingSet object by specifying feature lookups and a label DataFrame. It does not retrieve feature table metadata and does not accept a feature table name alone as shown. It’s for assembling training data, not inspecting table properties.

Incorrect. load_df() loads the feature table’s data into a Spark DataFrame. A DataFrame contains rows/columns of feature values, not the table’s metadata description. While you could inspect schema from the DataFrame, you cannot retrieve the Feature Store description from it.

Incorrect (incomplete). fs.get_table("new_table") returns the FeatureTable object, but by itself it does not “return the metadata description”; it returns the whole metadata object. To specifically return the description string, you must access the .description property (as in option C).

Incorrect. Type hints are not the defining advantage of pandas UDFs. Spark requires explicit return types for UDFs (and pandas UDFs) via Spark SQL types, and while Python type hints may be used in code style, they are not the core benefit tested. The key differentiator is vectorized execution with Arrow, not typing support.

Partially true but not the best answer. pandas UDFs do allow you to write logic using pandas/NumPy operations inside the function because inputs arrive as pandas objects. However, the exam question asks for a benefit compared to standard UDFs; the primary, canonical benefit is vectorized batch processing (and Arrow-based transfer), not merely that pandas APIs can be used.

Incorrect. Both standard PySpark UDFs and pandas UDFs operate on distributed Spark DataFrames; distribution is a property of Spark, not a unique advantage of pandas UDFs. pandas UDFs still run per partition/executor like other Spark transformations, so this does not distinguish them from standard UDFs.

Incorrect. pandas UDFs do not inherently guarantee in-memory processing or prevent spilling to disk. Spilling is determined by Spark’s execution plan, shuffle operations, partition sizes, and memory configuration. While Arrow can improve transfer efficiency, it does not change Spark’s fundamental memory management or eliminate disk spill behavior.

spark_df.describe() is a Spark DataFrame method that returns a new DataFrame containing basic descriptive statistics (count, mean, stddev, min, max) for numeric columns (and count/min/max for non-numeric). It does not generate visualizations such as histograms. It’s useful for quick numeric summaries but does not meet the requirement for “visual histograms.”

dbutils.data(spark_df).summarize() is not the correct API shape. In Databricks, dbutils.data is a utility namespace and summarize is called as a function (dbutils.data.summarize(df)), not by calling dbutils.data(df) as if it were a constructor returning an object with methods. This option is therefore syntactically/semantically incorrect for the intended summarize feature.

This task can be accomplished in a single line in Databricks notebooks using the built-in summarize helper. While in pure Spark (outside Databricks) you would typically need multiple steps (compute bins, then plot), Databricks provides a one-line EDA summary that includes histograms. So it is incorrect to claim it cannot be done in one line.

spark_df.summary() is another Spark DataFrame method that extends describe() by allowing additional statistics (e.g., percentiles) depending on parameters. Like describe(), it returns a DataFrame of summary statistics and does not automatically render histograms or other visual distribution plots. It helps with numeric profiling but not with visual histogram output.

Incorrect. Supplying all features in a Spark DataFrame is not what enables fs.score_batch to work with only customer_id. If you already have all feature columns, you could score with standard MLflow/Spark methods. fs.score_batch’s key value is automatic feature retrieval, which requires Feature Store model logging metadata.

Incorrect as the best answer. If the model truly uses only customer_id as its sole input feature, then the provided batch_df would indeed contain the required input column and scoring could work. However, this is a narrow special case and not the Feature Store-specific condition being tested by fs.score_batch. The generally correct condition is that the model was logged with Feature Store feature metadata, which allows Databricks to automatically look up additional features using customer_id.

Incorrect. The code can perform the desired inference in a valid and common situation: when the model is Feature Store–logged with feature lookups. In that case, fs.score_batch will enrich batch_df with the required features and return predictions.

Incorrect. Features do not need to be in a single Feature Store table. Feature Store supports training sets built from multiple feature tables via multiple lookups. What matters is that the model was logged with the feature lookup metadata so Feature Store knows which tables/features to retrieve and how to join them.

17.0 is the RMSE from the worst-performing validation fold only. Cross-validation is designed to summarize performance across all folds, not to report only the maximum error. While the worst fold can be useful for diagnosing instability, it is not the overall CV RMSE.

12.0 is the RMSE from one specific fold (and also the median of the three values). Although median can be used as a robust summary in some analyses, the conventional and expected definition of overall cross-validation RMSE for exam questions is the mean across folds, not the median or a single fold value.

39.0 is the sum of the fold RMSE values (10 + 12 + 17). Summing metrics across folds is not a standard way to report cross-validation performance because it scales with the number of folds and is not directly interpretable as an error measure for the model.

10.0 is the RMSE from the best-performing validation fold only. Selecting the minimum fold metric would overstate expected generalization performance and defeats the purpose of cross-validation. The overall CV RMSE should reflect performance across all folds, typically via the mean.

This is not why the code fails. While it is best practice to fit the imputer on the training set and then transform both training and test sets (to prevent leakage and ensure consistent medians), the immediate issue is that the code cannot impute even a single DataFrame because it never fits. The question asks why the block is not accomplishing imputation at all, not about workflow hygiene across splits.

inputCols and outputCols do not need to be exactly the same. Spark ML allows outputCols to be different so you can preserve original columns and create imputed versions (e.g., col -> col_imputed). You may choose to overwrite by setting outputCols equal to inputCols, but it’s optional. A mismatch between these lists only matters if their lengths differ or names are invalid, not because they must match exactly.

Calling fit() is required, but not “instead of transform” in the overall process. The correct sequence is fit() to create an ImputerModel, then transform() using that model. So transform is still needed to apply the learned medians to the DataFrame. The option is misleading because it implies transform should not be used, when in fact it must be used after fitting.