Databricks Certified Machine Learning Associate: Certified Machine Learning Associate

Incorrect. This syntax resembles pandas-style boolean indexing, not the standard Spark DataFrame filtering pattern expected here. In PySpark, df["price"] returns a Column object, but row filtering is typically done with filter() or where() rather than bracket-based boolean masking. Even if some shorthand may appear in certain contexts, it is not the canonical answer for Databricks certification questions. The exam expects the explicit Spark API method for filtering rows.

Incorrect. This is SQL syntax written as a bare statement, not valid PySpark DataFrame code by itself. To use SQL, the DataFrame would first need to be registered as a temporary view, and then the query would need to be executed with spark.sql(). As written, it does not create a new Spark DataFrame from spark_df in Python code. Therefore it does not directly accomplish the task in the form shown.

Incorrect. .loc is a pandas indexing feature and is not available on PySpark DataFrames. Spark DataFrames do not support label-based row and column indexing in this way because they are distributed datasets, not in-memory pandas objects. Attempting to use .loc on a Spark DataFrame will fail. In Spark, row filtering should be done with filter() or where().

Incorrect. This also uses pandas-style .loc syntax, which PySpark DataFrames do not support. In addition, the boolean expression is placed in the column-selection position, which is not how Spark selects columns or filters rows. Spark separates row filtering and column projection into methods like filter() and select(). As written, this is not valid Spark DataFrame code.

Incorrect. There is no requirement to keep max_evals under 10. The number of evaluations affects runtime and search quality, not correctness. If anything, more evaluations can improve the chance of finding good hyperparameters, assuming the tuning process is configured correctly for the type of objective function.

Incorrect. Hyperopt provides fmin (minimization). There is no fmax API in standard Hyperopt usage. To maximize a metric (e.g., AUC), you convert it to a loss by negating it or using (1 - metric), then still call fmin.

Incorrect. Removing trials=trials would make Hyperopt use the default Trials() object implicitly, which could work, but it does not represent the necessary explicit change asked for. The key fix is to avoid SparkTrials for Spark ML objectives; you should explicitly use Trials() for clarity and correctness.

Incorrect. algo=tpe.suggest selects the TPE Bayesian optimization algorithm and is valid for both Trials and SparkTrials. Removing it would fall back to a default algorithm (often random search), which changes optimization behavior but does not address the core issue of Spark ML incompatibility with SparkTrials.

Incorrect. A hyperparameter grid is typically built with ParamGridBuilder and used by tuning components like CrossValidator or TrainValidationSplit. The grid itself is not an Estimator; it’s a configuration object listing parameter combinations to try. Estimators are the algorithms being tuned (e.g., LogisticRegression), not the grid describing candidate settings.

Incorrect. Chaining multiple algorithms together to specify an ML workflow describes a Pipeline (an Estimator) or a PipelineModel (a Transformer) depending on whether it is fit yet. While a Pipeline is indeed an Estimator, the statement is not the definition of an Estimator in general; it describes a specific workflow container rather than the core Estimator concept.

Incorrect. This describes a Transformer (often a trained model) because it takes a DataFrame and outputs a new DataFrame with additional columns such as predictions, probability, or transformed features. In Spark ML, the trained model returned by an Estimator’s fit method is usually a Transformer (e.g., LogisticRegressionModel).

Incorrect. An evaluation tool is an Evaluator (e.g., RegressionEvaluator, MulticlassClassificationEvaluator, BinaryClassificationEvaluator). Evaluators compute metrics from predictions and labels but do not train models and do not produce Transformers. They are commonly used alongside Estimators in tuning workflows like CrossValidator.

Incorrect. .loc is a pandas DataFrame indexer used for label-based selection and boolean masking. Spark DataFrames do not support .loc, because Spark DataFrames are not indexed the same way and operate via distributed query plans. This option reflects pandas syntax, not PySpark/Databricks Spark DataFrame operations.

This explanation is inaccurate. In PySpark, DataFrame.__getitem__ supports passing a Column expression, so spark_df[spark_df["discount"] <= 0] can return a filtered DataFrame. Although filter()/where() is the more idiomatic and exam-expected syntax, saying this form is invalid or not a Spark API pattern is technically misleading.

Incorrect. This is another pandas-style .loc usage (and the parentheses/argument structure also resembles pandas). Spark DataFrames do not provide .loc for row/column selection. In Spark, you filter rows with filter()/where() and select columns with select(), not with pandas indexers.

Incorrect. mapInPandas is a DataFrame method used as df.mapInPandas(func, schema) to transform partitions and return pandas DataFrames, not a Column expression usable inside withColumn. Also, mapInPandas requires an explicit output schema and is not invoked as a standalone function in this context.

Incorrect. Iterator(spark_df) is not a valid way to supply Spark DataFrame data to a Pandas UDF. Spark controls the iterator of pandas batches internally during execution. You never manually wrap a Spark DataFrame in an Iterator to call a Pandas UDF; instead you pass Column expressions and Spark handles batching and distribution.

Incorrect. This mixes APIs incorrectly. mapInPandas is not used inside withColumn, and predict(spark_df.columns) is not a valid call because spark_df.columns is a Python list of strings, not Spark Columns. Even if predict were called, it would not return a Spark Column expression suitable for withColumn.

Incorrect. Passing spark_df.columns provides a single Python list argument (of strings) to the UDF, not multiple Spark Column arguments. Scalar Pandas UDFs require Spark Column inputs; they cannot accept a Python list of column names as a single argument in a DataFrame expression. The correct approach is to expand the list: predict(*spark_df.columns).

Incorrect. Databricks Feature Store exposes feature table metadata programmatically. The FeatureStoreClient.get_table method returns a FeatureTable object that includes metadata such as the description. This option may seem plausible if you assume descriptions are only visible in the UI, but they are accessible via the API.

Incorrect. create_training_set is used to create a TrainingSet object by specifying feature lookups and a label DataFrame. It does not retrieve feature table metadata and does not accept a feature table name alone as shown. It’s for assembling training data, not inspecting table properties.

Incorrect. load_df() loads the feature table’s data into a Spark DataFrame. A DataFrame contains rows/columns of feature values, not the table’s metadata description. While you could inspect schema from the DataFrame, you cannot retrieve the Feature Store description from it.

Incorrect (incomplete). fs.get_table("new_table") returns the FeatureTable object, but by itself it does not “return the metadata description”; it returns the whole metadata object. To specifically return the description string, you must access the .description property (as in option C).

RMSE (Root Mean Squared Error) is a regression metric used for continuous numeric predictions, not for binary classification outcomes. It measures the average magnitude of prediction errors in the same units as the target variable. Because the problem is to classify infection status (yes/no), RMSE is not an appropriate evaluation metric for this scenario.

Precision is TP / (TP + FP) and measures how reliable positive predictions are. It is most important when false positives are costly (e.g., expensive follow-up tests or unnecessary treatments). However, the question’s goal is to maximize the number of positive cases identified (reduce false negatives), which is better captured by recall rather than precision.

“Area under the residual operating curve” is not a standard classification metric; the common term is “Area Under the Receiver Operating Characteristic curve” (AUC-ROC). AUC-ROC evaluates ranking quality across thresholds, not directly maximizing captured positives at a chosen operating point. While AUC can be useful, the question asks which metric to use to maximize identified positives, which is recall.

Accuracy is (TP + TN) / (TP + TN + FP + FN). It can be misleading, especially with imbalanced classes common in healthcare (few infected patients). A model can achieve high accuracy by predicting most patients as negative, yet miss many true infections (high FN). Since leaders want to maximize identified positives, accuracy is not the best metric.

Categorical features generally should not be imputed with mean or median because those statistics assume numeric ordering and distance. For categorical variables, common approaches are imputing with the mode (most frequent category), adding a new category like "Unknown", or using algorithms/encoders that can handle missingness explicitly. Median vs mean is primarily a numeric-feature decision.

Boolean features are typically imputed with the mode (most common True/False) or a separate missing indicator, depending on semantics. Using mean/median on boolean data can be misleading: the mean becomes a proportion (e.g., 0.73) and is not a valid boolean without thresholding, and the median may collapse to 0/1 but still ignores meaning. So median-over-mean is not the key choice here.

If there are no outliers and the distribution is roughly symmetric, mean imputation is often fine and can be slightly more efficient statistically than median. Median is not harmful, but it’s not specifically preferable in this scenario. The question asks when median is preferable over mean; the absence of outliers removes the primary advantage of the median.

If there are no missing values, imputation is unnecessary. The choice between mean and median does not apply because you would not run an imputer at all. On exams, this option is a distractor testing whether you recognize that imputation is only relevant when missingness exists.

Incorrect. One-hot encoding is widely supported across ML libraries (Spark ML, scikit-learn, TensorFlow, PyTorch preprocessing, etc.). In Databricks, Spark ML provides OneHotEncoder and StringIndexer specifically for this purpose. Lack of library support is not a valid reason to avoid one-hot encoding in a feature repository.

Incorrect. One-hot encoding is not dependent on the target variable; it depends on the set of observed categories in the feature. Target/mean encoding is an example of an encoding that does depend on the label. This option confuses one-hot encoding with supervised encodings.

Incorrect. One-hot encoding can be computationally and memory intensive for high-cardinality features, but it is routinely performed at scale on full training datasets. The key issue is not that it should only be done on small samples; rather, it may be the wrong representation for some models or may create operational complexity when stored centrally.

Incorrect. One-hot encoding is one of the most common strategies for representing categorical variables numerically, especially for linear models and many general-purpose ML workflows. The question is not about popularity; it’s about whether it belongs in a shared feature repository.

One-hot encoding categorical features is generally efficient to distribute because, after determining the category-to-index mapping, each row can be transformed independently. The main cost is handling category metadata and potentially high dimensionality, but the transformation itself is parallelizable across partitions. Spark commonly represents one-hot outputs sparsely, which further helps with distributed execution. While high-cardinality features can increase memory usage, the operation is still more distribution-friendly than exact median computation.

Target encoding categorical features requires grouped aggregation, such as computing the average target value for each category, and then applying those mappings back to the data. This does involve shuffles and joins, so it is not as lightweight as purely row-wise transformations. However, groupBy and aggregate operations are standard distributed patterns that Spark handles efficiently relative to exact global order-statistic computations. The statistical care needed to avoid leakage does not make the core distributed computation less efficient than computing a true median.

Imputing missing feature values with the mean is usually efficient to distribute because the mean can be computed from partial sums and counts on each partition and then combined centrally. This is exactly the kind of associative aggregation that distributed systems are optimized for. Once the mean is computed, filling missing values is a simple parallel row-wise operation. As a result, mean imputation is much more scalable than exact median imputation.

Creating binary indicator features for missing values is one of the most efficient operations to distribute because it is a pure row-wise transformation. Each partition can independently check whether a value is null and emit a 0/1 indicator without any global coordination, shuffle, or aggregation. This makes it embarrassingly parallel and highly scalable in Spark. It is therefore far more efficient to distribute than computing a true median.

Model tuning is not something the data scientist must perform outside the AutoML experiment in the standard Databricks workflow. AutoML automatically searches across algorithms and hyperparameters to find strong candidate models for the selected prediction task. It records these trials and their metrics in MLflow so the user can inspect the tuning results. While additional manual tuning is possible later, it is not a required external step.

Model evaluation is included within the AutoML experiment rather than being strictly external to it. Databricks AutoML evaluates candidate models using task-appropriate metrics and validation procedures, then ranks the runs based on performance. The generated notebooks and MLflow tracking provide visibility into these evaluation results. Although a team may later perform extra business-specific validation, baseline evaluation is already handled by AutoML.

Exploratory data analysis is also supported by Databricks AutoML as part of the experiment experience. AutoML can generate data summaries and notebooks that help users understand distributions, missing values, and other dataset characteristics before or alongside model training. This means EDA is not the one step that must necessarily occur outside the AutoML workflow. Users may still do deeper manual exploration, but AutoML already covers the basic EDA component.

Incorrect. Type hints are not the defining advantage of pandas UDFs. Spark requires explicit return types for UDFs (and pandas UDFs) via Spark SQL types, and while Python type hints may be used in code style, they are not the core benefit tested. The key differentiator is vectorized execution with Arrow, not typing support.

Partially true but not the best answer. pandas UDFs do allow you to write logic using pandas/NumPy operations inside the function because inputs arrive as pandas objects. However, the exam question asks for a benefit compared to standard UDFs; the primary, canonical benefit is vectorized batch processing (and Arrow-based transfer), not merely that pandas APIs can be used.

Incorrect. Both standard PySpark UDFs and pandas UDFs operate on distributed Spark DataFrames; distribution is a property of Spark, not a unique advantage of pandas UDFs. pandas UDFs still run per partition/executor like other Spark transformations, so this does not distinguish them from standard UDFs.

Incorrect. pandas UDFs do not inherently guarantee in-memory processing or prevent spilling to disk. Spilling is determined by Spark’s execution plan, shuffle operations, partition sizes, and memory configuration. While Arrow can improve transfer efficiency, it does not change Spark’s fundamental memory management or eliminate disk spill behavior.

Both Spark ML and sklearn decision trees can consider all available features at a split, so this does not explain the discrepancy. The issue is not whether features are tested, but how candidate thresholds are generated. Spark's binning strategy changes the split search process. That is why this option is not the best answer.

Automatic pruning is not the defining reason Spark ML trees differ from sklearn trees in this context. Spark primarily limits tree growth through hyperparameters rather than relying on post-pruning behavior to explain output differences. The exam is targeting the split-candidate generation mechanism instead. Therefore this option is misleading.

Spark ML does not typically test more split candidates than sklearn for continuous features. Instead, it reduces the candidate set by using bins and representative thresholds to make training scalable. Sklearn often evaluates more exact candidate thresholds from the observed data. So this option reverses the actual implementation difference.

Randomly sampling a subset of features is characteristic of random forests and similar ensemble methods, not a standard single decision tree in Spark ML. A standalone Spark decision tree generally evaluates the full feature set at each node. Therefore this behavior would not explain the observed difference. The real cause is Spark's use of binned split candidates.

Imputing with the mean instead of the median does not preserve any additional information about which rows were missing; it only changes the imputed value. Mean vs. median is mainly about robustness to outliers and skew. If the colleague’s concern is “missingness carries signal,” switching the statistic does not address it and can even be worse under heavy skew.

Relying on the algorithm to handle missing values is not a general solution. Many Spark ML estimators and transformers expect no nulls/NaNs and will error or require preprocessing. Even for models that can route missing values (some tree methods), you still may lose the explicit missingness signal or have inconsistent behavior across algorithms and pipelines.

Removing all features that had missing values throws away potentially predictive variables and reduces the feature set unnecessarily. This is the opposite of “include as much information as possible.” Feature removal is only justified when missingness is extreme, the feature is unreliable, or it causes leakage/quality issues—none of which is implied here.

A constant feature representing the overall percent missing for a column is the same value for every row, so it provides no row-level signal to the model. It may be useful for dataset monitoring or feature quality reporting, but it generally won’t help prediction because it does not differentiate observations.

Adding a test set validation process is not the right fix because the issue is not insufficient evaluation, but the direction of optimization. The test set should be held out until final model assessment and should not be used during hyperparameter tuning. Using it inside the objective would introduce data leakage and bias the final evaluation. It would not correct the fact that fmin is minimizing the returned score.

Adding a random_state to RandomForestRegressor can improve reproducibility by making results more consistent across runs, but it does not address the core bug. Hyperopt would still minimize the returned R^2 value and therefore prefer worse-performing parameter combinations. Reproducibility is useful for debugging and comparison, but it does not change the optimization objective. The model would remain inaccurately tuned if the score sign is not corrected.

Removing the mean is incorrect because Hyperopt expects the objective to return a single scalar value or a properly structured result dictionary. cross_val_score returns one score per fold, and averaging those scores is the standard way to summarize cross-validation performance. Returning the full array would not improve model accuracy and may even break the optimization workflow. The mean is not the problem; the problem is that the scalar being returned has the wrong optimization direction.

Replacing fmin with fmax is not a valid solution because Hyperopt's standard optimization API is based on fmin. The intended pattern is to keep using fmin and convert any metric that should be maximized into a loss by negating it or otherwise transforming it. There is no standard Hyperopt workflow where you simply swap to fmax for this use case. This option misunderstands how Hyperopt is designed to optimize objectives.

TrainValidationSplit is an MLlib hyperparameter tuning estimator. It splits the provided dataset into train and validation portions internally to select the best model from a parameter grid. It is not used to directly and generally create separate training and test DataFrames for downstream use; it returns a fitted TrainValidationSplitModel after tuning.

DataFrame.where applies a boolean filter condition to keep only rows matching a predicate. While you could simulate a split by filtering on a random column, where itself is not the dedicated random splitting operation and would require additional steps (e.g., adding a random value column). The question asks specifically which Spark operation can randomly split a DataFrame.

CrossValidator is an MLlib hyperparameter tuning estimator that performs k-fold cross-validation over a parameter grid. It repeatedly splits data into folds internally for model selection and returns a CrossValidatorModel. It is not intended as a simple operation to produce a training DataFrame and a test DataFrame for downstream use.

TrainValidationSplitModel is the fitted model produced after running TrainValidationSplit. It contains the best model and metrics from tuning, not a mechanism to split a DataFrame. It cannot be used as an operation to randomly create train/test DataFrames.

Incorrect. MinMaxScaler is also an Estimator that learns min/max from the data during fit(). If you fit it on the full dataset (as in the original pattern), you still leak test-set information. The issue is not which scaler you use; it’s when and on what data you call fit().

Incorrect. Standardizing (or scaling) the test set according to its own min/max (or mean/std) is not appropriate for evaluation because it makes the test preprocessing depend on test-only statistics and makes train/test feature distributions incomparable. You must apply the training-fitted scaler to the test set.

Incorrect. Cross-validation does not eliminate the need for standardization, and it does not automatically prevent leakage unless preprocessing is included inside the CV workflow (e.g., a Pipeline inside CrossValidator). You still must fit scalers on training folds only.

Incorrect. This describes the wrong direction: using test summary statistics to standardize training data is a form of leakage and contaminates training with information from the test set. Proper practice is the opposite: compute statistics on training data and apply them to test data.

This is incorrect because transform is used after a model has already been trained. In Spark ML, LinearRegression is an Estimator, so the first step is calling fit on the training DataFrame to produce a LinearRegressionModel. Only then would transform be used on a DataFrame to generate predictions. Calling transform on train_df is not required to complete model training.

This is incorrect because the features column already appears to be in the expected Spark ML format. In Spark DataFrame schemas, a machine learning feature vector is represented as a UDT, specifically VectorUDT. Since LinearRegression expects a single vector column for features, the existing schema indicates that requirement has already been satisfied. No additional conversion is necessary.

This is incorrect because a Pipeline is optional, not mandatory, for fitting a Spark ML model. Pipelines are useful when chaining preprocessing stages such as StringIndexer, OneHotEncoder, or VectorAssembler together with a model. Here, the DataFrame already contains the final features vector and numeric label, so LinearRegression can be fit directly. Adding a Pipeline would be unnecessary overhead for this task.

This is incorrect because Spark ML's DataFrame-based API does not expect separate scalar columns to be passed directly into LinearRegression as independent features. Instead, it expects a single vector column referenced by featuresCol. Splitting the vector into multiple columns would move away from the required input format and would typically force the engineer to assemble them back again. The provided features column is already in the proper structure.

A is incorrect because 3 is only the number of cross-validation folds. Each fold does not represent the full set of model trainings, since every hyperparameter combination must be evaluated on every fold. With 6 parameter combinations, the fold count must be multiplied by the grid size. Therefore, 3 significantly undercounts the number of model fits.

B is incorrect because 5 does not correspond to any valid calculation from the given grid or cross-validation setup. The first hyperparameter has 3 values and the second has 2 values, which yields 6 combinations rather than 5. Cross-validation then increases the number of fits further rather than reducing it. As a result, 5 is not supported by the problem data.

C is incorrect because 6 is only the number of unique hyperparameter combinations in the grid. That would be the right count if the question asked only for the number of parameter settings to test without cross-validation. However, 3-fold cross-validation requires training one model per fold for each combination. This increases the total number of model trainings to 18, not 6.