Google Professional Machine Learning Engineer

A single Deep Learning VM with 4 V100s can meet the performance requirement, and local SSDs provide high throughput. However, it increases operational overhead (VM lifecycle, patching, GPU/CUDA/driver management) and can increase cost if the VM is left running between daily trainings. You also must manage data staging to local SSD each run and handle durability/backups yourself.

Reading/writing directly to Cloud Storage from a Deep Learning VM reduces local storage management, but you still own the VM operations and typically pay for idle GPUs unless you automate start/stop and image maintenance. For 1080p frames, Cloud Storage I/O is usually fine, but the primary requirement is minimizing operational overhead, which is better addressed by managed Vertex AI Training.

GKE with GPU node pools and a self-managed NFS server is the highest operational overhead option: cluster management, GPU scheduling, node autoscaling tuning, NFS reliability/backup, and security hardening. It’s appropriate for complex multi-tenant training platforms or many concurrent jobs, not a single daily <60-minute training run where simplicity and cost minimization are priorities.

Randomizing sensitive values is not deterministic unless you maintain a mapping table, and it is typically not reversible for audits. It also risks breaking referential integrity and downstream joins/validations because randomized outputs may not preserve the original format/length constraints (e.g., card number patterns). While Dataflow can scale to 250M rows, this approach does not meet the deterministic and decryptable requirements.

AES-256 with a per-row random salt makes the transformation non-deterministic (same input encrypts differently each time), which breaks the requirement for stable mapping needed for joins and consistent feature values. Additionally, standard ciphertext encoding (base64/hex) changes length and character set, violating format/length preservation. Building custom crypto also increases implementation risk compared to DLP’s managed FPE designed for this use case.

Dropping PII columns contradicts the requirement that every column is required as a feature for training. Authorized views can restrict access, but they do not reduce the sensitivity of the data used in training; the model pipeline would still either need the raw PII (violating security requirements) or lose critical features. This option addresses access control, not deterministic, reversible de-identification.

Incorrect. Feature selection may reduce sensitivity to some drifting inputs, but it does not fix the root cause: the meaning/units/sampling of inputs changed. Removing “low-importance” features also won’t address that 27% of temperature values are out of range or that multiple key features have shifted. You still need monitoring and likely preprocessing updates and retraining on correctly interpreted data.

Incorrect. Hyperparameter tuning (e.g., L2 regularization) addresses overfitting/underfitting, not a production data contract change. With unit changes and distribution shift, tuning regularization may slightly stabilize predictions but will not restore accuracy reliably. The correct approach is to detect drift, validate feature semantics, update transformations, and retrain with representative recent data.

Incorrect. A fixed monthly retraining schedule is weaker than event-driven monitoring and response. It can leave the system degraded for weeks after a sudden upstream change. Feature selection still doesn’t solve unit/sampling changes. Best practice is continuous monitoring with alerting and automated pipelines that retrain when drift/performance thresholds are exceeded, with evaluation gates before deployment.

Boosted trees can be powerful, but label encoding introduces artificial ordering for nominal categories (e.g., station_id=10 > station_id=2), which can distort splits and reduce generalization. Casting timestamp to a single Unix time value does not explicitly represent cyclical weekly/seasonal patterns; the model must infer periodicity indirectly, often leading to worse RMSE than using engineered time components.

Autoencoders are primarily unsupervised models used for dimensionality reduction or anomaly detection, not for directly optimizing supervised regression RMSE on demand forecasting. Label encoding has the same ordinal-risk issue as in option B. Normalizing timestamp to [0,1] still fails to capture cyclical structure (e.g., hour 23 is close to hour 0), so it’s not effective for weekly/seasonal patterns.

Matrix factorization is suited to collaborative filtering (user-item rating prediction) and latent factor discovery, not time-dependent demand regression with exogenous features. While interaction features (station_id x weather) can be useful, the underlying model type is mismatched for forecasting hourly demand and won’t naturally incorporate temporal seasonality without additional time feature engineering and a regression-appropriate algorithm.

A foundational LLM is not a search backend or index. While an LLM can interpret queries and generate responses, it does not natively provide document/listing indexing, deterministic filtering (price, rating thresholds), faceting, or scalable retrieval over 1.8M listings. You would still need a retrieval system (keyword, vector, or managed search) to ground results and meet latency/cost expectations.

Vertex AI Vector Search is excellent for approximate nearest neighbor retrieval over embeddings and can support semantic similarity at scale. However, for a 2-week POC with minimal integration, it is usually more work: you must build ingestion, generate embeddings for descriptions, design and maintain metadata filtering, and implement hybrid ranking and query orchestration. It’s a component, not a full search product.

Hosting a BERT-based model on a Vertex AI endpoint implies building a custom semantic retrieval/ranking system. You would need to handle embedding generation, indexing, retrieval, and filtering yourself, plus manage model serving, scaling, and updates. This is higher engineering effort and risk for a 2-week proof of concept, and it’s unnecessary when managed search capabilities exist.

Not ideal because it calculates descriptive statistics and runs statistical analyses inside a notebook after importing data. With 850 million rows, pulling large volumes into a user-managed notebook is expensive and slow, and may exceed memory/IO limits. Looker Studio is fine for visualization, but the heavy computations should be pushed down to BigQuery to minimize compute and leverage MPP execution.

Incorrect because it relies entirely on a Vertex AI Workbench user-managed notebook for importing and analyzing the full dataset. Notebooks are not designed as a primary engine for scanning and aggregating hundreds of millions of records; this typically requires large VM sizing, long runtimes, and high cost. It also reduces reproducibility and scalability compared to BigQuery-based processing.

Partially correct: BigQuery is the right place for descriptive statistics at scale, and Workbench can run complex hypothesis tests. However, using notebooks to generate all time plots is not the most resource-efficient approach for interactive hourly/daily visual exploration. Looker Studio can query BigQuery directly and offload visualization without keeping notebook compute running.

"You Might Also Like" is commonly used for discovery and can improve click-through rate on a home feed, but CTR is not the same as revenue. Home feed users may be in browsing mode, so incremental clicks may not translate into larger baskets or higher AOV. For an explicit AOV +6% goal in a short timeframe, cross-sell at high-intent moments is typically more effective than broad personalization on the home screen.

This reverses best practice. Recommendations AI Retail requires a product catalog to exist so events can be correctly attributed to items and enriched with metadata. Importing events first can lead to unmatched item IDs, reduced training quality, and delayed time-to-value. The system does not "backfill" missing metadata reliably from events; you should upload/maintain the catalog first (or in parallel) and then stream/import user events.

Creating placeholder SKUs with default categories/prices is explicitly counterproductive for recommendation quality and business outcomes. Recommendations AI relies on accurate item attributes (category, price, availability) to train and to filter/serve relevant results. Placeholders can cause irrelevant or misleading recommendations (e.g., wrong price or out-of-stock), harming user trust and conversion. It also complicates governance and measurement during A/B tests, producing noisy results.

Incorrect. The main problem is that this option relies on the scikit-learn prebuilt container and installs custom dependencies at runtime, which adds startup latency and variability that works against the under-2-minute requirement. It also does not properly address the need for the system library libgomp, because OS-level dependencies are not reliably handled through a Python source distribution or normal pip installation. In addition, bundling the dataset with the source package is not a good pattern for Vertex AI training inputs, since datasets should generally be stored separately in Cloud Storage so they can be updated and versioned independently of the training code.

Incorrect. Baking the dataset into the container can reduce data download time, but it makes the dataset difficult to update because every data refresh requires rebuilding and redeploying the container image. It also increases image size, which can increase image pull time and negate startup benefits. This conflicts with the requirement to keep the dataset easy to update.

Incorrect. Keeping data in Cloud Storage is good for updates, but installing dependencies at startup from a requirements file is the main problem: it adds significant latency and can be unreliable due to network/package repository issues. Additionally, libgomp is a system library and may require apt-get steps, further increasing startup time beyond the 2-minute requirement.

Skipping or commenting out components can reduce runtime, but it is a manual, error-prone workflow that changes the pipeline graph and may bypass important validations. It also doesn’t leverage Vertex AI Pipelines’ built-in orchestration best practices. In team settings, this harms reproducibility and makes it harder to compare runs because different runs execute different subsets of steps.

Dataflow can be excellent for large-scale ETL, but migrating feature processing to Dataflow is a redesign that adds operational overhead and may increase costs (Dataflow job charges, potential always-on resources, and additional integration). It also doesn’t inherently solve iteration speed if the pipeline still retriggers expensive processing; caching would still be needed.

Adding a T4 GPU increases cost and often provides little to no benefit for scikit-learn training, which is typically CPU-bound and not GPU-accelerated. Even if training sped up, the overall 80-minute runtime likely includes significant data ingestion/cleaning time, so a GPU would not address the main bottleneck and violates the “without increasing costs” requirement.

Natural Language API provides pretrained NLP capabilities (sentiment, entity extraction, and a limited content classification taxonomy). It is fast to integrate but offers minimal control over model architecture, training, and deployment. It also may not support custom 8-category warranty triage labels. This conflicts with the requirement to use TensorFlow and maintain full control over code, serving, and deployment, making it unsuitable here.

AutoML Natural Language can train a custom text classifier quickly from labeled data and is often a strong choice for rapid delivery. However, it is a managed training and serving solution where you do not maintain full control over the TensorFlow model code and deployment mechanics. While it can be orchestrated in pipelines, it violates the explicit requirement for full TensorFlow control, so it is not the best answer.

Using an established text classification model “as-is” is unlikely to work because the target labels are specific (8 warranty triage categories) and won’t match the pretrained model’s original label set or taxonomy. Even if the model outputs generic categories, it won’t map cleanly to your business classes without adaptation. The requirement emphasizes leveraging existing resources, but still implies customization; transfer learning is needed.

Dataproc/Spark can preprocess large datasets, but exporting transformed Parquet to Cloud Storage introduces extra data movement, storage management, and pipeline operations. It also risks training/serving skew because weekly inference is required to run directly in BigQuery; you would need to re-implement the same feature logic in BigQuery or continuously export new partitions. This is typically less cost-effective and less consistent than doing feature engineering in BigQuery.

Loading 2.6 TB (900M rows) into a local pandas DataFrame is infeasible due to memory and compute constraints and would be extremely slow and costly. It also breaks scalability and operational best practices for production ML. This option ignores distributed processing and BigQuery’s strengths, and it would not support ongoing weekly inference in BigQuery without duplicating preprocessing logic elsewhere.

Dataflow/Beam is strong for streaming and ETL, but writing preprocessed data as CSV is inefficient (large size, slow parsing, poor typing) and increases storage and pipeline overhead. Like option A, it also complicates training/serving consistency because inference must run in BigQuery; you would still need equivalent SQL feature logic or repeated exports for new partitions, increasing cost and operational complexity.

Incorrect. Fully switching to human moderation may reduce model-driven bias, but it is typically expensive, slow, and hard to scale for live classroom chat. It also removes the benefits of automation (latency and coverage) and is not aligned with the constraint that the team is overextended and budget is tight. A targeted model improvement is more practical.

Incorrect. Replacing with an off-the-shelf classifier is a major change with integration, evaluation, and retraining costs, and it may still exhibit similar bias or poor performance on domain-specific language (festival names, slang, classroom context). Even if adopted, you would still need representative data and slice-based evaluation, so it doesn’t solve the core issue efficiently.

Incorrect. Increasing the global threshold reduces overall flags but is a blunt instrument: it does not specifically reduce false positives for the affected subgroup and can significantly increase false negatives for genuinely toxic messages, undermining safety. It also fails the fairness objective because the disparity may persist even if the overall flag rate drops.

Mixing Vertex AI Pipelines for artifacts and MLflow for lineage adds unnecessary operational complexity and weakens standardization. MLflow tracking is not the native lineage store for Vertex Pipelines, so you would need custom integration to correlate pipeline steps, artifacts, and environments. This violates the “minimal custom code” and “simple to adopt via reusable templates” requirements and increases maintenance burden over time.

Vertex AI Experiments is primarily for experiment/run tracking (parameters, metrics, comparisons) and is not designed to provide full end-to-end lineage across multi-step pipelines and artifact dependencies. While ML Metadata can store lineage, using Experiments for artifacts is a mismatch: artifacts like datasets, checkpoints, and binaries are typically managed via pipeline artifacts and storage, with MLMD capturing their relationships automatically.

Using Cloud Composer to schedule lineage capture via Cloud Run functions is a custom-built metadata system. It requires significant bespoke code to infer lineage, map artifacts to steps, and maintain consistency across dev/stage/prod. This approach is harder to standardize, less reliable for audit-grade provenance, and does not leverage Vertex AI’s native MLMD integration, making it a poor fit for low-code, scalable governance.

Continuing with Vertex AI Training with checkpoints improves resiliency but does not, by itself, guarantee the required >=40% cost reduction. If the underlying compute remains on-demand T4 GPUs, spend will be similar to current levels. This option is attractive because it is operationally simple, but the question’s explicit cost target and interruption tolerance point to using discounted interruptible capacity rather than only improving fault tolerance.

Running distributed training without checkpoints is risky and conflicts with the stated tolerance for interruptions. Without checkpoints, any node preemption or failure can force restarting from scratch or losing significant progress, materially increasing wall-clock time and potentially increasing total cost. The small checkpoint overhead (<2%) is not a meaningful savings lever compared to GPU-hour costs, so removing checkpoints is a poor trade-off.

Using Spot/Preemptible GPU node pools can reduce cost, but doing so without checkpoints is likely to cause large recomputation after evictions and can significantly increase wall-clock time. In multi-node distributed training, a single node interruption can disrupt the whole job; without checkpoints, recovery is expensive and may negate savings. Given the pipeline already supports efficient checkpointing, omitting it is unjustified.

Incorrect. Tree-based models in scikit-learn are intended for tabular features; using raw pixels would create tens of millions of features per image, which is infeasible. Even with engineered features, spatial localization is lost. SHAP values can provide feature attributions, but for images this becomes computationally expensive and does not naturally yield clean pixel-level crack localization at 3840x3840 resolution in a production setting.

Incorrect. Partial dependence plots are global interpretability tools showing how changing one (or a few) features affects predictions on average. They are not designed for per-example, pixel-level attribution, and they assume a tabular feature representation. For image inspection requiring localized heatmaps, PDPs are the wrong explanation modality and do not meet the operational requirement to pinpoint hairline cracks on each tile.

Mostly incorrect for this scenario. While TensorFlow is appropriate for the vision model, sampled Shapley methods are typically too computationally heavy for very high-dimensional inputs like 3840x3840 RGB images, especially if explanations are needed routinely in operations. Shapley approximations can also be noisy and require many samples for stable pixel-level maps, making Integrated Gradients a more practical and commonly tested choice.

Incorrect. CPU and memory utilization help diagnose serving bottlenecks (e.g., saturation causing timeouts), but the question states autoscaling is sufficient, latency is low, and error rate is <0.1%. Resource bottlenecks typically manifest as elevated latency, throttling, or errors—not a clean drop in AUC computed from feedback labels. This is not the first troubleshooting step for predictive performance loss.

Incorrect as a first step. Explainable AI attributions can help understand which features influence predictions and can be useful after you identify drifted features or suspect spurious correlations. However, explanations do not directly measure distribution shift or confirm training-serving mismatches. Without first checking drift/skew, you may misinterpret changing attributions that are actually driven by changed input distributions.

Incorrect. Latency/throughput checks validate the serving SLO, but the question already provides healthy serving metrics (P95 ~110 ms, low error rate, sufficient replicas). Confirming SLO compliance won’t explain why AUC dropped. For ML systems, predictive quality issues are usually rooted in data/label changes, drift, skew, or concept drift—requiring model monitoring rather than performance monitoring.

Loading 2.3 TB into a pandas DataFrame is not feasible (memory limits, single-node bottleneck) and does not scale to 4–8 distributed workers. tf.data.Dataset.from_tensor_slices() is appropriate for small in-memory datasets or prototypes, not production-scale training. This approach also makes repeatability and fault tolerance difficult because the dataset must be rebuilt in memory each run.

Exporting to CSV in Cloud Storage improves decoupling from BigQuery, but CSV is inefficient for ML training at this scale. Text parsing is CPU-heavy, files are larger than binary formats, and schema/typing issues are common. While tf.data.TextLineDataset can be parallelized, overall throughput is typically worse than TFRecord, increasing the risk of input bottlenecks on multi-worker training.

Streaming directly from BigQuery during training can create ingestion bottlenecks due to BigQuery read throughput limits, concurrency/quotas, and per-request overhead, especially with multiple workers. It also couples training stability to BigQuery availability and query performance, reducing repeatability. While TensorFlow I/O can work for smaller datasets or experimentation, for TB-scale multi-worker training it is generally safer to materialize to Cloud Storage in an efficient format.

Lowering the sampling rate reduces the number of prediction requests used to estimate serving distributions, which can reduce monitoring cost and sometimes reduce noisy estimates. However, it does not correct a systematic skew caused by comparing against an outdated baseline. It can also increase variance and potentially miss true skew/drift, weakening reliability and observability.

Waiting for more production traffic can help if the issue is insufficient sample size or cold-start behavior. But here, the monitor already had three consecutive windows and still alerts after deployment, indicating a persistent distribution difference. If the baseline remains the June dataset, additional traffic won’t resolve the skew; it will likely confirm it more strongly.

Disabling alerts until another retrain is risky and contrary to good MLOps practices. It sacrifices monitoring coverage and can hide genuine data issues (pipeline bugs, upstream changes, seasonality impacts). Also, retraining again without fixing the monitoring baseline/configuration may still produce the same alert pattern, because the monitor would still be comparing to the wrong reference.

Separating features in Feature Store and retraining without demographics is a remediation attempt, not the requested analysis. It also risks “fairness through unawareness,” which often fails because proxy variables (e.g., ZIP code, school, essay topics) can still encode demographics. Additionally, removing sensitive features can make it harder to measure fairness and audit outcomes by protected class. The board asked for transparent insight and cohort fairness evaluation first.

Vertex AI Model Monitoring focuses on operational issues like training-serving skew, feature drift, and prediction drift. While valuable for production reliability, it does not provide decision transparency for specific cases nor does it directly identify fairness issues across demographic cohorts. Retraining with recent data may even perpetuate or amplify bias if the underlying labeling or historical decision process is biased. This option addresses a different problem than requested.

Vector Search nearest-neighbor retrieval can help find similar examples, but it is not a primary tool for explainability or fairness auditing. Similarity search depends on embeddings and distance metrics and may not reveal which features drove the model’s decision or quantify demographic impacts. It can be a supplemental qualitative investigation technique, but it does not provide the systematic, per-feature, cohort-based transparency and bias evidence the board requested.