Practice Test #3

Loading 2.3 TB into a pandas DataFrame is not feasible (memory limits, single-node bottleneck) and does not scale to 4–8 distributed workers. tf.data.Dataset.from_tensor_slices() is appropriate for small in-memory datasets or prototypes, not production-scale training. This approach also makes repeatability and fault tolerance difficult because the dataset must be rebuilt in memory each run.

Exporting to CSV in Cloud Storage improves decoupling from BigQuery, but CSV is inefficient for ML training at this scale. Text parsing is CPU-heavy, files are larger than binary formats, and schema/typing issues are common. While tf.data.TextLineDataset can be parallelized, overall throughput is typically worse than TFRecord, increasing the risk of input bottlenecks on multi-worker training.

Streaming directly from BigQuery during training can create ingestion bottlenecks due to BigQuery read throughput limits, concurrency/quotas, and per-request overhead, especially with multiple workers. It also couples training stability to BigQuery availability and query performance, reducing repeatability. While TensorFlow I/O can work for smaller datasets or experimentation, for TB-scale multi-worker training it is generally safer to materialize to Cloud Storage in an efficient format.

Vertex AI Pipelines is the right orchestrator, but configuring distributed Vertex AI Training jobs addresses model training scale, not the heavy preprocessing/validation workload. TFX preprocessing components (Transform, StatisticsGen, ExampleValidator) are Beam-based and need a scalable Beam runner. Without explicitly running Beam on Dataflow, preprocessing may run locally or on limited resources, becoming the bottleneck at 120 TB/day.

Dataproc can run Spark/Hadoop workloads and can host custom orchestration, but using a Beam TFX orchestrator on Dataproc is not the standard managed path for TFX on Google Cloud. It increases operational overhead (cluster lifecycle, scaling, upgrades) and weakens the “automatic” integration story with Vertex AI Pipelines, Experiments, and ML Metadata compared to the native Vertex AI Pipelines + Dataflow runner approach.

Running the TFX orchestrator itself on Dataflow is not the typical or recommended pattern. Dataflow is designed to execute Beam pipelines (the data processing steps), not to serve as the primary orchestrator for multi-step ML pipelines with artifact lineage and experiment tracking. This option also risks losing the managed orchestration, UI, and MLMD-first integration that Vertex AI Pipelines provides.

Not best as a first step for a predictive maintenance solution. Pure heuristics can be deployed quickly, but they are brittle, require constant tuning across trucks and seasons, and often generate many false positives/negatives. They also don’t leverage temporal dynamics well and may not provide reliable 24-hour early warning beyond simple threshold breaches.

Tempting, but risky. Training on heuristic-generated labels usually causes the model to replicate the heuristic rather than learn true failure precursors. This can create a false sense of model quality because offline metrics reflect the heuristic, not real failures. It can be useful later as weak supervision, but only after establishing validation with real confirmed events.

Impractical and not necessary as a first step. Manually labeling 14 months of high-frequency telemetry for 250 trucks is expensive, slow, and ambiguous without clear definitions of “failure” and lead-time windows. A better approach is to start with unsupervised detection to triage and prioritize a much smaller set of segments for expert review.

Incorrect. GKE plus Traffic Director can do sophisticated traffic management, but it introduces significant engineering overhead: cluster operations, service mesh/proxy configuration, scaling, and security hardening. It also duplicates capabilities already provided by Vertex AI Endpoints for multi-model deployments and traffic splitting. For an exam scenario emphasizing simplicity and existing Vertex AI deployment, this is over-architected.

Incorrect as the “simplest approach.” Exporting logs and building Looker Studio dashboards can work for analysis, especially for accuracy, but it adds extra steps and ongoing maintenance (log sinks, schema management, dashboard upkeep). It also doesn’t address traffic splitting by itself; you’d still need a routing mechanism. Vertex AI’s built-in endpoint traffic splitting and monitoring reduce the need for custom dashboards for core serving metrics.

Incorrect. Cloud Run traffic splitting across revisions is useful for web services, but it requires packaging and operating your own model server per version, handling autoscaling behavior, and ensuring consistent model loading and performance. Since the models are already in Vertex AI Model Registry and deployed to a Vertex endpoint, moving to Cloud Run increases engineering effort and loses Vertex AI’s purpose-built model deployment and management features.

Vertex AI Pipelines is strong for orchestration and repeatability, but it is not an analytics datastore. While pipeline runs can record metrics, querying complex trends (7-day rolling averages, top-N by mean over 30 days across many dimensions) via the Pipelines API is awkward and not what it’s optimized for. You would typically export pipeline metadata to BigQuery or another store for analytics rather than rely on the Pipelines API for reporting.

Cloud Monitoring custom metrics are designed for operational telemetry, dashboards, and alerting, not deep experiment analytics. High-cardinality labels (feature set, architecture, hyperparameter grid, run_id) can hit quotas and become expensive or unwieldy. Querying “top 10 configurations by mean F1 in last 30 days” and computing rolling averages across many dimensions is more natural in BigQuery than in Monitoring’s time-series model.

Workbench notebooks plus Google Sheets is highly manual and does not scale to 180 runs/day with reliable, consistent logging. Sheets lacks strong schema enforcement, lineage tracking, and efficient analytical querying for rolling windows and top-N across large datasets. It also introduces collaboration and data integrity risks. This option violates the requirement to minimize manual effort and is not aligned with production-grade experiment tracking.

Running scheduled inference on Compute Engine VMs is feasible but requires you to manage instance sizing, autoscaling, retries, logging, patching, and failure handling. You also need to build the data download/streaming logic and ensure throughput for 60 GB daily. This increases operational burden and is less aligned with managed best practices compared to Vertex AI batch prediction.

Triggering a Cloud Function per file write creates an event-driven fan-out that can be hard to control for large daily batches. Cloud Functions have execution time and resource constraints and are not ideal for heavy, long-running inference workloads. You’d also need aggregation/coordination across many files and handle retries/idempotency carefully, increasing complexity versus a single managed batch job.

Online prediction endpoints are optimized for low-latency, per-request serving. Invoking the endpoint per record for 1.5 million records adds significant request overhead, can run into quota/QPS limits, and is typically more expensive than batch prediction for large offline workloads. It also introduces unnecessary complexity (client batching, retries, throttling) when latency is not required.

Using experimental_distribute_dataset (or the modern strategy.distribute_datasets_from_function) is mainly relevant when you need the strategy to shard/distribute input across replicas. With MirroredStrategy and model.fit, Keras typically handles distribution automatically. Since profiling shows no input bottleneck and GPUs are underutilized due to small per-replica work, changing dataset distribution won’t materially reduce epoch time.

A custom training loop (GradientTape) can provide flexibility, but it does not inherently increase GPU utilization or reduce all-reduce overhead. If the root issue is that each replica only processes batch=16, the GPUs will still be underfed. Custom loops can also introduce extra Python overhead unless carefully graph-compiled, so this is not the best first action for performance.

Moving to TPU with TPUStrategy can accelerate training, but it’s not the targeted fix for the observed symptom. TPUs also require sufficiently large per-core batch sizes to be efficient, and migration adds operational complexity (TPU-compatible ops, input pipeline changes, XLA behavior). The question asks what to do now on 4 GPUs where utilization is low; batch scaling is the direct remedy.

Increasing the threshold to improve precision generally does not increase recall. Recall depends on capturing as many true positives as possible. A higher threshold makes the classifier stricter, so some borderline true positives will be classified as negative, reducing TP and increasing FN. That typically lowers recall rather than raising it, assuming the underlying model is unchanged.

Raising the threshold to increase precision usually decreases the number of false positives, not increases it. False positives occur when negative images are predicted as positive. A stricter threshold means fewer positives are predicted overall, so fewer negative examples will cross the threshold and be incorrectly flagged. Increasing false positives would generally reduce precision, the opposite of the goal.

Adjusting the threshold upward to increase precision typically increases false negatives, not decreases them. False negatives are positives predicted as negative; when the threshold is higher, some true logo images with moderate confidence will fall below the cutoff and be missed. Decreasing false negatives would usually require lowering the threshold, which tends to increase recall but can hurt precision.

Feature drift monitoring with sampling rate 1 (100%) will detect shifts in input feature distributions, but it won’t directly explain which features are driving predictions or capture changes in model reliance/interaction effects. It is also the most expensive option because it logs and analyzes all online predictions (~200k/day), increasing storage and monitoring costs. Weekly cadence helps, but full sampling still conflicts with “keep monitoring costs low.”

This option improves cost by sampling only 10% of predictions and running weekly, which is a good cost-control pattern. However, it still focuses on feature drift (input distribution changes) rather than how the model uses features. If feature interactions change without large marginal distribution shifts, feature drift may not alert appropriately, and it does not satisfy the requirement to understand which features drive predictions.

Feature attribution drift monitoring aligns with the need to understand which features drive predictions and to detect changes in model reliance over time. However, sampling rate 1 is unnecessarily costly for a high-volume online endpoint. Logging and computing attributions for every prediction increases monitoring and storage costs significantly. Weekly frequency reduces job runs, but full sampling still violates the “keep monitoring costs low” requirement.