Practice Test #2

AWS Data Exchange is a service for finding, subscribing to, and using third-party datasets in AWS. It does not provide workflow orchestration, scheduling, or DAG execution. Even though data pipelines may consume external datasets, Data Exchange is not an orchestrator and cannot satisfy requirements like running the same open-source DAG definitions on premises and in a managed cloud service.

Amazon SWF is an AWS-native workflow coordination service. While it can orchestrate tasks, it is not Apache Airflow and does not use Airflow DAG definitions. Using SWF would require redesigning the workflow logic and application integration, increasing vendor lock-in. It also does not provide a “fully managed equivalent” of an on-prem open-source DAG engine with shared DAG code.

AWS Glue is a serverless data integration (ETL/ELT) service with crawlers, jobs, and Glue Workflows. However, Glue is not Apache Airflow and does not allow running the same Airflow DAGs on premises and in AWS without rewriting. Glue also cannot be deployed as the same open-source engine on an on-prem Kubernetes cluster, so it fails the portability and lock-in requirements.

AWS Step Functions can orchestrate ETL steps, but it is a general-purpose workflow service rather than the most natural feature for a Glue-centric pipeline. In this scenario, the pipeline already depends on Glue-native capabilities such as crawling the JDBC source and running the ETL job, so adding Step Functions introduces an extra orchestration service that is not necessary. While Step Functions is serverless and low overhead, the exam-oriented best answer is the Glue-native orchestration feature when the workflow is primarily crawler-to-job sequencing. Therefore, Step Functions is viable but not the most cost-effective or direct fit among the listed choices.

AWS Glue Studio is a visual development interface for creating, editing, and monitoring Glue ETL jobs. It helps data engineers design transformations and generate Glue job code, but it is not the primary orchestration feature for chaining crawlers, jobs, and scheduled dependencies end to end. The question asks for the service or feature that will orchestrate the pipeline on a daily schedule with minimal overhead, which points to Glue workflows rather than the Studio UI. Choosing Glue Studio confuses job authoring with workflow orchestration.

Amazon MWAA provides managed Apache Airflow for complex DAG-based orchestration across many systems, but it is usually excessive for a single daily Glue-oriented ETL pipeline. MWAA requires a continuously running Airflow environment, which increases both cost and operational complexity compared with Glue workflows. The question emphasizes minimal managed service overhead and cost control, making MWAA a poor fit. It is better suited for organizations that already standardize on Airflow or need extensive custom orchestration beyond Glue-native capabilities.

Incorrect. Increasing shards raises aggregate capacity, but it does not inherently fix partition-key skew. If merchantId remains the partition key, the flash-sale merchant’s records will still hash to the same shard (or small subset after resharding), keeping those shards hot and throttled while other shards remain underutilized.

Incorrect. Throttling producers to 1,000 records/s reduces ingestion throughput and violates the requirement to keep the same overall throughput. The stream already has enough aggregate capacity; the issue is uneven distribution across shards, not insufficient total capacity.

Incorrect. Reducing record size can help if the shard is hitting the 1 MB/s limit, but hot shards can also be constrained by the 1,000 records/s per-shard limit. With 2 KB records, the dominant merchant can exceed the per-shard record-rate limit even though MB/s is low. The root cause is partition-key skew, not record size.

High operational overhead and unnecessary complexity. MWAA requires managing an Airflow environment (workers, schedulers, plugins, dependencies). Using both Lambda for correlation and Step Functions for orchestration duplicates orchestration responsibilities. Also, correlating 100+ GB with Lambda is not ideal due to runtime/memory limits and distributed join needs; Glue/Spark is a better fit for large-scale joins.

AppFlow + Glue is a strong ingestion/ETL approach, but adding MWAA increases operational overhead versus Step Functions for a once-nightly pipeline. MWAA still needs environment management, scaling, and DAG maintenance. Unless the company already standardizes on Airflow or needs complex DAG patterns/operators, Step Functions is typically the lower-ops orchestration choice.

Over-engineered for the requirement. Kinesis Data Streams and Managed Service for Apache Flink are designed for real-time streaming ingestion and stream processing. The question requires a nightly batch run, not continuous correlation. Additionally, pulling from PostgreSQL into Kinesis is non-trivial and often requires custom producers or CDC tooling, increasing operational burden compared to Glue JDBC extraction.

Amazon MWAA can certainly orchestrate SQL tasks on a schedule, but it requires creating and maintaining an Airflow environment, DAGs, IAM roles, and supporting configuration. That is significantly more operational overhead than the question requires for a straightforward recurring SQL maintenance task. The requirement explicitly says to avoid provisioning or managing orchestration infrastructure, and MWAA is still an orchestration platform even though AWS manages parts of it. Therefore, MWAA is functional but not the least-effort option.

Redshift Lambda UDFs are invoked from SQL statements during query execution and are not a native scheduling mechanism for recurring database maintenance. A UDF cannot independently wake up every 30 minutes and trigger refreshes on a timer inside Redshift Serverless. To make this work, the team would still need an external scheduler such as EventBridge or another orchestration service, which contradicts the stated requirement. In addition, using a UDF for this purpose is an unnatural design compared with built-in scheduled queries.

AWS Glue workflows and jobs can be scheduled and can connect to Amazon Redshift, but they are designed primarily for ETL and data integration pipelines rather than simple recurring SQL administration tasks. Implementing this would require creating and maintaining a Glue job or Python shell job, configuring connectivity, IAM permissions, and operational monitoring. That is more setup and more moving parts than necessary when Redshift already provides native scheduled query capability. As a result, Glue can meet the functional requirement but not the least-effort requirement.

Pandas drop_duplicates() is simple for exact duplicate rows, but it is not a managed, auto-scaling approach for 3 TB/week. You would need to run it on self-managed compute (EC2/ECS/EKS) or engineer a distributed approach, and Pandas is memory-bound (single-node) unless you add additional frameworks. This increases operational overhead and risks performance bottlenecks at 120M rows.

A custom Python ETL using the third-party dedupe library can perform probabilistic matching, but it violates the requirement to avoid third-party libraries. It also implies operational overhead for dependency management, packaging, runtime compatibility, and scaling the compute environment. This approach is harder to operationalize reliably at multi-terabyte weekly volumes without significant engineering effort.

Running the third-party dedupe library inside an AWS Glue job still violates the “avoid third-party libraries” requirement and adds operational burden (managing Python wheel/egg dependencies, Glue version compatibility, job bootstrap, and troubleshooting). While Glue provides serverless scaling, the dependency lifecycle and potential Spark/Python environment issues increase overhead compared to using Glue’s built-in FindMatches transform.

Not ideal. Implementing 30-minute time-based aggregations with 90 seconds of late arrivals in Lambda requires external state storage (e.g., DynamoDB) and custom windowing, watermarking, and deduplication logic. This increases operational overhead and complexity, and correctness becomes harder under retries and partial failures. Lambda is better for stateless transforms or lightweight enrichment, not robust event-time analytics.

Partially relevant because Managed Service for Apache Flink is the right engine for stateful aggregations. However, the option focuses on duplicates rather than the stated requirement of event-time windows with late arrivals. While Flink can handle duplicates (often via keys/ids and state), the core requirement is time-based analytics with event-time windowing; this option is less directly aligned than D.

Still not ideal. Even though it mentions tumbling windows and event timestamps, Lambda does not natively provide Flink-like event-time windowing semantics, watermarks, and managed state for long windows with late data. You would need to build and operate significant custom logic and state management to handle late arrivals and window finalization, which conflicts with the “minimum operational overhead” requirement.

Kinesis Data Firehose can deliver streaming data to Redshift, but it commonly uses S3 staging and COPY under the hood, operating in buffered micro-batches. Achieving consistent sub-60-second freshness can be challenging depending on buffering settings, COPY performance, and peak throughput. It’s lower-ops than custom consumers, but not as direct/low-latency as Redshift streaming ingestion for Kinesis Data Streams.

Landing to S3 and using the Redshift COPY command is a batch ingestion pattern. It requires building and operating a process to write files, manage partitions, trigger COPY jobs, handle retries, and tune file sizes/manifesting. This increases operational overhead and typically cannot guarantee sub-minute freshness because it depends on file creation and batch load scheduling.

Aurora zero-ETL integration with Amazon Redshift is specifically for replicating data from Amazon Aurora (MySQL/PostgreSQL) into Redshift with minimal ETL. The source in this scenario is Amazon Kinesis Data Streams, not Aurora, so this option does not address the ingestion requirement and cannot meet the stated streaming telemetry use case.

This is incorrect because the Lambda execution role trust policy determines which service can assume the execution role when the function runs. EventBridge does not assume the Lambda execution role in order to invoke the function. Invocation permission is controlled by the Lambda function's resource-based policy, so changing the execution role trust relationship would not resolve the AccessDeniedException. This option confuses runtime permissions with invoke permissions.

This is incorrect because placing a Lambda function in a private subnet affects the function's network access to resources inside or outside a VPC. It does not control whether EventBridge is authorized to invoke the function. EventBridge-to-Lambda invocation is governed by IAM and Lambda resource-based permissions, not subnet placement. Therefore, moving the function into a private subnet would not fix an authorization failure from Lambda.

This is incorrect because the EventBridge schema registry and event pattern mapping are related to event discovery and rule matching, not Lambda authorization. If the event pattern were wrong, the rule would simply not match or the target would not be invoked. The question specifically states that the invocation fails with AccessDeniedException from Lambda, which points to missing invoke permission on the Lambda side. Therefore, schema or pattern corrections would not address the root cause.