Over-Provisioning — AWS Data Engineer (DEA-C01)

Choose between a serverless ETL service (AWS Glue) and a managed-cluster service (Amazon EMR) when data volume and transformation complexity are moderate and the dominant constraint is operational simplicity with minimized cost — not peak throughput or custom runtime flexibility.

When a pipeline is short, linear, and lacks complex DAG requirements or cross-team scheduling needs, choose serverless-native orchestration over a managed Airflow cluster; 'managed' does not equal 'serverless,' and an always-on Airflow environment is over-provisioned for a workload Step Functions handles at zero infrastructure cost.

Choose the managed ingestion service whose capacity model and operational footprint match a periodic, schedule-driven SaaS pull rather than a continuously provisioned real-time stream.

Whether the orchestration requirements of a short, linear, event-driven ETL pipeline justify provisioning an always-on MWAA environment, or whether AWS Step Functions satisfies all requirements serverlessly at lower operational and infrastructure cost.

Whether serverless state-machine orchestration (Step Functions) or a managed Airflow environment (MWAA) is the correct fit when the pipeline has a low step count, no complex DAG branching, and an explicit serverless-preference plus cost-minimization constraint.

When a transformation workload's payload size and execution duration both fit within Lambda's runtime ceiling, choose Lambda deployed via SAM over a provisioned EMR cluster; EMR adds cluster lifecycle management and idle cost that the workload does not justify.

Whether a 30-minute scheduled SaaS-to-S3 poll warrants provisioned streaming shard infrastructure (Kinesis Data Streams) or whether a managed SaaS connector service (AppFlow) right-sizes the ingestion tier to match the periodic cadence and operational-overhead constraint.

When query frequency is low and the workload is intermittent, choose a serverless query engine billed per-TB-scanned over a provisioned cluster that charges for idle compute regardless of utilization.

Whether to rely on scheduled AWS Glue crawlers to discover new partitions versus using programmatic Glue partition API registration or Lake Formation governed tables, given that the partition scheme is fully predictable and frequent crawler runs constitute over-provisioned cataloging spend.

Select between a provisioned always-on cluster and a serverless pay-per-scan query engine when query frequency is low and intermittent, and the latency SLA is measured in minutes rather than seconds.

Which S3 storage class and lifecycle transition delivers the lowest per-GB cost for a seven-year retention archive with a sub-annual access frequency while still guaranteeing that standard retrieval completes within the 12-hour audit SLA window?

When a SQL-on-S3 workload is intermittent and low-frequency, serverless per-scan query execution (Athena) better matches cost to actual utilization than an always-on provisioned cluster (Redshift), making query cadence and utilization pattern the deciding factor over raw capability.

Whether to implement pre-load data quality validation using AWS Glue with a fixed DPU worker allocation and custom validation scripts, or AWS Glue DataBrew with declarative quality rules that scale on-demand to actual daily data volume.

Whether the intermittent, low-utilization query frequency tips the cost-efficiency decision toward serverless on-demand execution (Athena) rather than a provisioned always-on cluster (Redshift) whose idle hours accumulate fixed spend with no workload to justify it.

Whether to use a serverless, per-execution ETL service (AWS Glue) that charges only for active DPU-seconds versus provisioning a managed cluster (Amazon EMR) with fixed capacity that accrues cost during idle windows for a bursty, event-driven transformation workload.

Whether to use a serverless job-based profiling service (AWS Glue DataBrew) or provision a managed cluster (Amazon EMR) for an intermittent, once-per-month profiling workload where cluster idle cost and lifecycle overhead are explicitly disqualifying.

Whether to enforce least-privilege access across a multi-service analytics stack through centralized Lake Formation grants or through individually authored IAM resource-based policies—where the deciding factor is the operational cost of scaling access control as teams and services grow.

Select the log collection and query layer that satisfies end-to-end observability for an infrequent, event-driven pipeline workload without provisioning persistent cluster capacity.

Whether to provision an always-on search cluster or use serverless ad-hoc query capability against exported log data, determined by query frequency and the explicit no-persistent-infrastructure cost constraint.