When a data infrastructure bill gets large enough that someone notices, engineers look for the obvious candidates: forgotten S3 buckets, idle EC2 instances, snapshots from a project that ended two years ago. Those are worth cleaning up. They are also not where the money is going. The real cost drivers in modern data engineering are invisible: a JOIN running hourly without partition filters scanning petabytes it has no business touching, an always-on cluster provisioned for peak load that runs at five percent utilisation for twenty-three hours a day, data transfer fees that appear nowhere in the architecture diagram.

This is a systematic guide to finding the waste before it compounds. The 20-point audit works across cloud providers. The AWS, GCP, and Databricks sections cover the highest-impact cuts specific to each platform. The monitoring section covers the alerts that would have prevented the last three bill surprises.

Where the Money Goes

Five categories, ranked by waste potential. The order matters: if you start with storage, you are optimising the least important line item first.

The 20-Point Audit

Work through this in order. The highest-impact items are at the top of each category.

Compute and Warehouse

1. Identify the ten most expensive queries. Check INFORMATION_SCHEMA in the data warehouse. Find queries scanning the most bytes or consuming the most compute time. Action: Rewrite with partition filters or materialise the results. One query fix here typically exceeds weeks of storage optimisation in impact.
2. Check warehouse provisioning vs utilisation. AWS: CloudWatch Redshift CPU metrics. Action: Average CPU under 20% means the instance class is oversized. Scale down.
3. Audit development clusters. Any cluster with dev, test, or sandbox in the name that has been running for more than 24 hours is a candidate. Action: Terminate idle clusters. Set auto-suspend policies: 15 minutes of inactivity is sufficient for most development environments.
4. Cap auto-scaling limits. Serverless warehouses without a hard concurrency or node limit expose the infrastructure to unbounded spend under a bad query loop. Action: Set explicit maximums. A single unguarded query in a serverless warehouse can generate a four-figure bill before anyone notices.
5. Find orphaned snapshots. aws ec2 describe-snapshots --owner-ids self. Action: Delete manual snapshots older than 30 days.

Pipelines and Data Processing

6. Audit pipeline frequency. A slowly changing dimension table syncing every five minutes is generating compute and transfer costs for data that changes weekly. Action: Move batch pipelines from hourly to daily where real-time latency is not a genuine business requirement, not a stated preference.
7. Switch to Graviton instances for batch jobs. AWS Graviton (ARM) processors deliver approximately 20% lower cost with better performance for compatible workloads. Action: Migrate Glue and EMR task nodes to Graviton where supported.
8. Find duplicate pipelines. Two dbt models materialising the same base data is a codebase archaeology problem that compounds at scale. Action: Refactor into a single upstream model. This is a code review failure that becomes a cloud cost problem.
9. Review streaming infrastructure. Kafka or Kinesis running for workloads that update hourly and are consumed in batch is a pattern that appears frequently after architectural decisions made under pressure. Action: Audit latency requirements against actual consumption patterns. If no consumer needs sub-minute latency, batch is the correct answer.
10. Right-size orchestration environments. Managed Airflow environments are often over-provisioned relative to current DAG count and complexity. Action: Profile actual resource usage against current provisioning. Reduce if the gap is significant.

Storage and Data Transfer

11. Fix Cross-AZ data transfer. AWS Cost Explorer: DataTransfer-Regional-Bytes. Action: Keep workloads in a single AZ where possible. Use VPC Endpoints for AWS services to eliminate cross-AZ transfer costs on managed service traffic.
12. Replace NAT Gateway with VPC Endpoints for S3. S3 Gateway Endpoints are free. NAT Gateway charges per gigabyte of traffic routed through it. Action: Creating the VPC Endpoint takes approximately five minutes. Not creating it is paying per gigabyte for traffic that has a free routing option.
13. Set S3 Lifecycle policies. Data older than 90 days on S3 Standard that has not been accessed recently belongs on Glacier or Standard-IA. Action: Implement Intelligent-Tiering for data lakes. The monitoring fee is negligible compared to the resulting savings on large datasets.
14. Set CloudWatch log retention. Application logs accumulating indefinitely in CloudWatch Logs are a common discovery in cost audits. Action: Set retention policy to 14–30 days. Logs older than that are available in S3 if needed for compliance.
15. Audit Fivetran column selection. Syncing columns that no downstream model ever references is a common result of initial connector setup done quickly. Action: Block unused columns and tables from syncing. Fivetran pricing is based on rows synced for most connectors.

Architecture and Miscellaneous

16. Deprecate unused BI dashboards. Dashboards refreshing hourly that have not been viewed in 90 days are generating query costs for nobody. Action: Pull access logs. Delete dashboards that are not being used. Notify the stated owner before deletion.
17. Check EBS volume utilisation. Volumes at less than 10% utilisation are paying for capacity that is not needed. Action: Resize gp3 volumes to match actual usage plus a reasonable headroom margin.
18. Enforce resource tagging. Without consistent Environment, Project, and Owner tags, cost attribution by team or project is impossible and anomalies are invisible until they appear on the invoice. Action: Block resource creation without mandatory tags via Service Control Policies (SCPs) or equivalent.
19. Check multi-region replications. Cross-region S3 replication creates ongoing transfer costs and double the storage bill for replicated data. Action: Disable unless the replication serves a specific, documented disaster recovery requirement with a tested recovery objective.
20. Audit personal cloud storage. Users dumping CSVs and exports to personal cloud storage folders at scale creates both a cost and a data governance problem. Action: Implement quotas. Automated cleanup scripts for unaccessed personal exports after 30 days.

AWS: Eight Highest-Impact Cuts

GCP: Eight Highest-Impact Cuts

Databricks: Five Critical Controls

Monitoring Setup

A monthly review of the billing console is not a monitoring strategy. By the time an anomaly is visible on the monthly invoice, it has been running for weeks. The alerts below catch it in hours.

• Daily spend anomaly alert: Configure a budget alert in AWS Cost Explorer or GCP Billing at 110% of the 30-day daily average. A single bad query running overnight is visible the next morning, not at month-end.
• Per-service budget alerts: Set separate alerts for the top three cost drivers (typically compute, transfer, and managed services). A spike in one service is a different problem from a spike across all services.
• Query cost alerting: For BigQuery: monitor INFORMATION_SCHEMA.JOBS_BY_PROJECT on a schedule. For Snowflake: QUERY_HISTORY view. Route alerts for any single query exceeding a defined cost threshold.
• Idle cluster detection: A CloudWatch or GCP Monitoring metric alert on clusters with CPU under 5% for more than two hours during business hours. Idle clusters during off-hours are expected. Idle clusters at 2pm are a configuration problem.
• Data transfer spike detection: Alert on daily data transfer costs exceeding 150% of the rolling 14-day average. Transfer spikes usually indicate a pipeline change that introduced cross-region or cross-AZ traffic.

Nauman Shahid builds zero-dependency data infrastructure for organisations in the UAE and Gulf region. If the bill is large enough that a systematic audit is worth running as an engagement, the diagnostic starts at: www.mindflex.tech.