The data tooling market is built on a specific asymmetry: vendors know exactly what their products cost you at scale, and most engineering leaders do not. The sales cycle exploits that gap. This document exists to close it before a contract is signed.

What follows is an honest assessment of when the major data infrastructure categories are justified, what they cost at different volumes, and what the vendor is hoping you will not ask. The comparison tables use published pricing as of mid-2025. The decision flowchart applies one criterion: does your actual data volume and latency requirement justify this tool, or are you buying for anticipated scale that may never arrive?

The over-engineering checklist at the end is the fastest diagnostic. Tick more than three boxes and the problem is not tooling: it is that the architecture is serving resumes, not the business.

The Tool Evaluation Matrix

Each comparison shows the actual scale threshold where the more expensive option becomes justified, monthly costs at three volume tiers, operational overhead, vendor lock-in exposure scored from 0 to 10, and an honest recommendation. Read the recommendation column first.

Comparison	Scale threshold	Cost: <1M rows/day	Cost: 1–10M rows/day	Cost: 10M+ rows/day	Lock-in score	Honest recommendation
Snowflake vs DuckDB	100 GB+ active data, 5+ concurrent writers	Snowflake: ~$50 DuckDB: $0	Snowflake: ~$300 DuckDB: $0	Snowflake: $2,000+ DuckDB: $0–$50 (VM)	Snowflake: 8/10 DuckDB: 1/10	Use DuckDB on a single VM until it strains. You do not need Snowflake for a 10 GB database.
BigQuery vs DuckDB + S3	1 TB+ querying, complex ML integration	BQ: ~$5 DuckDB: $0	BQ: ~$50 DuckDB: $0	BQ: $1,000+ DuckDB: $0	BQ: 9/10 DuckDB: 1/10	BigQuery is cheap at low volume and aggressive at scale. Use DuckDB against local Parquet until analysts complain about query times.
Airflow vs cron + Python	20+ interdependent DAGs, frequent backfilling	Airflow: $50 (hosting) Cron: $0	Airflow: $150 Cron: $0	Airflow: $500+ Cron: maintenance burden	Airflow: 7/10 Cron: 0/10	If you have three Python scripts running nightly, use cron. Do not stand up Airflow for three scripts.
dbt Core vs raw SQL	5+ data modellers, deeply nested views	dbt: $0 SQL: $0	dbt: $0 SQL: $0	dbt: $0 SQL: maintenance burden	dbt Core: 4/10 Raw SQL: 0/10	Adopt dbt Core early if you have dedicated analysts. Avoid dbt Cloud until the Snowflake bill is already unavoidable.
Databricks vs DuckDB	Real-time streaming, multi-node processing	DBX: $200 DuckDB: $0	DBX: $800 DuckDB: $0	DBX: $5,000+ DuckDB: $0	DBX: 9/10 DuckDB: 1/10	Spark is a distributed system for distributed data. Most organisations do not have distributed data. DuckDB replaces ninety percent of local Spark use cases.
Kafka vs Postgres replication vs batch	Sub-second latency requirements	Kafka: $150+ Postgres: $0	Kafka: $400 Postgres: $0	Kafka: $2,000+ Postgres: $0	Kafka: 6/10 Postgres: 2/10	Batch is correct for ninety-nine percent of businesses. Real-time requirements are frequently an executive preference, not an operational need. Use Postgres logical replication before Kafka.
Redshift vs DuckDB + Parquet	Legacy AWS entrenchment, massive concurrent joins	Redshift: $180 DuckDB: $0	Redshift: $360 DuckDB: $0	Redshift: $2,000+ DuckDB: $0	Redshift: 9/10 DuckDB: 2/10	Redshift is aging. DuckDB querying Parquet over S3 is fast and effectively free. Migrate if you are not already committed.
Looker vs Evidence.dev vs Metabase	Non-technical users who need to build dashboards	Looker: $3,000+ Evidence/Metabase: $0	Looker: $3,000+ Evidence/Metabase: $0	Looker: $5,000+ Evidence/Metabase: $0	Looker: 10/10 Evidence/Metabase: 2/10	Metabase is the default. Evidence.dev is excellent if your consumers can read markdown. Looker is for when you have run out of better arguments.
Fivetran vs custom ETL	15+ distinct APIs with changing schemas	Fivetran: $500 Custom: $0	Fivetran: $1,500+ Custom: $0	Fivetran: $5,000+ Custom: $0	Fivetran: 8/10 Custom: 0/10	Fivetran is the cost of not writing API polling scripts. Worth it for Salesforce or Zendesk. Not worth it for your own internal Postgres database that your engineers control.

The Decision Flowchart

Four questions. Answer them in order. Stop at the first No.

The Over-Engineering Audit

Tick more than three and the architecture is not serving the business. Tick more than seven and the stack is a resume-driven development project.

1. You have a "Data Platform Team" but fewer than fifty total engineers in the company. (The overhead of the platform team exceeds the value it delivers at this scale.)
2. You run Apache Kafka but total message throughput is below 1,000 messages per second. (Kafka's operational complexity is unjustified below tens of thousands of messages per second.)
3. Your Airflow DAGs mostly consist of SELECT * FROM table_a and write to table_b. (Cron handles this. Airflow was not built for it.)
4. You are paying Fivetran to sync data from an internal database that your own engineers control. (Write the script.)
5. You use Kubernetes to run daily batch jobs. (The orchestration overhead exceeds the workload complexity.)
6. You have a "data mesh" strategy but less than one terabyte of total data. (Data mesh is an organisational solution to a scale problem you do not yet have.)
7. You use Databricks but ninety-five percent of jobs are simple SQL aggregations. (DuckDB runs these faster and costs nothing.)
8. Your monthly Snowflake bill exceeds your primary application database bill. (The analytics layer costs more than the system it analyses.)
9. You have implemented a reverse-ETL tool to push data back to a system with a simple REST API. (Write the Python script.)
10. You talk about "streaming pipelines" but users check the dashboards on Monday mornings. (Hourly batch is streaming for this use case.)
11. You use dbt but have fourteen layers of nested views for a five-table schema. (The transformation complexity exceeds the data model complexity.)
12. You are evaluating data catalogue tools for a team of three people. (A well-named dbt project is your data catalogue.)
13. Your deployment process for a new SQL query takes more than thirty minutes. (The process is the bottleneck, not the query.)
14. You use AWS Glue but scripts fit within standard Lambda memory limits. (Run a Lambda function.)
15. Multiple vendors in the stack do essentially the same thing. (One of them is not being used.)
16. You bought Looker because "we might need the semantic layer later." "Later" has not arrived in most organisations that say this.)
17. Engineers spend more time maintaining pipeline infrastructure than writing transformations. (The infrastructure is the product, not the data.)
18. You have implemented Change Data Capture for metrics that change weekly. (Batch handles this.)
19. Your architecture diagram has more than seven distinct vendor logos. (Each logo is a dependency and a renewal negotiation.)
20. When a stakeholder requests a new column, the answer involves the word "sprint." (The pipeline is too rigid for its purpose.)

The Zero-Dependency Baseline

What the stack should look like before a vendor contract is signed. These are not aspirational: they are the architectures that serve most real workloads at each tier.

Tier 1: Startup (under 1 million rows per day)

• Ingestion: Python scripts on cron or systemd timers on a single VM
• Storage: Postgres for transactional; S3 with Parquet for analytics
• Transformation: Python or raw SQL
• Orchestration: Cron
• Serving: Metabase (open-source) on a $20/month VM
• Monthly cost: approximately $100
• Maintenance: Two hours per month

Tier 2: Scale-up (1–10 million rows per day)

• Ingestion: Cloud Run or Fargate tasks on a schedule
• Storage: S3 with partitioned Parquet
• Transformation: dbt Core + DuckDB
• Orchestration: GitHub Actions or Dagster (open-source)
• Serving: Metabase or Evidence.dev against DuckDB
• Monthly cost: $300–$500
• Maintenance: One day per month

Tier 3: Enterprise (10 million+ rows per day)

At this scale, some managed tooling becomes defensible. Use it selectively.

• Ingestion: Managed connector (Fivetran or Airbyte) for complex third-party APIs only. Custom Python for internal databases.
• Storage: Snowflake or BigQuery for the curated presentation layer. S3 with Parquet for raw data.
• Transformation: dbt Core
• Orchestration: Airflow or Dagster
• Serving: Metabase or Superset
• Monthly cost: $3,000+
• Maintenance: One to two full-time engineers

Vendor Sales Phrase Translation

On the sales call, these phrases mean something different from what is being said. Translate them in real time.

Nauman Shahid builds zero-dependency data infrastructure for organisations in the UAE and the Gulf region. If your current data stack has more vendor dependencies than your workload requires, a diagnostic engagement identifies the exposure: www.mindflex.tech. The Vendor Lock-In Audit is at audit.nauman.cc.