At some point, every data team ends up in the same room having the same conversation.
Someone asks, "Which database should we use?"
Then the familiar answers arrive quickly.
"We used ClickHouse at my last company."
"DuckDB is supposed to be really fast."
"We already have Postgres running."
None of these are bad observations. In fact, they are often useful starting points. But they are not decisions. They are preferences, memories, and conveniences dressed up as conclusions.
Choosing a data query engine is one of the most consequential decisions a data team makes because it shapes how data is stored, queried, operated, scaled, and eventually replaced. The mistake is treating the choice like a search for the fastest engine.
There is no fastest engine.
There is only the engine that fits a particular workload, team, and set of constraints.
The real job is not to pick the best tool in the abstract. It is to make a choice that can still be explained a year later, when the data has grown, the team has changed, and someone inevitably asks, "Why did we choose this?"
Start With the Workload
Before naming a single technology, it helps to slow down and describe the workload in plain language.
What kind of questions will the system answer?
Are users looking up one record at a time, or are they scanning millions or billions of rows to build reports and dashboards? A system designed for point lookups is very different from one designed for large analytical scans.
How will data arrive?
Some workloads depend on frequent small writes. Others ingest data in large batches. Some data is append-only. Other data needs regular updates and deletes. This matters because mutability is where many engines start to reveal their limits.
What does "fast" actually mean?
For an API, fast might mean 10 milliseconds. For an internal dashboard, it might mean two seconds. For an overnight report, it might mean 10 minutes. Without a latency budget, "fast" is just a vague hope.
How many users or requests will the system handle at once?
Ten analysts running heavy queries is not the same problem as thousands of application requests per second. Engines that are excellent for one case often make tradeoffs that hurt the other.
And finally, how large is the data today, and how quickly is it growing?
Current size matters, but the growth curve matters more. A comfortable choice at one terabyte can become painful at one hundred.
Once these questions are written down, the list of possible engines usually gets smaller very quickly.
Understand the Shape of the Tools
Most query engines have a natural center of gravity.
Columnar OLAP engines, such as ClickHouse, are built for analytical workloads. They store data by column, which makes wide scans and aggregations much cheaper because the engine only reads the columns required by the query. If the common question is "group, filter, and aggregate a huge number of rows," this family is often a strong candidate.
But that strength comes with tradeoffs. High-frequency single-row updates, transactional behavior, and highly mutable data are usually not where these systems feel most natural.
Embedded and in-process engines, such as DataFusion or DuckDB depending on the architecture, solve a different problem. They can bring analytical query power directly into an application or workflow without operating a separate database cluster. This can reduce operational overhead and give the team more control, especially when the query engine is part of a larger system.
Search engines and key-value stores live in another world entirely. If the workload is full-text search, relevance ranking, or very high-rate point lookups, a columnar analytics database is usually the wrong shape. The right tool may look nothing like a traditional analytics engine.
This is why the workload has to come first. The engine is only "good" in relation to the problem it is being asked to solve.
Benchmark With Your Own Data
Published benchmarks are tempting because they appear objective. They have charts, numbers, and rankings. But they are usually measuring someone else’s data, someone else’s queries, and someone else’s hardware.
That does not make them useless, but it does make them incomplete.
A useful benchmark is less glamorous. Take a representative slice of your own data. Replay the real queries your users or applications will run. Test the concurrency you expect. Measure the numbers that actually matter: p95 latency, ingestion throughput, cost per query, memory usage, operational complexity, and tail behavior under load.
Most importantly, run the test on infrastructure you would actually be willing to pay for.
The goal is not to produce the most impressive number. The goal is to produce a decision that can be defended.
Count the Costs That Do Not Show Up in the Benchmark
Raw query speed is easy to compare. It is also rarely the whole story.
The hidden costs often matter more.
Who will operate the system when something breaks at 3am? A database that is 20% faster but requires a dedicated team may be the more expensive choice in practice.
Does the engine have a mature ecosystem? Are there stable drivers, client libraries, monitoring tools, deployment patterns, and people who already know how to run it? Boring and well supported often beats clever and obscure.
How easy is it to leave?
This question is uncomfortable, but important. Every database choice creates some amount of lock-in. Using open formats such as Parquet and Arrow can keep future options open. A good architecture should make it possible to change your mind later without rewriting the entire system.
Write the Decision Down
The final output of this process should not only be a database deployment. It should be a short decision record.
Write down the workload you were solving for. Write down the candidates you considered. Write down what you benchmarked, what you measured, what you chose, and what tradeoffs you accepted.
Then write the most important line: when should this decision be revisited?
Maybe the answer changes when the dataset grows by 10x. Maybe it changes when concurrency reaches a certain threshold. Maybe it changes when updates become more frequent, or when operational cost crosses a limit.
A decision with a revisit trigger is healthier than a decision pretending to be permanent.
In the end, choosing a data query engine is not about winning an argument over which technology is best. It is about matching a tool to a workload, making the tradeoffs explicit, and leaving behind reasoning that others can understand.
That is what good technical decision-making looks like.
And in a data team, that reasoning is part of the system too.