QuestDB Benchmark Compliance Issue: Query Result Caching Enabled by Default

[xe-nvdk]

Summary

QuestDB's ClickBench results appear to violate benchmark compliance rules by running with query result caching and intermediate data caching enabled by default. The benchmark scripts (benchmark.sh and run.sh) do not explicitly disable these caches, which are enabled in QuestDB's default configuration.

ClickBench Rules

From the official benchmark rules:

"If the system contains a cache for query results, it should be disabled."

"If the system contains a cache for intermediate data, that cache should be disabled if it is located near the end of the query execution pipeline, thus similar to a query result cache."

Evidence of Violation

1. Query Result Cache Enabled by Default

Source Code Evidence:

File: core/src/main/java/io/questdb/PropServerConfiguration.java:1812

this.httpSqlCacheEnabled = getBoolean(properties, env,
    PropertyKey.HTTP_QUERY_CACHE_ENABLED, true);

Default value: true (third parameter)

Configuration file: core/src/main/resources/io/questdb/site/conf/server.conf:155

# enables the query cache
#http.query.cache.enabled=true

The configuration line is commented out, meaning the code default of true applies.

Cache Implementation:

File: core/src/main/java/io/questdb/cutlass/http/HttpServer.java:89-92

if (serverConfiguration.isQueryCacheEnabled()) {
    this.selectCache = new ConcurrentAssociativeCache<>(
        serverConfiguration.getConcurrentCacheConfiguration());
} else {
    this.selectCache = NO_OP_CACHE;
}

File: core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessor.java:204-216

final RecordCursorFactory factory = context.getSelectCache().poll(state.getQuery());
if (factory != null) {
    // queries with sensitive info are not cached, doLog = true
    try {
        sqlExecutionContext.storeTelemetry(CompiledQuery.SELECT, TelemetryOrigin.HTTP_JSON);
        executeCachedSelect(state, factory);
    } catch (TableReferenceOutOfDateException e) {
        LOG.info().$safe(e.getFlyweightMessage()).$();
        compileAndExecuteQuery(state);
    }
} else {
    // new query
    compileAndExecuteQuery(state);
}

The query execution explicitly checks for cached query results and reuses them.

2. PostgreSQL Query Cache Also Enabled by Default

File: core/src/main/java/io/questdb/PropServerConfiguration.java:1806

this.pgSelectCacheEnabled = getBoolean(properties, env,
    PropertyKey.PG_SELECT_CACHE_ENABLED, true);

3. Symbol Table Cache Enabled by Default

File: core/src/main/java/io/questdb/PropServerConfiguration.java:1393

this.defaultSymbolCacheFlag = getBoolean(properties, env,
    PropertyKey.CAIRO_DEFAULT_SYMBOL_CACHE_FLAG, true);

Symbol tables are intermediate data structures that cache dictionary lookups, which should be disabled per benchmark rules.

4. Benchmark Scripts Do Not Disable Caches

File: questdb/benchmark.sh

The benchmark script only modifies two configuration parameters:

sed -i 's/query.timeout.sec=60/query.timeout.sec=500/' ~/.questdb/conf/server.conf
sed -i "s|cairo.sql.copy.root=import|cairo.sql.copy.root=$PWD|" ~/.questdb/conf/server.conf

Missing required configurations:

http.query.cache.enabled=false
pg.select.cache.enabled=false
cairo.default.symbol.cache.flag=false

These settings are completely absent from both benchmark.sh and run.sh, meaning QuestDB runs with all caches enabled by default.

Impact on Results

Suspicious Query Patterns

QuestDB has numerous queries where the "cold" run is barely slower than warm runs, suggesting query result caching is assisting:

• Q1: Cold 0.022s, Warm 0.000s (instant warm - cached?)
• Q20: Cold 0.034s, Warm 0.033s (1.03x - nearly identical)
• Q24: Cold 8.389s, Warm 0.539s → then Q24 run 2-3: 1.051s, 0.028s (massive drop suggests cache hit)
• Q25: Cold 0.055s, Warm 0.001s (55x faster - extreme caching benefit)
• Q27: Cold 0.007s, Warm 0.006s (1.17x - minimal difference)
• Q30: Cold 0.092s, Warm 0.007s (13x faster - caching benefit)

26 out of 43 queries show cold/warm ratios less than 2.0x, which is highly unusual for a properly flushed cache benchmark.

Additional Timing Measurement Issues

QuestDB's benchmark uses the "execute" timing from their API:

grep -P '"timings"|"error"|null' | \
sed -r -e 's/^.*"error".*$/null/; s/^.*"execute":([0-9]*),.*$/\1/' | \
awk '{ print ($1) / 1000000000 }'

File: core/src/main/java/io/questdb/cutlass/http/processors/JsonQueryProcessorState.java:1095

.putAsciiQuoted("execute").putAscii(':')
    .put(nanosecondClock.getTicks() - executeStartNanos)

This timing calculation happens during JSON serialization, which means:

• JSON serialization time is NOT included
• HTTP response writing is NOT included
• Network transmission is NOT included

Other databases (ClickHouse, DuckDB, Arc) measure full end-to-end client time including all overhead. QuestDB's reported times may be 10-30% lower than comparable measurements.

Recommendations

1. Add explicit cache disabling to benchmark scripts:

   echo "http.query.cache.enabled=false" >> ~/.questdb/conf/server.conf
   echo "pg.select.cache.enabled=false" >> ~/.questdb/conf/server.conf
   echo "cairo.default.symbol.cache.flag=false" >> ~/.questdb/conf/server.conf

2. Re-run benchmarks with caches properly disabled

3. Document timing methodology - clarify what overhead is included/excluded

4. Update configuration documentation to show which caches are enabled by default

Request for Response

@questdb team - Can you confirm whether these caches are disabled during ClickBench runs? If they are disabled, please show the configuration used. If not, the results should be re-run with proper compliance.

References

• QuestDB Source Code: https://github.com/questdb/questdb
• ClickBench Benchmark Rules: https://github.com/ClickHouse/ClickBench
• QuestDB benchmark.sh: https://github.com/ClickHouse/ClickBench/blob/main/questdb/benchmark.sh

---

Disclosure: I'm reporting this issue in good faith to ensure benchmark integrity. Arc explicitly disables all caches as required by benchmark rules, and we believe all participants should follow the same standards.

[xe-nvdk]

Also, the benchmark.sh is failing

/benchmark.sh: line 25: pigz: command not found
{"ddl":"OK"}curl: (26) Failed to open/read local data from file/application
waiting for rows to become readable...

[nwoolmer]

Query Result Cache Enabled by Default

QuestDB does not have a query result cache.

The query execution explicitly checks for cached query results and reuses them.

Nope.

Symbol Table Cache

You don't understand what this is.

Other databases (ClickHouse, DuckDB, Arc) measure full end-to-end client time including all overhead. QuestDB's reported times may be 10-30% lower than comparable measurements.

Citation needed. Looks like ClickHouse also report actual execution time.

ClickBench/clickhouse/run.sh

Line 20 in 5f07b86

RES=$(clickhouse-client --host "${FQDN:=localhost}" --password "${PASSWORD:=}" ${PASSWORD:+--secure} --time --format=Null --query="$query" --progress 0 2>&1 ||:)

I believe this is an example of using an LLM to try to get ahead, but not understanding the output.

[nwoolmer]

Also, the benchmark.sh is failing

/benchmark.sh: line 25: pigz: command not found
{"ddl":"OK"}curl: (26) Failed to open/read local data from file/application
waiting for rows to become readable...

Did you install pigz?

ClickBench/questdb/benchmark.sh

Lines 20 to 22 in 5f07b86

	sudo apt-get install -y pigz
	wget --continue --progress=dot:giga 'https://datasets.clickhouse.com/hits_compatible/hits.csv.gz'
	pigz -d -f hits.csv.gz

Subsequent failure is likely due to the file being missing.. because you didn't have pigz. Let me know if this resolves your problem 👍

[xe-nvdk]

Also, the benchmark.sh is failing

/benchmark.sh: line 25: pigz: command not found
{"ddl":"OK"}curl: (26) Failed to open/read local data from file/application
waiting for rows to become readable...

Did you install pigz?

ClickBench/questdb/benchmark.sh

Lines 20 to 22 in 5f07b86

sudo apt-get install -y pigz
wget --continue --progress=dot:giga 'https://datasets.clickhouse.com/hits_compatible/hits.csv.gz'
pigz -d -f hits.csv.gz
Subsequent failure is likely due to the file being missing.. because you didn't have pigz. Let me know if this resolves your problem 👍

According to Clickbench the benchmark.sh shouldn't require pre requisites. The benchmark.sh should run without any issues to reproduce results.

[xe-nvdk]

Query Result Cache Enabled by Default

QuestDB does not have a query result cache.

The query execution explicitly checks for cached query results and reuses them.

Nope.

Symbol Table Cache

You don't understand what this is.

Other databases (ClickHouse, DuckDB, Arc) measure full end-to-end client time including all overhead. QuestDB's reported times may be 10-30% lower than comparable measurements.

Citation needed. Looks like ClickHouse also report actual execution time.

ClickBench/clickhouse/run.sh

Line 20 in 5f07b86

RES=$(clickhouse-client --host "${FQDN:=localhost}" --password "${PASSWORD:=}" ${PASSWORD:+--secure} --time --format=Null --query="$query" --progress 0 2>&1 ||:)
I believe this is an example of using an LLM to try to get ahead, but not understanding the output.

As cache enabled, I run the test after fix the issues and the results are different. No LLM just estimation on the results that I got. In the code you have by default enable cache and in the benchmark.sh nothing is disabling those settings.

[xe-nvdk]

I dont need to be ahead, Im looking fairness, you accused me of cheating and mine was a honest mistake, in the other hand, a little bit of investigation show that you cache by default and cheat at different levels on the benchmark.

Evidences are there. Competition elevates us. I will not comment anymore, just focus on make Arc better everyday.

[puzpuzpuz]

QuestDB's ClickBench results appear to violate benchmark compliance rules by running with query result caching and intermediate data caching enabled by default.

QuestDB doesn't cache data or query results or any intermediate data.

echo "http.query.cache.enabled=false" >> ~/.questdb/conf/server.conf

This is HTTP API's prepared statements cache. A prepared statement is execution plan, not anything from the list above. It allows the database avoiding query compilation saving a few millis per query execution. Any SQL database does this.

echo "pg.select.cache.enabled=false" >> ~/.questdb/conf/server.conf

This is PGWire's (posgresql protocol) prepared statements cache.

echo "cairo.default.symbol.cache.flag=false" >> ~/.questdb/conf/server.conf

This is CACHE/NO CACHE default value for creating symbol columns in the CREATE TABLE statement. When enabled (the default), it turns on dictionary caching (not row data caching) for the given symbol column - this is metadata caching. The cache is used for the key code search: for the given string it gives you its code. This cache is used by our ILP (Influx Line Protocol) implementation to speed up the ingestion. For SQL queries it's not used since it provides no value there (even if it did, the best thing we can lookup is something like filtered symbol's code, i.e. single operation per query execution).

[nwoolmer]

@puzpuzpuz has added some specifics, I will also clarify a few points.

According to Clickbench the benchmark.sh shouldn't require pre requisites. The benchmark.sh should run without any issues to reproduce results.

The benchmark script includes the pigz install, I quoted it to you. Did you check that bit?

Alexey, let me know if it gets stuck for you, we'll check it if needed.

As cache enabled, I run the test after fix the issues and the results are different. No LLM just estimation on the results that I got. In the code you have by default enable cache and in the benchmark.sh nothing is disabling those settings.

Stop trusting AI slop. This is not comparable to the issues you faced with your own benchmark, which misrepresented data from hot runs as cold runs, removing the overhead of reading from the parquet file from your timings.

And if it is not clear enough, re-quoting the rules you mentioned:

"If the system contains a cache for query results, it should be disabled."

It doesn't.

"If the system contains a cache for intermediate data, that cache should be disabled if it is located near the end of the query execution pipeline, thus similar to a query result cache."

It doesn't. The query runs against the raw data each time.

I dont need to be ahead, Im looking fairness, you accused me of cheating and mine was a honest mistake, in the other hand, a little bit of investigation show that you cache by default and cheat at different levels on the benchmark.

Actually, neither @puzpuzpuz or myself accused you of cheating at all. He simply raised strange inconsistencies in your results when compared to DuckDB (which you use as your query engine), and also that in-memory databases were being included in cold run results (which is an apples-to-oranges comparison).

If you are upset at your results being called 'cheated', then please take that up with @alexey-milovidov - and don't take it out on us!

Evidences are there.

The evidence is not there, please can you clarify which of the 4 points you raised are accurate (if any), and which are not?

Competition elevates us. I will not comment anymore, just focus on make Arc better everyday.

Competition is great, and it should be done on a level playing field, else the measurement is somewhat useless. In that spirit, once the corrected results are live, will you revise your marketing posts?

[xe-nvdk]

The results are there expecting merging, and we don't have any issues correcting that we had a mistake... honesty should be first.

#661

About the blog, yes, I just updated this

@puzpuzpuz has added some specifics, I will also clarify a few points.

According to Clickbench the benchmark.sh shouldn't require pre requisites. The benchmark.sh should run without any issues to reproduce results.

The benchmark script includes the pigz install, I quoted it to you. Did you check that bit?

Alexey, let me know if it gets stuck for you, we'll check it if needed.

As cache enabled, I run the test after fix the issues and the results are different. No LLM just estimation on the results that I got. In the code you have by default enable cache and in the benchmark.sh nothing is disabling those settings.

Stop trusting AI slop. This is not comparable to the issues you faced with your own benchmark, which misrepresented data from hot runs as cold runs, removing the overhead of reading from the parquet file from your timings.

And if it is not clear enough, re-quoting the rules you mentioned:

"If the system contains a cache for query results, it should be disabled."

It doesn't.

"If the system contains a cache for intermediate data, that cache should be disabled if it is located near the end of the query execution pipeline, thus similar to a query result cache."

It doesn't. The query runs against the raw data each time.

I dont need to be ahead, Im looking fairness, you accused me of cheating and mine was a honest mistake, in the other hand, a little bit of investigation show that you cache by default and cheat at different levels on the benchmark.

Actually, neither @puzpuzpuz or myself accused you of cheating at all. He simply raised strange inconsistencies in your results when compared to DuckDB (which you use as your query engine), and also that in-memory databases were being included in cold run results (which is an apples-to-oranges comparison).

If you are upset at your results being called 'cheated', then please take that up with @alexey-milovidov - and don't take it out on us!

Evidences are there.

The evidence is not there, please can you clarify which of the 4 points you raised are accurate (if any), and which are not?

Competition elevates us. I will not comment anymore, just focus on make Arc better everyday.

Competition is great, and it should be done on a level playing field, else the measurement is somewhat useless. In that spirit, once the corrected results are live, will you revise your marketing posts?

When I run it I had this

/benchmark.sh: line 25: pigz: command not found
{"ddl":"OK"}curl: (26) Failed to open/read local data from file/application
waiting for rows to become readable...

Simple as that, have you checked that the installation works on your script?

Yes of course, we did a PR with the new values, we dont have fear of show in what we missed, and we fixed.

#661

And in our blog and landing, the same, we updated the article with the results.

https://basekick.net/blog/arc-fastest-timeseries-database-clickbench

Please, disable the caching and run the benchmark, if you do this right, you wll find that the values are a bit different. For example: in the query 31 was 246% slower without cache!

[nwoolmer]

The results are there expecting merging, and we don't have any issues correcting that we had a mistake... honesty should be first.

#661

About the blog, yes, I just updated this

@puzpuzpuz has added some specifics, I will also clarify a few points.

According to Clickbench the benchmark.sh shouldn't require pre requisites. The benchmark.sh should run without any issues to reproduce results.

The benchmark script includes the pigz install, I quoted it to you. Did you check that bit?

Alexey, let me know if it gets stuck for you, we'll check it if needed.

As cache enabled, I run the test after fix the issues and the results are different. No LLM just estimation on the results that I got. In the code you have by default enable cache and in the benchmark.sh nothing is disabling those settings.

Stop trusting AI slop. This is not comparable to the issues you faced with your own benchmark, which misrepresented data from hot runs as cold runs, removing the overhead of reading from the parquet file from your timings.

And if it is not clear enough, re-quoting the rules you mentioned:

"If the system contains a cache for query results, it should be disabled."

It doesn't.

"If the system contains a cache for intermediate data, that cache should be disabled if it is located near the end of the query execution pipeline, thus similar to a query result cache."

It doesn't. The query runs against the raw data each time.

I dont need to be ahead, Im looking fairness, you accused me of cheating and mine was a honest mistake, in the other hand, a little bit of investigation show that you cache by default and cheat at different levels on the benchmark.

Actually, neither @puzpuzpuz or myself accused you of cheating at all. He simply raised strange inconsistencies in your results when compared to DuckDB (which you use as your query engine), and also that in-memory databases were being included in cold run results (which is an apples-to-oranges comparison).

If you are upset at your results being called 'cheated', then please take that up with @alexey-milovidov - and don't take it out on us!

Evidences are there.

The evidence is not there, please can you clarify which of the 4 points you raised are accurate (if any), and which are not?

Competition elevates us. I will not comment anymore, just focus on make Arc better everyday.

Competition is great, and it should be done on a level playing field, else the measurement is somewhat useless. In that spirit, once the corrected results are live, will you revise your marketing posts?

When I run it I had this

/benchmark.sh: line 25: pigz: command not found
{"ddl":"OK"}curl: (26) Failed to open/read local data from file/application
waiting for rows to become readable...

Simple as that, have you checked that the installation works on your script?

Yes of course, we did a PR with the new values, we dont have fear of show in what we missed, and we fixed.

#661

And in our blog and landing, the same, we updated the article with the results.

https://basekick.net/blog/arc-fastest-timeseries-database-clickbench

Please, disable the caching and run the benchmark, if you do this right, you wll find that the values are a bit different. For example: in the query 31 was 246% slower without cache!

Indeed, you can change database configs and affect the results. For example, you could run the database in single-threaded mode, that would slow it down too. But using multiple threads isn't against the rules either.

Are you satisfied by the answers to your questions? If so, let's close this and call it a day!

[xe-nvdk]

Lets keep to the clickbench team decided that. For me, is ok, do whatever you want, but is not about single thread or multi thread is about disabling cache and show results that really are.

Have a great day

[rschu1ze]

Hi 👋,

I ran QuestDB's benchmark.sh script locally, it didn't complain about pigz not being available.

I could imagine that the QuestDB configuration docs were a source of confusion:

• http.query.cache.enabled is documented as "Enable or disable the query cache." whereas @puzpuzpuz mentioned above that prepared query plans are cached. The latter is fine according to the benchmark rules, and indeed this is a standard technique in databases (mostly OLTP systems though).

• pg.select.cache.enabled is likewise documented confusingly as "Enable or disable the SELECT query cache" (I'd get suspicious here as well). According to @puzpuzpuz this is the same thing as http.query.cache.enabled but for the Postgres wire protocol if I understood correctly, so it's fine.

• cairo.default.symbol.cache.flag is documented as "When true, symbol values will be cached on Java heap instead of being looked up in the database files.". Tbh, neither normal users nor database people will understand what that means. @puzpuzpuz explanation didn't help (me) either ... but this could because I'm not a QuestDB person. Metadata caching is alright.

I didn't compare QuestDB against itself with the three caches disabled because there is at the moment no credible evidence that it violates the rules.

[nwoolmer]

Thanks for reviewing this, @rschu1ze!

If it helps, SYMBOL is essentially an ENUM with a 32-bit key. We store the mapping on disk as column metadata.

If there are only a few keys, we cache them in a lookup table.

If there are many, it can eat a lot of RAM, so we read them on-demand from disk instead.