Fix O(n²) performance in ReindentFilter for large queries

[jonmmease]

After reviewing #16, I set claude loose on the repo and asked it to stress test other forms of large queries to look for similar cases. And it found 3. The issues and fixes are described below. I also had it added pytest-benchmark benchmarks for a bunch of cases and report the speedups.

---

Summary

Fix O(n²) performance bottlenecks in ReindentFilter that cause sqlparse.format(..., reindent=True) to hang on large SQL queries. After these fixes, the same queries format in seconds.

This is a companion to #16 which fixes parse-time O(n²) in group_tokens. This branch fixes the remaining format-time O(n²) in the reindent filter.

Changes

Fix 1: Replace _get_offset with backward tree walk

• Old: _flatten_up_to_token concatenated all tokens from the start of the statement, then took splitlines()[-1] to get the current line — O(statement_length) per call
• New: Walk backward from the target token through the tree until a newline is found — O(line_length) per call
• Added _reverse_leaves_before and _reverse_flatten for the backward walk
• Removed the now-unused _flatten_up_to_token method

Fix 2: Eliminate token_index calls in _process_identifierlist

• Old: insert_before(token, ...) called token_index(token) → list.index(token) = O(n) linear scan per insertion
• New: Pre-compute {id(token): index} mapping once, pass integer indices directly to insert_before/insert_after, and track cumulative shift from insertions

Fix 3: Eliminate O(n²) in _process_values for large INSERT statements

• Old: Same two O(n²) patterns as above in the VALUES processing loop
• New: Pass integer indices from token_next_by directly. Added _parent_idx hint parameter so the backward walk skips the O(n) index() call on the hot parent. Hoisted loop-invariant offset in comma_first mode.

Benchmark Results

Reproducible benchmark tests are included in the first commit (tests/test_benchmarks.py). Run with:

uv run --with pytest --with pytest-benchmark pytest tests/test_benchmarks.py -v

Reindent benchmarks

Benchmark	Example SQL	Before	After	Speedup
wide_select	SELECT col_0, col_1, ... col_4999 FROM t	0.66s	0.16s	4.1x
large_in_list	SELECT * FROM t WHERE id IN (0, 1, ... 99999)	DNF (>120s)	6.39s	FIXED
large_insert	INSERT INTO t VALUES (0, 1), (1, 2), ... (24999, 25000)	DNF (>120s)	2.56s	FIXED
deep_subqueries	SELECT * FROM (SELECT * FROM (... FROM t ...) s0) s11	0.00s	0.00s	—
many_joins	SELECT * FROM t0 JOIN t1 ON ... JOIN t2 ON ... (×500)	0.07s	0.07s	—
complex_where	SELECT * FROM t WHERE ((col_0 = 0 AND ...) OR ...) (depth=8, breadth=3)	16.56s	0.60s	27.6x
mixed_batch	CREATE TABLE ...; INSERT INTO ...; SELECT ...; UPDATE ... (×50)	0.20s	0.13s	1.5x
heavy_formatting	WITH cte AS (SELECT CASE WHEN col_0 > 0 ... (×200)) SELECT * FROM cte	0.20s	0.05s	4.0x

INSERT scaling (Fix 3)

Benchmark	Example SQL	Before	After	Speedup
insert 5k rows	INSERT INTO t VALUES (0, 1), ... (4999, 5000)	13.26s	0.37s	35.8x
insert 10k rows	INSERT INTO t VALUES (0, 1), ... (9999, 10000)	DNF (>120s)	0.86s	FIXED
insert 25k rows	INSERT INTO t VALUES (0, 1), ... (24999, 25000)	DNF (>120s)	2.47s	FIXED

Testing

• All 467 existing tests pass
• 11 pytest-benchmark tests included for reproducible performance measurement

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}