feat(sync-service): Scale SQLite connection pool to 0

[magnetised]

NimblePool allows for lazy initialization and scale down of SQLite connections. This PR adds support for that.

The scaling is disabled for exclusive_mode as an exclusive_mode db could be in-memory, and closing that would be catastrophic.

I've added the number of active connections to the exported data. Will be interesting to see what goes on there, in both the larger instances and the many less-active ones.

This disables sqlite statistics collection by default as the usefulness of the exported numbers is low without memory statistics enabled, and enabling memory stats is potentially causing issues in some deployments.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
2491	1	2490	1

View the top 1 failed test(s) by shortest run time

Elixir.Electric.ShapeCache.ShapeStatus.ShapeDbTest::test error handling crashing WriteBuffer restarts entire supervision tree

Stack Traces | 0.411s run time

23) test error handling crashing WriteBuffer restarts entire supervision tree (Electric.ShapeCache.ShapeStatus.ShapeDbTest)
     .../shape_cache/shape_status/shape_db_test.exs:651
     Assertion failed, no matching message after 400ms
     The following variables were pinned:
       buffer_ref = #Reference<0.3274322101.3512467457.120881>
       write_buffer_pid = #PID<0.12049.0>
     The process mailbox is empty.
     code: assert_receive {:DOWN, ^buffer_ref, :process, ^write_buffer_pid, :some_reason}
     stacktrace:
       .../shape_cache/shape_status/shape_db_test.exs:658: (test)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[claude]

Claude Code Review

Summary

Round 15: One new commit since round 14 — "Cleanup after rebase" (b281e9b). It removes the open_connection/2 helper functions (exclusive vs. non-exclusive stats connection dispatch), simplifies Sqlite3.release error handling in close/1, and carries memstat_available? through the task result tuple. The "SQLite statistics disabled" notice logged on every default startup is now gone — that is a clean resolution. One genuinely new concern has been introduced: the dbstat availability check now queries sqlite_master, which may not find SQLite's built-in eponymous dbstat virtual table.

What's Working Well

• "SQLite statistics disabled" notice removed — init/1 no longer logs about the disabled state on startup. Previous open issue fully resolved.
• first_run? moved out of state — Threading it as a parameter to read_stats/do_read_stats/initialize_connection is cleaner than storing it in state and toggling it via a then/1 post-processing step.
• open_connection/2 removal — The non-exclusive path was opening a raw connection outside the pool to avoid waking it. Removing this keeps the code simpler and the connection counter accurate during stats collection.
• memstat_available? correctly initialised to true — Matches the optimistic-until-proven-otherwise pattern used for dbstat_available?.

Issues Found

Critical (Must Fix)

None.

Important (Should Fix)

1. dbstat availability check via sqlite_master likely always returns false

File: packages/sync-service/lib/electric/shape_cache/shape_status/shape_db/statistics.ex:250-263

Issue: initialize_connection/4 now checks whether dbstat is available with:

SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'dbstat'

dbstat is an eponymous-only virtual table — compiled into SQLite via SQLITE_ENABLE_DBSTAT_VTAB and directly queryable (SELECT * FROM dbstat), but not registered in sqlite_master because it is never created with CREATE VIRTUAL TABLE. This query returns 0 rows regardless of whether dbstat is available, causing fetch_one to return :error, which sets dbstat_available?: false after the first stats run. Disk stats would then be silently disabled for the lifetime of the process.

Why the test does not catch this: "only exports disk usage by default" asserts if enabled.disk, do: assert(stats.disk_size > 0). When enabled.disk is false (as it would be here) the assertion is silently skipped and the test still passes.

Suggested fix: probe dbstat by querying it directly rather than via sqlite_master:

dbstat_available? =
  if dbstat_available? do
    case Connection.fetch_one(conn, "SELECT 1 FROM dbstat LIMIT 1", []) do
      {:ok, _} -> true
      :error   -> true   # 0 rows but table exists -- empty db
      _error   ->
        if first_run?,
          do: Logger.warning("SQLite disk size statistics will not be available.")
        false
    end
  else
    false
  end

(:error / 0 rows still means the table exists — an empty database produces no dbstat rows. Only an actual SQL error, e.g. "no such table: dbstat", indicates it is unavailable.)

Suggestions (Nice to Have)

2. Three dead state fields left over from open_connection/2 removal

File: packages/sync-service/lib/electric/shape_cache/shape_status/shape_db/statistics.ex:88-98

pool_opts, exclusive_mode?, and page_size: 0 were all used by the removed open_connection/2 functions. None are referenced anywhere in the current code. Safe to remove the three corresponding lines from init/1.

3. :ok = Sqlite3.release(conn, stmt) removes error observability

File: packages/sync-service/lib/electric/shape_cache/shape_status/shape_db/connection.ex:325-328

The cleanup commit replaced the previous case (which logged {:error, reason} with the statement name) with a bare match assertion. Sqlite3.release/2 is documented (and now spec'd) to potentially return {:error, term()}. If that ever occurs, the MatchError would crash close/1 before reaching Sqlite3.close(conn), leaving the connection open. Worth reverting this specific hunk back to the explicit case.

Issue Conformance

No linked issue — unchanged from all previous rounds.

Previous Review Status

• RESOLVED: Logger.notice("SQLite statistics disabled") on every default startup — Gone. init/1 no longer logs about the disabled state.
• OPEN: Blacksmith test failures — Latest CI run shows 2 failures in ShapeDbTest (down from 25 shell failures in an earlier run). Worth confirming they are pre-existing flakes before merging.
• DECLINED: Task.async crash propagation — Carried forward.
• DECLINED: Missing @impl GenServer on handle_info/handle_cast — Carried forward.

---

Review iteration: 15 | 2026-03-04

[alco]

@magnetised I've looked through the issues Claude flagged up, all for them look legit. Please ping me when ready for another review round.

[magnetised]

@alco think we can ignore claude's suggestions. ready for another look pls

[alco]

Looking good!

[blacksmith-sh]

Found 4 test failures on Blacksmith runners:

Failures

Test	View Logs
Elixir.Electric.ShapeCache.ShapeStatus.ShapeDbTest/ test error handling crashing WriteBuffer restarts entire supervision tree	View Logs
Elixir.Electric.ShapeCache.ShapeStatus.ShapeDbTest/ test error handling crashing WriteBuffer restarts entire supervision tree	View Logs
Elixir.Electric.ShapeCache.ShapeStatus.ShapeDbTest/ test error handling crashing WriteBuffer restarts entire supervision tree	View Logs
Elixir.Electric.ShapeCache.ShapeStatus.ShapeDbTest/ test error handling crashing WriteBuffer restarts entire supervision tree	View Logs