preserve optimistic mutations during truncate operations

[samwillis]

issue found by @balegas

Problem

When a collection receives a truncate operation (typically triggered by a "must-refetch" signal from the server), optimistic mutations could be permanently lost if their async handlers (onInsert, onUpdate, onDelete) completed during the truncate processing.

Root Causes Identified

Bug #1: Optimistic Data Loss During Truncate

• When an optimistic transaction's async handler completed during truncate processing, its data was lost
• The commitPendingTransactions logic was rebuilding optimistic state from "active" transactions only
• If a transaction changed from pending → completed during truncate processing, it was skipped during re-application
• Impact: Users' optimistic changes disappeared from the UI

Bug #2: Missing Delete Events for Optimistic-Only Data

• Truncate delete phase only iterated over syncedData.keys() to emit delete events
• Items that existed only in optimisticUpserts (not yet synced) didn't get delete events
• Impact: UI components didn't receive delete events for optimistic-only items during truncate

Solution

Snapshot-Based Approach

The fix implements a snapshot mechanism that captures optimistic state at the exact moment truncate() is called:

1. Capture snapshot when truncate is called (packages/db/src/collection/sync.ts)

   truncate: () => {
     // ... existing truncate logic ...
     
     // Capture optimistic state NOW
     pendingTransaction.optimisticSnapshot = {
       upserts: new Map(this.state.optimisticUpserts),
       deletes: new Set(this.state.optimisticDeletes),
     }
   }

2. Use snapshot for delete events (packages/db/src/collection/state.ts)

   // Emit deletes for ALL visible items (synced + optimistic)
   const visibleKeys = new Set([
     ...this.syncedData.keys(),
     ...(truncateOptimisticSnapshot?.upserts.keys() || []),
   ])

3. Restore snapshot, then overlay active transactions

   if (hasTruncateSync && truncateOptimisticSnapshot) {
     // Restore optimistic state from snapshot
     for (const [key, value] of truncateOptimisticSnapshot.upserts) {
       this.optimisticUpserts.set(key, value)
     }
     for (const key of truncateOptimisticSnapshot.deletes) {
       this.optimisticDeletes.add(key)
     }
   }
   
   // Always overlay currently active transactions
   // This captures late-arriving mutations that started after truncate()
   for (const transaction of this.transactions.values()) {
     if (![`completed`, `failed`].includes(transaction.state)) {
       // Re-apply active mutations
     }
   }

Why This Works

Timing Independence:

• Snapshot is taken at truncate() call time, before any transactions can complete
• Mutations that complete during processing are preserved in the snapshot
• Late-arriving mutations (started after truncate) are captured by the active overlay

Client Intent Preservation:

• Always overlaying active transactions ensures the newest client intent wins
• If same key is modified after snapshot, the latest mutation takes precedence
• Optimistic deletes prevent server re-insertions from being visible

Tests Added

Added 11 comprehensive tests in packages/db/tests/collection-truncate.test.ts:

Core Scenarios

1. ✅ Preserve optimistic inserts when truncate completes before async handler
2. ✅ Preserve optimistic inserts when handler completes during truncate processing (Bug ci: Introduce changesets #1)
3. ✅ Handle truncate on empty collection
4. ✅ Emit delete events for optimistic-only data during truncate (Bug Prep for initial release #2)
5. ✅ Preserve optimistic inserts started after truncate begins (late arrivals)

Edge Cases

6. ✅ Preserve optimistic delete when transaction active during truncate
   • Server re-inserts item but client delete is preserved
7. ✅ Preserve optimistic value over server value during truncate
   • Client intent wins when optimistic differs from server
8. ✅ Handle multiple consecutive truncate operations
   • Sequential truncates preserve all optimistic state
9. ✅ Handle new mutation on same key after truncate snapshot
   • Newest mutation wins when same key modified multiple times
10. ✅ Handle transaction completing between truncate and commit
    • Snapshot approach correctly handles this timing
11. ✅ Preserve all optimistic inserts when truncate occurs during async handler
    • Multiple items with delayed handlers all preserved

Test Results

• ✅ All 11 truncate tests pass
• ✅ All 221 collection tests pass (12 test files)
• ✅ No regressions introduced

Files Changed

Core Logic

• packages/db/src/collection/state.ts

  • Added optimisticSnapshot field to PendingSyncedTransaction interface
  • Modified truncate delete phase to include optimistic keys from snapshot
  • Changed re-application logic to restore snapshot then overlay active transactions

• packages/db/src/collection/sync.ts

  • Added snapshot capture when truncate() is called

Tests

• packages/db/tests/collection-truncate.test.ts (new file)

  • 11 comprehensive tests covering all scenarios

• packages/db/tests/collection-subscribe-changes.test.ts

  • Updated test expectation to match correct behavior (5 events including delete for optimistic-only item)

Performance Impact

Minimal overhead:

• Snapshot only created during truncate operations (rare)
• No overhead for normal insert/update/delete operations
• Snapshot memory is cleaned up after truncate commit
• Just 2 Maps/Sets during truncate processing

Breaking Changes

None. This is a bug fix that corrects existing behavior.

Migration Guide

No migration needed. The fix is transparent to existing code.

[changeset-bot]

🦋 Changeset detected

Latest commit: 01977a7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@tanstack/db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• todos
• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@659

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@659

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@659

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@659

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@659

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@659

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@659

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@659

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@659

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@659

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@659

commit: 01977a7

[github-actions]

Size Change: +59 B (+0.08%)

Total Size: 76.5 kB

Filename	Size	Change
./packages/db/dist/esm/collection/state.js	3.83 kB	+7 B (+0.18%)
./packages/db/dist/esm/collection/sync.js	1.7 kB	+52 B (+3.15%)

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/collection/change-events.js	963 B
./packages/db/dist/esm/collection/changes.js	1.01 kB
./packages/db/dist/esm/collection/events.js	660 B
./packages/db/dist/esm/collection/index.js	3.31 kB
./packages/db/dist/esm/collection/indexes.js	1.16 kB
./packages/db/dist/esm/collection/lifecycle.js	1.82 kB
./packages/db/dist/esm/collection/mutations.js	2.52 kB
./packages/db/dist/esm/collection/subscription.js	1.83 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3.1 kB
./packages/db/dist/esm/index.js	1.58 kB
./packages/db/dist/esm/indexes/auto-index.js	828 B
./packages/db/dist/esm/indexes/base-index.js	835 B
./packages/db/dist/esm/indexes/btree-index.js	2 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.21 kB
./packages/db/dist/esm/indexes/reverse-index.js	577 B
./packages/db/dist/esm/local-only.js	967 B
./packages/db/dist/esm/local-storage.js	2.33 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	3.86 kB
./packages/db/dist/esm/query/builder/functions.js	615 B
./packages/db/dist/esm/query/builder/index.js	4.04 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	938 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.55 kB
./packages/db/dist/esm/query/compiler/expressions.js	760 B
./packages/db/dist/esm/query/compiler/group-by.js	2.04 kB
./packages/db/dist/esm/query/compiler/index.js	2.04 kB
./packages/db/dist/esm/query/compiler/joins.js	2.52 kB
./packages/db/dist/esm/query/compiler/order-by.js	1.21 kB
./packages/db/dist/esm/query/compiler/select.js	1.28 kB
./packages/db/dist/esm/query/ir.js	785 B
./packages/db/dist/esm/query/live-query-collection.js	340 B
./packages/db/dist/esm/query/live/collection-config-builder.js	2.69 kB
./packages/db/dist/esm/query/live/collection-subscriber.js	1.92 kB
./packages/db/dist/esm/query/optimizer.js	3.08 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	3 kB
./packages/db/dist/esm/utils.js	1.01 kB
./packages/db/dist/esm/utils/browser-polyfills.js	365 B
./packages/db/dist/esm/utils/btree.js	6.01 kB
./packages/db/dist/esm/utils/comparison.js	754 B
./packages/db/dist/esm/utils/index-optimization.js	1.73 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 1.47 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	152 B
./packages/react-db/dist/esm/useLiveQuery.js	1.32 kB

_{compressed-size-action::react-db-package-size}

[KyleAMathews]