chore: Add memory reservation debug logging#3489
Merged
andygrove merged 4 commits intoapache:mainfrom Feb 12, 2026
Merged
Conversation
Add a new `spark.comet.debug.memory` config that wraps the native memory pool in a LoggingPool decorator, logging all grow/shrink/try_grow calls with task ID and consumer name. This helps diagnose memory reservation issues in production environments. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
comphead
reviewed
Feb 12, 2026
| use std::sync::Arc; | ||
|
|
||
| #[derive(Debug)] | ||
| pub(crate) struct LoggingPool { |
Contributor
There was a problem hiding this comment.
Suggested change
| pub(crate) struct LoggingPool { | |
| pub(crate) struct LoggingMemoryPool { |
comphead
approved these changes
Feb 12, 2026
Contributor
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
looks good to me, thanks @andygrove
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #3445
Rationale for this change
These changes were originally in #2521, which also included Python scripts for analyzing the data. This PR just adds the logging changes. I am hoping this makes review easier. We can discuss adding analysis scripts in a separate issue or PR in the future.
When debugging memory reservation issues in Comet, it is helpful to see all memory pool interactions (grow, shrink, try_grow) with task IDs and consumer names. This PR adds a lightweight opt-in logging decorator for the native memory pool.
What changes are included in this PR?
spark.comet.debug.memory(default:false)LoggingPoolthat wraps the existing memory pool when the config is enabledgrow,shrink, andtry_growcalls with task attempt ID, consumer name, and result (Ok/Err)Example output:
How are these changes tested?
Manual testing with TPC-DS/TPC-H benchmarks. The logging pool is a thin decorator that delegates all operations to the underlying pool, so it does not change any behavior.
🤖 Generated with Claude Code