feat: Add orchestrator queue health metrics#77
Draft
morgan-wowk wants to merge 1 commit intometrics-pipeline-outcomesfrom
Draft
feat: Add orchestrator queue health metrics#77morgan-wowk wants to merge 1 commit intometrics-pipeline-outcomesfrom
morgan-wowk wants to merge 1 commit intometrics-pipeline-outcomesfrom
Conversation
Collaborator
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Feb 2, 2026
morgan-wowk
force-pushed
the
metrics-pipeline-outcomes
branch
from
a26f61f to
91a8a84
Compare
morgan-wowk
force-pushed
the
metrics-queue-health
branch
2 times, most recently
from
79d46ca to
57037ef
Compare
morgan-wowk
force-pushed
the
metrics-pipeline-outcomes
branch
from
91a8a84 to
4153fe9
Compare
Tracks orchestrator queue processing activity and errors without costly COUNT(*) queries. Metrics added: - orchestrator_queue_processing_errors_total: Counter of queue processing errors by queue_type - orchestrator_executions_processed_total: Counter tracking queue sweeps by queue_type and whether work was found These metrics enable monitoring of orchestrator health, processing rates, and error rates without database overhead. Processing rates can be derived from the found_work labels to understand queue activity patterns.
morgan-wowk
changed the base branch from
metrics-pipeline-outcomes
to
graphite-base/77
morgan-wowk
force-pushed
the
graphite-base/77
branch
from
4153fe9 to
2b22982
Compare
morgan-wowk
force-pushed
the
metrics-queue-health
branch
from
57037ef to
0e2bde5
Compare
morgan-wowk
changed the base branch from
graphite-base/77
to
metrics-pipeline-outcomes
Ark-kun
reviewed
Feb 6, 2026
| if _orchestrator_processing_errors_counter is None: | ||
| _orchestrator_processing_errors_counter = meter.create_counter( | ||
| name="orchestrator_queue_processing_errors_total", | ||
| description="Total number of orchestrator queue processing errors", |
Contributor
There was a problem hiding this comment.
This resets after every restart/deploy, correct?
Contributor
There was a problem hiding this comment.
What if I wanted to see a graph of number of errors in last few months to look at the trends?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracks orchestrator queue processing activity and errors without costly COUNT(*) queries.
Metrics added:
These metrics enable monitoring of orchestrator health, processing rates, and error rates without database overhead.
Processing rates can be derived from the found_work labels to understand queue activity patterns.