📊 Missing Metrics Infrastructure for Agent Performance Analysis

[github-actions]

Problem

The Agent Performance Analyzer meta-orchestrator requires centralized metrics data to perform quantitative analysis of agent performance. The expected metrics infrastructure is missing from shared memory.

Expected Location

/tmp/gh-aw/repo-memory/default/metrics/
├── latest.json          # Most recent daily metrics snapshot
└── daily/               # Historical daily metrics for trend analysis
    ├── 2026-01-02.json
    ├── 2026-01-01.json
    └── ...

Required Metrics

Per-Workflow Metrics

• Safe output counts: Issues, PRs, comments, discussions created
• Workflow run statistics: Total runs, success rate, failure rate
• Execution metrics: Average duration, resource usage
• Engagement metrics: Reactions, comments, replies on outputs
• Quality indicators: PR merge rate, issue close time, resolution rate

Ecosystem Metrics

• Output volume: Total safe outputs per day/week/month
• Engine distribution: Usage of copilot vs claude vs codex
• Agent categories: Breakdown by workflow type
• Trend data: Week-over-week and month-over-month changes

Impact

Without Metrics Data:

• ❌ Cannot calculate agent quality scores (0-100)
• ❌ Cannot rank agents or identify top performers
• ❌ Cannot detect underperforming agents
• ❌ Cannot track trends over time
• ❌ Cannot measure effectiveness rates
• ❌ Cannot assess resource efficiency
• ❌ Cannot analyze collaboration patterns
• ❌ Cannot provide data-driven recommendations

Current State:

• ✅ Qualitative analysis possible (PR reviews, documentation quality)
• ✅ Baseline ecosystem structure documented
• ❌ Quantitative performance analysis blocked

Solution

Verify Metrics Collector Workflow

1. Check if Metrics Collector workflow exists:

   ls -la .github/workflows/ | grep -i "metric"

2. If exists, trigger a run:

   • Manually dispatch workflow
   • Verify it writes to /tmp/gh-aw/repo-memory/default/metrics/
   • Check output format matches expected schema

3. If missing, create Metrics Collector workflow:

   • Daily scheduled run (e.g., 00:00 UTC)
   • Queries GitHub API for safe output data
   • Aggregates per-workflow statistics
   • Writes JSON to shared memory
   • Uses repo memory persistence

Expected JSON Schema

{
  "collected_at": "2026-01-02T00:00:00Z",
  "period": {
    "start": "2026-01-01T00:00:00Z",
    "end": "2026-01-02T00:00:00Z"
  },
  "workflows": {
    "workflow-name": {
      "safe_outputs": {
        "issues_created": 5,
        "prs_created": 2,
        "comments_added": 10,
        "discussions_created": 1
      },
      "runs": {
        "total": 10,
        "successful": 8,
        "failed": 2,
        "success_rate": 0.80
      },
      "engagement": {
        "total_reactions": 15,
        "total_comments": 20,
        "avg_reactions_per_output": 3.0
      },
      "quality_indicators": {
        "pr_merge_rate": 0.75,
        "avg_close_time_hours": 48
      }
    }
  },
  "ecosystem": {
    "total_workflows": 176,
    "active_workflows": 165,
    "total_outputs": 50,
    "engines": {
      "copilot": 40,
      "claude": 5,
      "codex": 5
    }
  }
}

Integration with Meta-Orchestrators

All three meta-orchestrators depend on this data:

• Agent Performance Analyzer: Quality scores, rankings, effectiveness
• Campaign Manager: Campaign progress tracking, goal achievement
• Workflow Health Manager: Success rates, failure patterns, health scores

Acceptance Criteria

• Metrics Collector workflow exists and runs successfully
• Writes to /tmp/gh-aw/repo-memory/default/metrics/latest.json
• Writes daily snapshots to metrics/daily/YYYY-MM-DD.json
• JSON schema matches expected format
• Includes all required metrics sections
• Agent Performance Analyzer can load and parse data
• Data persists across workflow runs via git branch
• Historical data retained for at least 30 days

Priority

High - Blocks quantitative analysis for all meta-orchestrators.

Related

• #aw_pr_completion_barrier - PR completion barrier issue
• Discussion: Agent Performance Report - Baseline
• Workflow: Agent Performance Analyzer

---

Identified by Agent Performance Analyzer

AI generated by Agent Performance Analyzer - Meta-Orchestrator

[pelikhan]

/q review prompt and expected memory location compared with the repo-memory.md spec

[github-actions]

🎩 Mission equipment ready! Q has optimized your workflow. Use wisely, 007! 🔫

[github-actions]

Investigation Complete - Configuration Fix Applied

Root Cause Identified

The issue was not missing metrics infrastructure but rather misconfigured repo-memory settings in the meta-orchestrator workflows.

Problem

The workflows were configured with:

repo-memory:
  branch-name: memory/meta-orchestrators
  file-glob: "memory/meta-orchestrators/metrics/**"

But without an explicit id field, the system defaults to id: default, which creates:

• Memory directory: /tmp/gh-aw/repo-memory/default/
• Expected branch: memory/default (not memory/meta-orchestrators)

Issues Found

1. Branch name mismatch: Branch was memory/meta-orchestrators but should be memory/default for default ID
2. File glob anti-pattern: Pattern included branch name prefix (memory/meta-orchestrators/) which violates the spec - globs should be relative to artifact root
3. Path inconsistency: Agents wrote to /tmp/gh-aw/repo-memory/default/ but data would be pushed to wrong branch

Fix Applied

Created PR with corrections to align with specs/repo-memory.md:

Modified workflows:

• metrics-collector.md
• agent-performance-analyzer.md
• campaign-manager.md
• workflow-health-manager.md

Changes:

• ✅ Added explicit id: default
• ✅ Changed branch-name to memory/default
• ✅ Fixed file-glob patterns (removed branch name prefix)

Expected Outcome

After merge:

• ✅ Metrics will be correctly written to memory/default branch
• ✅ Agent Performance Analyzer will correctly read metrics
• ✅ Campaign Manager will correctly access campaign state
• ✅ Workflow Health Manager will correctly access health data

Verification

The spec at line 41 explicitly states the prompt shows:

You have access to a persistent repo memory folder at `/tmp/gh-aw/repo-memory/default/`

And line 93 shows the directory construction:

memoryDir := fmt.Sprintf("/tmp/gh-aw/repo-memory/%s/", memory.ID)

With id: default, this creates /tmp/gh-aw/repo-memory/default/, which must use branch memory/default per the spec.

---

The metrics infrastructure exists and works correctly - it just needed the configuration to match the spec's path conventions.

AI generated by Q

[github-actions]

🍪 Issue Monster has assigned this to Copilot!

I've identified this issue as a good candidate for automated resolution and assigned it to the Copilot agent.

The Copilot agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

AI generated by Issue Monster