Skip to content

Fix repo-memory file-glob patterns to match spec (exclude branch names)#8578

Merged
pelikhan merged 3 commits intomainfrom
copilot/add-metrics-infrastructure
Jan 2, 2026
Merged

Fix repo-memory file-glob patterns to match spec (exclude branch names)#8578
pelikhan merged 3 commits intomainfrom
copilot/add-metrics-infrastructure

Conversation

Copy link
Contributor

Copilot AI commented Jan 2, 2026

Fix repo-memory file-glob patterns and campaign lock file purge

This PR addresses two issues:

1. Repo-Memory File-Glob Path Consistency

Meta-orchestrator workflows were using file-glob patterns that incorrectly included the git branch name (memory/meta-orchestrators/**), violating the repo-memory spec which requires patterns to match relative paths from the artifact directory.

Changes:

  • metrics-collector.md: memory/meta-orchestrators/metrics/**metrics/**
  • agent-performance-analyzer.md: memory/meta-orchestrators/**/***
  • campaign-manager.md: memory/meta-orchestrators/**/***
  • workflow-health-manager.md: memory/meta-orchestrators/**/***
  • docs/workflows/metrics-collector.md: Fixed all path references

2. Campaign Lock File Purge Bug

The compile --purge command was incorrectly deleting campaign orchestrator lock files (.campaign.g.lock.yml) during recompilation.

Root Cause: The purgeOrphanedLockFiles function used a glob pattern *.lock.yml that matched all lock files including campaign orchestrator lock files. Since campaign lock files weren't in the expected lock files list (which only contained regular workflow lock files), they were incorrectly identified as orphaned and deleted.

Fix: Skip .campaign.g.lock.yml files in purgeOrphanedLockFiles - these are handled by the dedicated purgeOrphanedCampaignOrchestratorLockFiles function.

Testing:

  • Added TestCompileWorkflows_PurgeCampaignLockFiles to verify campaign lock files are preserved
  • Updated TestCompileWorkflows_PurgeCampaignOrchestrators to reflect that .campaign.g.md files are kept as debug artifacts
  • All purge and campaign tests pass

Context: Per repo-memory spec, when using object config:

  • Memory ID is implicitly default
  • Runtime path is /tmp/gh-aw/repo-memory/default/
  • Branch name (memory/meta-orchestrators) only affects git storage
  • File-glob patterns match artifact-relative paths, not branch names

Files written to /tmp/gh-aw/repo-memory/default/ are automatically pushed to the configured git branch.

Original prompt

This section details on the original issue you should resolve

<issue_title>📊 Missing Metrics Infrastructure for Agent Performance Analysis</issue_title>
<issue_description>## Problem

The Agent Performance Analyzer meta-orchestrator requires centralized metrics data to perform quantitative analysis of agent performance. The expected metrics infrastructure is missing from shared memory.

Expected Location

/tmp/gh-aw/repo-memory/default/metrics/
├── latest.json          # Most recent daily metrics snapshot
└── daily/               # Historical daily metrics for trend analysis
    ├── 2026-01-02.json
    ├── 2026-01-01.json
    └── ...

Required Metrics

Per-Workflow Metrics

  • Safe output counts: Issues, PRs, comments, discussions created
  • Workflow run statistics: Total runs, success rate, failure rate
  • Execution metrics: Average duration, resource usage
  • Engagement metrics: Reactions, comments, replies on outputs
  • Quality indicators: PR merge rate, issue close time, resolution rate

Ecosystem Metrics

  • Output volume: Total safe outputs per day/week/month
  • Engine distribution: Usage of copilot vs claude vs codex
  • Agent categories: Breakdown by workflow type
  • Trend data: Week-over-week and month-over-month changes

Impact

Without Metrics Data:

  • ❌ Cannot calculate agent quality scores (0-100)
  • ❌ Cannot rank agents or identify top performers
  • ❌ Cannot detect underperforming agents
  • ❌ Cannot track trends over time
  • ❌ Cannot measure effectiveness rates
  • ❌ Cannot assess resource efficiency
  • ❌ Cannot analyze collaboration patterns
  • ❌ Cannot provide data-driven recommendations

Current State:

  • ✅ Qualitative analysis possible (PR reviews, documentation quality)
  • ✅ Baseline ecosystem structure documented
  • ❌ Quantitative performance analysis blocked

Solution

Verify Metrics Collector Workflow

  1. Check if Metrics Collector workflow exists:

    ls -la .github/workflows/ | grep -i "metric"
  2. If exists, trigger a run:

    • Manually dispatch workflow
    • Verify it writes to /tmp/gh-aw/repo-memory/default/metrics/
    • Check output format matches expected schema
  3. If missing, create Metrics Collector workflow:

    • Daily scheduled run (e.g., 00:00 UTC)
    • Queries GitHub API for safe output data
    • Aggregates per-workflow statistics
    • Writes JSON to shared memory
    • Uses repo memory persistence

Expected JSON Schema

{
  "collected_at": "2026-01-02T00:00:00Z",
  "period": {
    "start": "2026-01-01T00:00:00Z",
    "end": "2026-01-02T00:00:00Z"
  },
  "workflows": {
    "workflow-name": {
      "safe_outputs": {
        "issues_created": 5,
        "prs_created": 2,
        "comments_added": 10,
        "discussions_created": 1
      },
      "runs": {
        "total": 10,
        "successful": 8,
        "failed": 2,
        "success_rate": 0.80
      },
      "engagement": {
        "total_reactions": 15,
        "total_comments": 20,
        "avg_reactions_per_output": 3.0
      },
      "quality_indicators": {
        "pr_merge_rate": 0.75,
        "avg_close_time_hours": 48
      }
    }
  },
  "ecosystem": {
    "total_workflows": 176,
    "active_workflows": 165,
    "total_outputs": 50,
    "engines": {
      "copilot": 40,
      "claude": 5,
      "codex": 5
    }
  }
}

Integration with Meta-Orchestrators

All three meta-orchestrators depend on this data:

  • Agent Performance Analyzer: Quality scores, rankings, effectiveness
  • Campaign Manager: Campaign progress tracking, goal achievement
  • Workflow Health Manager: Success rates, failure patterns, health scores

Acceptance Criteria

  • Metrics Collector workflow exists and runs successfully
  • Writes to /tmp/gh-aw/repo-memory/default/metrics/latest.json
  • Writes daily snapshots to metrics/daily/YYYY-MM-DD.json
  • JSON schema matches expected format
  • Includes all required metrics sections
  • Agent Performance Analyzer can load and parse data
  • Data persists across workflow runs via git branch
  • Historical data retained for at least 30 days

Priority

High - Blocks quantitative analysis for all meta-orchestrators.

Related

  • #aw_pr_completion_barrier - PR completion barrier issue
  • Discussion: Agent Performance Report - Baseline
  • Workflow: Agent Performance Analyzer

Identified by Agent Performance Analyzer

AI generated by Agent Performance Analyzer - Meta-Orchestrator

Comments on the Issue (you are @copilot in this section)

@pelikhan /q review prompt and expected memory loca...

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
Copilot AI changed the title [WIP] Add missing metrics infrastructure for agent performance analysis Fix repo-memory file-glob patterns to match spec (exclude branch names) Jan 2, 2026
Copilot AI requested a review from mnkiefer January 2, 2026 06:13
@pelikhan
Copy link
Contributor

pelikhan commented Jan 2, 2026

@copilot the "compile" command should also recompile campaigns and not delete the lock.yml files of campaign when doing --purge

Copy link
Contributor Author

Copilot AI commented Jan 2, 2026

@copilot the "compile" command should also recompile campaigns and not delete the lock.yml files of campaign when doing --purge

I've identified the issue. The purgeOrphanedLockFiles function uses *.lock.yml glob which incorrectly includes .campaign.g.lock.yml files. These campaign orchestrator lock files should only be handled by the separate purgeOrphanedCampaignOrchestratorLockFiles function.

I'll fix this by excluding .campaign.g.lock.yml files from the general lock file purge.

@pelikhan
Copy link
Contributor

pelikhan commented Jan 2, 2026

@copilot recompile and recompile campaigns

The purgeOrphanedLockFiles function was incorrectly deleting .campaign.g.lock.yml
files because its glob pattern (*.lock.yml) matched all lock files including campaign
orchestrator lock files. Campaign lock files should only be handled by the dedicated
purgeOrphanedCampaignOrchestratorLockFiles function.

Changes:
- Skip .campaign.g.lock.yml files in purgeOrphanedLockFiles function
- Add test to verify campaign lock files are preserved during purge
- Update existing test to reflect that .campaign.g.md files are kept as debug artifacts

Fixes #3704580322

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan pelikhan marked this pull request as ready for review January 2, 2026 06:41
@pelikhan pelikhan merged commit 6477056 into main Jan 2, 2026
4 of 5 checks passed
@pelikhan pelikhan deleted the copilot/add-metrics-infrastructure branch January 2, 2026 06:41
Copy link
Contributor Author

Copilot AI commented Jan 2, 2026

@copilot recompile and recompile campaigns

Verified with make recompile. Both campaign specs compiled successfully and their lock files were preserved:

  • ✓ docs-quality-maintenance-project67.campaign.g.lock.yml (82K)
  • ✓ go-file-size-reduction-project64.campaign.g.lock.yml (81K)

The purge logic now correctly skips .campaign.g.lock.yml files. Fixed in commit c041e68.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

📊 Missing Metrics Infrastructure for Agent Performance Analysis

3 participants