Add wasm compiler integration tests with golden file comparison by Mossaka · Pull Request #16643 · github/gh-aw

Mossaka · 2026-02-18T21:06:48Z

Summary

Add comprehensive golden-file integration tests for the wasm compiler to verify YAML output correctness
Cover all 3 engine types (copilot, claude, codex) and various workflow configurations (16 fixtures total)
Add both Go-level string API tests and Node.js-level wasm binary tests
Integrate into Makefile and CI pipeline

What's included

Go golden tests (`pkg/workflow/wasm_golden_test.go`)

Test	Purpose
`TestWasmGolden_CompileFixtures`	Compiles 16 workflow fixtures via the string API (same code path as wasm) and compares against golden files
`TestWasmGolden_CompileWithImports`	Tests import resolution via filesystem
`TestWasmGolden_RoundTrip`	Verifies compilation determinism (compiles 3 times, compares)
`TestWasmGolden_NativeVsStringAPI`	Informational comparison of native vs string API output differences
`TestWasmGolden_AllEngines`	Verifies all 3 engine types compile successfully

16 test fixtures covering

Engines: copilot, claude, codex
Triggers: workflow_dispatch, issues, push, pull_request, schedule
Features: tools, safe-outputs, permissions, GitHub expressions, imports, network config, strict mode, dispatch inputs

Node.js wasm binary test (`scripts/test-wasm-golden.mjs`)

Builds the actual .wasm binary, loads it in Node.js using Go's wasm_exec.js, compiles all fixtures, and compares output against the Go golden files.

Makefile targets

make test-wasm-golden     # Run Go golden tests
make test-wasm            # Build wasm + run Node.js comparison
make update-wasm-golden   # Regenerate golden files

CI integration

Added to the existing build-wasm CI job:

Go string API golden test step
Node.js wasm binary golden test step

Key findings from NativeVsStringAPI comparison

The string API (wasm) path diverges from the native file-based path in expected ways:

Pre-activation job: String API includes a pre_activation job not present in native output
GH_AW_REQUIRED_ROLES: Empty in string API, populated in native (env var dependent)
Safe-outputs: Native path generates additional safe-output processing jobs

These differences are expected — the golden tests verify the string API/wasm path is self-consistent, not identical to the native path.

Test plan

All 16 fixtures compile successfully via string API
Golden files generated and verified (round-trip match)
Determinism test passes (3 compilations produce identical output)
Import resolution works via filesystem
All 3 engine types compile correctly
No regressions in existing unit tests (go test ./... -short passes)
Node.js wasm binary test (requires make build-wasm in CI)
CI pipeline validation

🤖 Generated with Claude Code

Changeset

Type: patch
Description: Add wasm compiler golden-file verification tests and CI steps so the wasm build stays validated against the fixtures.

Generated by Changeset Generator

Add a comprehensive test suite to verify the wasm compiler produces correct YAML output by comparing against golden files. ## What's included ### Go golden tests (pkg/workflow/wasm_golden_test.go) - TestWasmGolden_CompileFixtures: Compiles 16 workflow fixtures via the string API (same code path as wasm) and compares against golden files - TestWasmGolden_CompileWithImports: Tests import resolution via filesystem - TestWasmGolden_RoundTrip: Verifies compilation determinism (3 runs) - TestWasmGolden_NativeVsStringAPI: Informational comparison of native vs string API output, logging differences without failing - TestWasmGolden_AllEngines: Verifies copilot, claude, codex engines compile ### Test fixtures (16 workflows covering) - All 3 engine types (copilot, claude, codex) - Multiple trigger types (workflow_dispatch, issues, push, PR, schedule) - Tool configurations, safe-outputs, permissions, expressions, imports - Network configurations, strict mode, dispatch inputs ### Node.js wasm binary test (scripts/test-wasm-golden.mjs) - Builds the actual wasm binary, loads in Node.js, compiles fixtures - Compares wasm binary output against Go golden files ### Makefile targets - make test-wasm-golden: Run Go golden tests - make test-wasm: Build wasm + run Node.js comparison - make update-wasm-golden: Regenerate golden files ### CI integration - Added golden test + Node.js wasm test steps to build-wasm CI job Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds comprehensive golden-file integration tests for the wasm compiler to verify YAML output correctness. The implementation includes both Go-level string API tests and Node.js-level wasm binary tests, covering 16 workflow fixtures across 3 engine types (copilot, claude, codex) with various configurations.

Changes:

Added Go test suite (wasm_golden_test.go) with 5 test functions covering compilation, imports, determinism, API comparison, and all engines
Created Node.js test runner (test-wasm-golden.mjs) for wasm binary validation against golden files
Added 16 test fixtures covering different engines, triggers, and features (tools, safe-outputs, permissions, imports, expressions, strict mode)
Integrated tests into Makefile with new targets (test-wasm-golden, update-wasm-golden, test-wasm) and CI pipeline

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`pkg/workflow/wasm_golden_test.go`	Main Go test suite with 5 test functions for wasm compiler validation
`scripts/test-wasm-golden.mjs`	Node.js script to test actual wasm binary output against golden files
`pkg/parser/virtual_fs_test_helpers.go`	Test helper for overriding file reading function
`pkg/workflow/testdata/wasm_golden/fixtures/*.md`	16 workflow fixtures covering various engine types, triggers, and features
`pkg/workflow/testdata/wasm_golden/fixtures/shared/tools.md`	Shared import fixture for testing import resolution
`pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/*.golden`	Golden output files for each fixture
`Makefile`	Added test targets for wasm golden tests and updated help documentation
`.github/workflows/ci.yml`	Integrated wasm golden tests into CI pipeline

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Go 1.24+ moved wasm_exec.js from misc/wasm/ to lib/wasm/. Update the test script to check lib/wasm/ first with fallback to misc/wasm/ for older Go versions, matching the pattern already used in bundle-wasm-docs.sh. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…orts - Wasm API: accept optional filename parameter (3rd arg) so GH_AW_WORKFLOW_FILE matches golden files - JS test: pass fixture filename and virtual files for import resolution - Lint: remove unused abs() function, add nolint for golden comparison (exact string match is intentional for golden tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pelikhan · 2026-02-19T00:50:50Z

@copilot keep working

Copilot · 2026-02-19T00:50:56Z

@pelikhan I've opened a new pull request, #16694, to work on those changes. Once the pull request is ready, I'll request review from you.

github-actions · 2026-02-19T01:20:40Z

🧪 Smoke Project is now testing project operations...

github-actions · 2026-02-19T01:20:43Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-02-19T01:20:46Z

📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing...

github-actions · 2026-02-19T01:20:46Z

💫 TO BE CONTINUED... Smoke Claude failed! Our hero faces unexpected challenges...

github-actions · 2026-02-19T01:20:48Z

🧪 Smoke Temporary ID is now testing temporary ID functionality...

github-actions · 2026-02-19T01:21:50Z

❌ Smoke Temporary ID encountered failures. Check the logs for details.

github-actions · 2026-02-19T01:22:19Z

📰 DEVELOPING STORY: Smoke Copilot reports failed. Our correspondents are investigating the incident...

github-actions · 2026-02-19T01:23:31Z

✅ Smoke Project completed successfully. All project operations validated.

github-actions · 2026-02-19T01:26:40Z

Smoke test summary
PRs: #16699 Remove GitHub API call from CLI initialization hot path
PRs: #16690 Add blocked pattern matching for add-labels and remove-labels
GitHub MCP: ✅
Serena MCP: ✅
Playwright: ✅
File write/read: ✅
Build (make build): ✅
Overall: PASS

🔮 The oracle has spoken through Smoke Codex for issue #16643

Copilot AI review requested due to automatic review settings February 18, 2026 21:06

Copilot started reviewing on behalf of Mossaka February 18, 2026 21:07 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

This was referenced Feb 18, 2026

Smoke Test: Copilot - 22157553390 #16650

Closed

Smoke Test: Claude - 22157554498 #16652

Closed

github-actions bot mentioned this pull request Feb 18, 2026

Smoke Test: Copilot - 22160562130 #16666

Closed

Merge branch 'main' into feat/wasm-integration-tests

6385847

Copilot AI mentioned this pull request Feb 19, 2026

[WIP] Add wasm compiler integration tests with golden file comparison #16694

Closed

Merge branch 'main' into feat/wasm-integration-tests

b8c8816

github-actions bot mentioned this pull request Feb 19, 2026

Smoke Test: Copilot - 22164477402 #16701

Closed

Mossaka added the smoke label Feb 19, 2026

Mossaka merged commit 02de894 into main Feb 19, 2026
88 of 92 checks passed

Mossaka deleted the feat/wasm-integration-tests branch February 19, 2026 01:21

github-actions bot added smoke-codex and removed smoke labels Feb 19, 2026

Mossaka mentioned this pull request Feb 19, 2026

Fix wasm/native compiler divergence, add smoke workflow golden tests #16705

Merged

4 tasks

Conversation

Mossaka commented Feb 18, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Go golden tests (pkg/workflow/wasm_golden_test.go)

16 test fixtures covering

Node.js wasm binary test (scripts/test-wasm-golden.mjs)

Makefile targets

CI integration

Key findings from NativeVsStringAPI comparison

Test plan

Changeset

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

pelikhan commented Feb 19, 2026

Uh oh!

Copilot AI commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Mossaka commented Feb 18, 2026 •

edited by github-actions bot

Loading

Go golden tests (`pkg/workflow/wasm_golden_test.go`)

Node.js wasm binary test (`scripts/test-wasm-golden.mjs`)

github-actions bot commented Feb 19, 2026 •

edited

Loading

github-actions bot commented Feb 19, 2026 •

edited

Loading