Arm backend: Rework "no_target" test collection by martinlsm · Pull Request #18493 · pytorch/executorch

martinlsm · 2026-03-25T12:43:28Z

Prior to this patch, test_arm_baremetal.sh would consider all tests containing the substring no_target in their names to be tests that lacks a TOSA target. This patch changes this concept such that tests that do not match against any target ("u55", "u85", "tosa" and "vgf" as of now), will be considered to have no target; therefore, targetless tests must no longer include the no_target substring in their names.

This patch fixes a problem where a test could be skipped because it wasn't matching against any target substring (including no_target). Such tests will now instead be considered targetless and always run.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell

Prior to this patch, test_arm_baremetal.sh would consider all tests containing the substring `no_target` in their names to be tests that lacks a TOSA target. This patch changes this concept such that tests that do not match against any target ("u55", "u85", "tosa" and "vgf" as of now), will be considered to have no target; therefore, targetless tests must no longer include the no_target substring in their names. This patch fixes a problem where a test could be skipped because it wasn't matching against any target substring (including no_target). Such tests will now instead be considered targetless and always run. Signed-off-by: Martin Lindström <Martin.Lindstroem@arm.com> Change-Id: I1cff769ce6bd5595db9f26c1506adcae564f4ea8

pytorch-bot · 2026-03-25T12:43:32Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18493

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 3 Unrelated Failures

As of commit 362cae6 with merge base 28b4813 ():

NEW FAILURES - The following jobs have failed:

pull / unittest / macos / macos-job (gh)
export/tests/test_target_recipes.py::TestTargetRecipes::test_linear_model
pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_all_models_with_recipes
trunk / test-arm-backend-vkml (test_pytest_ops_vkml) / linux-job (gh)
RuntimeError: Command docker exec -t a4b97cd7b0a8e88403285e0eeceea1385385a9854935b3f8316e76ca9a50c9e8 /exec failed with exit code 1
trunk / test-qnn-model (fp32, mb) / linux-job (gh)
RuntimeError: Command docker exec -t 7b64a744c917c985802d64dc4efeeb20a4d47cea213614f0be551c93b16f7caf /exec failed with exit code 9

CANCELLED JOB - The following job was cancelled. Please retry:

trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh)
##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / windows / windows-job (gh) (trunk failure)
##[error]The operation was canceled.
pull / unittest-editable / windows / windows-job (gh) (trunk failure)
##[error]The operation was canceled.
trunk / unittest-release / windows / windows-job (gh) (trunk failure)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

martinlsm · 2026-03-25T12:43:59Z

@pytorchbot label ciflow/trunk

martinlsm · 2026-03-25T12:44:09Z

@pytorchbot label "partner: arm"

martinlsm · 2026-03-25T12:44:20Z

@pytorchbot label "release notes: none"

Copilot

Pull request overview

This PR updates the Arm baremetal test runner’s definition of “target-less” pytest tests so they’re selected by not matching any known target substring (u55/u85/tosa/vgf), preventing un-tagged tests from being skipped by all suites.

Changes:

Introduces a shared pytest -k expression (EXCLUDE_TARGET_EXPR) to select tests that don’t match any known target substring.
Updates the “no_target” pytest suites to use that exclusion expression instead of -k no_target.
Cleans up formatting/alignment of the testing command table in backends/arm/README.md.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`backends/arm/test/test_arm_baremetal.sh`	Switches target-less test selection from `-k no_target` to an exclusion-based `-k` expression covering all current targets.
`backends/arm/README.md`	Adjusts the markdown table formatting in the testing section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-25T12:47:09Z

backends/arm/README.md

-| `test_arm_baremetal.sh test_model_smollm2-135M`      | Runs some models with Corstone FVP.          |
-| `test_arm_baremetal.sh test_smaller_stories_llama`   | Runs E2E model tests on Corstone FVP.        |
+| `test_arm_baremetal.sh test_run_vkml`                | Runs end-to-end unit tests for VGF specific use-cases.       |
+| `test_arm_baremetal.sh test_model_smollm2-135M`      | Runs some models with Corstone FVP.                          |


The documented test suite name uses a hyphen (test_model_smollm2-135M), but the actual function in backends/arm/test/test_arm_baremetal.sh is test_model_smollm2_135M (underscore). Running the command as documented will fail; please update the README command to match the script function name.

Suggested change

| `test_arm_baremetal.sh test_model_smollm2-135M` | Runs some models with Corstone FVP. |

| `test_arm_baremetal.sh test_model_smollm2_135M` | Runs some models with Corstone FVP. |

zingo · 2026-03-25T14:15:22Z

backends/arm/test/test_arm_baremetal.sh


 TEST_SUITE_NAME="$(basename "$0") ${TEST_SUITE}"

+EXCLUDE_TARGET_EXPR="(not u55) and (not u85) and (not tosa) and (not _vgf_)"


Assuming the "_" on vgf (only) are on purpose.

Yes, it was the exact token I just copied over to that expression.

martinlsm · 2026-03-27T13:16:37Z

Failing tests were unrelated. The failing Arm test is seen on main also.

martinlsm requested a review from digantdesai as a code owner March 25, 2026 12:43

Copilot AI review requested due to automatic review settings March 25, 2026 12:43

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 25, 2026

pytorch-bot bot added the ciflow/trunk label Mar 25, 2026

Copilot started reviewing on behalf of martinlsm March 25, 2026 12:44 View session

pytorch-bot bot added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Mar 25, 2026

pytorch-bot bot added the release notes: none Do not include this in the release notes label Mar 25, 2026

Copilot AI reviewed Mar 25, 2026

View reviewed changes

zingo approved these changes Mar 25, 2026

View reviewed changes

Merge branch 'main' into remove-no-target

362cae6

martinlsm merged commit 93960e5 into pytorch:main Mar 27, 2026
556 of 571 checks passed

martinlsm deleted the remove-no-target branch March 27, 2026 13:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arm backend: Rework "no_target" test collection#18493

Arm backend: Rework "no_target" test collection#18493
martinlsm merged 2 commits intopytorch:mainfrom
martinlsm:remove-no-target

martinlsm commented Mar 25, 2026 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Mar 25, 2026 •

edited

Loading

Uh oh!

martinlsm commented Mar 25, 2026

Uh oh!

martinlsm commented Mar 25, 2026

Uh oh!

martinlsm commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

zingo Mar 25, 2026 •

edited

Loading

Uh oh!

martinlsm Mar 26, 2026

Uh oh!

Uh oh!

martinlsm commented Mar 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	\| `test_arm_baremetal.sh test_model_smollm2-135M` \| Runs some models with Corstone FVP. \|
	\| `test_arm_baremetal.sh test_model_smollm2_135M` \| Runs some models with Corstone FVP. \|


		TEST_SUITE_NAME="$(basename "$0") ${TEST_SUITE}"

		EXCLUDE_TARGET_EXPR="(not u55) and (not u85) and (not tosa) and (not _vgf_)"

Conversation

martinlsm commented Mar 25, 2026 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18493

❌ 4 New Failures, 1 Cancelled Job, 3 Unrelated Failures

Uh oh!

martinlsm commented Mar 25, 2026

Uh oh!

martinlsm commented Mar 25, 2026

Uh oh!

martinlsm commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

zingo Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martinlsm Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

martinlsm commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martinlsm commented Mar 25, 2026 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Mar 25, 2026 •

edited

Loading

zingo Mar 25, 2026 •

edited

Loading

martinlsm commented Mar 27, 2026 •

edited

Loading