Skip to content

Add count column to markdown results summary#1187

Open
Bortlesboat wants to merge 1 commit intohuggingface:mainfrom
Bortlesboat:bortlesboat/issue-804-markdown-count
Open

Add count column to markdown results summary#1187
Bortlesboat wants to merge 1 commit intohuggingface:mainfrom
Bortlesboat:bortlesboat/issue-804-markdown-count

Conversation

@Bortlesboat
Copy link

Summary

  • add per-task
    _samples to the exported evaluation results dictionary
  • render a new Count column in make_results_table markdown output
  • add unit tests covering populated and missing count behavior

Why

Issue #804 asks for the markdown summary to show how many questions were evaluated so low scores are easier to interpret.

Validation

  • python3 -m py_compile src/lighteval/logging/evaluation_tracker.py src/lighteval/utils/utils.py tests/unit/utils/test_utils.py
  • Full unit test run was not possible in this environment because WSL Python is missing pip/pytest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant