Skip to content

[FIX] Integrate never-worked baseline handling into CI reporting#1058

Draft
Rahul-2k4 wants to merge 5 commits intoCCExtractor:masterfrom
Rahul-2k4:feat/never-worked-baseline-integration
Draft

[FIX] Integrate never-worked baseline handling into CI reporting#1058
Rahul-2k4 wants to merge 5 commits intoCCExtractor:masterfrom
Rahul-2k4:feat/never-worked-baseline-integration

Conversation

@Rahul-2k4
Copy link

@Rahul-2k4 Rahul-2k4 commented Mar 11, 2026 •
edited
Loading

Please prefix your pull request with one of the following: [FEATURE] [FIX] [IMPROVEMENT].

In raising this pull request, I confirm the following (please check boxes):

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • I have considered, and confirmed that this submission will be valuable to others.
  • I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
  • I give this submission freely, and claim no ownership to its content.

My familiarity with the project is as follows (check one):

  • I have never used the project.
  • I have used the project briefly.
  • I have used the project extensively, but have not contributed previously.
  • I am an active contributor to the project.

Summary

  • integrate never-worked baseline handling into completed test reporting
  • restrict baseline refresh to trusted main-repo commit runs
  • make never-worked reporting platform-specific and tighten migration backfill behavior

Linked Issue

Closes #1057

Test Plan

  • ./.venv/bin/python -m nose2 -v tests.test_ci.test_controllers.TestControllers.test_refresh_baseline_statuses_for_test_uses_full_result_status tests.test_ci.test_controllers.TestControllers.test_refresh_baseline_statuses_for_test_skips_pull_request_runs tests.test_ci.test_controllers.TestControllers.test_comment_info_separates_never_worked_failures tests.test_ci.test_controllers.TestControllers.test_comment_info_tracks_never_worked_per_platform tests.test_ci.test_controllers.TestControllers.test_progress_type_request_completed_refreshes_baseline_status tests.test_ci.test_controllers.TestControllers.test_progress_type_request_completed_does_not_refresh_baseline_for_pull_requests tests.test_ci.test_controllers.TestControllers.test_comments_successfully_in_passed_pr_test tests.test_ci.test_controllers.TestControllers.test_comments_successfuly_in_failed_pr_test tests.test_ci.test_controllers.TestControllers.test_comment_info_handles_variant_files_correctly tests.test_ci.test_controllers.TestControllers.test_comment_info_handles_invalid_variants_correctly tests.test_ci.test_controllers.TestControllers.test_update_build_badge tests.test_regression.test_baseline_status
  • ./.venv/bin/python -m nose2 -v tests.test_ci.test_controllers tests.test_regression.test_baseline_status (still has one unrelated existing failure in this environment: test_start_ci_empty_token, caused by blocked api.github.com access before token validation)

Rahul-2k4 and others added 5 commits March 11, 2026 12:54
@Rahul-2k4
feat(regression): add BaselineStatus to distinguish never-worked tests
c8fa870 
Adds a baseline_status field to RegressionTest with three states:
- unknown      - newly added test, no run history
- never_worked - has run history, but never passed on any CCExtractor version
- established  - has at least one passing run (a failure here is a regression)

This makes it possible to distinguish true regressions from samples that
have never produced correct output, addressing the 'Never worked' test
state goal from the Sample Platform NG project brief.

Includes:
- BaselineStatus DeclEnum in mod_regression/models.py
- RegressionTest.update_baseline_status(passed) state-machine method
- RegressionTest.is_regression property
- Flask-Migrate migration (d1f3a9c2e8b7) with upgrade/downgrade
- 12 unit tests covering all state transitions and DB persistence
@Rahul-2k4
Integrate never-worked baseline handling
38534ab
Initial plan
1f688b4
@Rahul-2k4
Fix variable shadowing and missing docstring in baseline integration
a0ab2e1 
Co-authored-by: Rahul-2k4 <216878448+Rahul-2k4@users.noreply.github.com>
@Rahul-2k4
Merge pull request #2 from Rahul-2k4/copilot/review-upstream-necessity
15d34d7 
Fix variable shadowing and missing docstring in never-worked baseline integration
@sonarqubecloud
Copy link

sonarqubecloud bot commented Mar 11, 2026

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@Rahul-2k4 Rahul-2k4 changed the title Integrate never-worked baseline handling into CI reporting [FIX] Integrate never-worked baseline handling into CI reporting Mar 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@canihavesomecoffee canihavesomecoffee Awaiting requested review from canihavesomecoffee canihavesomecoffee will be requested when the pull request is marked ready for review canihavesomecoffee is a code owner

@thealphadollar thealphadollar Awaiting requested review from thealphadollar thealphadollar will be requested when the pull request is marked ready for review thealphadollar is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] never-worked baseline handling is not fully integrated into trusted CI reporting

2 participants

@Rahul-2k4