refactor: use jubilant for main integration tests

[imanenami]

Issue

Migration of tests to jubilant. All test files in the tests/integration root folder are migrated. HA & new_relations tests are skipped for now, to see how comfortable we are with this approach...

Solution

Experimented with an Adapter-based approach. Quick and easy. Each test file took 5-10 min. to migrate once the adapters module was mature.

Findings:

• The model.block_until method of python-libjuju is hard to adapt, the current implementation tries to leverage jubilant.Juju.wait, but comes with a performance penalty.

Checklist

• I have added or updated any relevant documentation.
• I have cleaned any remaining cloud resources from my accounts.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.43%. Comparing base (cda19e7) to head (4193ae9).

Additional details and impacted files

@@           Coverage Diff            @@
##           16/edge    #1507   +/-   ##
========================================
  Coverage    70.43%   70.43%           
========================================
  Files           15       15           
  Lines         4282     4282           
  Branches       694      694           
========================================
  Hits          3016     3016           
  Misses        1057     1057           
  Partials       209      209           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[taurus-forever]

Wow, this is massive, but promising.
LGTM from my side, but some questions first.
The decision here is on @marceloneppel

[taurus-forever]

Also, I have noticed this backups:Restore failed: Unit cannot restore backup as it was not elected the leader unit yet

It looks like we are restoring backup having two units /0 and /1 ... once /1 is a leader the test failed.
The test must scaledown to one unit before restoring backup... is it a bug in test or in Jubilant/fixtures/scale-down? @imanenami , Tnx!

P.S. I do not recall such error in non-Jubilant tests

[marceloneppel]

I suspect the issue @taurus-forever commented about (Unit cannot restore backup as it was not elected the leader unit yet) may be due to a bug in ModelAdapter.block_until (adapters.py:251-257):

def block_until(self, *conditions, timeout=None, wait_period=0.5):
    self._juju.wait(
        lambda status: all(conditions),
        ...
    )

conditions is a tuple of callable objects (due to *conditions packing — see Python docs). The built-in all() checks the truthiness of each element (Python docs), and since function objects are always truthy (Python docs — "By default, an object is considered true")), all(conditions) is always True regardless of what the conditions would actually return.

This would cause juju.wait() to return immediately without ever polling, so the scale-down wait in backup_helpers.py would complete instantly and the restore would run before the second unit is actually removed. The test may still pass sometimes due to incidental latency from Juju API calls and action dispatch giving Patroni enough time to converge, which would explain the flaky behavior.

If that's the case, the fix would be to actually call each condition:

lambda status: all(c() for c in conditions),

[imanenami]

@taurus-forever @marceloneppel

Thanks Alex for spotting that, and special thanks to Marcello for finding the root cause, which I also think is the issue. I'll fix that 👍

[imanenami]

@marceloneppel @dragomirp

Sorry for making the reviews stale, I just had to rebase and sign the commits, and did some final, small tidy ups in this commit: 4193ae9

I'd also like to thank all the reviewers who provided valuable input.

No need to mention that the decision to merge this or not is completely on the repo's maintainers, specially @marceloneppel. I have plans to make the migration full and remove pytest-operator dependency altogether in a follow-up PR (The same way I did in this Kafka PR) so I'd highly appreciate if we keep the branch and not delete it ❤️

[marceloneppel]

Thanks for the work on this, @imanenami! I've re-reviewed after the rebase, and the changes look good. Approving. Looking forward to the follow-up PR to go fully Jubilant-native, like the Kafka one! If the branch gets auto-deleted after the merge, we can restore it.

[taurus-forever]

Let's give it a try as 1) it works 2) build more experience and confidence in Jubilant 3) can help us merge PG14+PG16 tests.

We are open for AI tests rewrite to native Jubilant. @tonyandrewmeyer can you help us make it success story?

[tonyandrewmeyer]

We are open for AI tests rewrite to native Jubilant. @tonyandrewmeyer can you help us make it success story?

Happy to help. I'll take a closer look tomorrow.

[taurus-forever]

LGTM. Let's give it a try! Thank you Imam!