Archive fieldcompare diff files in fieldcompare-diffs/ (fixes #441)

changelog-entries/441.md

@@ -0,0 +1 @@
+- Archive fieldcompare diff VTK files into a `fieldcompare-diffs/` folder in each systemtest run directory so they are easy to find in CI artifacts when investigating comparison failures (fixes [#441](https://github.com/precice/tutorials/issues/441)).

tools/tests/README.md

@@ -105,7 +105,7 @@ In this case, building and running seems to work out, but the tests fail because
 
 The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub.
 If this does not provide enough hints, the next step is to download the generated `system_tests_run_<run_id>_<run_attempt>` artifact. Note that by default this will only be generated if the systemtests fail.
-Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation.
+Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation. When fieldcompare runs with `--diff`, it writes VTK diff files; these are copied into a `fieldcompare-diffs/` subfolder in the same run directory so you can open them (e.g. in ParaView) to see where results differ from the reference.
 
 ## Adding new tests
 

tools/tests/systemtests/Systemtest.py

@@ -6,6 +6,8 @@
 from pathlib import Path
 from paths import PRECICE_REL_OUTPUT_DIR, PRECICE_TOOLS_DIR, PRECICE_REL_REFERENCE_DIR, PRECICE_TESTS_DIR, PRECICE_TUTORIAL_DIR
 
+FIELDCOMPARE_DIFFS_DIR = "fieldcompare-diffs"
+
 from metadata_parser.metdata import Tutorial, CaseCombination, Case, ReferenceResult
 from .SystemtestArguments import SystemtestArguments
 
@@ -413,6 +415,24 @@ def _run_field_compare(self):
             elapsed_time = time.perf_counter() - time_start
             return FieldCompareResult(1, stdout_data, stderr_data, self, elapsed_time)
 
+    def __archive_fieldcompare_diffs(self):
+        """
+        Copy fieldcompare diff VTK files from precice-exports into fieldcompare-diffs/
+        so they are easy to find in CI artifacts (issue #441).
+        """
+        precice_exports = self.system_test_dir / PRECICE_REL_OUTPUT_DIR
+        if not precice_exports.exists():
+            return
+        diff_files = list(precice_exports.glob("diff_*")) + list(precice_exports.glob("*_diff.*"))
+        if not diff_files:
+            return
+        dest_dir = self.system_test_dir / FIELDCOMPARE_DIFFS_DIR
+        dest_dir.mkdir(exist_ok=True)
+        for f in diff_files:
+            if f.is_file():
+                shutil.copy2(f, dest_dir / f.name)
+        logging.debug(f"Archived {len(diff_files)} fieldcompare diff file(s) to {dest_dir} for {self}")
+
     def _build_docker(self):
         """
         Builds the docker image
@@ -563,6 +583,7 @@ def run(self, run_directory: Path):
                 fieldcompare_time=0)
 
         fieldcompare_result = self._run_field_compare()
+        self.__archive_fieldcompare_diffs()
         std_out.extend(fieldcompare_result.stdout_data)
         std_err.extend(fieldcompare_result.stderr_data)
         if fieldcompare_result.exit_code != 0: