Python test code refactoring - part 2

.github/workflows/build_test.yaml

@@ -52,7 +52,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -109,7 +109,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -171,7 +171,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -225,7 +225,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -271,7 +271,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -320,7 +320,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -366,7 +366,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -424,7 +424,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |

.github/workflows/build_test_py.yaml

@@ -25,8 +25,10 @@ jobs:
       - name: Update apt packages
         run: sudo apt-get update
 
-      - name: Install Python
-        run: sudo apt-get install python3 python-is-python3
+      - name: Setup Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |

.github/workflows/build_test_rv.yaml

@@ -30,15 +30,22 @@ jobs:
       - name: Install ImageMagick
         run: sudo apt-get install imagemagick
 
-      - name: Install Python
-        run: sudo apt-get install python3 python-is-python3 python3-numpy python3-pillow
-
       - name: Install clang-20 and RISC-V binutils
         run: sudo apt-get install clang-20 g++-riscv64-linux-gnu
 
       - name: Install RISC-V qemu
         run: sudo apt-get install qemu-user-static
 
+      - name: Setup Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.12'
+
+      - name: Get Python modules
+        run: |
+          python -m pip install --upgrade pip
+          pip install numpy Pillow
+
       - name: Build release
         run: |
           export CXX=clang++-20

.github/workflows/post_weekly_release.yaml

@@ -96,7 +96,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -143,7 +143,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -182,7 +182,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.11'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |
@@ -243,7 +243,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v6
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Get Python modules
         run: |

.pylintrc

@@ -315,7 +315,7 @@ max-locals=16
 max-parents=7
 
 # Maximum number of positional arguments for function / method.
-max-positional-arguments=5
+max-positional-arguments=7
 
 # Maximum number of public methods for a class (see R0904).
 max-public-methods=20

Docs/Testing.md

@@ -1,113 +1,115 @@
 # Testing astcenc
 
-The repository contains a small suite of tests which can be used to sanity
-check source code changes to the compressor. It must be noted that this test
-suite is relatively limited in scope and does not cover every feature or
-bitrate of the standard.
+The repository contains a suite of tests which can be used to validate source code changes to the compressor. The test suite is split into three main test
+types:
 
-# Required software
+* Python tests that drive the command line functional interface tests.
+* Python tests that drive command line performance and image quality tests.
+* C++ unit tests for some internal codec components.
 
-Running the tests requires Python 3.7 to be installed on the host machine, and
-an `astcenc-avx2` release build to have been previously compiled and installed
-into an directory called `astcenc` in the root of the git checkout. This
-can be achieved by configuring the CMake build using the install prefix
-`-DCMAKE_INSTALL_PREFIX=../` and then running a build with the `install` build
-target.
+There is currently no test suite for the library API, and we try to exercise all of the functionality via the command line functional interface tests. This
+gives good coverage, but is not able to reach all of the API.
 
-# Running C++ unit tests
+In addition to the tests, we have automated Python script lint, typing, and
+style checkers.
 
-We support a small (but growing) number of C++ unit tests, which are written
-using the `googletest` framework and integrated in the CMake "CTest" test
-framework.
+## Prerequisites
 
-To build unit tests pull the `googletest` git submodule and add
-`-DASTCENC_UNITTEST=ON` to the CMake command line when configuring.
+Running the tests requires Python 3.12 to be installed on the host machine.
 
-To run unit tests use the CMake `ctest` utility from your build directory after
-you have built the tests.
+# Running Python-based CLI functional tests
 
-```shell
-cd build
-ctest --verbose
-```
+Compile the build of `astcenc` you wish to test, and install it into the project `./bin` directory. This can be achieved by configuring the CMake build using the install prefix `-DCMAKE_INSTALL_PREFIX=../` and then running a build with the `install` build target.
 
-# Running command line tests
+Run the functional tests against to appropriate encoder variant using:
 
-To run the command line tests, which aim to get coverage of the command line
-options and core codec stability without testing the compression quality
-itself, run the command line:
+```sh
+python3 ./Test/astc_test_functional.py -v --encoder <encoder_variant>
+```
 
-    python3 -m unittest discover -s Test -p astc_test*.py -v
+# Running Python-based CLI performance and quality tests
 
-# Running image tests
+Compile the build of `astcenc` you wish to test and profile, and install it into the project `./bin` directory. This can be achieved by configuring the CMake build using the install prefix `-DCMAKE_INSTALL_PREFIX=../` and then running a build with the `install` build target.
 
 To run the image test suite run the following command from the root directory
 of the repository:
 
-    python3 ./Test/astc_test_image.py
+```sh
+python3 ./Test/astc_test_image.py
+```
 
 This will run though a series of image compression tests, comparing the image
 PSNR against a set of reference results from the last stable baseline. The test
-will fail if any reduction in PSNR above a set threshold is detected. Note that
-performance information is reported, but regressions will not flag a failure.
+will fail if any reduction in PSNR above a set threshold is detected.
 
-For debug purposes, all decompressed test output images and result CSV files
-are stored in the `TestOutput` directory, using the same test set structure as
-the `Test/Images` folder.
+The `--test-set` parameter can be passed to select which test images you use.
+The `--color-format`, `--block-size`, and `--test-quality` options allow you
+to subset which compressor options are tested. The `-j` option lets you control
+how many threads the compressor is allowed to use.
 
-## Test selection
+By default output images are discarded during testing. To store images for
+debug purposes, the decompressed output images can be kept by passing
+`--keep-output`. The resulting images are stored in the `TestOutput`
+subdirectory, using the same test set directoy structure as the `Test/Images`
+folder.
 
-The runner supports a number of options to filter down what is run, enabling
-developers to focus local testing on the parts of the code they are working on.
+## Benchmarking
 
-* `--encoder` selects which encoder to run. By default the `avx2` encoder is
-  selected. Note that some out-of-tree reference encoders (older encoders, and
-  some third-party encoders) are supported for comparison purposes. These will
-  not work without the binaries being manually provided; they are not
-  distributed here.
-* `--test-set` selects which image set to run. By default the `Small` image
-  test set is selected, which aims to provide basic coverage of many different
-  color formats and color profiles.
-* `--block-size` selects which block size to run. By default a range of
-  block sizes (2D and 3D) are used.
-* `--color-profile` selects which color profiles from the standard should be
-  used (LDR, LDR sRGB, or HDR) to select images. By default all are selected.
-* `--color-format` selects which color formats should be used (L, XY, RGB,
-  RGBA) to select images. By default all are selected.
+All runs of this test suite will report performance data. To improve result
+stability, the `--repeats` option makes the test suite process each image multiple times and return the best measured performance.
 
-## Performance tests
+Performance regressions will not automatically cause test failure, and must be
+manually reviewed.
 
-To provide less noisy performance results the test suite supports compressing
-each image multiple times and returning the best measured performance. To
-enable this mode use the following options:
-
-* `--repeats <M>` : Run M test compression passes which are timed.
-
-**Note:**  The reference CSV contains performance results measured on an Intel
-Core i5 9600K running at 4.3GHz, running each test 5 times.
+**Note:** The upstream reference data contains performance results measured on
+an Intel Core i5 9600K running at 4.3GHz, running each test 5 times.
 
 ## Updating reference data
 
-The reference PSNR and performance scores are stored in CSVs committed to the
-repository. This data is created by running the tests using the last stable
+The reference quality and performance scores are stored in CSVs committed to
+the repository. This data is created by running the tests using the last stable
 release on a standard test machine we use for performance testing builds.
 
 It can be useful for developers to rebuild the reference results for their
-local machine, in particular for measuring performance improvements. To build
-new reference CSVs, download the current reference `astcenc` binary (1.7) from
-GitHub for your host OS and place it in to the `./Binaries/1.7/` directory.
-Once this is done, run the command:
+local machine, in particular for measuring performance improvements.
+
+To build new reference CSVs for an official release, checkout and build a
+release build of the baseline release you want to compare against and place it
+in to the `./Binaries/<version>/` directory, where `<version>` must be the
+`major.minor` number of the release, e.g. `5.3`.
+
+Once this is done, run the command (e.g.):
 
-    python3 ./Test/astc_test_image.py --encoder 1.7 --test-set all --repeats 5
+```sh
+python3 ./Test/astc_test_image.py --encoder ref-5.3-avx2 ...
+```
 
-... to regenerate the reference CSV files.
+... for whichever test set you want to build a reference for. This can take
+some time to complete, especially if your command line enables many test sets,
+high quality levels, or uses many repeats to improve result stability.
 
-**WARNING:** This can take some hours to complete, and it is best done when the
+**WARNING:** This may take some hours to complete, and it is best done when the
 test suite gets exclusive use of the machine to avoid other processing slowing
 down the compression and disturbing the performance data. It is recommended to
 shutdown or disable any background applications that are running.
 
-## ASAN and UBSAN
+# Running C++ unit tests
+
+We support a small (but growing) number of C++ unit tests, which are written
+using the googletest framework and integrated in the CMake CTest test runner.
+
+To build unit tests pull the `googletest` git submodule and add
+`-DASTCENC_UNITTEST=ON` to the CMake command line when configuring.
+
+To run unit tests use the CMake `ctest` utility from your build directory after
+you have built the tests.
+
+```shell
+cd build
+ctest --verbose
+```
+
+# Using ASAN and UBSAN
 
 Running builds compiled with ASAN (address sanitizer) and UBSAN (undefined
 behavior sanitizer) is one way to check that we have not introduced memory
@@ -120,7 +122,7 @@ Build a release build with debug information using
 
 Run test commands as normal with the generated binaries.
 
-## Valgrind memcheck
+# Using Valgrind memcheck
 
 Running builds using the Valgrind memcheck tool is another way to validate that
 we have not introduced memory errors.