Skip to content

Rewrite tests to python for windows runtime testing#215

Open
julek-wolfssl wants to merge 24 commits intowolfSSL:mainfrom
julek-wolfssl:windows-ci
Open

Rewrite tests to python for windows runtime testing#215
julek-wolfssl wants to merge 24 commits intowolfSSL:mainfrom
julek-wolfssl:windows-ci

Conversation

@julek-wolfssl
Copy link
Copy Markdown
Member

No description provided.

@julek-wolfssl julek-wolfssl self-assigned this Apr 1, 2026
Copilot AI review requested due to automatic review settings April 1, 2026 14:56
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the wolfCLU test suite from shell scripts to Python unittest modules to enable reliable runtime testing on Windows, while also updating Windows build configuration and CI workflows.

Changes:

  • Replaced many tests/**/*.sh scripts with equivalent tests/**/*-test.py Python unittest modules and added a Windows-friendly tests/run_tests.py runner.
  • Updated Automake integration to recognize .py tests and added Python detection in configure.ac.
  • Updated Windows Visual Studio project/solution and CI workflows; added Windows networking initialization and SNI handling in the client setup.

Reviewed changes

Copilot reviewed 93 out of 94 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
wolfclu/clu_header_main.h Undefines CRL reason macros before wolfSSL headers (Windows build hygiene).
wolfCLU.vcxproj.filters Adds new source files to the Visual Studio filters list.
wolfCLU.vcxproj Updates toolset and enables multiprocessor compilation.
wolfclu.sln Updates VS version metadata and fixes x64 config mapping.
tests/x509/x509-verify-test.sh Removed (replaced by Python unittest).
tests/x509/x509-verify-test.py New Python unittest for wolfssl verify.
tests/x509/x509-req-test.sh Removed (replaced by Python unittest).
tests/x509/x509-process-test.sh Removed (replaced by Python unittest).
tests/x509/x509-ca-test.sh Removed (replaced by Python unittest).
tests/x509/include.am Registers Python x509 tests in Automake.
tests/x509/CRL-verify-test.sh Removed (replaced by Python unittest).
tests/x509/CRL-verify-test.py New Python unittest for wolfssl crl.
tests/wolfclu_test.py Shared helper to locate wolfssl binary and run commands.
tests/testEncDec/test_aesctr.sh Removed (replaced by Python unittest).
tests/testEncDec/test_aescbc_3des_cam.sh Removed (replaced by Python unittest).
tests/testEncDec/README.md Removed (test approach changed to self-contained Python).
tests/testEncDec/include.am Registers new enc/dec Python test in Automake.
tests/testEncDec/encdec-test.py New encrypt/decrypt round-trip tests for multiple ciphers.
tests/server/server-test.sh Removed (replaced by Python unittest).
tests/server/server-test.py New Python unittest for s_server/s_client handshake.
tests/server/include.am Registers Python server test in Automake.
tests/run_tests.py New Windows-oriented Python test runner discovering *-test.py.
tests/rand/rand-test.sh Removed (replaced by Python unittest).
tests/rand/rand-test.py New Python unittest for wolfssl rand.
tests/rand/include.am Registers Python rand test in Automake.
tests/pkey/rsa-test.sh Removed (replaced by Python unittest).
tests/pkey/rsa-test.py New Python unittest for RSA key conversion/validation.
tests/pkey/pkey-test.sh Removed (replaced by Python unittest).
tests/pkey/pkey-test.py New Python unittest for pkey public/private conversions.
tests/pkey/include.am Registers Python pkey tests in Automake.
tests/pkey/ecparam-test.sh Removed (replaced by Python unittest).
tests/pkey/ecparam-test.py New Python unittest for curve enumeration and key generation.
tests/pkcs/pkcs8-test.sh Removed (replaced by Python unittest).
tests/pkcs/pkcs8-test.py New Python unittest for PKCS8 conversion and error cases.
tests/pkcs/pkcs7-test.sh Removed (replaced by Python unittest).
tests/pkcs/pkcs7-test.py New Python unittest for PKCS7 conversions and printing certs.
tests/pkcs/pkcs12-test.sh Removed (replaced by Python unittest).
tests/pkcs/pkcs12-test.py New Python unittest for PKCS12 extraction options.
tests/pkcs/include.am Registers Python pkcs tests in Automake.
tests/ocsp/ocsp-test.sh Removed (replaced by Python unittest).
tests/ocsp/ocsp-test.py New combined OCSP interop test suite in Python.
tests/ocsp/include.am Registers Python OCSP test in Automake.
tests/ocsp-scgi/scgi_params Removed (nginx dependency eliminated).
tests/ocsp-scgi/ocsp-scgi-test.py New Python HTTP-to-SCGI proxy test replacing nginx.
tests/ocsp-scgi/include.am Registers Python OCSP-SCGI test in Automake.
tests/hash/include.am Switches hash tests from .sh to .py.
tests/hash/hash-test.sh Removed (replaced by Python unittest).
tests/hash/hash-test.py New Python unittest for -hash and shortcut hash commands.
tests/genkey_sign_ver/include.am Switches genkey/sign/verify test registration to Python.
tests/genkey_sign_ver/genkey-sign-ver-test.sh Removed (replaced by Python unittest).
tests/genkey_sign_ver/genkey-sign-ver-test.py New Python unittest for keygen + sign/verify workflows incl. PQC.
tests/encrypt/include.am Switches encrypt tests from .sh to .py.
tests/encrypt/enc-test.sh Removed (replaced by Python unittest).
tests/encrypt/enc-test.py New Python unittest for enc/dec, interop, and legacy names.
tests/dsa/include.am Switches DSA tests from .sh to .py.
tests/dsa/dsa-test.sh Removed (replaced by Python unittest).
tests/dsa/dsa-test.py New Python unittest for dsaparam behaviors.
tests/dh/include.am Switches DH tests from .sh to .py.
tests/dh/dh-test.sh Removed (replaced by Python unittest).
tests/dh/dh-test.py New Python unittest for dhparam behaviors.
tests/dgst/include.am Switches dgst tests from .sh to .py.
tests/dgst/dgst-test.sh Removed (replaced by Python unittest).
tests/dgst/dgst-test.py New Python unittest for signature verification + large-file cases.
tests/client/include.am Switches client test from .sh to .py.
tests/client/client-test.sh Removed (replaced by Python unittest).
tests/client/client-test.py New Python unittest for external TLS connection + x509 extraction.
tests/bench/include.am Switches bench test from .sh to .py.
tests/bench/bench-test.sh Removed (replaced by Python unittest).
tests/bench/bench-test.py New Python unittest for -bench smoke tests.
tests/base64/include.am Switches base64 test from .sh to .py.
tests/base64/base64-test.sh Removed (replaced by Python unittest).
tests/base64/base64-test.py New Python unittest for base64 encode/decode + stdin coverage.
src/x509/clu_x509_sign.c Adjusts unsupported-hash handling in cert signing code path.
src/tools/clu_funcs.c Gates AES-CTR help output on wolfSSL version macro.
src/server/clu_server_setup.c Calls StartTCP() before running server test (Windows networking).
src/ocsp/clu_ocsp.c Calls StartTCP() before OCSP client/responder execution.
src/crypto/clu_decrypt.c Adjusts file-read loop behavior for decrypt path.
src/client/clu_client_setup.c Adds SNI args and calls StartTCP() before running client test.
src/client/client.c Implements Windows checkStdin() logic and enables it on Windows.
Makefile.am Adds .py test extensions and includes new testEncDec include.am.
ide/winvs/user_settings.h Enables SNI/TLS13/PKCS/CRL/OCSP features for Windows builds.
configure.ac Adds Python discovery/substitution for Automake test harness.
.gitignore Ignores Visual Studio build outputs and Python cache dirs.
.github/workflows/windows-check.yml Modernizes Windows build + runs new Python test runner.
.github/workflows/ubuntu-check.yml Removed (replaced by consolidated CI workflow).
.github/workflows/macos-check.yml Removed (replaced by consolidated CI workflow).
.github/workflows/fsanitize-check.yml Removed (replaced by consolidated CI workflow matrix).
.github/workflows/ci.yml New consolidated Ubuntu/macOS matrix build + make check.
.gitattributes Enforces LF for PEM/config files; marks DER as binary.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 1, 2026 15:36
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 93 out of 94 changed files in this pull request and generated 11 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 2, 2026 08:38
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 94 out of 95 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

src/crypto/clu_decrypt.c:166

  • The read loop now treats only XFREAD(...) > 0 as success and logs "Input file does not exist" for a 0-byte read. Because feof(inFile) is checked before attempting the read, EOF will typically be detected only after XFREAD returns 0; with this change that becomes a hard error instead of being handled as EOF. Consider checking feof()/ferror() after the read (or handling ret == 0 && feof(inFile) separately) so valid EOF conditions don’t get reported as a missing file.
            }
            else {
                ret = (int)XFREAD(input, 1, MAX_LEN, inFile);
                if (ret > 0) {
                    tempMax = ret;
                    ret = 0; /* success */
                }
                else {
                    wolfCLU_LogError("Input file does not exist.");
                    ret = FREAD_ERROR;
                }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 2, 2026 16:56
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 95 out of 96 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 2, 2026 17:27
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 95 out of 96 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 3, 2026 13:36
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 95 out of 96 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings April 7, 2026 15:07
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 97 out of 98 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (2)

wolfclu.sln:1

  • The solution file now starts with an empty first line, and VisualStudioVersion includes a non-numeric suffix (stable). .sln parsing is picky: the first line is expected to be the header, and VisualStudioVersion is typically a numeric version string. Please remove the leading blank line and ensure VisualStudioVersion is in the standard numeric format to avoid VS/MSBuild failing to load the solution.
    tests/ocsp-scgi/ocsp-scgi-test.py:1
  • The SCGI/HTTP proxy tests use fixed ports (6961/8089). This can cause intermittent CI failures if those ports are already in use on the runner (or if the suite is run concurrently). Prefer binding to ephemeral ports (bind to port 0, then read back the assigned port) and pass the chosen ports through to both the proxy and the wolfssl responder invocation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Add shared test helper and cross-platform test runner
- Extract binary lookup and run_wolfssl() into tests/wolfclu_test.py so all test files share the same logic for finding the wolfssl binary across Linux (./wolfssl) and Windows (x64/Debug/wolfssl.exe etc.).
- Add tests/run_tests.py which discovers and runs all *-test.py files, intended for Windows where `make check` is not available.
- Enable PKCS7 and CRL in MSVC
- Fix three Windows bugs uncovered by the test:
  - Add StartTCP() (WSAStartup) in client and server setup so Winsock is initialized before gethostbyname/connect calls
  - Pass SNI hostname (-S flag) to the underlying client_test so modern TLS servers accept the connection
  - Implement checkStdin() for Windows using WaitForSingleObject so s_client exits promptly when stdin is a closed pipe
- Update user_settings.h with defines required for TLS 1.3 and full wolfCLU functionality: WOLFSSL_TLS13, HAVE_HKDF, WC_RSA_PSS, HAVE_SUPPORTED_CURVES, HAVE_FFDHE_2048, HAVE_SNI.
- Fix run_wolfssl() to use stdin=DEVNULL when no input is provided, preventing subprocesses from blocking on inherited stdin (e.g. over SSH/network sessions on Windows).
- Also add WOLFCLU_SKIP_SLOW_TESTS env var to skip slow tests on Windows, and optimize large file creation to a single write.
- Add HAVE_PKCS12 to Windows user_settings.h. Skip binary DER stdin test on Windows where pipe binary mode is unreliable.
- Combine ocsp-test.sh and ocsp-interop-test.sh into a single Python test module that tests all client/responder combinations (wolfssl and openssl).
- Add StartTCP() in OCSP setup for Winsock initialization on Windows.
- Add HAVE_OCSP and HAVE_OCSP_RESPONDER to Windows user_settings.h.
- Rewrite base64 test from bash to Python unittest
- Rewrite bench test from bash to Python unittest
- Rewrite client test to Python and fix Windows networking bugs
- Replace tests/client/client-test.sh with a cross-platform Python unittest.
- Rewrite dgst test from bash to Python unittest
- Rewrite dh test from bash to Python unittest
- Rewrite dsa test from bash to Python unittest
- Rewrite enc test from bash to Python unittest
- Rewrite genkey sign/verify test from bash to Python unittest
- Rewrite hash test from bash to Python unittest
- Rewrite pkcs7/pkcs8/pkcs12 tests from bash to Python unittest
- Rewrite pkey/rsa/ecparam tests from bash to Python unittest
- Rewrite rand test from bash to Python unittest
- Rewrite server test from bash to Python unittest
- Rewrite encdec test from bash to Python unittest
- Rewrite x509/CRL tests from bash to Python unittest
- Rewrite OCSP tests from bash to Python unittest
- Rewrite OCSP SCGI test from bash to Python, drop nginx dependency
- Replace nginx + bash with a pure-Python HTTP-to-SCGI proxy using stdlib http.server and raw sockets for the SCGI netstring protocol. No external dependencies needed.
- Remove nginx from CI apt-get installs since it is no longer required for testing.
Replace fsanitize-check.yml, macos-check.yml, and ubuntu-check.yml
with a single ci.yml that uses a matrix of OS, config, and sanitizer
dimensions. Update all checkout actions to v4 and setup-msbuild to v2.
Simplify windows-check.yml: remove ${{env.GITHUB_WORKSPACE}} indirection,
drop PlatformToolset/WindowsTargetPlatformVersion overrides, add timeout,
pin wolfSSL ref, consolidate env vars. Update VS project to v145 toolset,
add missing source files to filters, enable multi-processor compilation.
- Use $(DefaultPlatformToolset) in vcxproj instead of hard-coded v145
- Remove toolset detection step from Windows CI (no longer needed)
- Disable Python tests when Python 3 not found instead of failing build
- Add -noservername flag to s_client (SNI on by default, matching openssl)
- Fix wolfclu_test.py paths to be __file__-relative for location independence
- Add trailing newline to pkcs12 stdin password inputs
- Redirect s_server stdout/stderr to DEVNULL to prevent pipe deadlock
Add test_main() helper in wolfclu_test.py that runs unittest with
exit=False and translates the result: all-skipped/no-tests -> exit 77
(automake SKIP), failures -> exit 1, otherwise -> exit 0.
wc_CamelliaSetKey was passed `block` (cipher block size, always 16)
instead of `size / 8` (key size in bytes). The `size` parameter is in
bits (128/192/256) so it must be divided by 8. The old code happened
to work for camellia-128 since block == 128/8 == 16.
- Fix decrypt lastLoopFlag to use payload size excluding salt+IV,
  preventing spurious EOF error when payload < MAX_LEN
- Handle WAIT_FAILED/WAIT_ABANDONED in Windows checkStdin()
- Catch TimeoutExpired in server-test.py cleanup, fall back to kill()
- Also exit 77 when all individual tests are skipped (not just class)
- Increase MAX_CLIENT_ARGS from 17 to 32 for worst-case arg expansion
- Remove trailing whitespace on #undef lines in clu_header_main.h
- Move SNI arg insertion to after option parsing so -noservername
  works regardless of argument order relative to -connect
- Validate input file is large enough for salt+IV before computing
  lastLoopFlag to prevent negative values on truncated files
s_client may return non-zero when peer verification fails (no CA
bundle loaded), but the connection still succeeds and the server
certificate is printed to stdout. Assert on the certificate being
present rather than requiring exit code 0.
- Use absolute paths with forward slashes for all temp files via
  _tmp() helper so wolfSSL recognizes them correctly
- Use explicit UTF-8 encoding with LF newlines when writing config
  files so wolfSSL's NCONF_load can parse them on Windows
- Remove $dir variable expansion in config templates; use direct
  absolute paths to CERTS_DIR instead
- Pass bare filenames to ca -out (not absolute paths) since wolfSSL
  prepends new_certs_dir and doesn't recognize drive-letter paths
  as absolute
- Export PROJECT_ROOT from wolfclu_test.py for test file path helpers
- Fix AES-CTR version check from > to >= 0x05009000
- Add HAVE_CAMELLIA to Windows user_settings.h
- Override XINET_PTON to use InetPtonA (narrow-string) instead of
  wolfSSL's default which incorrectly casts char* to PCWSTR
- Remove IPv6 SAN test skips now that InetPtonA works correctly
Adds WOLFSSL_SHA3, WOLFSSL_SHAKE128, WOLFSSL_SHAKE256,
HAVE_DILITHIUM, and WOLFSSL_WC_DILITHIUM to user_settings.h.
This enables ML-DSA (dilithium) keygen/sign/verify tests on Windows
using wolfSSL's native implementation (no liboqs needed).
- Check for drive-letter (X:) and backslash (\) paths as absolute in
  ca -out handling, matching OpenSSL's ossl_is_absolute_path() logic
  with #ifdef _WIN32 guard
- Add Windows-specific tests for drive-letter and backslash absolute
  paths in TestCAOutdirPath
- Enable WOLFSSL_DUAL_ALG_CERTS for chimera/altextend CA tests
- Fix MSVC VLA error: use #define for LARGE_TEMP_SZ instead of
  const int (MSVC requires compile-time constant for array sizes)
- Replace CSR attributes test: use new attributes-supported-csr.pem
  with only wolfSSL-supported attributes (challengePassword,
  unstructuredName), add test asserting unsupported attributes fail
Use load_tests protocol to filter out _OCSPInteropBase which has
CLIENT_BIN=None. Previously it showed as a confusing "binary not
configured" skip.
- Default to P-256 when no curve name is given, so the EC_KEY group
  NID is always set before EC_KEY_generate_key(). Without this,
  wolfSSL_EC_KEY_new() leaves the group as WC_NID_undef and the
  generated key's DER export is malformed.
- Increase test sign data from 15 to 20 bytes to meet wolfSSL's
  WC_MIN_DIGEST_SIZE (16) requirement in wc_ecc_sign_hash().
Copilot AI review requested due to automatic review settings April 8, 2026 10:19
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 97 out of 98 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

src/crypto/clu_decrypt.c:172

  • When XFREAD() returns 0, the code reports "Input file does not exist.". At this point the file was already opened successfully, so this message is misleading and will obscure real causes (unexpected EOF vs read error). Update this branch to emit an accurate read/EOF error and handle ret == 0 (EOF) distinctly from ret < 0/error.
                ret = (int)XFREAD(input, 1, MAX_LEN, inFile);
                if (ret > 0) {
                    tempMax = ret;
                    ret = 0; /* success */
                }
                else {
                    wolfCLU_LogError("Input file does not exist.");
                    ret = FREAD_ERROR;
                }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

/* Validate file is large enough for salt + IV header */
if (length < saltAndIvSize) {
wolfCLU_LogError("Input file too small (missing salt/IV).");
XFCLOSE(inFile);
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The early-return path when the input file is smaller than saltAndIvSize closes inFile but leaves outFile open (and already created). This can leak a handle and also leave an empty/partial output file behind on failure. Close outFile as well (and consider deleting the output path) before returning DECRYPT_ERROR here.

Suggested change
XFCLOSE(inFile);
XFCLOSE(inFile);
XFCLOSE(outFile);

Copilot uses AI. Check for mistakes.
* Matches OpenSSL's ossl_is_absolute_path() behaviour. */
if (out[0] == '/'
#ifdef _WIN32
|| out[0] == '\\' || (out[0] != '\0' && out[1] == ':')
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Windows absolute-path detection for out doesn’t match the stated intent (“Matches OpenSSL's ossl_is_absolute_path()”). In particular, treating any path with out[1] == ':' as absolute will also classify drive-relative paths like C:relative\file as absolute. Consider tightening this to isalpha(out[0]) && out[1]==':' && (out[2]=='\\' || out[2]=='/'), while keeping the \\ (UNC) case.

Suggested change
|| out[0] == '\\' || (out[0] != '\0' && out[1] == ':')
|| out[0] == '\\'
|| ((((out[0] >= 'A') && (out[0] <= 'Z')) ||
((out[0] >= 'a') && (out[0] <= 'z'))) &&
out[1] == ':' &&
(out[2] == '\\' || out[2] == '/'))

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants