LLVM 22 clang-cl on Windows fails for jit builds

[chris-eibl]

Bug report

Bug description:

When using https://github.com/llvm/llvm-project/releases/tag/llvmorg-22.1.1 and

build.bat  --experimental-jit-off -p x64 "/p:PlatformToolset=ClangCL" "/p:LLVMInstallDir=D:\dev\cpython\externals\llvm-22.1.1" "/p:LLVMToolsVersion=22"

to build the interpreter, building the stencils fails:

C:\Program Files\Microsoft Visual Studio\18\Professional\VC\Tools\MSVC\14.50.35717\include\vcruntime.h(399,51): error G58EA42A4: unknown type name 'uintptr_t'; did you mean 'intptr_t'? [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files\Microsoft Visual Studio\18\Professional\VC\Tools\MSVC\14.50.35717\include\vcruntime.h(400,67): error G58EA42A4: unknown type name 'uintptr_t'; did you mean 'intptr_t'? [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files\Microsoft Visual Studio\18\Professional\VC\Tools\MSVC\14.50.35717\include\vcruntime.h(404,8): error G58EA42A4: unknown type name 'uintptr_t'; did you mean 'intptr_t'? [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt.h(380,16): error G58EA42A4: unknown type name 'uintptr_t'; did you mean 'intptr_t'? [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(277,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(286,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(295,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(303,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(317,39): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(332,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(348,43): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(365,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(379,39): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(393,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(406,39): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(420,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(435,43): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(451,49): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\ucrt\corecrt_wstdio.h(464,39): error G58EA42A4: unknown type name 'va_list' [D:\dev\cpython\PCbuild\pythoncore.vcxproj]
EXEC : fatal error : too many errors emitted, stopping now [-ferror-limit=] [D:\dev\cpython\PCbuild\pythoncore.vcxproj]

This is because during building the interpreter, the respective include path of clang is smuggled into the INCLUDE environment variable¹:

INCLUDE=D:\dev\cpython\externals\llvm-22.1.1\llvm-21.1.4.0\lib\clang\22\include;<other include paths>

and the clang

cpython/Tools/jit/_llvm.py

Lines 13 to 14 in 2f4eb34

	_LLVM_VERSION = "21"
	_EXTERNALS_LLVM_TAG = "llvm-21.1.4.0"

used to build the jit stencils picks it up. Since these are two different versions, the hiccup is no surprise (but that it manifests in errors in vcruntime and ucrt is).

Instead of tying the two LLVM versions together like discussed in #138452, I now rather think it would be better to just empty² the INCLUDE variable when running Tools\jit\build.py. Either by adding EnvironmentVariables="INCLUDE=" to

cpython/PCbuild/regen.targets

Line 132 in 9f9faa2

<Exec Command='$(PythonForBuild) "$(PySourcePath)Tools\jit\build.py" $(JITArgs) --output-dir "$(GeneratedJitStencilsDir)" --pyconfig-dir "$(PySourcePath)PC"'/>

or by adapting Tools\jit\build.py.

@zooba, @Fidget-Spinner, @savannahostrowski: WDYT? I'd happily create a PR if this seems the correct approach to you.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

• GH-146210: Fix building the jit stencils on Windows when the interpreter is built with a different clang version #146338

Footnotes

1. I think `MSBuild support for LLVM (clang-cl) toolset does this ↩

2. I've tested locally that this fixes building the jit stencils. ↩

[zooba]

I now rather think it would be better to just empty the INCLUDE variable when running Tools\jit\build.py.

Yeah, if the JIT stencils don't rely on including anything from the compiler headers, making it ignore them seems like the best way forward. I'd put it as close as possible to where the compiler is launched from.

[chris-eibl]

Yeah, if the JIT stencils don't rely on including anything from the compiler headers

Oh, they do, the problem is that if clang 21 sees clang 22 headers "first", then all hell breaks loose: the error is just a very confusing side effect. clang "internally" finds the MS crt headers without setting INCLUDE.

I'd put it as close as possible to where the compiler is launched from.

Ok, will then send a PR shortly to pass os.environ without the INCLUDE environment variable in

cpython/Tools/jit/_llvm.py

Lines 45 to 51 in a57209e

	try:
	process = await asyncio.create_subprocess_exec(
	*command, stdout=subprocess.PIPE
	)
	except FileNotFoundError:
	return None
	out, _ = await process.communicate()