[SPARK-56763][SPARK-56535][SPARK-56810][INFRA][3.5] Recover branch-3.5 CI

[sarutak]

What changes were proposed in this pull request?

Fix the broken Base image build CI workflow on branch-3.5 by addressing multiple issues in dev/infra/Dockerfile and python/mypy.ini:

1. Stale apt package index: Add apt-get update before each apt-get install that runs in a separate RUN layer from the initial update, preventing 404 errors when packages are superseded on the Ubuntu 20.04 archive.
2. EOL Python get-pip.py URLs: Use version-specific get-pip.py URLs (pip/3.9/get-pip.py, pip/3.8/get-pip.py).
3. Pin scipy build dependencies for PyPy 3.8: Pin beniget==0.4.1 and pyproject-metadata==0.8.1 to fix scipy build failures on PyPy. beniget 0.4.2+ requires gast>=0.5.4 but pythran 0.12.x constrains gast<=0.5.3. pyproject-metadata 0.9.0 has breaking API changes incompatible with the older meson-python version.
4. Pin plotly<6.0: Plotly 6.0 introduced breaking changes in datetime handling that cause PySpark plot-related test failures. Same fix as master in SPARK-51143 / [SPARK-51143][PYTHON] Pin plotly<6.0.0 and torch<2.6.0 #49863.
5. Pin Flask==1.1.2 and Werkzeug==2.1.2: Flask is a transitive dependency of mlflow. Newer Flask requires jinja2 3.x+, conflicting with jinja2<3.0.0 installed for lint tools in the workflow. Werkzeug is a transitive dependency of Flask; newer Werkzeug requires MarkupSafe 2.1+, conflicting with markupsafe==2.0.1 pinned in the workflow.
6. Fix mypy lint failures:
   • Add has_numpy: bool = False type annotation in pyspark/sql/utils.py to fix Cannot determine type errors.
   • Add follow_imports = skip for pydantic and sqlalchemy in mypy.ini to prevent mlflow's transitive dependencies from affecting mypy output.
   • Update test expectations in test_feature.yml and test_functions.yml for mypy output format changes (def __init__ instead of class name, positional-only argument __cols display).
7. Fix cleanClosure for R 4.4+ (SPARK-56810): Skip primitive functions in SparkR's cleanClosure/processClosure. R 4.4+ made environment<- on primitive functions a hard error (previously a warning), causing SparkR RDD operations to fail.
8. Fix R CRAN check NOTE threshold: R 4.4+ checkRd now reports "Lost braces" in Rd files (e.g., \url{URL}{text}) as a NOTE. Since SparkR is deprecated as of Spark 4.0, raise the NOTE tolerance from 2 to 3 rather than fixing the Rd files.
9. Pin ragg==1.2.5 in the workflow: ragg 1.5.x added libwebp as a new dependency and its configure script fails to find freetype2 headers on Ubuntu 20.04. ragg 1.2.5 works with the existing system libraries using pkg-config.
10. Misc: Update FULL_REFRESH_DATE to force a full image rebuild.

Why are the changes needed?

The branch-3.5 CI has been broken for an extended period. The Ubuntu 20.04 (focal) base image is aging, and upstream package repositories have rotated or removed packages that the Dockerfile previously fetched without version pins. Additionally, the CI now uses R 4.4+ which introduced a breaking change affecting SparkR's closure serialization. Multiple unrelated failures compound to make the CI completely non-functional.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Base image build workflow passes on GitHub Actions.
• docker build dev/infra succeeds locally.

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Opus 4.6

[sarutak]

Once Base image build passes on GA, I'll fix Linters, licenses, dependencies and documentation generation workflow.

cc: @dongjoon-hyun @gaogaotiantian @zhengruifeng

[gaogaotiantian]

I don't quite understand

RUN printf 'beniget==0.4.1\npyproject-metadata<0.9.0\n' > /tmp/pypy-constraints.txt && \
    PIP_CONSTRAINT=/tmp/pypy-constraints.txt pypy3 -m pip install numpy scipy coverage matplotlib && \
    SETUPTOOLS_USE_DISTUTILS=stdlib pypy3 -m pip install 'pandas<=2.0.3' && \
    rm /tmp/pypy-constraints.txt

Why pip constraint? Are these package needed by some other packages? Can we just pin the package we need to a specific version? It's an infra docker.

[sarutak]

Why pip constraint? Are these package needed by some other packages? Can we just pin the package we need to a specific version? It's an infra docker.

scipy will be built using meson and beniget==0.4.1 and pyproject-metadata<0.9.0 are required during the build.

[gaogaotiantian]

Usually the reason CI broke is that a new version of something used some new stuff. It's an old branch so it has to be working at certain point of time. Could you show some evidence to support your fix? Like why the specific version 0.4.1 and 0.9.0? What is the dependency chain? What did not work when we didn't pin the version?

[sarutak]

I'll pin pyproject-metadata==0.8.1.

Could you show some evidence to support your fix? Like why the specific version 0.4.1 and 0.9.0? What is the dependency chain? What did not work when we didn't pin the version?

Regarding beniget

scipy uses pythran (>=0.12.0,<0.13.0) during its build process. pythran 0.12.x requires gast<=0.5.3 and beniget~=0.4.0 as dependencies.

• pythran 0.12.2 requirements.txt: https://github.com/serge-sans-paille/pythran/blob/0.12.2/requirements.txt

beniget 0.4.2 and later added code to handle Python 3.10 match statement AST nodes (MatchStar, MatchAs), but these nodes were only implemented in gast 0.5.4. Since pythran constrains gast to <=0.5.3, the combination of beniget 0.4.2 + gast 0.5.3 results in the following error:

AttributeError: module 'gast' has no attribute 'MatchStar'

• beniget issue (bug report): AttributeError: module 'gast' has no attribute 'MatchStar', when gast < 0.5.4 installed serge-sans-paille/beniget#108
• gast 0.5.4 diff where MatchStar was added: serge-sans-paille/gast@0.5.3...0.5.4#diff-44b67094998a7df1e5ce843b5a2faa44950f5896ac9e07ab99d09e2b24c90143R328

Upgrading gast to 0.5.4 or later is not possible because it violates pythran's constraint. Pinning beniget==0.4.1 is the only solution.

Regarding pyproject-metadata

scipy's build system, meson-python, uses pyproject-metadata for metadata processing. pyproject-metadata 0.9.0 introduced breaking changes for PEP 639 support and implicitly requires packaging>=23.2 (this requirement is not correctly recorded in the package metadata due to a bug).

The older version of meson-python used in the Dockerfile environment is not compatible with the API changes in pyproject-metadata 0.9.0, causing the build to fail at the metadata generation stage. Pinning pyproject-metadata==0.8.1 maintains compatibility with the existing meson-python version.

• meson-python changelog (notes that 0.18.0 bumped the minimum pyproject-metadata requirement to 0.9.0, and mentions the packaging version requirement bug in 0.9.1): https://mesonbuild.com/meson-python/changelog.html

[sarutak]

As of Spark 4.0, SparkR is deprecated. So just applying this change to branch-3.5 in this PR.
Do we need to apply this change for other branches including master?
CRAN seems to provide only the latest version of R. So it's difficult to pin to a older version, and this change is necessary at least for branch-3.5 for CI.

[dongjoon-hyun]

cc @holdenk since she is the next release manager for Apache Spark 3.5.x

[sarutak]

For reviewers:

There is another PR which is trying to address the same issue as this PR with different approach.
#55432

[holdenk]

Mind sharing why this one @sarutak ? I'm tempted to just go with the other fix which I wrote and started before and also deals with the flakiness we've seen with the PPAs the last few weeks

[sarutak]

Hi @holdenk, I'm thinking it better to recover CI as soon as possible. So, rather than trying to create a perfect solution from the start, I prioritized creating a solution with minimal difference from the original CI configurations.

[holdenk]

That does seem beneficial, we could layer the fixes I suppose. I am cautious because of the PPA flakes though.

[holdenk]

Looking at my OG pr it seems the only failure is transient so I'll probably merge that one, but I'll cherry pick in some of the pining here to re-enable Py3.8 back pining if that works for you @sarutak (will show co-authored by) unless you want to rebase to re-enable Py3.8 yourself?

[sarutak]

Either is fine. I’ll leave it up to the reviewer’s discretion. If I have one concern, it’s that your PR hasn’t passed the R tests.

[holdenk]

True I punted that one to a follow up instead and did the PPA issue first disabling R with a follow up to switch R build back to the supported version of R. ugh the merge conflict is going to happen either way. Ok I'll do something today to get these in.

[gaogaotiantian]

Wait what happened on the type tests? Are those related? My main concern is that we should not need the pip restriction file. We should be able to just pin those like others.

[holdenk]

Type tests came from non-pinned dependencies changing their typing and we depend on an old version of mypy (upgrading mypy would involve large changes). I'm going to rebase this on top of the other fix to re-enable R (working on it over at #55886 )

I think we should just disable Py3.8 type tests personally.

[gaogaotiantian]

Do we want to apply all the fixes in a single PR to recover the 3.5 CI? Maybe the up side is that we can confirm the CI is working properly? It would probably be more managable if we can fix issues iteratively - if we had issues in the future, revert would be simpler.

[holdenk]

So I was doing that then two other folks made incompatible PRs so I'm going to iteratively land the fixes today

[holdenk]

I think provided we don't merge non CI fixes until all of CI is fixed the approach should be fine

[gaogaotiantian]

Yeah I agree that we should fix the CI before we merge other changes. I also think that with this example, we probably should not reduce the CI coverage to this level even if the branch is stable. I think we are having plenty of troubles releasing 3.5. We should try to avoid that for future releases.

[sarutak]

Oh, you've merged that PR without getting approval, even though that wasn't a small change.

[holdenk]

Oh, you've merged that PR without getting approval, even though that wasn't a small change.

I am running the 3.5 release right now, it's been open for almost a month without meaningful comment, and we keep seeing other people raising different conflicting solutions. If you think it should be reverted though let me know.

[holdenk]

Yeah I agree that we should fix the CI before we merge other changes. I also think that with this example, we probably should not reduce the CI coverage to this level even if the branch is stable. I think we are having plenty of troubles releasing 3.5. We should try to avoid that for future releases.

I think it's a good thing we should probably pay attention to the periodic CI failures on "old" branches more going forward yes. I think this was especially bad because of the combination of upstream infra issues (PPA breakage) non-pinned language version (R is floating unlike Python Scala and Java). For R I have a PR which would pin to a particular version of R and we could consider similar in other branches but didn't want to block CI restoration on that since I think that's a larger conversation.