GH-46600: [C++][CI] Add job with ARROW_LARGE_MEMORY_TESTS enabled

[raulcd]

Rationale for this change

Now that we have self-hosted runners with AWS we should test the ARROW_LARGE_MEMORY_TESTS on CI.

What changes are included in this PR?

Added new runner for ARROW_LARGE_MEMORY_TESTS.
Fix parquet-writer-test to generate huge expected page using huge max_rows_per_page instead of default.

Are these changes tested?

Yes via CI

Are there any user-facing changes?

No

• GitHub Issue: [C++][CI] Have a job with ARROW_LARGE_MEMORY_TESTS enabled #46600

[github-actions]

⚠️ GitHub issue #46600 has been automatically assigned in GitHub to PR creator.

[raulcd]

@rok I tried with /spot=capacity-optimized, without anything and also forcing /spot=false all seem to fail due to quota:

Error: Failed to launch runner: failed to create fleet: [{"ErrorCode":"VcpuLimitExceeded","ErrorMessage":"You have requested more vCPU capacity than your current vCPU limit of 0 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit."

Should we request an increase of quota for x8i.xlarge?

[rok]

Let me take a look.

[rok]

Requested 8 vcpus for spot and 8 for on-demand.

[raulcd]

Requested 8 vcpus for spot and 8 for on-demand.

nice! Thanks @rok

[rok]

Requests were approved. Change usually needs some 10 minutes to propagate.

[raulcd]

I tried with both 64GB and 128GB machines to validate that it wasn't a RAM issue. There are a couple of test failures due to timeout:

	 96 - parquet-arrow-reader-writer-test (Timeout)
	106 - gandiva-projector-test (Timeout)

And one due to what seems like a bug on Parquet (WriteLargeDictEncodedPage):

 [ RUN      ] TestColumnWriter.WriteLargeDictEncodedPage
/arrow/cpp/src/parquet/column_writer_test.cc:1100: Failure
Expected equality of these values:
  page_count
    Which is: 7501
  2
[  FAILED  ] TestColumnWriter.WriteLargeDictEncodedPage (19975 ms)

I'll see if I can reproduce the timeouts locally.

[raulcd]

I have 64GB of RAM locally.
The TestHugeProjector.SimpleTestSumHuge from gandiva-projector-test takes more than 15 minutes locally for me when running with a Debug build.
When running on release build it takes ~3 minutes but is failing, see:

[----------] 1 test from TestHugeFilter
[ RUN      ] TestHugeFilter.TestSimpleHugeFilter
/home/raulcd/code/arrow/cpp/src/gandiva/tests/huge_table_test.cc:157: Failure
Value of: (exp)->Equals(selection_vector->ToArray(), arrow::EqualOptions().nans_equal(true))
  Actual: false
Expected: true
expected array: [
  4,
  5,
  9,
  11,
  12,
  13,
  19,
  21,
  25,
  26,
  ...
  2147483625,
  2147483627,
  2147483629,
  2147483630,
  2147483636,
  2147483637,
  2147483641,
  2147483643,
  2147483644,
  2147483645
] actual array: [
  0,
  1,
  2,
  3,
  6,
  7,
  8,
  10,
  14,
  15,
  ...
  2147483634,
  2147483635,
  2147483638,
  2147483639,
  2147483640,
  2147483642,
  2147483646,
  2147483647,
  2147483648,
  2147483649
]

[  FAILED  ] TestHugeFilter.TestSimpleHugeFilter (153849 ms)
[----------] 1 test from TestHugeFilter (153850 ms total)

For the parquet-arrow-reader-writer-test the problem is with TestArrowReaderAdHoc.LargeStringColumn, when running locally it takes on a release build ~10 minutes (haven't tested on debug but I am not sure I want to :P)

[ RUN      ] TestArrowReaderAdHoc.LargeStringColumn
[       OK ] TestArrowReaderAdHoc.LargeStringColumn (602823 ms)

The parquet-writer-test TestColumnWriter.WriteLargeDictEncodedPage and TestColumnWriter.ThrowsOnDictIndicesTooLarge also fail locally for me:

[ RUN      ] TestColumnWriter.WriteLargeDictEncodedPage
/home/raulcd/code/arrow/cpp/src/parquet/column_writer_test.cc:1100: Failure
Expected equality of these values:
  page_count
    Which is: 7501
  2

[  FAILED  ] TestColumnWriter.WriteLargeDictEncodedPage (2190 ms)
[ RUN      ] TestColumnWriter.ThrowsOnDictIndicesTooLarge
/home/raulcd/code/arrow/cpp/src/parquet/column_writer_test.cc:1147: Failure
Expected: try { ([&]() { file_writer->Close(); })(); } catch (const ParquetException& err) { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::MakePredicateFormatterFromMatcher((::testing::Property(&ParquetException::what, ::testing::HasSubstr("exceeds maximum int value"))))("err", err))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "/home/raulcd/code/arrow/cpp/src/parquet/column_writer_test.cc", 1147, gtest_ar.failure_message()) = ::testing::Message(); throw; } throws an exception of type ParquetException.
  Actual: it throws nothing.

[  FAILED  ] TestColumnWriter.ThrowsOnDictIndicesTooLarge (23736 ms)

My takes from this. We can enable a job that test the memory large tests, currently there seems to be some bugs on them, both for Gandiva and Parquet. We probably want to run on CI with a release build, in order to shorten execution time but even with that we will require like a 15 minutes timeout on individual tests.
@pitrou what are your thoughts?

Should I open individual issues for those tests?

[rok]

+1 to increasing timeouts and including them either into extras and/or release.

[raulcd]

It seems to require a really long timeout:

 [  FAILED  ] 1 test, listed below:
[  FAILED  ] TestHugeFilter.TestSimpleHugeFilter
 1 FAILED TEST
/build/cpp/src/gandiva/tests
107/107 Test  #96: parquet-arrow-reader-writer-test .............***Timeout 1200.10 sec
Running parquet-arrow-reader-writer-test, redirecting output into /build/cpp/build/test-logs/parquet-arrow-reader-writer-test.txt (attempt 1/1)
        Start  96: parquet-arrow-reader-writer-test

[pitrou]

1. We want to test in debug mode to keep all runtime checks, assertions etc., activated, but we can enable some optimizations, see [C++][CI] Have a job with ARROW_LARGE_MEMORY_TESTS enabled #46600 (comment)
2. We can just disable Gandiva, it's not really maintained anyway

[raulcd]

I've updated to use debug build and -O1 optimizations.
parquet-arrow-reader-writer-test doesn't seem to be on the original list of checked large memory tests. It takes ~25 minutes. I've updated the timeout to 30 minutes and the job was successful on CI. I've disabled Gandiva Large memory tests, they are currently failing.
The job is still failing on a couple of parquet-writer-test:

 [ RUN      ] TestColumnWriter.WriteLargeDictEncodedPage
/arrow/cpp/src/parquet/column_writer_test.cc:1100: Failure
Expected equality of these values:
  page_count
    Which is: 7501
  2
[  FAILED  ] TestColumnWriter.WriteLargeDictEncodedPage (4223 ms)
[ RUN      ] TestColumnWriter.ThrowsOnDictIndicesTooLarge
/arrow/cpp/src/parquet/column_writer_test.cc:1147: Failure
Expected: try { ([&]() { file_writer->Close(); })(); } catch (const ParquetException& err) { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::MakePredicateFormatterFromMatcher((::testing::Property(&ParquetException::what, ::testing::HasSubstr("exceeds maximum int value"))))("err", err))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "/arrow/cpp/src/parquet/column_writer_test.cc", 1147, gtest_ar.failure_message()) = ::testing::Message(); throw; } throws an exception of type ParquetException.
  Actual: it throws nothing.
[  FAILED  ] TestColumnWriter.ThrowsOnDictIndicesTooLarge (28941 ms)

I've created an issue to fix those:

• [C++][Parquet] parquet-writer-test Large memory tests fail on TestColumnWriter.WriteLargeDictEncodedPage and TestColumnWriter.ThrowsOnDictIndicesTooLarge #49502

[raulcd]

CI failures are unrelated and currently being worked on other issues.

[raulcd]

@wgtmac does this test fix sound reasonable for the LARGE_MEMORY_TEST?