API: Fix AllowMissing bug and Add Unit tests for StructProjection class

[Govindarajan-D]

Added Unit tests that improves the reliability of StructProjection class.

1. Subset projection from a flat schema
2. Nested projection across multiple levels
3. Optional field projection (schema evolution)
4. Map projection with nested value and Primitive Types
5. List projection with nested value and Primitive Types
6. Map and List Partial Projection
7. And other small functions

Code coverage from 0% to 100% (Based on Jacoco report)

Fixes #16123

[pvary]

Thanks @Govindarajan-D for the PR. Left some minor formatting comments

[Govindarajan-D]

@pvary Resolved the mentioned formatting corrections. Thanks much!

[nastra]

let's omit test as a prefix in new test names as they don't add any value

[nastra]

Suggested change

	StructProjection projectedStructure = StructProjection.create(dataSchema, projectedSchema);
	StructProjection projection = StructProjection.create(dataSchema, projectedSchema);

[nastra]

this tests only the happy path. Let's also add something where we try to access a position that isn't valid and such. Another test method could also call set() and verify that the expected exception is thrown

[nastra]

we should probably also check projectedFieldIds() in all of the tests

[nastra]

Suggested change

	StructType coordinateStruct =
	StructType coordinatesStruct =

[nastra]

A few more things that would be good to cover as well:

• create(Schema, Set<Integer>) / copyFor(StructLike) are currently not tested
• Missing required field (without allowMissing) should throw IllegalArgumentException
• Partial map-value/list-element projection should throw IAE and we should most likely test this

[Govindarajan-D]

Will add more tests as planned

[Govindarajan-D]

@nastra @pvary Added more tests covering map and list with partial projection and testing few other small functions. The code coverage is at 100% as per JaCoCo report.

[Govindarajan-D]

Fixes #16123

[nastra]

let's omit test as a prefix in new test names as they don't add any value

[Govindarajan-D]

@nastra Removed the test prefix

[pvary]

In many cases the projected schema could be calculated by TypeUtil.select/selectNot.

For example:

Schema projectedSchema = TypeUtil.select(dataSchema, Sets.newHashSet(2, 5));

or

Schema projectedSchema = TypeUtil.selectNot(dataSchema, Sets.newHashSet(1));

Please check other places too?
I think the selectNot is actually quite intuitive in projection scenarios

[Govindarajan-D]

But we do have tests that are not exact subset of dataSchema right. So I would have to use a mix of select/selectNot with normal definition which would not be uniform. Let me know what you think.

[pvary]

In most of the test either the schema, or the projected schema could be generated by a single select or selectNot from the other. select is easier if we want only a few columns, selectNot is easier if we just want to remove a few columns.

Just try it in a few places and see if you like it, and based on the results work on the other ones

[pvary]

What is the difference between the dataSchema and the projectedSchema?

[Govindarajan-D]

There is no difference. This is for testing the nested structure, so re-used the same structure with nested Map.

[pvary]

So we can do a:

StructProjection projection = StructProjection.create(dataSchema, dataSchema);