fix(sparksql): Default ignoreNulls to true for collect_set backward compatibility

[yaooqinn]

Summary

Fixes a backward compatibility bug introduced in PR #16416.

The ignoreNulls_ field in SparkCollectSetAggregate was defaulting to false (RESPECT NULLS). When the 1-arg signature collect_set(T) is used, setConstantInputs() does not receive a boolean constant, so the default value is used — which must match Spark's default behavior of ignoring nulls (true).

Root cause

// Before (broken): includes nulls by default
bool ignoreNulls_{false};

// After (fixed): ignores nulls by default (Spark's default)
bool ignoreNulls_{true};

Impact

Without this fix, any downstream consumer (e.g., Gluten) using the native collect_set with the 1-arg signature would get null elements in the output array, causing NullPointerException during Spark's result projection.

Testing

Verified in Gluten with VeloxAggregateFunctionsDefaultSuite — all 16 collect_set/collect_list tests pass after this fix.

Related: Gluten PR apache/gluten#11837

[netlify]

✅ Deploy Preview for meta-velox ready!

Name	Link
🔨 Latest commit	889d8d5
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/69cd37db07bb570008b483cc
😎 Deploy Preview	https://deploy-preview-16947--meta-velox.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

Build Impact Analysis

Selective Build Targets (building these covers all 5 affected)

cmake --build _build/release --target spark_aggregation_fuzzer_test velox_functions_spark_aggregates_test velox_spark_query_runner_test velox_sparksql_coverage

Total affected: 5/556 targets

Affected targets (5)

Directly changed (2)

Target	Changed Files
velox_functions_spark_aggregates	CollectSetAggregate.cpp
velox_functions_spark_aggregates_test	CollectSetAggregateTest.cpp

Transitively affected (3)

• spark_aggregation_fuzzer_test
• velox_spark_query_runner_test
• velox_sparksql_coverage

---

Fast path • Graph from main@24e6ab97ba8015c3b6fae82e8184047cadf521df

[jinchengchenghh]

Please don't include non-related change

[yaooqinn]

Rebased on latest main — removed unrelated changes from the diff. Now only the 1-file fix (CollectSetAggregate.cpp).

[rui-mo]

Can you please add test for verify the default behavior? Thanks.

[yaooqinn]

Added defaultIgnoreNulls test verifying the 1-arg collect_set(c0) ignores nulls by default (Spark's default behavior). All 10 tests pass. Thanks @rui-mo!

[rui-mo]

Intermediate results already have null filtering applied by the partial step.

Velox supports flushing during partial aggregation. When this happens, the intermediate results are left unaggregated, with the final aggregation step responsible for processing them. Could this cause any result issues?

[yaooqinn]

Good point! This is safe because:

1. When partial flushes (toIntermediate is called), null filtering is already applied there:

   • ignoreNulls_=true: null inputs become empty arrays (size=0), so the intermediate output contains no null elements
   • ignoreNulls_=false: Base::toIntermediate wraps each value (including nulls) into [value] arrays

2. Final step receives pre-filtered data: Since toIntermediate already handles null filtering, addIntermediateResults just needs to merge arrays — using addValues (preserve everything) is correct for both cases.

3. No behavior change: Before this fix, the final/intermediate nodes had ignoreNulls_={false} by default, which also always used addValues in the intermediate path. My change makes this explicit and removes the dead addNonNullValues branch that was never reachable in the final step.

[rui-mo]

Thanks.

[meta-codesync]

@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this in D99321794.

[meta-codesync]

@bikramSingh91 merged this pull request in 1dfcfbb.