[GLUTEN-11862][VL] Work around GMT session timezone validation failure on macOS#11869
Open
malinjawi wants to merge 3 commits intoapache:mainfrom
Open
[GLUTEN-11862][VL] Work around GMT session timezone validation failure on macOS#11869malinjawi wants to merge 3 commits intoapache:mainfrom
malinjawi wants to merge 3 commits intoapache:mainfrom
Conversation
malinjawi
changed the title
|
|
||
| void WholeStageResultIterator::addIteratorSplits(const std::vector<std::shared_ptr<ResultIterator>>& inputIterators) { | ||
| GLUTEN_CHECK(!allSplitsAdded_, "Method addIteratorSplits should not be called since all splits has been added to the Velox task."); | ||
| GLUTEN_CHECK( |
Contributor
There was a problem hiding this comment.
Please remove unrelated changes
| const auto sessionTimezone = | ||
| normalizeSessionTimezone(it == confMap.end() ? std::string_view("UTC") : std::string_view(it->second)); | ||
| std::unordered_map<std::string, std::string> configs{ | ||
| {velox::core::QueryConfig::kSparkPartitionId, "0"}, {velox::core::QueryConfig::kSessionTimezone, "GMT"}}; |
Contributor
There was a problem hiding this comment.
update the GMT to other value is enough
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Fixes #11862.
This PR works around a macOS-specific native validation failure in the Velox backend when the effective Spark session timezone is
GMT.Based on the discussion in #11862, the underlying behavior is likely related to upstream Folly/Velox timezone handling on macOS rather than a general Spark semantics problem. This patch keeps the workaround on the Gluten side so Spark session timezone values do not block native validation and execution in the Velox path.
The changes in this PR are:
GMTandGMT+/-offsetsession timezone values toUTC-equivalent forms before passing them into native Velox query configSubstraitToVeloxPlanValidatorinstead of hardcodingGMTspark.sql.session.timeZone=GMTHow was this patch tested?
session 'session_timezone' set with invalid value 'GMT'MiscOperatorSuite:cast date to timestamp with GMT session timezoneWas this patch authored or co-authored using generative AI tooling?
Generated-by: IBM BOB