Conversation
…its statistics, not by listing its location Electing the big table for an auto sort-merge join sizes every candidate. For a partitioned table that loop called getSize(conf, partition), which falls through to fs.getContentSummary(partition.getDataLocation()) when the partition holds no totalSize. A storage handler keeps no partition parameters, so every partition took that path - and a synthesised partition's location is effectively the whole table, making it a full recursive listing per pruned partition. SizeBasedBigTableSelectorForAutoSMJ now asks the handler: - getSize(conf, Table) takes the handler's own size for a non-native table rather than listing its path; - a new getSize(conf, Table, List<Partition>) batches the pruned partitions through HiveStorageHandler.getAggrBasicStatsFor, falling back to the table's own size when any partition cannot be sized - more than the scan reads, never less, so the elected big table is never under-sized; - TableSizeBased* and AvgPartitionSizeBased* use that overload in place of the per-partition loop. Sizes are parsed through NumberUtils.toLong, which returns the default for a null string, so neither the size nor the storage handler needs a null guard: every caller of handlerSize is already isNonNative(), and getStorageHandler() resolves or throws for such a table. ConvertJoinMapJoin no longer pays for that sizing when the join cannot be an SMB at all. checkConvertJoinSMBJoin is split into checkJoinSMBFeasibility, which decides from traits alone and so does not depend on which side is elected, and is consulted before the election; the numBuckets derivation that used the elected position moves to the caller, and the full-outer joinCandidates isEmpty() guard moves ahead of the selector lookup. Testing: new TestSizeBasedBigTableSelectorForAutoSMJ covers handler sizing without filesystem access, batched partition sizing, the unsized-partition fallback, and the unchanged native path. No plan output changes. Claude-Session: https://claude.ai/code/session_013Di92MyTywhTgbc5ZMwxWx
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Sort-merge join should size a storage-handler table from its statistics, not by listing its location
Why are the changes needed?
Electing the big table for an auto sort-merge join sizes every candidate. For a partitioned table that loop called getSize(conf, partition), which falls through to fs.getContentSummary(partition.getDataLocation()) when the partition holds no totalSize. A storage handler keeps no partition parameters, so every partition took that path - and a synthesised partition's location is effectively the whole table, making it a full recursive listing per pruned partition.
Does this PR introduce any user-facing change?
How was this patch tested?
TestSizeBasedBigTableSelectorForAutoSMJ