HIVE-30026: (ACID Compaction) Extend the query for finding potential compactions with ordering

[kuczoram]

What changes were proposed in this pull request?

I propose to extend the SQL queries which are fetching the potential compaction with an oder by CTC_TXNID. Since the TXNID is an always increasing number, this way we can be sure that the older entries in the COMPLETED_TXN_COMPONENTS tables are fetched first. This way we can make sure that all tables will get into the compaction cycle at once.
Also propose to apply separate limit for the aborted txn cleaup. If the fetch limit is smaller then the compaction candidates, the aborted txns will never be fetched.

Why are the changes needed?

After HIVE-27444, if the number of candidates is bigger than the compactor.fetch.size value, it can happen that the Initiator doesn't schedule all tables for compaction.

For example: we have 4 tables, which are all eligible for compaction, but the fetch size is 2.
In the first initiator run, it will pick two of these tables. It would be expected that during the second run of the initiator it would pick the other two tables, so all 4 tables have been scheduled for compaction.

Since there is no ordering when selecting the tables for compaction, this was not always the case. It could happen that some tables never got compacted.

Also if the number of candidates is bigger than the compactor.fetch.size limit, the aborted txn candidates will never be picked up by the initiator. Because the "remaining budget" will always be 0 in the FindPotentialCompactionsFunction.

Set candidates = new HashSet<>(jdbcResource.execute(new CompactionCandidateHandler(lastChecked, fetchSize)));
int remaining = fetchSize - candidates.size();
if (collectAbortedTxns && remaining > 0) {
candidates.addAll(jdbcResource.execute(new AbortedTxnHandler(abortedTimeThreshold, abortedThreshold, remaining)));
}
This affects only the cases when the aborted txn clean-up is done by the compaction cycle and not just the cleaner.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

[kasakrisz]

Can abortedThreshold be set to 0?

[kuczoram]

Yes, it can. We can use 0 and in this case one aborted transaction would be enough. I changed a code like this.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud