HDDS-15535. Container Balancer should validate configuration and report startup failures to user

[sreejasahithi]

What changes were proposed in this pull request?

Currently, Container Balancer could start with configuration that made balancing impossible or misleading, with no clear reporting to the user.
This change adds startup time validation and ensures configuration errors are returned to the client with a useful message.
Problem :
Users could start the balancer with invalid or self-defeating configuration, for example:

• non existent container IDs
• all include-datanodes covered by exclude-datanodes, or all include-containers covered by exclude-containers
• fewer than two eligible healthy in-service datanodes after applying include/exclude filters
• max-datanodes-percentage-to-involve-per-iterationthat rounds down to fewer than two datanodes
• per iteration size limits where max-size-entering-target or max-size-leaving-source exceeds max-size-moved-max-per- iteration
• max-datanodes-percentage-to-involve-per-iteration set to 0, which cannot support any balancing iteration

This patch adds validation in ContainerBalancer before the balancer task starts. Failures throw InvalidContainerBalancerConfigurationException, so ozone admin containerbalancer start reports the reason to the user.

Note: The datanode check validates that at least two eligible datanodes exist and that the configured percentage allows at least two datanodes to be involved per iteration. It does not validate whether those datanodes can actually form a valid source or target, because that would require reusing much of the balancer iteration logic. Failures of that kind may still appear via ozone admin containerbalancer status (stop reason / message).

What is the link to the Apache JIRA

HDDS-15535

How was this patch tested?

Added tests
Manually tested in docker ozone cluster:

bash-5.1$ ozone admin containerbalancer start --include-datanodes ozone-balancer-datanode1-1.ozone-balancer_default,ozone-balancer-datanode2-1.ozone-balancer_default --exclude-datanodes ozone-balancer-datanode2-1.ozone-balancer_default,ozone-balancer-datanode1-1.ozone-balancer_default  -t 0.001
Failed to start Container Balancer. include-datanodes is a subset of exclude-datanodes, no datanode can participate in balancing.
bash-5.1$
bash-5.1$ ozone admin containerbalancer start --include-containers 1,2,3,4 --exclude-containers 1,2,3,4 -t 0.001
Failed to start Container Balancer. include-containers is a subset of exclude-containers, no container can be selected for balancing.
bash-5.1$
bash-5.1$ ozone admin containerbalancer start --max-datanodes-percentage-to-involve-per-iteration 20 -t 0.001
Failed to start Container Balancer. max-datanodes-percentage-to-involve-per-iteration=20 allows at most 1 datanode(s) per iteration with 6 eligible datanode(s), but at least 2 are required for a source and target datanode pair.
bash-5.1$
bash-5.1$ ozone admin containerbalancer start --max-datanodes-percentage-to-involve-per-iteration 0 -t 0.001
Max Datanodes Percentage To Involve Per Iteration must be greater than zero.
bash-5.1$

Green CI : https://github.com/sreejasahithi/ozone/actions/runs/29627948987

[sreejasahithi]

Open question for reviewers:
Should include-* and exclude-* be mutually exclusive for both datanodes and containers, or should we keep allowing both together?
According to me they could be made mutually exclusive.
The only case I can think of where both will be used at same time would be a scenario like:
automation generates include - e.g. “all nodes in rack A should be included” -> dn1,dn2,dn3,dn4,dn5. But user decides to skip a node from participating in balancing, so user adds --exclude-datanodes dn3 instead of editing the script for changing the include list.

[sreejasahithi]

@ashishkumar50 could you please review this PR.

[ashishkumar50]

@sreejasahithi Thanks for the patch, please find comments inline.

[ashishkumar50]

(int)(0.20×eligibleCount)≥2==>0.20×eligibleCount≥2==>eligibleCount≥10
So a cluster with 6 healthy in-service datanodes passes Check 1 (6 ≥ 2) but fails Check 2, because (int)(0.20 × 6) = 1 < 2.
Any cluster with fewer than 10 eligible datanodes will fail in Check 2 and fails to start, even though it has well more than 2 nodes.
If at least 2 nodes are available we should allow balancer to start and it should work.

[sreejasahithi]

The behavior when these checks are removed at startup. With 6 eligible datanodes and the default 20%, (int)(0.20 × 6) = 1, so maxDatanodesToInvolve = 1.

The balancer still starts, but in ContainerBalancerTask.doIteration() the first call to adaptWhenNearingIterationLimits() fires immediately (count=0, 0+1==1), which calls resetPotentialTargets(selectedTargets) with an empty set. That clears all potential targets before any source/target pairing happens so no moves are scheduled. The iteration ends with CAN_NOT_BALANCE_ANY_MORE and the balancer stops with "No more eligible container moves were found".

Since balancing requires at least one source and one target (2 datanodes), a cap of 1 makes balancing impossible under the configured limit. Rejecting this at startup with an explicit config error is better than starting and immediately stopping.

For clusters with fewer than 10 eligible datanodes, the user needs to raise --max-datanodes-percentage-to-involve-per-iteration. the percentage is a hard cap, and if the cap is below the minimum needed for a source/target pair the config is invalid for that cluster size.

[ashishkumar50]

Yes in this case it makes sense to stop at the start.
But in large cluster when after some iteration of balancing if eligible nodes comes down to below 10, balancer should keep running(I think this should be already working?).

[sreejasahithi]

No, if eligible datanodes drop below 10 at 20%, the balancer stops on the next iteration with CAN_NOT_BALANCE_ANY_MORE. That is pre existing behaviour in ContainerBalancerTask.

[sreejasahithi]

Correction on my earlier comment, I mixed eligible datanodes (total healthy in-service pool) with unbalanced nodes. In a large cluster unbalanced nodes dropping below 10 does not stop the balancer as the 20% cap is still computed against the full pool.

[ashishkumar50]

Should include-* and exclude-* be mutually exclusive for both datanodes and containers, or should we keep allowing both together?
Ideally should be mutually exclusive, it's good to handle but not important, we can ignore.

[sarvekshayr]

Thanks @sreejasahithi for working on this. I've given minor comments below -

[sreejasahithi]

@ashishkumar50 and @sarvekshayr I have made the changes requested, could you please review it.

[sarvekshayr]

Thanks @sreejasahithi for updating the patch. LGTM.

[ashishkumar50]

LGTM

[sarvekshayr]

Thanks @sreejasahithi for the patch and @ashishkumar50 for the review.