HBASE-29364 Fix FAILED_OPEN state during master restart

[SwaraliJoshi]

https://issues.apache.org/jira/browse/HBASE-29364

[Apache9]

Is it possible to reproduce the problem in a real mini cluster with a real TRSP?
We need to confirm that this could happen in real production.

[SwaraliJoshi]

Yes, I did have a test to confirm this issue before fixing it. Here is the code :

1. Mini Cluster test -
   TestFailedOpenRestoredAsOpenOnFailover.java

2. Replication of exact bug manually -
   TestOpenRegionProcedureRestoreFailedOpen.java

These two assert the pre-fix (buggy) behavior — that a FAILED_OPEN ends up OPEN. With the fix currently in place, they will fail. They're bug-demonstration artifacts, not regression guards.

[Apache9]

Let's include them in this PR too.

[SwaraliJoshi]

But these tests will fail. Or should I instead have assertEquals(State.OPENING, regionState(newMaster, hri)); ?

Update : I have added the test to assert the state is marked as OPENING

[virajjasani]

Build failure:

Banned imports detected:
Reason: Use junit5 instead
	in file: org/apache/hadoop/hbase/master/assignment/TestOpenRegionProcedureRestoreFailedOpen.java
		static org.junit.Assert.assertNotEquals (Line: 20, Matched by: static org.junit.Assert.**)

[virajjasani]

Instead of this log, we can assert the region state with expected value

[SwaraliJoshi]

@virajjasani Updated sir, could you please check now?

[Apache9]

I think the real harm here is that, the region is recorded as OPEN on the target regionserver but actually it is not? So the end to end test should make sure that the region is finally online and we can read/write it? Or we get the region location in meta, and then check with the region server directly about whether the region is on it.