Skip to content

HDDS-15949. Handle split MPU part counting for abort batch sizing and container-key-mapping CLI - #10849

Merged
sarvekshayr merged 10 commits into
apache:masterfrom
spacemonkd:HDDS-15949
Jul 28, 2026
Merged

HDDS-15949. Handle split MPU part counting for abort batch sizing and container-key-mapping CLI#10849
sarvekshayr merged 10 commits into
apache:masterfrom
spacemonkd:HDDS-15949

Conversation

@spacemonkd

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

HDDS-15949. Handle split MPU part counting for abort batch sizing and container-key-mapping CLI

Please describe your PR in detail:

Schema-aware MPU part counting for abort batch sizing (OmMetadataManagerImpl.getExpiredMultipartUploads(), KeyLifecycleService)

For split-schema MPUs (schemaVersion == 1), part count now comes from multipartPartsTable via OMMultipartUploadUtils.countParts() instead of getPartKeyInfoMap().size() (which is always 0 for split MPUs).

Without fix the expired MPU cleanup and lifecycle AbortIncompleteMultipartUpload treated every split MPU as having 0–1 parts when sizing abort batches. Batches could exceed the intended part limit (maxParts / mpuAbortLimitPerTask), causing larger-than-planned abort work per iteration. The abort logic itself was already schema-aware; only batch sizing was wrong.

Shared countParts() helper (OMMultipartUploadUtils)

Added countParts(omMetadataManager, uploadId) wrapping the existing cache-aware scanParts() logic.
Without fix it is the same as above, plus duplicated/incorrect counting if each call site reimplemented split-table reads differently from abort/complete paths.

Schema-aware split MPU handling in debug CLI (ContainerToKeyMapping)

processMultipartUpload branches on schema version:

  • legacy MPUs still use embedded PartKeyInfo
  • split MPUs scan multipartPartsTable by uploadId and map block containers from each part’s locations.
    Without fix the ozone debug om container-key-mapping --in-progress reported no open MPU keys for split-schema uploads, even when their parts used the queried containers, this causes misleading output for debugging container to key mappings on upgraded clusters.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15949

How was this patch tested?

Patch was tested via unit tests

@spacemonkd spacemonkd self-assigned this Jul 23, 2026
@spacemonkd
spacemonkd marked this pull request as ready for review July 23, 2026 10:22

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @spacemonkd for handling this case. Added a few comments, pls check it.

@spacemonkd
spacemonkd requested a review from rakeshadr July 23, 2026 12:55
@spacemonkd
spacemonkd requested a review from sumitagrawl July 24, 2026 07:13

@sarvekshayr sarvekshayr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @spacemonkd for working on this.

Comment on lines +358 to +359
String uploadId = multipartInfoDbKey.substring(
multipartInfoDbKey.lastIndexOf(OM_KEY_PREFIX) + OM_KEY_PREFIX.length());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the below approach instead of manual substring parsing -

Suggested change
String uploadId = multipartInfoDbKey.substring(
multipartInfoDbKey.lastIndexOf(OM_KEY_PREFIX) + OM_KEY_PREFIX.length());
String uploadId;
try {
uploadId = OmMultipartUpload.from(multipartInfoDbKey).getUploadId();
} catch (IllegalArgumentException e) {
err().println("Invalid multipartInfoTable key " + multipartInfoDbKey + ", " + e);
return matchedContainers;
}

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few minor comments. Please look at other comments and resolve once its fixed. Thanks!


@Test
public void testContainerToKeyMappingWithSplitSchemaMPU() {
int exitCode = execute("--containers", String.valueOf(CONTAINER_ID_4), "--in-progress");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a bug, but using a distinct CONTAINER_ID_5 for the new tests would make the split-schema test self-contained and easier to reason about in future.

CONTAINER_ID_4 for both legacy and split MPU testContainerToKeyMappingWithSplitSchemaMPU

// per-MPU skip — bucket loop continues
}
}
assertEquals(2, list.size());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add comments to help maintainers.

 // A (2 parts) and C (1 part) are added; B is skipped due to IOException
 assertEquals(2, list.size()); // uploadA and uploadC only
 assertEquals(3, list.getPartCount()); // 2 (uploadA) + 1 (uploadC)

@sarvekshayr sarvekshayr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@rakeshadr rakeshadr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1LGTM, thanks @spacemonkd for taking care this case.

@sarvekshayr
sarvekshayr merged commit ca0715a into apache:master Jul 28, 2026
45 checks passed
@sarvekshayr

Copy link
Copy Markdown
Contributor

Thanks @spacemonkd for the patch and @rakeshadr for the review.

@spacemonkd

Copy link
Copy Markdown
Contributor Author

Thanks @sarvekshayr for the reviews and merging this PR and @rakeshadr for the reviews.

@spacemonkd
spacemonkd deleted the HDDS-15949 branch July 28, 2026 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants