Skip to content

Add S3 integration tests - #2067

Open
PeterDowdy wants to merge 2 commits into
samtools:developfrom
PeterDowdy:develop
Open

Add S3 integration tests#2067
PeterDowdy wants to merge 2 commits into
samtools:developfrom
PeterDowdy:develop

Conversation

@PeterDowdy

@PeterDowdy PeterDowdy commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

This PR adds S3 integration tests through minIO as a mocked S3 provider. It covers most of the happy and sad paths in the project, with moderate rigour. It doesn't check transient errors (since the library just dies on them anyways), TLS verification (since that seems like it's drifting out of testing S3), or very large synthetic files (to keep runtime short).

The minIO test depends on an env var so developers without minIO can safely bypass it.

This effort surfaced a few fairly small gaps in S3 file-handling that could be filled:

  • No retries (as mentioned)
  • A single failure in multipart aborts the whole multipart operation (related)
  • SigV2 write unimplemented (seems pretty legacy though)
  • CRAM UR: tag can't reach S3
  • CRAM reference caching only ever uses local disk (not so much a gap as a possible useful feature)
  • bgzip CLI can't write to S3
  • Loading a .crai always downloads it to disk

Assisted-by: Claude:claude-sonnet-5

… running these tests against minio and added a ci step

Signed-off-by: Peter Dowdy <peter.dowdy@gmail.com>
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: Peter Dowdy <peter.dowdy@gmail.com>
Assisted-by: Claude:claude-sonnet-5
@whitwham

Copy link
Copy Markdown
Member

Thank you for this. I won't have time to look at it for the next few weeks but I will get round to it.

@PeterDowdy

Copy link
Copy Markdown
Contributor Author

Thank you for this. I won't have time to look at it for the next few weeks but I will get round to it.

No rush!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants