Apply CURL_CA_BUNDLE environment variable in the S3 backend too

[jmarshall]

Refactor the repeated trivial CURLOPT setting into a subroutine, and additionally reinstate setting CURLOPT_CAINFO via the curl(1) tool's CURL_CA_BUNDLE environment variable, which the S3 code previously listened to via hfile_libcurl.c code.

[jmarshall]

Added a second one-liner to make hfile_libcurl.c handle curl_global_init() failure the same way as hfile_s3.c. These days easy_errno() uses curl_easy_strerror() in a log message, which actually caused a crash in conjunction with some upcoming pysam changes!

[daviesrob]

Thanks. Out of interest, how did curl_easy_strerror() fail?

[jmarshall]

I suspected that might pique your interest… 😄

Python wheels like to bundle the more exotic system libraries to make themselves portable across Linux distros. For a constellation of reasons (not least that this bundling is IMHO misguided), I'm changing pysam to omit libcurl and libcrypto — and the dozen others they drag in — from its wheels by dynamically loading them. (Sadly even building hfile_libcurl.so et al as htslib plugins wasn't enough to defeat the bundling machinery.)

So I have a wrapper around curl_global_init() that dlopens libcurl and libcrypto and sets up a table of function pointers for all the library routines that hfile_*.c are going to call. But when that wrapper fails and returns CURLE_FAILED_INIT (because it was dlopening libcrypto.so.3 on a platform that only had libcrypto.so.1.1… d'oh!), hfile_libcurl.c's subsequent error handling called curl_easy_strerror which at that point was a function pointer that was NULL… 💥

So technically it wasn't the real curl_easy_strerror that actually failed… but the point remains that the libcurl documentation says not to do this.

(BTW I've since refactored it to set up libcurl and libcrypto separately.)

This means that I won't be merging pysam-developers/pysam#1415 so pysam doesn't have an urgent need for the first commit of this PR after all. However it's still good to reinstate the CURL_CA_BUNDLE functionality for S3.

[daviesrob]

Ah, yes. I thought it might be something related to library loading.