feat(minio): Allow for proxies

[eddiebergman]

Reference Issue

Fixes #1159

What does this PR implement/fix? Explain your changes.

The Minio client does not automatically resolve proxies as is currently being done for requests. This PR uses requests strategy to automatically resolve the correct proxy and use this with the Minio client. With the shift to .pq files which are communcated with by Minio, this should help transition away from arff with less complaints.

Previously, Minio would do repeated calls to the download server but have long timeouts and ignore openml.config settings before finally failing and throwing an error.

How should this PR be tested?

I don't know, if you have a proxy, please use it along with this following script:

# Runs fine now
python test-proxy.py

# Will hang as we disable proxies which is apparently common in curl,wget,requests
no_proxy="*" python test-proxy.py

# test-proxy.py
import openml
import shutil
from pathlib import Path

did = 2
path = Path(openml.config.get_cache_directory()) / "datasets" / str(did)
if path.exists():
    shutil.rmtree(str(path))
    
assert not path.exists()

d = openml.datasets.get_dataset(did, download_data=True)
print(d)

assert path.exists()

I'm unfamiliar with python's networking libraries and how to test proxies in general so I excluded tests.

Any other comments?

• Proxies are handled without user knowledge with requests and so Minio will do so too. I think this makes sense but if you wish to make it more explicit somehow, please let me know
• I left a parameter proxy: str | bool in the _download_minio_file specifically to document the behavior and the user has no way to interact with this beyond environment variables (as is requests)
• I didn't think this required an example as there is non for requests and proxies, they are now alligned.
• Uni-freiburg and its cluster users would really appreciate a release with these changes! 🥂

[PGijsbers]

I'm unfamiliar with python's networking libraries and how to test proxies in general so I excluded tests.

Same here, unfortunately. A manual test at least confirms the defaults don't break when not behind a proxy 😅

Uni-freiburg and its cluster users would really appreciate a release with these changes!

I was planning a release today, let's get it in! (btw, is it correct you are not on OpenML Slack?)

[eddiebergman]

I put in your "auto" suggestion and it's looking a little cleaner, thanks for that!

Same here, unfortunately. A manual test at least confirms the defaults don't break when not behind a proxy sweat_smile

That's good at least! I would be lying if I said I actually tried it without a proxy, I was hoping the tests would all catch that :) Brief look at the failing tests makes me think it's unrelated?

(btw, is it correct you are not on OpenML Slack?)
I'm not actually, if it's relatively active I'd be thrilled to join to know what's going on!

[PGijsbers]

I'm not actually, if it's relatively active I'd be thrilled to join to know what's going on!

Not always, but it can be very convenient to quickly discuss things or ask a question :) (for example, the server issue last week, or me notifying you of the new openml-python release so you can do a quick test later)