Initial guidance on a PR to resolve the Accept-Ranges TODO

[itcarroll]

I'd like to help resolve a TODO which could help earthaccess and the NASA Ocean Biology DAAC play nicely, at least if the resolution isn't beyond my skills (which do fall short of understanding async very well). Are there thoughts on a resolution here:

filesystem_spec/fsspec/implementations/http.py

Lines 837 to 840 in 527728d

	# TODO:
	# recognise lack of 'Accept-Ranges',
	# or 'Accept-Ranges': 'none' (not 'bytes')
	# to mean streaming only, no random access => return None

The TODO has come to my attention because this DAAC does not accept ranges ... and is kinda aggressive 'bout it. If you try to GET a range, the response status is 416 (even if the range is within the file size). Everything goes downhill from there, including HTTPFile.async_fetch_range quietly returning b"". I don't have an informed opinion about the proper solution, but would try to help implement if there's any guidance. In the end, I'd like the following (which uses HTTPFileSystem) to automatically avoid range requests:

import earthaccess

earthaccess.login()
results = earthaccess.search_data(short_name="VIIRSJ1_L2_OC", count=1)
paths = earthaccess.open(results)
paths[0].read(8)

It looks like the plan is to use HTTPStreamFile rather than HTTPFile when Accept-Ranges is missing or none. Is there some subtlety to implementing that solution I should understand?

[itcarroll]

If a more targeted "fix" (for my use case) is advised, retrying with an HTTPStreamFile from here seems reasonable:

filesystem_spec/fsspec/implementations/http.py

Line 650 in 527728d

if r.status == 416:

[martindurant]

I don't expect servers to necessarily tell the truth about whether they accept range requests or not :|

[itcarroll]

What about respecting "Accept-Ranges: 'none'" but not changing anything based on the lack of "Accept-Ranges"?

[martindurant]

That sounds reasonable. Trust the message when it is explicit.