Calling HTTPFileSystem.get on large files looks problematic

[Koncopd]

The problem is that when HTTPFileSystem.get is called, it checks if it is a directory here, but to check that this is a directory, it downloads the whole body of the file here, and then again streams the file.

[martindurant]

If you know for sure it is a file, you can always use get_file() instead, or pass a one-item list with recursive=False to get(), I think that should work. Regardless, you make a good point, that checking for directoriness like this can be pathological - the .read() should be limited to a reasonable size, and maybe only attempted on responses that claim to be HTML or at least text-like.

Does this sound like something you'd like to fix?

[Koncopd]

Thank you, get_file() is inconvenient for a number of reasons, but passing a one-item list with recursive=True sounds like a good solution for now.
Yes, i can look into that when i have time.

btw i also see a separate problem of the default aiohttp timeout of 5 min making it impossible to download large files.

[martindurant]

Should be recursive=False, I think, to prevent the listing.

i also see a separate problem of the default aiohttp timeout of 5 min making it impossible to download large files.

If you manage to trigger this, do let me know. You can configure aiohttp via the HTTPFileSystem constructor, of course, but large files should download in blocks anyway and not hit the timeout except on problematic connections.

[martindurant]

(also: setting good defaults that work for everyone is hard!)