Support loading from URLs in Python

[hyanwong]

For teaching purposes, and possibly for other reasons too, I think it would be really useful to be able to load a tree sequence from a URL. IMO the nicest way to make this easy for a user would be to have a url argument to tskit.load:

tskit.load(url="https://tskit.dev/tutorials/data/basics.trees")

I would hope that this would mainly be used for small tree sequences, although I suppose it could be an easy way for someone to load up larger ones e.g. from zenodo - depends how long that would take I guess.

I suspect implementing this using the urllib.request library would be quite easy, although I don't know how we would unit test it - probably mock the urllib.request.urlopen function when testing, or something.

[hyanwong]

NB: this would mean that content in e.g. jupyterbooks could simply be pasted into the user's jupyter session and (as long as the right libraries were installed), the examples should all run. It's less error-prone and fiddly than trying to run the jupyterbook e.g. in binder (see tskit-dev/tutorials#114)

[jeromekelleher]

The right way to do this I think would be to make the string form of the file input support URLs, which we intercept in Python. BUT this would be quite a lot of work to do well (it's simple to do badly) so I'm not super keen on it right now.

[hyanwong]

Just to say that I have wanted to do this for a while, for other purposes too (not just the tutorials), so I am keen to get something like this done. But maybe others could weigh in if they have needed to do this in the past (as it could just be me).

Out of interest, why would it be a lot of work to do well? I would have thought the standard URL libraries would do most of the hard lifting?

[hyanwong]

Presumably the workaround for the moment is simply e.g.

import urllib.request
file = urllib.request.urlopen('file:///Users/yan/Documents/GitHub/tutorials/data/basics.trees')
ts = tskit.load(file)

Is there any reason that we shouldn't be doing this as a workaround?

[hyanwong]

I've been thinking about this, and actually I do think that the form load(url="http://...") is worth considering, for two reasons. Firstly the silly one, that you can still load a file starting with "http" in this case. Secondly, perhaps a more sensible one, that it makes it much easier to document and code without complications, as you can simply pass the url parameter to urllib.request.urlopen. In fact, I think that makes it about a 4 line addition to tskit, and to unit test it you can simply try it with a file:/// url, which should be pretty simple. Then we simply say "anything passed as a url parameter is passed on to urllib.request to open" (or something like that. I think this would tick the box of "doing it well" without having to check on the format of the string, which, as @jeromekelleher points out, is a bit icky.

[jeromekelleher]

It's about much more than just the parameters @hyanwong - once you start interacting with the network all sorts of extra complications come in, like http proxies, error handling, retries, etc etc etc. The nominal case is trivial - it's the error and edge cases that are really tricky. These are hard to implement and especially hard to test. I've done it several times and don't particularly want to spend the time doing it now.

[jeromekelleher]

import urllib.request
file = urllib.request.urlopen('file:///Users/yan/Documents/GitHub/tutorials/data/basics.trees')
ts = tskit.load(file)

This seems like a perfectly reasonable thing to do to me (assuming it works).

[hyanwong]

It's about much more than just the parameters @hyanwong - once you start interacting with the network all sorts of extra complications come in, like http proxies, error handling, retries, etc etc etc. The nominal case is trivial - it's the error and edge cases that are really tricky. These are hard to implement and especially hard to test. I've done it several times and don't particularly want to spend the time doing it now.

Right, that sounds reasonable. I thought that we could simply say that we are handing all this stuff over to urllib.request.urlopen and therefore if it doesn't work (e.g. problems with proxies etc) it's not something we are responsible for. But maybe that's a cop out.

[benjeffery]

I'd be much more motivated to do this if tskit had any sense of being able to use part of a file, as zarr for example does. tskit needs the whole file to do anything, so you have to download it all anyway. The tutorials should also be using the most common way of loading a .trees - from disk. Adding url arguments at that point adds confusion.

Pointing out that urlopen can be used in the load docs seems a good idea though.