Importing `utils_test` opens ports

[max-sixty]

This is related to #6806 & #6897, but I thought it was separate enough to open an issue.

We import utils_test in xarray's test suite. Currently importing distributed.utils_test does work, including opening ports.

I'm guessing this is caused by

distributed/distributed/utils_test.py

Line 113 in 8b3dcad

_offload_executor.submit(lambda: None).result() # create thread during import

Beyond just doing more work than I'd expect on import — on a MacOS machine, it pops open a system dialog asking whether that's OK, every time. And if someone is running something on each file save, this happens on every file save. By default, VSCode will search for tests on every file save, so will trigger this.

I don't have enough context to know for sure whether there's some immovable constraint that requires this. But would it be possible to put that in a fixture?

Thanks as ever!

[max-sixty]

Here's a screenshot to make this more concrete!

[jacobtomlinson]

I don't see this is macOS. I'm curious what Python.app is, how did you install Python?

My guess is that this doesn't happen for folks using Python installed via conda and so this hasn't come up before.

[jrbourbeau]

When I tried this initially I also wasn't able to reproduce. But when I went to my "Security & Privacy" settings I saw that I had my firewall turned off. When I enabled the firewall and then ran pytest distributed I got the network connection popups

[fjetter]

When I enabled the firewall and then ran pytest distributed I got the network connection popups

I'm not surprised about this. For instance we have stuff like

distributed/distributed/comm/tests/test_ucx_config.py

Line 18 in be7d8f8

HOST = get_ip()

in the tests that will talk to the internet

distributed/distributed/utils.py

Lines 178 to 185 in be7d8f8

	def get_ip(host="8.8.8.8", port=80):
	"""
	Get the local IP address through which the *host* is reachable.
	*host* defaults to a well-known Internet host (one of Google's public
	DNS servers).
	"""
	return _get_ip(host, port, family=socket.AF_INET)

However, I don't see anything in utils_test that would do something like that

[max-sixty]

Thanks for repro-ing @jrbourbeau

As an outside observer — I think it's fine to do these where necessary — but in a fixture with e.g. scope="session". That way, someone importing the library isn't doing work, it's only creating resources when required. Particularly for modules that other "friends of dask" are expected to import.

To answer the question above — I have python installed with brew.

[fjetter]

As an outside observer — I think it's fine to do these where necessary — but in a fixture with e.g. scope="session". That way, someone importing the library isn't doing work, it's only creating resources when required. Particularly for modules that other "friends of dask" are expected to import.

I think we all agree here. I don't think this was put in on purpose but rather due to negligence/ignorance. I'm very open to change how the tests are setup.

Would you be interested in helping out cleaning this up a bit?

[max-sixty]

Thanks @fjetter

Would you be interested in helping out cleaning this up a bit?

I appreciate the question and empathize with the sentiment. Unfortunately between xarray & PRQL I have my hands full (and, to be transparent, can get around the problem when running xarray tests by uninstalling dask...)