client.scatter a pandas DataFrane

[YarShev]

Hi there,

Since client.scatter distributes data in a round-robin fashion grouping by number of cores, I wonder how a pandas DataFrame is split up between workers. Suppose I have 3 workers and the following script.

from distributed.client import Client
import pandas

client = Client()
df = pandas.DataFrame([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

client.scatter(df)

Could you tell me how the df is split up between 3 workers?

Thanks in advance!

[fjetter]

It's not split up. We treat data that is passed to scatter as opaque python objects and scatter the objects as is.
If you want to split up a pandas dataframe, I suggest to look at the dask.DataFrame API, specifically from_pandas

[YarShev]

It works now. Thanks! I wonder why both the partitions are saved in the same worker? I thought partitions of a Dask DataFrame would be distributed in a round-robin fashion between workers.

fs = distributed.futures_of(ddf)
client.who_has(fs)
{"('from_pandas-4baea7a71ed57934d2b90c0100c6f228', 0)": ('tcp://127.0.0.1:35009',),
 "('from_pandas-4baea7a71ed57934d2b90c0100c6f228', 1)": ('tcp://127.0.0.1:35609',)}

[YarShev]

@jrbourbeau, @fjetter, just a friendly reminder on my question above.

[YarShev]

cc @martindurant

[martindurant]

The do not look like the same worker to me!!

[YarShev]

@martindurant, sorry, my bad. Did read incorrectly.