Parallel Mode: AssertionError: No examples were yielded.

[emfebert]

Hi,

I'd like to run dataset generation using beam and I'm getting the following error:

shard_boundaries = _get_shard_boundaries(num_examples, num_shards)

File "/home/febert/anaconda3/envs/rlds_env/lib/python3.9/site-packages/tensorflow_datasets/core/writer.py", line 142, in _get_shard_boundaries
raise AssertionError("No examples were yielded.")
AssertionError: No examples were yielded.

Dataset generation in a single process works fine.

[kpertsch]

For multi-threaded generation, please use the multithreading branch of the repo -- you can just copy your logic for parsing an example into the dataset generator there and control the level of parallelism with the N_WORKERS variable:

rlds_dataset_builder/example_dataset/example_dataset_dataset_builder.py

Line 68 in d38f6d3

N_WORKERS = 10 # number of parallel workers for data conversion