Evaluate forecasting models locally

[slies]

Thank you all for creating this benchmark!

I’ve encountered some issues while using it. I downloaded the FEV benchmark from Hugging Face and saved the datasets locally; however, when I attempt to load the local datasets, the loading speed is very slow. In particular, it seems that for window in task.iter_windows(): consumes a considerable amount of time. Is this expected behavior? BTW, as I'm loading the benchmark datasets locally, how can I get the configuration of the forecasting tasks?

[shchur]

Hi @slies, can you please check what version of the package you are using? We made some optimizations in fev-0.7.0 that should significantly reduce the runtime.

If you are using the latest version and still encounter these issues, it would be helpful if you share the exact code that you use, tell us how long it takes to perform certain opeartions, and describe your system (OS, # of cores, available RAM).

[slies]

Hi @shchur, I confirm that the version of fev is 0.7.0, where this code consumes 80 seconds:

task = fev.Task(dataset_path="/data/fev_datasets/ercot/1D/train-00000-of-00001.parquet")

for window in task.iter_windows():
    past_data, future_data = window.get_input_data()
    break

Regarding the system information, it is running Ubuntu 22.04.3 LTS with 128 CPUs and 450 GB of available RAM, so I don’t believe the issue is system-related. Additionally, it appears that the datasets package is attempting to establish an HTTP connection, which results in a very long wait time.

[shchur]

Thanks for sharing the details, this looks very surprising - even on my laptop this takes <1 second (with HF datasets cache disabled). Can you please share the output of pip freeze or at least the versions of datasets, pyarrow and huggingface-hub?

Alternatively, could you try creating a new virtual environment, installing these dependency versions, and checking if the problem persists?

aiobotocore==2.26.0
aiohappyeyeballs==2.6.1
aiohttp==3.13.3
aioitertools==0.13.0
aiosignal==1.4.0
annotated-types==0.7.0
anyio==4.12.1
attrs==25.4.0
botocore==1.41.5
certifi==2026.1.4
charset-normalizer==3.4.4
click==8.3.1
datasets==3.6.0
dill==0.3.8
fev==0.7.0
filelock==3.20.3
frozenlist==1.8.0
fsspec==2025.3.0
h11==0.16.0
hf-xet==1.2.0
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==1.3.7
idna==3.11
jmespath==1.1.0
multidict==6.7.1
multiprocess==0.70.16
numpy==2.4.2
packaging==26.0
pandas==3.0.0
propcache==0.4.1
pyarrow==23.0.0
pydantic==2.12.5
pydantic-core==2.41.5
python-dateutil==2.9.0.post0
pyyaml==6.0.3
requests==2.32.5
s3fs==2025.3.0
scipy==1.17.0
shellingham==1.5.4
six==1.17.0
tqdm==4.67.2
typer-slim==0.21.1
typing-extensions==4.15.0
typing-inspection==0.4.2
urllib3==2.6.3
wrapt==1.17.3
xxhash==3.6.0
yarl==1.22.0

[shchur]

@slies you can cache all datasets locally using this command before running the evaluation completely offline

python -c "
from datasets import load_dataset, get_dataset_config_names, disable_progress_bar
from concurrent.futures import ThreadPoolExecutor
disable_progress_bar()
ds='autogluon/fev_datasets'
with ThreadPoolExecutor(8) as e: list(e.map(lambda c: load_dataset(ds,c,split='train'), get_dataset_config_names(ds)))
"

This will cache all datasets from https://huggingface.co/datasets/autogluon/fev_datasets under ~/.cache/huggingface/datasets and make them available for loading with load_dataset(...) when you instantiate the tasks.

[slies]

@shchur Thanks for your suggestion! I will try it later ;)