issues in model loading

[kayano-mi]

I have encountered an error when running the evaluate.py script.

The error pertains to the loading of the model from the provided model data file located at https://huggingface.co/ridger/MLP-SNN/blob/main/model.pth.tar. The error message indicates that the hyperparameter “hparams” is empty upon model loading.

Upon inspecting the contents of the model.pth.tar file, I found that the dictionary includes keys such as epoch, arch, state_dict, optimizer, version, args, amp_scaler, and metric, but hparams is not listed.

I want to know if this model file corresponds to the evaluate.py script, or how I should run the evaluate.py scriptcorrectly。

[ridgerchu]

Hi,

I think this hparam should be loaded from the config file, could you please check if you can load hparam from here?

[kayano-mi]

Thank you for your reply,

The config files only contain YAML parameters for Perception, Planning, and Prediction, and there are no file for the evaluate.py script. If I only want to run the evaluate.py script, do I also need these three YAML files?

[kayano-mi]

It seems I didn't understand the logic of your program. I'd like to ask if I can directly use evaluate.py to assess the dataset without training.

[ridgerchu]

Thank you for following up with these questions. Yes, you can directly evaluate the dataset without training by running the evaluation script. The easiest way is to use the provided evaluation shell script:

./scripts/eval_plan.sh

[kayano-mi]

Thank you for your response.
I am currently using a Windows 11 system. so, I directly ran the command python evaluate.py --checkpoint $1 --dataroot $2 from ./scripts/eval_plan.sh in terminal. The checkpoint $1 points to the location https://huggingface.co/ridger/MLP-SNN/blob/main/model.pth.tar, and dataroot $2 points to the location where the dataset is downloaded. When running, I encountered the problem I initially mentioned, which is that the program throws an error when loading the model model.pth.tar, stating 'the hyperparameter "hparams" is empty.'

Could you please help me understand why this error might be occurring and how to resolve it?

[ridgerchu]

Hmmm I guess it may cause by the pytorch_lightning version... Do you use my conda yaml to set up the environment?

[kayano-mi]

yeah.. I set up the environment in environment.yaml.

[kayano-mi]

I noticed that in the main function of evaluate.py, the input parameter for the checkpoint is expected to be a ckpt file. However, the model you provided is a pth file. Won't this cause any issues?

[kayano-mi]

Hi, regarding the previous issue, I reinstalled the environment on a different device, but I still encountered the same problem. I hope you can help me identify the issue.

[kayano-mi]

The following images are respectively the version of my PyTorch Lightning, the model model.pth.tar I am using, and the location in the main function where the model is called. Could you please take a look and see if there are any issues?

[ridgerchu]

Hi @kayano-mi,

Thank you for your continued patience and for thoroughly documenting the issues you've encountered. I've been trying to recover the original ckpt from our cluster to better understand how the model was initially configured. Unfortunately, our cluster experienced a crash some time ago that resulted in the loss of that data...

I should also clarify that, as mentioned in our README, the model.pth.tar file is only our pre-trained encoder, not the complete model checkpoint.

Given these circumstances, we will be taking down this model from HuggingFace and updating the README accordingly. I sincerely apologize for any inconvenience this has caused you and appreciate you bringing these issues to our attention, and if possible you may try to retrain this model...

[kayano-mi]

Thank you for your reply. I will try to train this model myself.

[BaophanN]

Hi @kayano-mi, how did you manage to train the model?