1gpu vs multi-gpu: good convergence, but no results when test single image

[RSly]

Hi,

I have a question maybe it is a bug or maybe I am missing something:

Case-1: I train a model using latest digits+nv-caffe with batch_size 1 and 1 gpu
=> the curves are great, loss is low, and when tested on the images (test one), the objects are correctly recognized

Case-2: same config as above, same data, same network, etc. I use batch_size=3 in train and batch_size=1 for test and run it on 3 gpus
=> the curves are great, loss is low BUT when tested using 'test single image' no object is recognized, even after many epochs, etc.

I am surprised that the curves show same 'low loss', but the actual test on the images gives nothing!
any suggestions?

[ontheway16]

Hi, are all 3 gpus are the same?

[RSly]

yes, would that matter?

[ontheway16]

Although I could not get an answer on selection of gpu for inference yet in #1418 , it appears only the gpu-0 utilized for infer. Just thought whether swapping gpus may help, in case of different gpu boards.