cross validation is slow

[asbudhkar]

Hi

I am working with public benchmarks for IT and V4 layers. My goal is to get scores for a modified 2-block resnet architecture for each layer on the benchmarks to select layers to run on private benchmarks.
It is taking me around 7 hours for encoder.conv1 layer out of which cross validation takes ~5-6 hours and similar for other layers as well.
Attaching screenshot for reference.

[mschrimpf]

Thanks for making this issue.

Is it possible that your ResNet early layer is larger than usual? It looks like extracting activations is fairly quick (2min21s). At the next step in the process, it should test the alignment of every layer to the public neural data in a cross-validated way. For this, it uses a linear regression metric so I'm wondering if the size of the activations matrix is very large and slowing things down.

[asbudhkar]

That's the model I am using. I do not think early layer is different than ResNet. But if you find anything off please let me know.

[mschrimpf]

That doesn't look so big. Do you know how many units are in each layer with the pixel input size you're using?

If you exclude the first layer from the layer selection, does it run much faster?

The other thing could be a slow CPU or small memory size requiring activations to be swapped in and out. What's your hardware like?

[asbudhkar]

These are the timings for some of the early layers

I am using gpu.
I can try on cpu but that was slow as well so I tried gpu.
I will let you know the specifics about hardware and pixel input size soon.

[asbudhkar]

Input image is 64x64x3.
Hardware:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
cpu MHz : 2500.000
cache size : 28160 KB
core id : 0
cpu cores : 20

NVIDIA-SMI 515.43.04
Driver Version: 515.43.04
CUDA Version: 11.7
V100

[mschrimpf]

could you also share information on RAM / memory size? Your processors look great and like they should run through this much more quickly.

[asbudhkar]

With free command I get this

[mschrimpf]

Okay so you also have more than enough memory available. I am not sure what is causing this.

You might be able to use profile and debug through the steps, but if you can just run this once and re-use the results, I'm not sure it's worth it. By default, the layer scoring results are cached and re-used, so after you ran it once, it should be very quick the next time.