Intel i5-5300U HD Graphics doesnt finish running

[rajgott]

I have an i5-5300U and want to do inference on the integrated GPU. I can detect the GPU using clDeviceQuery. I compiled and installed Greentea with Intel OpenCL 1.2.
clDeviceQuery.txt

I can test my model on CPU in ~1 second per image. When i switch to GPU mode the inference doesnt finish, i have waited 6+ hours. It shows CPU is running at close to 100% during this run.

Is this normal? Has anyone go it to work on intel integrated graphics?

[naibaf7]

@rajgott
Can you show me your Makefile.config and your network prototxt? In some instances, it could be that you run out of memory for the GPU, or that the convolution engine is not suitable for your GPU.
On the 5300U, you should test if either enabling INTEL_SPATIAL or LIBDNN in the Makefile.config will work instead.

Also try to run:

./build/tools/caffe device_query

and post the result here. It is the "built-in clinfo" for OpenCL Caffe.

[rajgott]

Here is my Makefile.config
Makefile.config.txt

This happens with standard models from bvlc, for example googlenet. INTEL_SPATIAL was enabled, now i enabled LIBDNN also. Even with this the problem remains.

Output of ./build/tools/caffe device_query:
I0708 07:01:31.117813 6125 common.cpp:373] Total devices: 2
I0708 07:01:31.118115 6125 common.cpp:374] CUDA devices: 0
I0708 07:01:31.118127 6125 common.cpp:375] OpenCL devices: 2
I0708 07:01:31.118134 6125 common.cpp:399] Device id: 0
I0708 07:01:31.118144 6125 common.cpp:401] Device backend: OpenCL
I0708 07:01:31.118168 6125 common.cpp:403] Backend details: Intel(R) Corporation: OpenCL 1.2
I0708 07:01:31.118181 6125 common.cpp:405] Device vendor: Intel(R) Corporation
I0708 07:01:31.118191 6125 common.cpp:407] Name: Intel(R) HD Graphics
I0708 07:01:31.118198 6125 common.cpp:409] Total global memory: 3427585229
I0708 07:01:31.118208 6125 common.cpp:399] Device id: 1
I0708 07:01:31.118216 6125 common.cpp:401] Device backend: OpenCL
I0708 07:01:31.118224 6125 common.cpp:403] Backend details: Intel(R) Corporation: OpenCL 1.2
I0708 07:01:31.118240 6125 common.cpp:405] Device vendor: Intel(R) Corporation
I0708 07:01:31.118249 6125 common.cpp:407] Name: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
I0708 07:01:31.118435 6125 common.cpp:409] Total global memory: 7245664256

Thanks

[naibaf7]

Can you try with LIBDNN enabled but INTEL_SPATIAL disabled and also with both disabled? Just to be sure. From your information I can't think of another problem than a convolution that is stalling.
Otherwise, can you try to run the network on --gpu1 instead of --gpu0 and see if at least OpenCL on CPU works?
Make sure to do clean-builds (make clean, make all) before testing.
You can also try to run
./build/test/test_all.testbin 0
./build/test/test_all.testbin 1
to see if a certain layer gets stalled.

[rajgott]

I tried all combinations with LIBDNN and INTEL_SPATIAL, but inference still stalls.

./build/test/test_all.testbin 1. For example BatchNormLayerTest/2:
[----------] 3 tests from BatchNormLayerTest/2, where TypeParam = caffe::GPUDevice
[ RUN ] BatchNormLayerTest/2.TestForward
[ OK ] BatchNormLayerTest/2.TestForward (2 ms)
[ RUN ] BatchNormLayerTest/2.TestForwardInplace
[ OK ] BatchNormLayerTest/2.TestForwardInplace (1 ms)
[ RUN ] BatchNormLayerTest/2.TestGradient
[ OK ] BatchNormLayerTest/2.TestGradient (15257 ms)
[----------] 3 tests from BatchNormLayerTest/2 (15260 ms total)
This segfaults after passing several tests. The output:
test1.txt

./build/test/test_all.testbin 0 is much slower. For example BatchNormLayerTest/2:
[----------] 3 tests from BatchNormLayerTest/2, where TypeParam = caffe::GPUDevice
[ RUN ] BatchNormLayerTest/2.TestForward
[ OK ] BatchNormLayerTest/2.TestForward (970 ms)
[ RUN ] BatchNormLayerTest/2.TestForwardInplace
[ OK ] BatchNormLayerTest/2.TestForwardInplace (68 ms)
[ RUN ] BatchNormLayerTest/2.TestGradient
[ OK ] BatchNormLayerTest/2.TestGradient (369108 ms)
[----------] 3 tests from BatchNormLayerTest/2 (370146 ms total)
This may take a while to finish more tests.

Thanks

[naibaf7]

@rajgott
Hmm, seems like INTEL_SPATIAL fails on your GPU (because it's not having the expected Skylake OpenCL features it seems).

I can't really find out what's wrong, seems you ran the runtest with INTEL_SPATIAL compiled. You should really compile it in the default configuration (LIBDNN and INTEL_SPATIAL off) to find out more...

[rajgott]

I have INTEL_SPATIAL and LIBDNN commented out as in the default Makefile.config

CMakeCache.txt shows this:
//No help, variable specified on the command line.
USE_INTEL_SPATIAL:UNINITIALIZED=OFF
//Build Caffe with OpenCL libdnn
USE_LIBDNN:BOOL=OFF

How else to confirm these two are disabled/enabled?

[mattg-sp]

I have the same CPU, and I experience the exact same failure on device 1. However, that's the CPU cores, which we don't plan to use. If I run that test case on device 0 (the real HD Graphics GPU), it passes. I am using the default setting of USE_INTEL_SPATIAL.

The problem exposed by the tests is that the GPU backend (device 0) runs many at a speed somewhere between 60x and 270x slower than device 1. During this time, the test is using about 6 % of a CPU core and the rest of its time seems to be io_wait.

[naibaf7]

@mattg-sp
OK. I got an Intel test platform now and will investigate this further.
As a benchmark, can you please run this:
./build/tools/caffe time -model models/bvlc_alexnet/benchmark64.prototxt -gpu=0 -iterations=5

[mattg-sp]

First, thanks for all your help!

Second, that command has been running for 75 minutes, using 99% of a CPU core (virtually all user time). I've attached a profile and the callstacks of all the threads.

caffe_profile-time_gpu0_bvlc_alexnet-benchmark64.txt
caffe_callstacks-time_gpu0_bvlc_alexnet-benchmark64.txt

[naibaf7]

@mattg-sp
Uh, that's not supposed to happen, especially not if you run it on the iGPU. Are you both using Macbooks by any chance?
Can I have these details please:

• Operating system
• clinfo
• ./build/tools/caffe device_query
• Laptop type and brand

It does not stall like that on either the i7-3632QM or i7-6560U integrated GPUs I use for testing (both on beignet-1.1 and Fedora 23/24).

@gongzg ideas?

[mattg-sp]

sumac:~/caffe # uname -a

Linux sumac 4.1.20-11-default #1 SMP PREEMPT Fri Mar 18 14:42:07 UTC 2016 (0a392b2) x86_64 x86_64 x86_64 GNU/Linux

sumac:~/caffe # cat /etc/os-release

NAME="SLES"
VERSION="12-SP1"
VERSION_ID="12.1"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP1"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp1"

I'm having issues attaching files to this post, so I'll attach the rest of the requested details in a separate post.

[gongzg]

@naibaf7 @mattg-sp @rajgott , beignet seems have a cpu side performance issue with the gradient test case. It runs slower and slower during the iteration and seems to be stalled. Beignet team is investigating it. But you will not have that issue if you run the convnet-benchmark or the case Fabian mentioned above with INTEL_SPATIAL enabled. And for BDW, the recommended kernel version is 4.4 or newer. For SKL the recommended kernel version is 4.6 or newer.

[mattg-sp]

caffe-device_query.txt
clinfo.txt

The hardware is actually an Intel NUC (model: NUC5i5MYHE) with 16 GB of RAM.

Actually, here's the top of /proc/meminfo:

MemTotal:       16300032 kB
MemFree:         6359192 kB
MemAvailable:   13040116 kB

[mattg-sp]

@gongzg thanks, but would that explain the behavior of caffe time -model models/bvlc_alexnet/benchmark64.prototxt?

Also, how do I know whether we're using beignet? The runtime I'm using is intel-linux-media-ocl_generic_16.4.4-47109_64bit.tar.gz, which I downloaded from Intel's website. Is that built on beignet?