Got stuck when integrating IR camera

[phucvinh57]

I am going to integrate IR camera as a part of Biopass's antispoofing feature. My current device is a Lenovo ThinkPad P1 Gen 6.

In an infrared imaging system, there are usually two main components that work together like a camera and a flashlight.
The first is the IR camera, which is a specialized image sensor designed to detect light in the infrared spectrum. The second is the IR emitter, typically an infrared LED. Because the environment usually does not provide enough infrared light for a small sensor to capture a clear image, the camera depends on this built-in infrared light source.

My issue is that when I access the IR camera through v4l2, the IR emitter flashes only once and then turns off. As a result, the captured image becomes black. Based on my research, it seems that this IR camera may be hardware-restricted for use only with Windows Hello and is not exposed for general-purpose access on Linux.

Here is my camera information:

╭─npvinh@thinkpad-p1-gen-6 ~  
╰─➤  v4l2-ctl --list-devices                                                                                      130 ↵
Integrated Camera: Integrated C (usb-0000:00:14.0-8):
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/media0
	/dev/media1

╭─npvinh@thinkpad-p1-gen-6 ~  
╰─➤  v4l2-ctl -d /dev/video2 -l                

User Controls

   region_of_interest_rectangle 0x00981ae1 (unknown): type=107 value=unsupported payload type flags=has-payload, 0x00001000
  region_of_interest_auto_ctrls 0x00981ae2 (bitmask): max=0x00000001 default=0x00000001 value=0 flags=0x00001000

Camera Controls

                        privacy 0x009a0910 (bool)   : default=0 value=0 flags=read-only
╭─npvinh@thinkpad-p1-gen-6 ~  
╰─➤  v4l2-ctl -d /dev/video0 -l

User Controls

                     brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=128 flags=0x00001000
                       contrast 0x00980901 (int)    : min=0 max=100 step=1 default=32 value=32 flags=0x00001000
                     saturation 0x00980902 (int)    : min=0 max=100 step=1 default=64 value=64 flags=0x00001000
                            hue 0x00980903 (int)    : min=-180 max=180 step=1 default=0 value=0 flags=0x00001000
        white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                          gamma 0x00980910 (int)    : min=90 max=150 step=1 default=120 value=120 flags=0x00001000
           power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1 (50 Hz)
      white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=10 default=4600 value=4600 flags=inactive, 0x00001000
                      sharpness 0x0098091b (int)    : min=0 max=7 step=1 default=3 value=3 flags=0x00001000
         backlight_compensation 0x0098091c (int)    : min=0 max=2 step=1 default=1 value=1 flags=0x00001000
   region_of_interest_rectangle 0x00981ae1 (unknown): type=107 value=unsupported payload type flags=has-payload, 0x00001000
  region_of_interest_auto_ctrls 0x00981ae2 (bitmask): max=0x00000001 default=0x00000001 value=0 flags=0x00001000

Camera Controls

                  auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=3 (Aperture Priority Mode)
         exposure_time_absolute 0x009a0902 (int)    : min=2 max=1250 step=1 default=156 value=156 flags=inactive, 0x00001000
     exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1
                        privacy 0x009a0910 (bool)   : default=0 value=0 flags=read-only

I have researched how Howdy and other similar Linux authentication tools handle IR camera integration. However, I realized that they just open the IR camera as a linux device and capture images. I do not think their IR camera integrations work in most cases.

I really need your help. Please give me some ideas and suggestions for this feature!

[notherealmarco]

In most laptops just opening /dev/video2 (or whatever the IR device is) just works. In some laptops this is required too: https://github.com/EmixamPP/linux-enable-ir-emitter

on arch linux you can just install aur/linux-enable-ir-emitter and then it should work.

[blauer4]

I think this is the right path for you @phucvinh57, it seems that your sensor is not enabled on your linux distro

[phucvinh57]

It really works! Thank you so much!

[blauer4]

I have an Asus G16, and I can read the IR stream correctly. Have you tried running the command ffplay /dev/video2? That gives a continuous stream out of the camera, with the diode flashing on my IR camera.

Also, I'm using cachyos but I think it will work on other distros.

[phucvinh57]

I sat in the dark room and use IR camera. The diode flashed only one, and only at that moment, I can see my face.
The rest of video is dark.

I would try linux-enable-ir-emitter.

[phucvinh57]

@notherealmarco @blauer4 Thank you all for your helps. A new release has been launched https://github.com/TickLabVN/biopass/releases/tag/1.1.0.

[phucvinh57]

Please run linux-enable-ir-emitter run and show me the log. By the way, it is not a part of Biopass. I got the same issue with the beta version of linux-enable-ir-emitter, you should use a stable version of it.

[kunningKing11]

@phucvinh57 somehow I was able to make it work by using the older v6.1.2 version

[blauer4]

I think just increasing number of retries and set retry gap to few tens of ms will help you in your case.

I increased the number of retries to 20 and the time between retries to 625ms. It works, but it could be better.
Couldn't we think of a "frame skip" for images that are too dark? Because the images we obtain are almost pitch black.
Like if the mean of the image is < a threshold, ignore the frame. That could help the instances where the IR camera takes a bit to warm up.

[kunningKing11]

@blauer4 @phucvinh57 Agreed; this seems like a good idea: for me, the first try nearly always fails which adds some latency so if we could implement this (which I don't think would be that difficult), we could probably reduce latency by ~300 ms.

[phucvinh57]

I set number of buffer frames to 5, but seem like it is not enough. Let me increase it.

I increased the number of retries to 20 and the time between retries to 625ms

By the way, I set the number of retries to 20 and the retry gap to 50ms :)) It would be better and works well. I do not think performance is a problem, because the camera session, which is the heaviest part, is reused between each retry in the latest release.