Extract individual frames from the stream

[xandrew94x]

Hi @Eittipat
I have recently started using RTMP.
My goal is to take frames from multiple live streams camera (Android) and analyze them via OpenCV.
Using your example code:

            ** your previous code 
            elif isinstance(message, VideoMessage):  
                ** my code here **
                # Write video data to file  
                flv.write(message.timestamp, message.payload, FLVMediaType.VIDEO) 
            ** your next code

I tried to intercept the payload and convert it into an image via OpenCV:

## my code ##
nparr = np.frombuffer(message.payload, np.uint8)
img_np = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

but img_np return always None.

For the android Client part I'm currently using this code by @TakuSemba.

I tried to save the flv video without changing anything and the file is saved correctly.

Can you suggest any solutions?

Thanks.

[Eittipat]

Hi @xandrew94x, Thanks for using my lib.

The message.payload contains H264 video stream. This is not an image. I think that's why your cv code returns None.

Instead of giving image as input. You should try feeding video stream into OpenCV, grab video frames, and manipulate them the way you want.

Thanks.

[xandrew94x]

Thanks @Eittipat for the answer.

I found a little trick to read frames from OpenCV, even if it doesn't look optimal.

At the moment I have created a script that reads the temporary .flv file that you save from your code (I also gave a fixed name to the saved video):

 ** your prev code
  elif isinstance(message, NSPublish):
      # create flv file at temp
      flv = FLVFile(os.path.join(tempfile.gettempdir(), message.publishing_name)) # <-- temp file
      session.write_chunk_to_stream(StreamBegin(stream_id=1))
      session.write_chunk_to_stream(message.create_response())
      await session.drain()
      logger.debug("Response to NSPublish")
  ** your next code

My code for now looks like this:

#path folder where save images
folder = os.path.join(os.getcwd(), "rtmp_images")
print("path to save folder:",folder)
#path to file video
path = os.path.join(tempfile.gettempdir(), "rtmp_video.flv")
print("Path to file:",path)
try:
    cap = cv2.VideoCapture(path)
    count_img = 0
    try:   
        while True:
            ret, frame = cap.read()
            if np.shape(frame) == ():
                continue
            cv2.imwrite(os.path.join(folder, "rtmp_img_"+str(count_img)+".jpg"),frame)
            count_img+=1
    except Exception as e:
        print("Something went wront!\n", str(e))
    finally:
        cap.release()
except Exception as e:
    print("Cannot open file!\n", str(e))

But at some point some error seems to happen, maybe something with the input packets failing to update the .flv file.
At some point I get these messages from the terminal:

....
[flv @ 0x555cfdd364c0] Packet mismatch 369104654 11 82106
[flv @ 0x555cfdd364c0] Packet mismatch 587209215 11 96370
[flv @ 0x555cfdd364c0] Packet mismatch -402646518 11 152074
[flv @ 0x555cfdd364c0] Packet mismatch 570432013 11 155044
[flv @ 0x555cfdd364c0] Packet mismatch 503323151 11 156760
[flv @ 0x555cfdd364c0] Packet mismatch 553654835 11 247433
[flv @ 0x555cfdd364c0] Packet mismatch 570432061 11 270643
[flv @ 0x555cfdd364c0] Packet mismatch 553654877 11 373317
[flv @ 0x555cfdd364c0] Packet mismatch 1560287849 11 440769
[flv @ 0x555cfdd364c0] Packet mismatch 452991622 11 497496
[flv @ 0x555cfdd364c0] Packet mismatch 520100488 11 507605
[flv @ 0x555cfdd364c0] Packet mismatch 1023416968 11 516955
[flv @ 0x555cfdd364c0] Packet mismatch 570432172 11 614470
.....

What happens here in your opinion?

---

Regarding your answer, what do you mean by: You should try feeding video stream into OpenCV

Directly connect OpenCV to stream via RTMP link?

Because I tried to insert the RTMP link directly on VideoCapture() (As suggested here, and also in the OpenCV documentation it seems to be feasible) but I don't receive anything.

##Sample code##
myrtmp_addr = "rtmp://{my_server_address}:1935/test/sample # <-my link
cap = cv2.VideoCapture(myrtmp_addr)
if not cap.open:
    print ("Not open!")
else:
    print("Open")
while True:
    ret, frame = cap.read()
    if(frame != None): print("Frame is here")

Maybe I am using wrong link? Or do I need to change something in your code first?

Thanks for your attention and patience.

[Eittipat]

Did you try using cv::CAP_FFMPEG on VideoCapture.

About your problem, You cannot process each chunk individually. Sometimes multiple chunks are required in order to correctly decode a video. If directly connecting OpenCV with RTMP URL didn't work, try feeding video chunks into named piped and read it back via OpenCV

Cheers,

[xandrew94x]

Hi @Eittipat

I momentarily found a solution.

I used FFMPEG to receive the rtmp streaming stream and then OpenCV to extract the frames (and eventually save them).

input = "rtmp://{address:port}/live/test"
count_img = 0
try:
    FFMPEG_BIN = "ffmpeg"
    command = [ FFMPEG_BIN, 
            '-hide_banner',           # global options
                                      # input options (none in this case)
            '-i',                     # input
            input,                    # input var
            '-pix_fmt',               # set pixel formats
            'bgr24',                  # opencv requires bgr24 pixel format.
            '-vcodec',                # Set the video codec 
            'rawvideo',               # video codec format
            '-filter:v',
            'fps=3',
            '-an',                    # remove audio
            '-sn',                    # remove subtitle
            '-vsync',
            'drop',
            '-f',                     # Force input or output file format.
            'image2pipe',             # file format
            '-']                      # none output file
    print("FFMPEG started!")
    pipe = sp.Popen(command, stdout = sp.PIPE, bufsize=10**8)
    while True:
        # Capture frame-by-frame
        raw_image = pipe.stdout.read(1280*720*3) #your img dim
        if raw_image != None and len(raw_image) > 0:
            image = np.frombuffer(raw_image, dtype='uint8')
            image = image.reshape((1280,720,3))
            cv2.imwrite(os.path.join(folder, "rtmp_img_ffmpeg_"+str(count_img)+".jpg"),image)
            count_img+=1
        pipe.stdout.flush()
except Exception as e:
    print("Error:",e)

I changed also the client app for the android device and finally i used this docker for the server.

Thank you for your time.