Gstreamer

We use gstreamer to stream video feeds to the Oden Voysys software. It is a powerfull but complex piece of software for for creating video and audio processing pipelines. It has a C API for creating dedicated programs, or you can create pipelines from a command line, which is what we use.

I found this conference talk to have some good tricks to getting started with using gstreamer.

Basically, a pipeline is a series of functional blocks linked together, where data flows from a "source" to a "sink". Each functional block typically takes data in, does some transformation on the data, and spits new data out. Syntactically, elements are sparated with the ! character, which you can think of as like the '|' pipe in Unix shells. Here is an example pipeline so we can go through what each component does.

Sender

gst-launch-1.0 -v videotestsrc \
    ! jpegenc \
    ! rtpjpegpay \
    ! udpsink host=127.0.0.1 port=5300

Receiver

gst-launch-1.0 -v udpsrc port=5300 \
    ! "application/x-rtp, media=video, clock-rate=90000, encoding-name=JPEG, payload=26" \
    ! rtpjpegdepay \
    ! jpegdec \
    ! autovideosink

If you run the receiver first and then the sender, a multi coloured spash screen should pop up.

Lets go through this from sender to receiver:

The first part is the videotestsrc. This is just a component for testing that generates a simple video feed. Later we'll look at getting a web cam feed running.

The next part is jpegenc, which encodes the videotestsrc video in a jpeg format. Encoding is just picking some standard for how to orgainze data in bit/byte patterns, but typically in video streaming it also does some sort of compression to reduce the bandwith of the feed. Some other common encodings are MPEG-4, H.264 and vp8.

rtpjpegdepay takes in jpeg encoded data and creates rtp payloads from it. RTP is Real Time Protocol, and is good for sending data over a network deterministically in real time (as it's name would suggest).

udpsink takes data (in this case our rtp payloads) and sends it over a UDP socket.

Now onto the receiver.

udpsrc receives data over a UDP socket (in our case, sent by the udpsink).

The next line is parameters to gstreamer to help it determine how to parse the incoming udp data. We didn't specify these in the sending script, these are just the default values. When you run gstreamer with the -v flag, it will output verbose messages that tells you what settings the udpsink is using, so that you can then use those settings for the udpsrc.

Next is rtpjpegdepay, which is just like rtpjpegpay but in reverse (takes rtp payloads containing jpeg encoded video and returns just the jpeg encoded video).

Next is jpegdec, which as you probably guessed decodes the jpeg encoded video.

Finaly is the autovideosink, which just takes the raw video and displays it in a window.

To replace the test source with a web cam on linux, use the following sender script:

gst-launch-1.0 -v v4l2src device=/dev/video0 \
    ! video/x-raw, width=640, height=480, format=YUY2 \
    ! videoconvert \
    ! jpegenc \
    ! rtpjpegpay \
    ! udpsink host=127.0.0.1 port=5300

Where /dev/video0 is a path to your webcam. The second line is just parameters for the v4l2src plugin, just like we did with udpsrc. The third line is videoconvert which is some magic thing that you need after a video source if gstreamer throws errors at you (if someone bothers to look up what it actually does, please update this page :P).

If you ever need to debug gstreamer pipelines, fakesink is like a void sink that takes any data at all and just eats it. Start from the source and pipe it into fakesink and if there are no errors, move on to the next part of the pipeline, pipe it into fakesink and keep going until you isolate the component that is causing difficulty (the errors you get from this method are usually different from what you get from just running the whole pipeline, and may be more useful).

If you're not sure what plugins/modules are available, gst-inspect-1.0 lists all available plugins. Combine with grep to search for things:

$ gst-inspect-1.0 | grep 264
typefindfunctions: video/x-h264: h264, x264, 264
uvch264:  uvch264src: UVC H264 Source
uvch264:  uvch264mjpgdemux: UVC H264 MJPG Demuxer
videoparsersbad:  h264parse: H.264 parser
libav:  avmux_ipod: libav iPod H.264 MP4 (MPEG-4 Part 14) muxer
libav:  avdec_h264: libav H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 decoder
x264:  x264enc: x264enc
rtp:  rtph264pay: RTP H264 payloader
rtp:  rtph264depay: RTP H264 depayloader

Searches for h264 plugins.