-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathNEWS
2307 lines (1558 loc) · 103 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
GStreamer 1.24 Release Notes
GStreamer 1.24.0 was originally released on 4 March 2024.
The latest bug-fix release in the stable 1.24 series is 1.24.6 and was released on 29 July 2024.
See https://gstreamer.freedesktop.org/releases/1.24/ for the latest version of this document.
Last updated: Monday 29 July 2024, 14:30 UTC (log)
## Introduction
The GStreamer team is proud to announce a new major feature release in the stable 1.x API series of your favourite
cross-platform multimedia framework!
As always, this release is again packed with many new features, bug fixes and other improvements.
## Highlights
- New Discourse forum and Matrix chat space
- New Analytics and Machine Learning abstractions and elements
- Playbin3 and decodebin3 are now stable and the default in gst-play-1.0, GstPlay/GstPlayer
- The va plugin is now preferred over gst-vaapi and has higher ranks
- GstMeta serialization/deserialization and other GstMeta improvements
- New GstMeta for SMPTE ST-291M HANC/VANC Ancillary Data
- New unixfd plugin for efficient 1:N inter-process communication on Linux
- cudaipc source and sink for zero-copy CUDA memory sharing between processes
- New intersink and intersrc elements for 1:N pipeline decoupling within the same process
- Qt5 + Qt6 QML integration improvements including qml6glsrc, qml6glmixer, qml6gloverlay, and qml6d3d11sink elements
- DRM Modifier Support for dmabufs on Linux
- OpenGL, Vulkan and CUDA integration enhancements
- Vulkan H.264 and H.265 video decoders
- RTP stack improvements including new RFC7273 modes and more correct header extension handling in depayloaders
- WebRTC improvements such as support for ICE consent freshness, and a new webrtcsrc element to complement webrtcsink
- WebRTC signallers and webrtcsink implementations for LiveKit and AWS Kinesis Video Streams
- WHIP server source and client sink, and a WHEP source
- Precision Time Protocol (PTP) clock support for Windows and other additions
- Low-Latency HLS (LL-HLS) support and many other HLS and DASH enhancements
- New W3C Media Source Extensions library
- Countless closed caption handling improvements including new cea608mux and cea608tocea708 elements
- Translation support for awstranscriber
- Bayer 10/12/14/16-bit depth support
- MPEG-TS support for asynchronous KLV demuxing and segment seeking, plus various new muxer features
- Capture source and sink for AJA capture and playout cards
- SVT-AV1 and VA-API AV1 encoders, stateless AV1 video decoder
- New uvcsink element for exporting streams as UVC camera
- DirectWrite text rendering plugin for windows
- Direct3D12-based video decoding, conversion, composition, and rendering
- AMD Advanced Media Framework AV1 + H.265 video encoders with 10-bit and HDR support
- AVX/AVX2 support and NEON support on macOS on Apple ARM64 CPUs via new liborc
- GStreamer C# bindings have been updated
- Rust bindings improvements and many new and improved Rust plugins
- Rust plugins now shipped in packages for all major platforms including Android and iOS
- Lots of new plugins, features, performance improvements and bug fixes
## Major new features and changes
### Discourse forum and Matrix chat space
- The new Discourse forum and Matrix chat space are now our preferred communication channels for support and developer chat.
- The mailing lists and IRC channel are on their way to being phased out, but Discourse can be used via e-mail as well.
- For release announcements please subscribe to the News + Announcements category on Discourse, although we will continue to
also send announcements to the mailing list for the time being.
### Playbin3, decodebin3 now stable and default
- After a year of stability, testing and more improvements, playbin3, and its various components (uridecodebin3, decodebin3
and urisourcebin), are now the recommended playback components.
- Some playback components have now switched to defaulting to playbin3: gst-play-1.0 and the GstPlay / GstPlayer libraries.
Application developers are strongly recommended to switch to using those components instead of the legacy playbin and
(uri)decodebin.
Improvements in this cycle:
- Better support missing/faulty decoders, attempt to use another one or gracefully un-select the stream.
- Many fixes for more complex gapless and instant-switching scenarios
- Lower latency for live pipelines
- Fix for “chained” streams (ex: Ogg, or PMT update in MPEG-TS)
- Fixes for hardware-accelerated playback with subtitles (provided the sink can handle offloading composition). This was also
partly due to a historical confusion between subtitle “decoders” (which decode the format to text and “parsers” (which only
do timing detection and optional seeking).
### GstMeta serialization/deserialization and other GstMeta improvements
- GstMeta serialization/deserialization allows metas to be transmitted or stored. This is used by the unixfd and cudaipc
plugins for inter-process communication (IPC). Implemented so far for GstCustomMeta, GstVideoMeta, GstAudioMeta and
GstReferenceTimestampMeta.
- Simplified GstCustomMeta registration with gst_meta_register_custom_simple() for the simple case where tags and transform
functions are not needed.
- GstMetaClearFunction clears the content of the meta. This will be called by buffer pools when a pooled buffer is returned to
the pool.
- Add gst_meta_info_new() and gst_meta_info_register() to register a GstMeta in two steps for easier extensibility.
### New unixfd plugin for efficient 1:N inter-process communication on Linux
- unixfdsink and unixfdsrc are elements that, inspired by shmsink andn shmsrc, send UNIX file descriptors (e.g. memfd, dmabuf)
from one sink to multiple source elements in other processes on Linux.
- The unixfdsink proposes a memfd/shm allocator to upstream elements which allows for example videotestsrc to write directly
into memory that can be transfered to other processes without copying.
### New GstMeta for SMPTE ST-291M HANC/VANC Ancillary Data
- Previously only various specific GstMeta for ancillary data were provided, such as GstVideoCaptionMeta and GstVideoAFDMeta.
The new GstAncillaryMeta allows passing arbitrary ancillary data between elements, including custom and non-standard
ancillary data. See GstAncillaryMeta for details.
- Add with gst_buffer_add_ancillary_meta() and retrieve with gst_buffer_get_ancillary_meta() or
gst_buffer_iterate_ancillary_meta().
- Supported by the newly added AJA sink and source elements
### DSD audio support
- DSD audio is a non-PCM raw audio format representation and the GstAudio library gained support for this in form of new
GstDsdInfo and GstDsdFormat API.
- Support for DSD audio has been implemented in alsasink as well as the GstAudioSink and GstAudioRingBuffer base classes, and
the gst-libav plugin to enable FFmpeg-based DSD elements and functionality.
### Analytics and Machine Learning
- A new library, GstAnalytics, has been added. It defines a GstAnalyticsRelationMeta that can efficiently hold a large number
of observations from a data analysis process, for example from machine learning. It also contains a matrix of the
relationship between those observations.
- Three types of metadata are already defined in the library: object detection, classification and tracking.
- A new objectdetectionoverlay element has been merged that draws the bounding boxes and the classes from the object detection
and classification metadata types.
- The onnxinference element has been split into two parts. The first part works with the ONNX Runtime library to do the actual
inference, while the second part called ssdobjectdetector interprets the produced tensor. This new element creates
GstAnalyticsRelationMeta.
- The onnxinference element now accepts video frames without transformation if the module declares that it accepts the “Image”
type and the format is something that GStreamer knows.
- In the next release, tensor decoders such as ssdobjectdetector will live outside of the ONNX plugin so they can be used with
other machine learning acceleration frameworks.
### Qt5 + Qt6 QML integration improvements
- The Qt5 qmlglsink, qmlgloverlay, qmlglmixer received support for directly consuming BGRA and YV12 video frames without a
prior glcolorconvert.
- New qml6glsrc, qml6glmixer, and qml6gloverlay elements as Qt6 counterparts to the existing Qt5 elements, also with support
for directly consuming BGRA and YV12 video frames without a prior glcolorconvert.
- qml6d3d11sink is a new Direct3D11 Qt6 QML sink for Windows as an alternative to the existing qml6glsink.
### DRM Modifier Support for dmabufs on Linux
The Linux dmabuf subsystem provides buffer sharing across different hardware device drivers and subsystems, and is used
extensively by the DRM subsystem to exchange buffers between processes, contexts, and library APIs within the same process, and
also to exchange buffers with other subsystems such as Linux Media.
In GStreamer, it’s used on the capture side (v4l2src, pipewire), hardware-accelerated video decoders and encoders, OpenGL
integration, Wayland renderers, etc.
GStreamer has had support for dmabufs for a long time and was able to negotiate “zero-copy” paths between different components,
however it only supported and assumed simple linear formats and was not able to negotiate complex non-linear formats. This meant
that dmabuf support actually had to be disabled in many scenarios to avoid “garbled video”.
With GStreamer 1.24 there is now full DRM modifier support and complex non-linear formats can be supported and negotiated
between components.
This is achieved with an extra drm_format field in video/x-raw(memory:DMABuf), format=(string)DMA_DRM caps, e.g.
drm-format=(string)NV12:0x0x0100000000000001.
See the GStreamer DMA buffers design documentation for more details.
This is used in the VA-API va plugin, waylandsink, the MSDK plugin, and the OpenGL integration. Video4Linux support is expected
to land in one of the next minor releases.
New API has been added for easy handling of these new caps:
- GstVideoInfoDmaDrm plus associated functions, similar to GstVideoInfo, including conversion to and from GstVideoInfo with
gst_video_info_dma_drm_from_video_info() and gst_video_info_dma_drm_to_video_info()
- GST_VIDEO_DMA_DRM_CAPS_MAKE
- GST_VIDEO_FORMAT_DMA_DRM
### OpenGL integration enhancements
- When using EGL, if both OpenGL ES and OpenGL are available, OpenGL ES is preferred over OpenGL. OpenGL ES supports some
necessary features required for dmabuf support. This does not apply if an external library/application chooses an OpenGL API
first.
- Improved support for dmabuf use cases. The glupload element now supports the new and improved dmabuf negotiation with
explicit modifiers.
- Base classes for mixing with OpenGL are now public API. GstGLBaseMixer and GstGLMixer are exposed matching the existing
filter-based GstGLBaseFilter and GstGLFilter base classes. The new OpenGL mixer base classes are based on
GstVideoAggregator.
- Add support for a ‘surfaceless’ EGL context using EGL_MESA_platform_surfaceless.
- Expose Vivante Framebuffer build-related files (pkg-config, gir) as public API
- Add support for more video formats:
- A420 8/10/12/16-bit.
- A422 8/10/12/16-bit.
- A444 8/10/12/16-bit.
- I420 10/12 bit.
- RBGA.
- Add support for tiled video formats
- NV12_16L32S (Mediatek format)
- NV12_4L4 (Verisilicon Hantro format)
- glcolorconvert now has API for retrieving shader strings for:
- swizzling (reordering components).
- YUV->RGB conversion.
- RGB->YUV conversion.
- Add more helpers for information about a particular video and/or GL format e.g. number of components, bytes used, or pixel
ordering.
- glvideomixer has new sink pad properties sizing-policy, xalign, yalign matching compositor.
- GstGLBufferPool now has a configuration option for allowing a number of buffers to be always outstanding allowing for
reducing the potential synchronisation delay when reusing OpenGL memory backed buffers.
### Vulkan integration enhancements
- Add support for the Vulkan H.264 and H.265 decoders.
- Add support for timeline semaphores.
- Optionally use newer Vulkan functions for format selection.
- Add support for GPU-assisted validation.
- Vulkan/Wayland: add support for xdg_wm_base protocol for creating a visible debug window. Required as the previous wl_shell
interface is being removed from compositors.
### CUDA / NVCODEC integration and feature additions
- New cudaipcsrc and cudaipcsink elements for zero-copy CUDA memory sharing between processes
- New nvJPEG library based nvjpegenc JPEG encoder element
- The NVIDIA desktop GPU decoder nvh264sldec, nvh265sldec, nvvp8sldec, and nvvp9sldec elements were renamed to nvh264dec,
nvh265dec, nvvp8dec, and nvvp9dec, respectively.
- GStreamer NVIDIA H.264 and H.265 encoders except for nvh264enc and nvh265enc gained support for CEA708 Closed Caption
inserting.
- OpenGL memory support is added to nv{cuda,autogpu}h264enc and nv{cuda,autogpu}h265enc elements
- CUDA stream integration: As of 1.24, CUDA stream synchronization is an application’s responsibility, and GStreamer will not
execute unnecessary synchronization operations. If an application needs direct access to CUDA memory via GST_MAP_CUDA map
flag, cuStreamSynchronize() or gst_cuda_memory_sync() call is required unless application-side CUDA operation is executed
with the GstCudaMemory’s associated CUDA stream.
### RTP stack improvements
- New rtppassthroughpay element which just passes RTP packets through unchanged, but appears like an RTP payloader element.
This is useful for relaying an RTP stream as-is through gst-rtsp-server, which expects an RTP payloader with certain
properties at the end of an RTSP media sub-pipeline.
- New “timeout-inactive-rtp-sources” property on rtpbin, sdpdemux and rtpsession to allow applications to disable automatic
timeout of sources from which no data has been received for a while.
- rtpvp8pay, rtpvp9pay: expose “picture-id” as a property, and add a “picture-id-offset” property to the VP9 payloader to
bring it in line with the VP8 payloader.
- rtpjitterbuffer has seen improved media clock handling (clock equality and clock setting/resetting), as well as two new
properties that allow reconstruction of absolute PTP timestamps without actually syncing to the PTP clock, which can be
useful in scenarios where one wants to reconstruct the absolute PTP clock timestamps on a machine that doesn’t have access
to the network of the PTP clock provider. The two new properties are:
- “rfc7273-use-system-clock”: allows the jitter buffer to assume that the system clock is synced sufficiently close to the
media clock used by an RFC7273 stream. By default the property is disabled and the jitter buffer will create a media
clock and try to sync to it, but this is only required to determine in which wraparound period from the media clock’s
Epoch the current RTP timestamps refer to (and thus to reconstruct absolute time stamps from them). If the property is
enabled the wraparound period and current offset from the Epoch will be determined based on the local system clock,
which means that no direct network connection to the media clock provider is needed to reconstruct absolute timestamps.
There is also no start-up delay, because there’s no clock sync that needs to be established first.
- “rfc7273-reference-timestamp-meta-only”: If this property is enabled then the jitter buffer will do the normal timestamp
calculations for the output buffers according to the configured mode instead of making use of the RFC7273 media clock
for that. It will still calculate RFC7273 media clock timestamps, but only attach them to the output buffers in form of
a clock reference meta.
- RTP payloaders and depayloaders now have an “extensions” property for retrieving the list of currently enabled RTP header
extensions.
- rtpbin and webrtcbin no longer blindly set properties on the jitter buffer assuming it’s a standard rtpjitterbuffer, but
instead check if the property is available first, to better support non-standard jitterbuffers or even an identity element
in lieu of a jitter buffer.
- RTP header extension handling fixes for depayloaders that aggregate multiple input buffers into a single output buffer.
Before, only the last RTP input buffer was checked for header extensions. Now the depayloader remembers all RTP packets
pushed before an output buffer is produced and checks all RTP input buffers for header extensions.
- Affected depayloaders: rtph264depay, rtph265depay, rtpvp8depay, rtpvp9depay, rtpxqtdepay, rtpasfdepay, rtpmp4gdepay,
rtpsbcdepay, rtpvorbisdepay, rtpmp4vdepay, rtptheoradepay, rtpsv3vdepay, rtpmp4adepay, rtpklvdepay, rtpjpegdepay,
rtpj2kdepay, rtph263pdepay, rtph263depay, rtph261depay. rtpgstdepay.
### WebRTC improvements
- Add support for ICE consent freshness (RFC 7675). This requires libnice >= 0.1.22.
- Advertise the local side of an end-of-candidates with an empty candidate string.
- Add the number of Data Channels opened and closed to webrtcbin’s statistics.
- Various improvements and feature additions in the Rust webrtc plugin, which provides webrtcsrc and webrtcsink elements as
well as specific elements for different WebRTC signalling protocols. See the Rust plugins section below for more details.
### Adaptive Streaming improvements and Low-Latency HLS (LL-HLS) support
- hlsdemux2 now supports Low-Latency HLS (LL-HLS)
- hlsdemux2 asynchronous playlist download and update improves responsiveness and bandwith usage.
- hlsdemux2 handles fallback variant URLs.
- hlsdemux2 is more responsive and accurate when handling seeks.
- dashdemux2 and hlsdemux2 have a new “start-bitrate” property, improving the decision for which initial stream variant that
will be used.
- dashdemux2, hlsdemux2, mssdemux2 have received many improvements regarding seeking, along with support for “early-seek”
which allows playback to start immediately from the requested position without any previous download.
- dashdemux2, hlsdemux2, mssdemux2 better handle errors on or near the live edge.
- dashsink can now use the dashmp4mux muxer from the Rust plugins and will also produce better and RFC 6381-compatible codec
strings. The “suggested-presentation-delay” property allows to set the suggested presentation delay in the MPD.
- No development took place on the legacy demuxers (dashdemux, hlsdemux, mssdemux). Application developers are reminded to use
the new demuxers instead. They are automatically picked up when using urisourcebin, uridecodebin3 or playbin3.
### W3C Media Source Extensions library
- A new GStreamer library (mse) implementing the W3C Media Source Extensions specification was added.
- Applications can embed this library along with GStreamer in order to integrate software that uses the Media Source APIs
without relying on a web browser engine. Typically an application consuming this library will wrap the C API with JavaScript
bindings that match the Media Source API so their existing code can integrate with this library.
### Closed Caption handling improvements
- ccconverter supports converting between the two CEA-608 fields.
- New cea608mux element for muxing multiple CEA-608 streams together.
- Various improvements and feature additions in the Rust-based closed caption elements. Check out the Rust plugins section
below for more details.
### Precision Time Protocol (PTP) clock improvements
- Many fixes and compatibility/interoperability improvements.
- Better support for running on devices with multiple network interfaces.
- Allow sync to master clock on same host.
- PTP clock support is now also available on Windows.
- The standalone ptp-helper binary has been rewritten in Rust for portability and security. This works on Linux, Android,
Windows, macOS, FreeBSD, NetBSD, OpenBSD, DragonFlyBSD, Solaris and Illumos. Newly supported compared to the C version is
Windows. Compared to the C version various error paths are handled more correctly and a couple of memory leaks are fixed.
Otherwise it should work identically. The minimum required Rust version for compiling this is 1.48, i.e. the version
currently in Debian oldstable. On Windows, Rust 1.54 is needed at least.
- New ptp-helper Meson build option so PTP support can be disabled or required.
- gst_ptp_init_full() allows for a more fine-grained and extensible configuration and initialization of the GStreamer PTP
subsystem, including TTL configuration.
### Bayer 10/12/14/16-bit depth support
- bayer2rgb and rgb2bayer now support bayer with 10/12/14/16 bit depths
- v4l2src and videotestsrc now support bayer with 10/12/14/16 bit depths
- imagefreeze gained bayer support as well
### MPEG-TS improvements
- mpegtsdemux gained support for
- segment seeking for seamless non-flushing looping, and
- synchronous KLV
- mpegtsmux now
- allows attaching PCR to non-PES streams
- allows setting of the PES stream number for AAC audio and AVC video streams via a new “stream-number” property on the
muxer sink pads. Currently the PES stream number is hard-coded to zero for these stream types.
- allows writing arbitrary Opus channel mapping families and up to 255 channels
- separate handling of DVB and ATSC AC3 descriptors
## New elements and plugins
- analyticsoverlay visualises object-detection metas on a video stream.
- autovideoflip and autodeinterlace are two new auto elements.
- AJA source and sink elements plus device provider for AJA capture and playout cards, including support for HANC/VANC
ancillary data.
- New cea608mux element for muxing multiple CEA-608 streams together.
- The codec2json plugin adds av12json, h2642json, h2652json and vp82json elements which convert AV1, H.264, H.265 and VP8
frame parameters into human readable JSON data, which is useful for debugging and testing purposes.
- New lc3 plugin with a decoder and encoder for the Bluetooth LC3 audio codec.
- New onnxinference element to run ONNX inference models on video buffers.
- New rtppassthroughpay element which just passes RTP packets through unchanged, but appears like an RTP payloader element.
This is mostly useful for medias that simply pass through an existing RTP stream in gst-rtsp-server.
- Qt6: qml6glsrc, qml6glmixer, qml6gloverlay, and qml6d3d11sink
- New SVT-AV1 encoder plugin, imported from SVT-AV1 but with many fixes.
- Many exciting new Rust elements, see Rust section below.
- New DirectWrite text rendering and Direct3D12 plugins (see Windows section below).
- New vaav1enc element for encoding video in AV1 (See VA-API section)
- New uvcsink element for exporting streams as UVC camera
## New element features and additions
- alphacombine supports I420_10LE now for 10-bit WebM/alpha support.
- The amfcodec for hardware-accelerated video encoding using the Advanced Media Framework (AMF) SDK for AMD GPUs gained some
new features:
- 10-bit and HDR support for H.265 / HEVC and AV1 video encoders
- B-frame support in the H.264 encoder
- Initial support of pre-analysis and pre-encoding
- Initial support of Smart Access Video for optimal distribution amongst multiple AMD hardware instances.
- appsink: new “propose-allocation” signal so applications can provide a buffer pool or allocators to the upstream elements,
as well as “max-time” and “max-buffers” properties to configure the maximum size of the appsink-internal queue in addition
to the existing “max-bytes” property.
- autovideoconvert exposes colorspace and scaler elements for well know elements
- avtp: add AVTP Raw Video Format payload and de-payload support.
- cacasink’s output driver can now be selected via the “driver” property.
- camerabin: various fixes and stability improvements
- clocksync: “QoS” property to optionally send QoS events upstream like a synchronising sink would.
- cutter: can add GstAudioLevelMeta on output buffers, which can be enabled via the new “audio-level-meta” property.
- dashdemux2 has a new “start-bitrate” property.
- dashsink can now use the dashmp4mux muxer from the Rust plugins and will also produce better and RFC 6381-compatible codec
strings. The “suggested-presentation-delay” property allows to set the suggested presentation delay in the MPD.
- deinterlace: Add support for 10/12/16-bit planar YUV formats
- The dvdspu subpicture overlay now implements GstVideoOverlayComposition support to make it work better with hardware
decoders where the video data should ideally stay on the GPU/VPU and the overlay blitting be delegated to the renderer.
- encodebin now automatically autoplugs timestamper elements such as h264timestamper or h265timestamper, based on new
“Timestamper” element factory type and rank.
- New fakevideodec element (see debugging section below).
- filesink: “file-mode” property to allow the ability to specify rb+ file mode, which overwrites an existing file. This is
useful in combination with splitmuxsink so that files can be pre-allocated which can be useful to reduce disk fragmentation
over time.
- flvmux: add “enforce-increasing-timestamps” property to allow disabling a hack that was added back in the day because
librtmp as used in rtmpsink would get confused by timestamps going backwards, but this is no longer required with rtmpsink2.
If set to true (still the default, for backwards compatibility), flvmux will modify buffers timestamps to ensure they are
always strictly increasing, inside one stream and also between the audio and video streams.
- giostreamsink: Add a property to close the stream on stop().
- h264parse improved its AU boundary detection.
- h264parse, h265parse, mpegvideoparse now support multiple unregistered user data SEI messages.
- insertbin is now a registered element and available via the registry, so can be constructed via parse-launch and not just
via the insertbin API.
- jack: libjack is now loaded dynamically at runtime instead of linking it at build time. This means the plugin can be shipped
on Windows and macOS and will work if there’s a user-installed JACK server/library setup.
- jpegparse now has a rank so it will be autoplugged if needed.
- kmssink: Add auto-detection for NXP i.MX8M Plus LCDIFv3, ST STM32 LTDC, and Texas Instruments TIDSS display controllers.
- matroskademux and matroskamux gained support for more raw video formats, namely RGBA64_LE, BGRA64_LE, GRAY10_BE32, GRAY16_LE
- mpg123audiodec’s rank was changed from MARGINAL to PRIMARY so it’s now higher than avdec_mp3, as it works better with
“freeformat” MP3s.
- msdk:
- DRM modifier support on Linux
- only expose codecs and capabilities actually supported by the platform
- msdkvpp video post-processing:
- new “hdr-tone-mapping” property to enable HDR-to-SDR tone mapping
- new compute scaling mode
- Decoders sport D3D11 and VA integration, and the VP9 decoder supports certain resolution changes.
- Encoders:
- msdkh264enc, msdkh265enc**: “pic-timing-sei” property to insert pic timing SEI
- msdkh264enc, msdkh265enc**: Add properties to allow different max/min-qp values for I/P/B frames
- msdkh264enc: Added BGRx format DMABuf support
- Advertise special image formats in low power mode
- mxfdemux gained support for FFV1 demuxing
- opusenc, opusdec now support decoding and encoding more than 8 channels, and can also handle unknown/unpositioned channel
layouts.
- The oss plugin gained a device provider for audio device discovery
- pcapparse learned how to handle the Linux “cooked” capture encapsulation v2
- Intel Quick Sync plugin improvements:
- qsvh264enc gained more encoding options
- qsvh265dec now supports GBR decoding and HEVC RExt profiles
- qtdemux now adds audio clipping meta when playing gapless m4a content, supports CENC sample grouping, as well as the SpeedHQ
video codec.
- ristsrc gained support for dynamic payloads via the new “caps” and “encoding-name” properties. These can be used to make the
ristsrc receive other payload types than MPEG-TS.
- rtmp2src: a new “no-eof-is-error” property was added: There is currently no way for applications to know if the stream has
been properly terminated by the server or if the network connection was disconnected, as an EOS is sent in both cases. With
the property set, connection errors will be reported as errors, allowing applications to distinguish between both scenarios.
- rtspsrc: new “extra-http-request-headers” property for adding custom http request headers when using http tunnelling.
- sdpdemux now supports SDP source filters as per RFC 4570; audio-only or video-only streaming can be selected via the new
“media” property, and RTCP feedback can be disabled via the “rtcp-mode” property.
- splitmuxsrc uses natural ordering to sort globbed filenames now, i.e. 0, 1, 2, 10, 11 instead of 0, 1, 10, 11, 2, …
- srt: Add more fields to the statistics to see how many packets were retransmitted and how many were dropped.
- switchbin: many improvements, especially for caps handling and passthrough.
- taginject: a “scope” property was added to allow injection of global tags in addition to the current default which is stream
tags.
- timeoverlay: add buffer-count and buffer-offset time modes.
- udpsrc: new “multicast-source” property to support IGMPv3 Source Specific Muliticast (SSM) as per RFC 4604.
- videoconvertscale, videoconvert: add a “converter-config” property to allow fine-tuning conversion parameters that are not
exposed directly as property.
- videoflip: many orientation tag handling fixes and improvements
- videorate: add “drop-out-of-segment” property to force dropping of out-of-segment buffers.
- volume now supports arbitrarily-large positive gains via a new “volume-full-range” property (it was not possibly to just
allow a bigger maximum value for the existing “volume” property for GstController-related backwards-compatibility reasons).
- waylandsink, gtkwaylandsink: improved frame scheduling reducing frame drops and improve throughput.
- webpenc now has support for animated WebP which can be enabled via the new “animated” property. By default it will just
output a stand-alone WebP image for each input buffer, just like before.
- wpe: added a WebProcess crash handler; gained WPEWebKit 2.0 API support.
- x264enc gained support for 8-bit monochrome video (GRAY8).
- ximagesrc gained navigation support (mouse and keyboard events).
- y4mdec now parses extended headers to support high bit depth video.
## Plugin and library moves
- The AMR-NB and AMR-WB plugins have been moved from -ugly to -good.
## Plugin and element removals
- The entire gst-omx package and plugin has been retired. See the OMX section below for more details.
- The RealServer RTSP extension, RDT handling and PNM source have been removed from the realmedia plugin.
- The kate subtitle plugin has been removed.
## Miscellaneous API additions
GStreamer Core
- gst_pipeline_get_configured_latency() and gst_pipeline_is_live() convenience functions to query liveness and configured
latency of a pipeline.
- Plugins can now provide status info messages for plugins that will be displayed in gst-inspect-1.0 and is useful for dynamic
plugins that register features at runtime. They are now able to provide information to the user why features might not be
available. This is now used in the amfcodec, nvcodec, qsv, and va plugins.
- GST_OBJECT_AUTO_LOCK() and GST_PAD_STREAM_AUTO_LOCK() are g_autoptr(GMutexLocker)-based helpers for GstPad and GstObject
that unlock the mutex automatically when the helper goes out of scope. This is not portable so should not be used in
GStreamer code that needs to be portable to e.g. Windows with MSVC.
- gst_clear_context(), gst_clear_promise(), gst_clear_sample()
- gst_util_ceil_log2() and gst_util_simplify_fraction() utility functions
- New TAG_CONTAINER_SPECIFIC_TRACK_ID tag for container specific track ID as used in an HTML5 context, plus basic support in
matroskademux, qtdemux, dashdemux and dashdemux2
- New utility functions to create a stream-id without a pad for elements:
- gst_element_decorate_stream_id()
- gst_element_decorate_stream_id_printf_valist()
- gst_element_decorate_stream_id_printf()
- GstQueueArray gained API for sorting and sorted insertion
- Add strict GstStructure serialisation with gst_structure_serialize_full() in combination with GST_SERIALIZE_FLAG_STRICT
which only succeeds if the result can later be fully deserialised again.
- GstBaseSrc enhancements: the “automatic-eos” property can be used to do the equivalent to gst_base_src_set_automatic_eos().
gst_base_src_push_segment() sends a segment event right away which can be useful for subclasses like appsrc which have their
own internal queuing.
- GstBaseSink gained a new custom GST_BASE_SINK_FLOW_DROPPED flow return which can be used by subclasses from the virtual
::render method to signal to the base class that a frame is not being rendered. This is used in e.g. waylandsink and ensures
that elements such as fpsdisplaysink will correctly report the rate of frames rendered and dropped.
GstDiscoverer
- New “load-serialized-info” signal to retrieve a serialized GstDiscovererInfo
GstSDP
- Add gst_sdp_message_remove_media()
Video Library
DRM Modifier Support for dmabufs on Linux
See section above.
List of Video Formats for Passthrough
New helper API was added to get a list of all supported video formats, including DMA_DRM formats, and can be used to advertise
all supported formats for passthrough purposes:
- GST_VIDEO_FORMATS_ANY_STR, GST_VIDEO_FORMATS_ANY
- gst_video_formats_any() which can be used by bindings or for code that prefers GstVideoFormat values instead of strings.
New Video Formats
- 12-bit and 16-bit A420 / A422 / A444 (YUV with alpha channel) variants:
- A444_12BE, A444_12LE
- A422_12BE, A422_12LE
- A420_12BE, A420_12LE
- A444_16BE, A444_16LE
- A422_16BE, A422_16LE
- A420_16BE, A420_16LE
- 8-bit A422 / A444 (YUV with alpha channel) variant:
- A422
- A444
- Planar 16-bit 4:4:4 RGB formats:
- GBR_16BE
- GBR_16LE
- RBGA, intended to be used by hardware decoders where VUYA is only supported 4:4:4 decoding surface but the stream is encoded
with GBR color space, such as HEVC and VP9 GBR streams for example.
- Two tiled Mediatek 10-bit formats:
- MT2110T
- MT2110R
- Tiled 10-bit NV12 format NV12_10LE40_4L4 (Verisilicon Hantro)
## Miscellaneous performance, latency and memory optimisations
- liborc 0.4.35 (latest: 0.4.38) adds support for AVX/AVX2 and contains improvements for the SSE backend.
- liborc 0.4.37 adds support for NEON on macOS on Apple ARM64 CPUs.
- Most direct use of the GLib GSLice allocator has been removed, as there is little evidence that it actually still provides
much advantage over the standard system allocator on Linux or Windows in 2024. There is strong evidence however that it
causes memory fragmentation for standard GStreamer workloads such as RTSP/RTP/WebRTC streaming.
- As always there have been plenty of performance, latency and memory optimisations all over the place.
## Tracing framework and debugging improvements
- The gst-stats tool can now be passed a custom regular expression
- gst-debug-viewer from the devtools module has seen minor improvements and fixes
New tracers
- None in this release.
Debug logging system improvements
- Nothing major in this cycle.
Fake video decoder
- The new fakevideodec element does not decode the input bitstream, it only reads video width, height and framerate from the
caps and then pushes out raw video frames of the expected size in RGB format.
- It draws a snake moving from left to right in the middle of the frame, which is reasonably light weight and still provides
an idea about how smooth the rendering is.
## Tools
- gst-launch-1.0 gained a new --prog-name command line option to set the program name, which will be used by GTK and GStreamer
to set the class or app-id.
- gst-play-1.0 now defaults to using playbin3, but can still be made to use the old playbin by passing the --use-playbin2
command line argument.
## GStreamer FFmpeg wrapper
- New avvideocompare element to compare two incoming video buffers using a specified comparison method (e.g. SSIM or PSNR).
- Prefer using FFmpeg Musepack decoder/demuxer over musepackdec as they work better with decodebin3 and playbin3 which likes
to have parsers and decoders separate.
- Added codec mappings for AV1, MxPEG, FFVHuff video
- Added raw video format support for P010, VUYA, Y410, P012, Y212 and Y412.
- Newer, non-deprecated APIs are used by the plugin when built with FFmpeg 6.0 or newer.
- The FFmpeg meson subproject wrap has been updated to v6.1
- Note: see Known Issues section below for known issues with FFmpeg 6.0.0 and the latest FFmpeg 7.x release
## GStreamer RTSP server
- New “ensure-keyunit-on-start” property: While the suspend modes NONE and PAUSED provided a low startup latency for
connecting clients, it did not ensure that streams started on fresh data. With this new property it is possible to maintain
the low startup latency of those suspend modes while also ensuring that a stream starts on a key unit. Furthermore, by
setting the new “ensure-keyunit-on-start-timeout” property it is also possible to accept a key unit of a certain age, but
discard it if too much time has passed and instead force a new key unit.
- rtspclientsink: apply “port-range” property for RTCP port selection as well
## GStreamer VA-API support
GstVA
- vah264dec, vah265dec, vavp8dec, vavp9dec, vampeg2dec and vaav1dec were promoted to rank PRIMARY+1 on Linux
- Improved support for dmabuf use cases. All va elements now negotiate the new and improved dmabuf capabilities with explicit
modifiers. This supports both import and export of dmabufs.
- Added vaav1enc element, available in recent Intel and AMD GPUs
- Added support for the experimental VA-Win32 backend. It needs at least libva 1.18
- Improved handling of multi-GPU systems. Still, sharing buffers among them is not advised.
- Bumped minimum libva version to 1.12
- Enhanced support for RadeonSI Mesa driver for 10bit decoding
- Register elements only for allowed drivers (Intel and Mesa, for the moment)
GStreamer-VAAPI
- The new GstVA elements (see above) should be preferred when possible.
- Ranks of decoders were demoted to NONE so they won’t be used automatically by playbin and similar elements anymore.
- Clean-ups and minimal fixes.
- gstreamer-vaapi should be considered deprecated and may be discontinued as soon as the va plugin is fully feature
equivalent. Users who rely on gstreamer-vaapi are encouraged to migrate and test the va elements at the earliest
opportunity.
## GStreamer Video4Linux2 support
- New uvcsink element, based on v4l2sink allow streaming your pipeline as a UVC camera using Linux UVC Gadget driver.
- v4l2src now supports 10/12/14/16-bit bayer formats.
- Stateful decoders now pass too large encoded frames over multiple buffers.
- AV1 Stateless video decoder.
- Stateless decoders now tested using Virtual driver (visl), making it possible to run the tests in the cloud based CI
## GStreamer OMX
- The gst-omx module has been removed. The OpenMAX standard is long dead and even the Raspberry Pi OS no longer supports it.
There has not been any development since 1.22 was released. Users of these elements should switch to the Video4Linux-based
video encoders and decoders which have been the standard on embedded Linux for quite some time now.
- Hardware vendors which still use OpenMAX are known to have non-standard forks and it is recommended that they maintain it
while planning their move to the Video4Linux API.
## GStreamer Editing Services and NLE
- Implement a gesvideoscale effect which gives user the ability to chooses where a clip has to be scaled in the chain of
effects. By default scaling is done in the compositor.
- Add support for gessrc as sub-timeline element so third party can implement their own formatters and use their timelines as
sub-timelines. Before that, only timelines serialized as files on the filesystem could be loaded as sub-timelines (using
gesdemux).
- Implement a new GESDiscovererManager singleton object making management of the discoverers used to discoverer media files
cleaner and allowing to expose the following APIs:
- load-serialize-info signal so GstDiscovererInfo can be serialized by users the way they like and load them without
requiring discovering the file when reloading a project.
- source-setup signal so user can tweak source elements during discovery
- Expose GESFrameCompositionMeta in public API so user can implement their own effects targetting GES which take into account
that meta.
- Expose audioconvert:mix-matrix property in audio sources
- Port GESPipeline rendering to use encodebin2. This allows rendering timelines directly with a muxing sink (like hlssinkX
etc..) and leverage all new features of that new element.
ges-launch
- Fix setting keyframes
- Add an ignore-eos option
- Allow overriding container profile so that the user can build encoding profiles following the media format of a specific
media file, for example, but ensuring it is muxed using a specific format
- Ensure sink elements are inside a GstBin and never in a GstPipeline
- Move +effect stack effects from source to last effect added, so it feels more natural to user as adding them at the
beginning of the chain while the syntax is +effect felt wrong
## GStreamer validate
- In action types, add a way to avoid checking property value after setting it, in case elements do it async for example.
- Add a vmethod to free GstValidateActionParameters to be more binding friendly.
- Allow scenarios to define the pipeline state target in the metadata instead of assuming PLAYING state.
- Add support to run sub-pipelines/scenarios
- Added support to forward buffers from appsink to appsrc
- Add a way to set pipeline base-time, start-time and force using the system clock.
- Add a ‘fill-mode’ to the appsrc-push action type so we can create some type of streams easily using an appsrc, giving
control when writing scenarios without requiring files with the content.
- Add a “select-streams” action type to test “stream aware” elements.
- Add a way to wait for a property to reach a specified value before executing an action. For example it is possible to wait
for a pad to get some specific caps set before executing an action.
- validate: Add support to replace variables in deeply nested structures in particular for more complex action types where
some of the properties are inside structures.
- Fixed compatibility with Python 3.12.
## GStreamer Python Bindings
gst-python is an extension of the regular GStreamer Python bindings based on gobject-introspection information and PyGObject,
and provides “syntactic sugar” in form of overrides for various GStreamer APIs that makes them easier to use in Python and more
pythonic; as well as support for APIs that aren’t available through the regular gobject-introspection based bindings, such as
e.g. GStreamer’s fundamental GLib types such as Gst.Fraction, Gst.IntRange etc.
- Added a GstTagList override that makes a tag list act like a dict
- Fix build and usage in Windows
- Various fixes for Python >= 3.12
- Rework libpython loading to be relocatable
- Fix libpython dlopen on macOS
## GStreamer C# Bindings
- The GStreamer C# bindings have been updated to a more recent version of GtkSharp and the bindings have been regenerated with
that version.
- GStreamer API added in recent GStreamer releases is now available
- GstRtspServer bindings have been added, plus an RTSP server example
## GStreamer Rust Bindings and Rust Plugins
The GStreamer Rust bindings and plugins are released separately with a different release cadence that’s tied to the twice-a-year
GNOME release cycle.
The latest release of the bindings (0.22) has already been updated for the new GStreamer 1.24 APIs, and works with any GStreamer
version starting at 1.14.
gst-plugins-rs, the module containing GStreamer plugins written in Rust, has also seen lots of activity with many new elements
and plugins. The GStreamer 1.24 binaries track the 0.12 release series of gst-plugins-rs, and fixes from newer versions will be
backported as needed to the 0.12 brach for future 1.24.x bugfix releases.
Rust plugins can be used from any programming language. To applications they look just like a plugin written in C or C++.
### WebRTC
- New element webrtcsrc that can act as a recvonly WebRTC client. Just like the opposite direction, webrtcsink, this can
support various different WebRTC signalling protocols. Some are included with the plugin and provide their own element
factory for easier usage but it is also possible for applications to provide new signalling protocol implementations.
- webrtcsink now exposes the signaller as property and allows implementing a custom signaller by connecting signal handlers to
the default signaller.
- A new signaller and webrtcsink implementation for Janus’ VideoRoom implementation. The corresponding webrtcsrc signaller
implementation is currently in a merge request in GitLab.
- New whepsrc element that can receive WHEP WebRTC streams. This is currently not based on webrtcsrc but in the future a new
element around webrtcsrc will be added.
- New whipserversrc element around webrtcsrc for ingesting WHIP WebRTC streams in GStreamer.
- New whipclientsink element around webrtcsink for publishing WHIP WebRTC streams from GStreamer. This deprecates the old
whipsink element.
- A new signaller and webrtcsink implementation for LiveKit. The corresponding webrtcsrc signaller implementation was merged
into the git repository recently.
- A new signaller and webrtcsink implementation for AWS Kinesis Video Streams
- webrtcsink has a new payloader-setup signal to allow the application more fine grained control over the RTP payloader
configuration, similar to the already existing encoder-setup signal for encoders.
- webrtcsrc gained support for a custom navigation event protocol over the data channel, which is compatible with the
navigation event protocol supported by webrtcsink.
- webrtcsink supports encoded streams as input. Using encoded streams will disable webrtcsinks congestion control changing any
encoded stream parameters.
- webrtcsink and webrtcsrc have a new signal ‘request-encoded-filter’ to allow transformations of the encoded stream. This can
be used, for example, for the same use-cases as the WebRTC Insertable Streams API.
- gstwebrtc-api: JavaScript API for interacting with the default signalling protocol used by webrtcsink / webrtcsrc.
… and various other smaller improvements!
### RTSP
- New rtspsrc2 element. Only a subset of RTSP features are implemented so far:
- RTSP 1.0 support
- TCP, UDP, UDP-Multicast lower transports
- RTCP SR, RTCP RR, RTCP-based A/V sync
- Tested for correctness in multicast cases too
- Lower transport selection and order (NEW!)
- The existing rtspsrc has a hard-coded order list for lower transports
- Many advanced features are not implemented yet, such as non-live support. See the README for the current status.
### GTK4
- Support for rendering GL textures on X11/EGL, X11/GLX, Wayland, macOS, and WGL/EGL on Windows.
- Create a window for testing purposes when running in gst-launch-1.0 or if GST_GTK4_WINDOW=1 is set.
- New background-color property for setting the color of the background of the frame and the borders, if any. This also allows
setting a fully transparent background.
- New scale-filter property for defining how to scale the frames.
- Add Python example application to the repository.
- Various bugfixes, including support for the new GTK 4.14 GL renderer. The plugin needs to be built with at least the
gtk_v4_10 feature to work with the new GTK 4.14 GL renderer, and will work best if built with the gtk_v4_14 feature.