Recently we encountered an issue when decoding a h.263 file:
FFmpeg will freeze when decoding h.263 video with NVDEC. Turns out this is not directly related to NVDEC but is a problem that shows with several other HW decoders like VDPAU, though the exact kind of error is different (either error messages or freezing[1]). The root cause is that ff_thread_finish_setup() is called twice per frame from ff_h263_decode_frame(). This is not supported by ff_thread_finish_setup() and specifically checked for and warned against in the functions code. The issue is also specific to hw accelerated decoding only as the second call to ff_thread_finish_setup() is only issued when hw acceleration is on. The fix is simple: add a check that the first call is only send when hw acceleration is off, and the second call only when hw acceleration is on (see attached patch). This works fine as far as I was able to test with vdpau and nvdec/nvcuvid hw decoding. The patch also adds NVDEC to the hw config list if available.
I also noticed a secondary issue when browsing through the code which is that, according to documentation, ff_thread_finish_setup() should only be called if the codec implements update_thread_context(), which h263dec does not. The patch does not address this and I'm not sure any action needs to be taken here at all.
[1] This is depending on whether or not the hw decoder sets the HWACCEL_CAPS_ASYNC_SAFE flag
Signed-off-by: Timo Rothenpieler <timo@rothenpieler.org>
The alloc_size attribute is valid only on functions that return a
pointer. GCC 9 (not yet released) warns about invalid usage:
./libavutil/mem.h:342:1: warning: 'alloc_size' attribute ignored on a function returning int' [-Wattributes]
342 | av_alloc_size(2, 3) int av_reallocp_array(void *ptr, size_t nmemb, size_t size);
| ^~~~~~~~~~~~~
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4361293fcf)
cbs_h2645_read_more_rbsp_data does not handle malformed input very well:
1. If there were <= 8 bits left in the bitreader, these bits were read
via show_bits. But show_bits requires the number of bits to be read to
be > 0 (internally it shifts by 32 - number of bits to be read which is
undefined behaviour if said number is zero; there is also an assert for
this, but it is only an av_assert2). Furthermore, in this case a shift
by -1 was performed which is of course undefined behaviour, too.
2. If there were > 0 and <= 8 bits left and all of them were zero
(this can only happen for defective input), it was reported that there
was further RBSP data.
This can lead to an infinite loop in H.265's cbs_h265_read_extension_data
corresponding to the [vsp]ps_extension_data_flag syntax elements. If the
relevant flag indicates the (potential) occurence of these syntax elements,
while all bits after this flag are zero, cbs_h2645_read_more_rbsp_data
always returns 1 on x86. Given that a checked bitstream reader is used,
we are also not "saved" by an overflow in the bitstream reader's index.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit d4035ca849)
The frame_centre_horizontal/vertical_offset values contained in picture
display extensions are actually signed values (i.e. it is possible to
indicate that the display device should add black bars/pillars).
The files sony-ct3.bs and tcela-6.bits (which are both used in fate
tests for mpeg2_metadata) contain picture display extensions; the former
even contains a negative frame_centre_vertical_offset. Fortunately, the
old code did not damage the picture display extensions when one did a
cycle of reading and writing. For the same reason the fate tests needn't
be updated either.
Furthermore these fields now use the trace output for matrices.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit de58803839)
MPEG-2 contains several elements that mustn't be zero according to the
specifications: horizontal/vertical_size_value, aspect_ratio_information,
frame_rate_code, the quantiser matrices, the colour_description
elements, picture_coding_type, the f_code[r][s] values and
quantiser_scale_code. It is now checked that the invalid values don't
occur.
The colour_description elements are treated specially in this regard:
Given that there are files in the wild which use illegal values for the
colour_description elements (some of them created by mpeg2_metadata),
they will be corrected to the value meaning "unknown" (namely 2) during
reading. This has been done in such a way that trace_headers will
nevertheless report the original value, together with a message about
the fixup.
Furthermore, the trace_headers output of user_data has been beautified.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
(cherry picked from commit 9c3f2a8894)
cbs_mpeg2_free_slice() calls av_buffer_unref() on extra_information_ref,
meaning allocating with av_malloc() was not the intention.
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit d903c09d9a)
Silences a warning with clang:
libavcodec/cbs_vp9_syntax_template.c:220:17: warning: implicit conversion from 'int' to 'int8_t' (aka 'signed char')
changes value from 255 to -1
(cherry picked from commit de441ad52a)
The spec defines it as a signed value.
Reviewed-by: Mark Thompson <sw@jkqxz.net>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit 9bf520d04d)
The earlier code used the most recent non-auxiliary slice to determine
whether an auxiliary slice has the syntax of an IDR slice, even when
the most recent slice was from a slice of a redundant frame. Now only
slices of the primary coded picture are used, as the specifications
mandate.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@googlemail.com>
(cherry picked from commit 8d1cf2d894)
Instead we ask for a sample, its unclear what to do in this case.
Fixes: index 30 out of bounds for type 'int8_t [30][64]'
Fixes: 15339/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_QDM2_fuzzer-5749441484554240
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ae021c1239)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This checks the ham value much stricter and avoids hitting cases which cannot be reached
with data from the libavformat demuxer.
Fixes: out of array access
Fixes: 15320/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_IFF_ILBM_fuzzer-5080476840099840
Fixes: 15423/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_IFF_ILBM_fuzzer-5630765833912320
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit f76d7352e0)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: signed integer overflow: 6494225984479297536 - -6043795377581187040 cannot be represented in type 'long'
Fixes: 15285/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FFWAVESYNTH_fuzzer-5632780307791872
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e9dd3c7126)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: negation of -9223372036854775808 cannot be represented in type 'int64_t' (aka 'long'); cast to an unsigned type to negate this value to itself
Fixes: 15289/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FFWAVESYNTH_fuzzer-5709034499342336
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8c02209935)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: 15360/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FLIC_fuzzer-5653837190266880
Fixes: 15412/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_FLIC_fuzzer-5740537648250880
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 37708cbae8)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
lpc_quant of 0 produces undefined behavior, thus disallow this.
If valid samples use this then such a sample would be quite
usefull to confirm the correct&lossles handling of this.
Fixes: libavcodec/alac.c:218:25: runtime error: shift exponent -1 is negative
Fixes: 15273/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALAC_fuzzer-5656388535058432
Fixes: 15276/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALAC_fuzzer-5761238417539072
Fixes: 15315/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALAC_fuzzer-5767260766994432
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a6474b899c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: Various anomalies
Fixes: 14493/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DXV_fuzzer-5071018000908288
Fixes: 14630/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_DXV_fuzzer-5714888963391488
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e96b7a8ba6)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: out of array access
Fixes: 15277/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALS_fuzzer-5184853437317120
Fixes: 15280/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALS_fuzzer-5741062137577472
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0794494c8f)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes: left shift of negative value -6
Fixes: 15275/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALS_fuzzer-5742361767837696
Fixes: signed integer overflow: 41582592 * 256 cannot be represented in type 'int'
Fixes: 15296/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALS_fuzzer-5739558227935232
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit e131568752)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
