203 Commits

Author SHA1 Message Date
Michael Niedermayer
f0ae57aca4 avcodec/hevc: Check max ctb addresses for WPP
Fixes out of array read
Fixes: 2f95ddd996db8a6281d2e18c184595a7/asan_heap-oob_192fe91_3330_58e4441181e30a66c19f743dcb392347.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit dad354f38ddc9bfc834bc21358a1d0ad41532ca0)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Conflicts:

	libavcodec/hevc.c

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:16:35 +01:00
Michael Niedermayer
5c0be549ef avcodec/hevc: Fix integer overflow of entry_point_offset
Fixes out of array read
Fixes: d41d8cd98f00b204e9800998ecf8427e/signal_sigsegv_321165b_7641_077dfcd8cbc80b1c0b470c8554cd6ffb.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 214085852491448631dcecb008b5d172c11b8892)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 02:08:26 +01:00
Michael Niedermayer
d8c0546ecd avcodec/hevc: allocate entries unconditionally
Fixes out of array access
Fixes: 08664a2a7921ef48172f26495c7455be/asan_heap-oob_23036c6_3301_523388ef84285a0270caf67a43247b59.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d85aa76115214183e7e3b7d65e950da61474959a)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

Conflicts:

	libavcodec/hevc.c
2015-12-05 23:48:52 +01:00
Michael Niedermayer
c6b81e252e avcodec/hevc: Check entry_point_offsets
Fixes out of array read
Fixes: 007c4a36608ebdf27ee260ad60a81184/asan_heap-oob_32076b4_2243_116b1cb29d91cc4974d6680e3d10bd91.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ef9f7bbfa47317f9d46bf46982a394d2be78503c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 23:48:52 +01:00
Hendrik Leppkes
7b6fced6b4 hevc: properly handle no_rasl_output_flag when removing pictures from the DPB
Fixes ticket .

Reviewed-By: Mickael Raulet <Mickael.Raulet@insa-rennes.fr>
Signed-off-by: Hendrik Leppkes <h.leppkes@gmail.com>
(cherry picked from commit 0118158efa8e45761f9f65a3bb74f33907bd2aec)
2015-10-09 21:57:05 +02:00
Ronald S. Bultje
cac62c8ad9 hevc: fix wpp threading deadlock.
Fixes ticket 4258.
(cherry picked from commit 74e4948235bc8f8946eeca20525258bbf383f75d)
2015-10-09 21:56:52 +02:00
Michael Niedermayer
1bf416dfc3 avcodec/hevc: Fix typo in num_entry_point_offsets check
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3051e7fa712dfe2136f19b7157211453895f2a3c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-21 20:43:39 +02:00
wm4
02c1074f18 hevc: make avcodec_decode_video2() fail if get_format() fails
Personally, I need the decoder to back out if get_format() returns no
usable pixel format. This didn't work because the error code was not
propagated down the call chain. This in turn happened because the
variable declaration removed in this patch shadowed the variable, whose
value is returned at the end of the function. Consequently, failures of
decode_nal_unit() were ignored in this place.

Reviewed-by:  Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit cc5e4bb48476a89cc8ce0c41bc2bd2e8fda9b37c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-21 20:43:39 +02:00
Michael Niedermayer
58f310f496 avcodec/hevc: Check num_entry_point_offsets
Fixes CID1239099 part 2

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1c6ae98d4a9ff9ea607df87908393eda4ebdf4e8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-21 20:43:38 +02:00
Michael Niedermayer
0230a8efc6 avcodec/hevc: Check offset_len
Fixes CID1239099 part 1

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3e9d5e16ad9799f6b6faae4f21120d23146b84c9)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-21 20:43:38 +02:00
Vittorio Giovara
0bdc64e8b9 hevc: always clip luma_log2_weight_denom
Its value shall be between 0 and 7 according to the specifications.

Bug-Id: CID 1257502
2015-02-12 17:10:35 +01:00
Michael Niedermayer
4f8814964c avcodec/hevc: Fix handling of skipped_bytes() reallocation failures
Fixes CID1260704

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e172f5e53ae4dbbcdcf81c9a3b962dc9f5a8a98d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-12 17:10:35 +01:00
Michael Niedermayer
e911f125fc avcodec/hevc: clear filter_slice_edges() on allocation
This avoids use of uninitialized memory
Fixes: asan_static-oob_17aa046_582_cov_212287884_DBLK_G_VIXS_1.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8aa8d12554868c32436750f881954193087219c8)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-22 03:17:55 +01:00
Michael Niedermayer
148506c965 Merge commit 'cd975d5658a1cbe99939df75db59d5ae9fbcb4e0'
* commit 'cd975d5658a1cbe99939df75db59d5ae9fbcb4e0':
  hevc: Spin the mv_mpv_mode calls in a stand alone function

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-22 01:32:55 +01:00
Luca Barbato
cd975d5658 hevc: Spin the mv_mpv_mode calls in a stand alone function
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2014-11-21 23:17:37 +01:00
Michael Niedermayer
91a9ae5b6b Merge commit 'eac3ac1fe0774b65316852616b2672702dbc3f31'
* commit 'eac3ac1fe0774b65316852616b2672702dbc3f31':
  hevc: eliminate an unneeded intermediate variable

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 12:13:09 +01:00
Michael Niedermayer
0d5af820f7 Merge commit 'eb335f3c5ce37f2b93c993e28404d113bee843bc'
* commit 'eb335f3c5ce37f2b93c993e28404d113bee843bc':
  hevc: reduce variable scope

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 12:07:49 +01:00
Michael Niedermayer
c23d7de22e Merge commit '84c0ece5fd9569c0f31804f02a199ecd0e7d13d8'
* commit '84c0ece5fd9569c0f31804f02a199ecd0e7d13d8':
  hevc: further reduce code duplication in hls_prediction_unit()

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 12:01:47 +01:00
Michael Niedermayer
e078549421 Merge commit 'a7b365ae191f45a0d7ed7b34033d5d0cbdd47139'
* commit 'a7b365ae191f45a0d7ed7b34033d5d0cbdd47139':
  hevc: reduce code duplication in hls_prediction_unit()

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 11:54:41 +01:00
Michael Niedermayer
c192be1968 Merge commit '16c01fb4347312b6d29a6498dad627665b96a20e'
* commit '16c01fb4347312b6d29a6498dad627665b96a20e':
  hevc: remove an unused function parameter

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 11:30:41 +01:00
Michael Niedermayer
f1b20930f2 Merge commit '2c6a7f9348378f887066fb1669c46b9485e8ef3e'
* commit '2c6a7f9348378f887066fb1669c46b9485e8ef3e':
  hevc: do not store rqt_root_cbf in the context

Conflicts:
	libavcodec/hevc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 11:20:32 +01:00
Michael Niedermayer
7cbe1e0447 Merge commit '920bca3e2332dced9c78bd14cfc2ff138188bd57'
* commit '920bca3e2332dced9c78bd14cfc2ff138188bd57':
  hevc: do not store pcm_flag in the context

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-17 11:06:13 +01:00
Anton Khirnov
eac3ac1fe0 hevc: eliminate an unneeded intermediate variable 2014-11-17 09:26:45 +01:00
Anton Khirnov
eb335f3c5c hevc: reduce variable scope
Also, collapse the array into a scalar, since only one value is needed
at a time.
2014-11-17 09:26:45 +01:00
Anton Khirnov
8b573ddda7 hevc: remove superfluous assignments and checks 2014-11-17 09:26:45 +01:00
Anton Khirnov
84c0ece5fd hevc: further reduce code duplication in hls_prediction_unit() 2014-11-17 09:25:15 +01:00
Anton Khirnov
a7b365ae19 hevc: reduce code duplication in hls_prediction_unit() 2014-11-17 09:25:12 +01:00
Anton Khirnov
16c01fb434 hevc: remove an unused function parameter 2014-11-17 09:25:12 +01:00
Anton Khirnov
2c6a7f9348 hevc: do not store rqt_root_cbf in the context
It does not need to be accessed outside of hls_coding_unit().
2014-11-17 09:25:12 +01:00
Anton Khirnov
920bca3e23 hevc: do not store pcm_flag in the context
It does not need to be accessed outside of hls_coding_unit().
2014-11-17 09:25:12 +01:00
Anton Khirnov
84b9463984 hevc: remove a redundant line
pred_mode is overwritten a few lines immediately below.
2014-11-17 09:25:12 +01:00
Michael Niedermayer
76fa78911f Merge commit 'b1b1a7370e141c912e3d0bbaa668dcee05c3ad67'
* commit 'b1b1a7370e141c912e3d0bbaa668dcee05c3ad67':
  display: fix order of operands

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-13 12:53:31 +01:00
Vittorio Giovara
b1b1a7370e display: fix order of operands
CC: libav-stable@libav.org
Bug-Id: CID 1238828 / CID 1238832
2014-11-13 01:41:25 +01:00
Changjiang Wei
6f2068e626 avcodec/hevc.c: for big negative mvy value, should wait line 0 of ref frame due to edge extending
Reviewed-by: Mickaël Raulet <mraulet@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-11 11:45:20 +01:00
Michael Niedermayer
17085a0251 Merge commit '7ea1b3472a61de4aa4d41b571e99418e4997ad41'
* commit '7ea1b3472a61de4aa4d41b571e99418e4997ad41':
  lavc: deprecate the use of AVCodecContext.time_base for decoding

Conflicts:
	libavcodec/avcodec.h
	libavcodec/h264.c
	libavcodec/mpegvideo_parser.c
	libavcodec/utils.c
	libavcodec/version.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-15 15:49:31 +02:00
Anton Khirnov
7ea1b3472a lavc: deprecate the use of AVCodecContext.time_base for decoding
When decoding, this field holds the inverse of the framerate that can be
written in the headers for some codecs. Using a field called 'time_base'
for this is very misleading, as there are no timestamps associated with
it. Furthermore, this field is used for a very different purpose during
encoding.

Add a new field, called 'framerate', to replace the use of time_base for
decoding.
2014-10-15 06:37:43 +00:00
Michael Niedermayer
df8394c693 avcodec/hevc: fix chroma transform_add size
Fixes accessing misaligned address
Fixes: signal_sigsegv_1feb99c_10_signal_sigsegv_2d1d35c_79_cov_691940146_EXT_A_ericsson_3.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-04 17:51:52 +02:00
Christophe Gisquet
9a3653c9ec hevc: reuse edge emu buffer for coefficients
Kind of hackish but...

Reviewed-by: Mickael Raulet <Mickael.Raulet@insa-rennes.fr>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-01 12:12:28 +02:00
Christophe Gisquet
f183febcda hevc: remove CodingTree
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-09-27 11:50:41 +02:00
Christophe Gisquet
3fe962940f hevc: move intermediate bidir buffer
Other buffers are already there.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-09-24 01:26:12 +02:00
Mickaël Raulet
684d0a0b23 avcodec/hevc: fix dead code
fix CID 1231985

cherry picked from commit 745a35a777eaa5f77b8660b44098110a29916aae
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-09-06 20:36:27 +02:00
Christophe Gisquet
dad7f15567 hevcdsp: remove more instances of compile-time-fixed parameters
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-22 15:22:42 +02:00
Christophe Gisquet
d4f44b66d3 hevcdsp: remove compilation-time-fixed parameter
The dststride parameter is always MAX_PB_SIZE.

Reviewed-by: Mickaël Raulet <mraulet@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-22 14:57:37 +02:00
Michael Niedermayer
2278146a0c Merge commit 'd1b1c3bb5ef1bee409c06c3bcacfb8674e1cb574'
* commit 'd1b1c3bb5ef1bee409c06c3bcacfb8674e1cb574':
  hevc: reorder loops

Conflicts:
	libavcodec/hevc.c

See: eca1957c4cbee66a0b1f3dcabaffe68d61885f16
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-12 15:59:33 +02:00
Christophe Gisquet
cf6090dc62 hevc: use intreadwrite
When dealing with MVs, both components may be processed at a time.

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2014-08-12 10:03:20 +00:00
Christophe Gisquet
d1b1c3bb5e hevc: reorder loops
iterate over memory in a more continuous order

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2014-08-12 09:57:05 +00:00
Christophe Gisquet
5ec85c9750 hevc: do generic validation of bitstream
After finishing parsing VPS/SPS/PPS/slice header, check remaining bits,
and if an overconsumption occurred, report invalid data.

Liked-by: BBB
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-11 02:47:05 +02:00
Ronald S. Bultje
bfffce4d08 hevc: add missing comma in log message.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-11 02:27:27 +02:00
Christophe Gisquet
7117547298 hevc: fix incorrect sao buffer size
It previously used the output, cropped size, causing overreads/writes.

Fixes ticket .

This issue was introduced by d249e682, which is not part of any release

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-10 12:34:57 +02:00
Michael Niedermayer
3f2495d98c Merge commit '70211539a39ca3854f8a9e97d51dc27caa079943'
* commit '70211539a39ca3854f8a9e97d51dc27caa079943':
  hevc: deobfuscate slice/tile boundary handling for DBF

Conflicts:
	libavcodec/hevc.c
	libavcodec/hevc_filter.c

See: a5c621aa852522c79146035b2db4b89d2e096d3c
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-09 21:01:55 +02:00