From 510d88ae939f80a0e69188135d9ca7824a2c1075 Mon Sep 17 00:00:00 2001 From: Andreas Cadhalpun Date: Sun, 8 Nov 2015 15:15:24 +0100 Subject: [PATCH] softfloat: assert when the argument of av_sqrt_sf is negative The correct result can't be expressed in SoftFloat. Currently it returns a random value from an out of bounds read. Reviewed-by: Michael Niedermayer Signed-off-by: Andreas Cadhalpun (cherry picked from commit f3866a14c3c2949fad16267e9f2977ba9d7b5504) Signed-off-by: Andreas Cadhalpun --- libavutil/softfloat.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/libavutil/softfloat.h b/libavutil/softfloat.h index facfbc2424..5b285e3d9b 100644 --- a/libavutil/softfloat.h +++ b/libavutil/softfloat.h @@ -179,6 +179,8 @@ static av_always_inline SoftFloat av_sqrt_sf(SoftFloat val) if (val.mant == 0) val.exp = MIN_EXP; + else if (val.mant < 0) + av_assert0(0); else { tabIndex = (val.mant - 0x20000000) >> 20;